Skip to content

transform: mutation coverage for a model's tests #232

Description

@marcociav-exmergo

Depends on: #215, #231

Problem

"Do these tests actually test anything" is unanswered for dbt projects. A model can
carry unit tests, generic tests, and singular tests, pass all of them, and be wrong
in a way none of them can detect. Test count is not test strength, and there is no
tool in the ecosystem that measures the difference.

Proposal

Perturb the model's own SQL in a library of standard analytics-defect mutations,
run the model's tests against each mutant on a dev target, and report which
mutations survive
, meaning which defects the tests cannot detect.

A starting mutation library, drawn from the defect classes that recur in real
projects:

Mutation Defect it stands for
flip a comparison operator boundary and threshold errors
negate or drop a where predicate a filter that stops filtering
swap a join type (inner for left, left for inner) silent row loss or unintended fanout
drop a case branch an unhandled category
swap a numerator and denominator inverted rates
shift a window frame bound off-by-one in rolling calculations
replace an aggregate (sum for max, max for sum) wrong aggregation over a cumulative measure

Each surviving mutant is reported as a gap, described in terms of the defect rather
than the syntax, so the output is "your tests would not catch an inner join here"
rather than a diff.

Constraints this must respect

  • Mutants never touch the project. They build in a throwaway copy, the way
    semantic validation already does for dbt-parser checks.
  • Dev target only, and priced as a batch before anything runs, since this is N
    builds rather than one.
  • A hard cap on mutants per run, reported, with the elision stated.
  • Depends on transform build --verify: run the sweep over the nodes just built #231 for the build-and-check harness.

Why it is worth doing despite the cost

It is the only item in this list that answers a question nobody else answers, and
it turns the whole defect taxonomy that motivates the rest of these issues into
something a project can be measured against.

Acceptance

  • A model with a strong test suite reports few or no surviving mutants.
  • A model with only a not_null test reports most mutants surviving.
  • No mutant is ever written into the project directory.
  • The batch is priced and confirmed before execution.
  • The cap is enforced and reported.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions