You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"Do these tests actually test anything" is unanswered for dbt projects. A model can
carry unit tests, generic tests, and singular tests, pass all of them, and be wrong
in a way none of them can detect. Test count is not test strength, and there is no
tool in the ecosystem that measures the difference.
Proposal
Perturb the model's own SQL in a library of standard analytics-defect mutations,
run the model's tests against each mutant on a dev target, and report which
mutations survive, meaning which defects the tests cannot detect.
A starting mutation library, drawn from the defect classes that recur in real
projects:
Mutation
Defect it stands for
flip a comparison operator
boundary and threshold errors
negate or drop a where predicate
a filter that stops filtering
swap a join type (inner for left, left for inner)
silent row loss or unintended fanout
drop a case branch
an unhandled category
swap a numerator and denominator
inverted rates
shift a window frame bound
off-by-one in rolling calculations
replace an aggregate (sum for max, max for sum)
wrong aggregation over a cumulative measure
Each surviving mutant is reported as a gap, described in terms of the defect rather
than the syntax, so the output is "your tests would not catch an inner join here"
rather than a diff.
Constraints this must respect
Mutants never touch the project. They build in a throwaway copy, the way
semantic validation already does for dbt-parser checks.
Dev target only, and priced as a batch before anything runs, since this is N
builds rather than one.
A hard cap on mutants per run, reported, with the elision stated.
It is the only item in this list that answers a question nobody else answers, and
it turns the whole defect taxonomy that motivates the rest of these issues into
something a project can be measured against.
Acceptance
A model with a strong test suite reports few or no surviving mutants.
A model with only a not_null test reports most mutants surviving.
No mutant is ever written into the project directory.
The batch is priced and confirmed before execution.
Depends on: #215, #231
Problem
"Do these tests actually test anything" is unanswered for dbt projects. A model can
carry unit tests, generic tests, and singular tests, pass all of them, and be wrong
in a way none of them can detect. Test count is not test strength, and there is no
tool in the ecosystem that measures the difference.
Proposal
Perturb the model's own SQL in a library of standard analytics-defect mutations,
run the model's tests against each mutant on a dev target, and report which
mutations survive, meaning which defects the tests cannot detect.
A starting mutation library, drawn from the defect classes that recur in real
projects:
wherepredicatecasebranchsumformax,maxforsum)Each surviving mutant is reported as a gap, described in terms of the defect rather
than the syntax, so the output is "your tests would not catch an inner join here"
rather than a diff.
Constraints this must respect
semantic validation already does for dbt-parser checks.
builds rather than one.
transform build --verify: run the sweep over the nodes just built #231 for the build-and-check harness.Why it is worth doing despite the cost
It is the only item in this list that answers a question nobody else answers, and
it turns the whole defect taxonomy that motivates the rest of these issues into
something a project can be measured against.
Acceptance
not_nulltest reports most mutants surviving.