Why
An ad-hoc sweep this week found tests that passed for the wrong reason: where several code paths
converge on one response, a test asserting only the status keeps passing after the guard it was
written for is deleted. Four such gaps were fixed in 033734f1, and the pitfall is recorded in
backend/tests/README.md.
That sweep was ad hoc and partial by construction. It matched literal
HTTPException(status_code=..., detail=...) constructs — 53 raise sites — while the codebase raises
through 113 custom exception classes (38 of them: BadRequestError, ConflictError,
DependentModelOwnershipError, ...) that the scan never saw. So roughly two thirds of the error
surface has not been checked.
A better regex is the wrong fix. This wants a real mutation-testing tool on a schedule, where a long
runtime does not matter.
What to investigate
mutmut vs cosmic-ray against backend/, run nightly or weekly rather than per-PR — the suite
is ~16s but a full mutation run is hours.- Scope it to
app/api/ and app/core/ first; skip generated code and migrations.
- Decide how results surface: a failing scheduled job is noisy for a metric that will never reach
zero. An artifact plus a tracked survivor count is probably better than a red build.
- Baseline the survivor list so only new survivors need triage.
Prior findings to carry over
The last sweep left three survivors that are deliberate, not gaps — they should be baselined,
not "fixed":
images.py re-checks for empty bytes that validate_upload_size has already rejected. Kept as
defence in depth on the untrusted device-upload path.images.py raises when a stored preview thumbnail's URL cannot be computed.account_security.py raises when the request context is missing.
Each guards a state that cannot currently occur, so a test for one would only pin a mock.
Reproducing the ad-hoc version
Replace a guard with if False: and run tests/unit tests/integration. A guard whose removal
breaks nothing is either untested, or covered only by an assertion another path also satisfies. Run
both tiers — several guards in routers and dependencies are only reachable from integration.
Why
An ad-hoc sweep this week found tests that passed for the wrong reason: where several code paths
converge on one response, a test asserting only the status keeps passing after the guard it was
written for is deleted. Four such gaps were fixed in
033734f1, and the pitfall is recorded inbackend/tests/README.md.That sweep was ad hoc and partial by construction. It matched literal
HTTPException(status_code=..., detail=...)constructs — 53 raise sites — while the codebase raisesthrough 113 custom exception classes (38 of them:
BadRequestError,ConflictError,DependentModelOwnershipError, ...) that the scan never saw. So roughly two thirds of the errorsurface has not been checked.
A better regex is the wrong fix. This wants a real mutation-testing tool on a schedule, where a long
runtime does not matter.
What to investigate
mutmutvscosmic-rayagainstbackend/, run nightly or weekly rather than per-PR — the suiteis ~16s but a full mutation run is hours.
app/api/andapp/core/first; skip generated code and migrations.zero. An artifact plus a tracked survivor count is probably better than a red build.
Prior findings to carry over
The last sweep left three survivors that are deliberate, not gaps — they should be baselined,
not "fixed":
images.pyre-checks for empty bytes thatvalidate_upload_sizehas already rejected. Kept asdefence in depth on the untrusted device-upload path.
images.pyraises when a stored preview thumbnail's URL cannot be computed.account_security.pyraises when the request context is missing.Each guards a state that cannot currently occur, so a test for one would only pin a mock.
Reproducing the ad-hoc version
Replace a guard with
if False:and runtests/unit tests/integration. A guard whose removalbreaks nothing is either untested, or covered only by an assertion another path also satisfies. Run
both tiers — several guards in routers and dependencies are only reachable from integration.