Skip to content

fix(test): a partial test run says what it did not cover - #204

Merged
TinDang97 merged 1 commit into
mainfrom
fix/partial-run-visible
Aug 12, 2026
Merged

fix(test): a partial test run says what it did not cover#204
TinDang97 merged 1 commit into
mainfrom
fix/partial-run-visible

Conversation

@TinDang97

Copy link
Copy Markdown
Collaborator

A partial test run says what it did not cover

Across a full session I reported "suite green" from pytest add-method/tests/ while add-method/tooling/ — 8 checks including the ENGINE_MD5 pins CI also runs — was red. Every green was true of the suite I ran and silent about the one I didn't, and a red branch reached the point of merge on the strength of it.

The repo was not at fault. pytest.ini already made a bare pytest from add-method/ collect everything, and its comment already explained why tooling/ had to stay collectable. Nothing here stops a narrow run, and nothing should — running one directory while iterating is usually the right thing to do. What's fixed is the ambiguity:

================================= PARTIAL RUN ==================================
not collected: tooling — this run says nothing about it.
full suite:    python3 -m pytest -q      (from add-method/, what CI runs)

Design

  • Roots are discovered, never listed. A hand-maintained list would have omitted tooling/ for the same reason a person does: it doesn't look like a test directory, it looks like the engine's home.
  • No off switch. The one thing this must survive is somebody finding it inconvenient.
  • Never fails the run. An error would only teach people to silence it. It informs; it does not gate.
  • Silent on a complete run. A notice that appears every time is one nobody reads.

Three real defects, each caught by the new checks rather than by reading

  1. tests/eval had never run — not in a bare run, not in CI. norecursedirs matches a basename, so the entry meant for the top-level eval/ fixture project also swallowed the eval driver that pytest.ini's own comment names. Two tests, absent from every count this repo has ever reported.
  2. The first fix broke position-independence. --ignore=eval resolves against rootdir, so it stopped applying the moment pytest ran from the repo root — caught by this task's own receipt run. Now collect_ignore anchored to the conftest's directory, which holds from either invocation point.
  3. import conftest silently resolved to tests/engine/conftest.py on a full run. It passed in isolation and failed 700 tests in — the same class of mistake this task exists to fix: a narrow run agreeing with you and a full run not.

Verification

  • Collection 715 → 721
  • 714 passed, 7 skipped, verified from both add-method/ and the repo root
  • partial-run-visible gate PASS @ plan, receipt runs/2.md, freshness: fresh
  • Re-frozen once to widen scope to pytest.ini, after the reachability check found the uncollected suite on its first execution

Across a whole session I reported "suite green" from `pytest add-method/
tests/` while `add-method/tooling/` — 8 checks including the ENGINE_MD5 pins
that CI also runs — was red. Every green was true of the suite I ran and
silent about the one I did not, and a red branch reached the point of merge
on the strength of it.

The repo was not at fault: pytest.ini already made a bare `pytest` from
add-method/ collect everything, and its comment already explained why
tooling/ had to stay collectable. Nothing here stops a narrow run, and
nothing should — running one directory while iterating is usually right.
What is fixed is the ambiguity: a run that covered part of the suite now
ends by naming the part it missed, plus the command that covers it.

    ================================= PARTIAL RUN ==================================
    not collected: tooling — this run says nothing about it.
    full suite:    python3 -m pytest -q      (from add-method/, what CI runs)

Roots are discovered from the tree, never listed — a hand-maintained list
would have omitted tooling/ for the same reason a person does: it does not
look like a test directory, it looks like the engine's home. The notice has
no off switch, because the one thing it must survive is being found
inconvenient, and it never changes the exit code, because an error would
only teach people to silence it.

Three real defects, each caught by the new checks rather than by reading:

1. tests/eval had NEVER run — not in a bare run, not in CI. `norecursedirs`
   matches a BASENAME, so the entry meant for the top-level eval/ fixture
   project also swallowed the eval driver that pytest.ini's own comment
   names. Two tests, absent from every count this repo has reported.
2. The first fix, `--ignore=eval`, resolves against ROOTDIR — so it stopped
   applying the moment pytest ran from the repo root. Its own receipt run
   caught that. Now `collect_ignore` anchored to the conftest's directory,
   which holds from either invocation point.
3. `import conftest` in the new test silently resolved to
   tests/engine/conftest.py on a full run. It passed alone and failed 700
   tests in — the same class of mistake this task is about.

Collection 715 -> 721; suite 714 passed / 7 skipped, verified from BOTH
add-method/ and the repo root.

Task partial-run-visible: gate PASS @ plan, receipt runs/2.md, freshness
fresh. Re-frozen once to widen scope to pytest.ini, after the reachability
check found the uncollected suite on its first execution.

author: Tin Dang
@TinDang97
TinDang97 merged commit 8783b22 into main Aug 12, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant