fix(test): a partial test run says what it did not cover - #204
Merged
Conversation
Across a whole session I reported "suite green" from `pytest add-method/
tests/` while `add-method/tooling/` — 8 checks including the ENGINE_MD5 pins
that CI also runs — was red. Every green was true of the suite I ran and
silent about the one I did not, and a red branch reached the point of merge
on the strength of it.
The repo was not at fault: pytest.ini already made a bare `pytest` from
add-method/ collect everything, and its comment already explained why
tooling/ had to stay collectable. Nothing here stops a narrow run, and
nothing should — running one directory while iterating is usually right.
What is fixed is the ambiguity: a run that covered part of the suite now
ends by naming the part it missed, plus the command that covers it.
================================= PARTIAL RUN ==================================
not collected: tooling — this run says nothing about it.
full suite: python3 -m pytest -q (from add-method/, what CI runs)
Roots are discovered from the tree, never listed — a hand-maintained list
would have omitted tooling/ for the same reason a person does: it does not
look like a test directory, it looks like the engine's home. The notice has
no off switch, because the one thing it must survive is being found
inconvenient, and it never changes the exit code, because an error would
only teach people to silence it.
Three real defects, each caught by the new checks rather than by reading:
1. tests/eval had NEVER run — not in a bare run, not in CI. `norecursedirs`
matches a BASENAME, so the entry meant for the top-level eval/ fixture
project also swallowed the eval driver that pytest.ini's own comment
names. Two tests, absent from every count this repo has reported.
2. The first fix, `--ignore=eval`, resolves against ROOTDIR — so it stopped
applying the moment pytest ran from the repo root. Its own receipt run
caught that. Now `collect_ignore` anchored to the conftest's directory,
which holds from either invocation point.
3. `import conftest` in the new test silently resolved to
tests/engine/conftest.py on a full run. It passed alone and failed 700
tests in — the same class of mistake this task is about.
Collection 715 -> 721; suite 714 passed / 7 skipped, verified from BOTH
add-method/ and the repo root.
Task partial-run-visible: gate PASS @ plan, receipt runs/2.md, freshness
fresh. Re-frozen once to widen scope to pytest.ini, after the reachability
check found the uncollected suite on its first execution.
author: Tin Dang
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A partial test run says what it did not cover
Across a full session I reported "suite green" from
pytest add-method/tests/whileadd-method/tooling/— 8 checks including theENGINE_MD5pins CI also runs — was red. Every green was true of the suite I ran and silent about the one I didn't, and a red branch reached the point of merge on the strength of it.The repo was not at fault.
pytest.inialready made a barepytestfromadd-method/collect everything, and its comment already explained whytooling/had to stay collectable. Nothing here stops a narrow run, and nothing should — running one directory while iterating is usually the right thing to do. What's fixed is the ambiguity:Design
tooling/for the same reason a person does: it doesn't look like a test directory, it looks like the engine's home.Three real defects, each caught by the new checks rather than by reading
tests/evalhad never run — not in a bare run, not in CI.norecursedirsmatches a basename, so the entry meant for the top-leveleval/fixture project also swallowed the eval driver thatpytest.ini's own comment names. Two tests, absent from every count this repo has ever reported.--ignore=evalresolves against rootdir, so it stopped applying the moment pytest ran from the repo root — caught by this task's own receipt run. Nowcollect_ignoreanchored to the conftest's directory, which holds from either invocation point.import conftestsilently resolved totests/engine/conftest.pyon a full run. It passed in isolation and failed 700 tests in — the same class of mistake this task exists to fix: a narrow run agreeing with you and a full run not.Verification
add-method/and the repo rootpartial-run-visiblegate PASS @plan, receiptruns/2.md,freshness: freshpytest.ini, after the reachability check found the uncollected suite on its first execution