diff --git a/DESIGN.md b/DESIGN.md index d4d15ea..4474e7d 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -56,8 +56,10 @@ Shared configs: `configs/ruff-base.toml` (C901 ≤ 10, PLR0915 ≤ 50, S battery BLE ban — the ratified budgets), `configs/mypy-base.toml` (strict-leaning profile, see §10), `configs/jscpd.src.json` / `configs/jscpd.tests.json` (the two-profile clone carve-out, see §7). Tool-behavior ground truth lives in -`docs/tool-spikes.md` — complexipy 5.5.0 and mypy-baseline 0.7.4 semantics -were observed on fixtures, not read off READMEs. +`docs/tool-spikes.md` — complexipy and mypy-baseline semantics were observed on +fixtures, not read off READMEs (spiked on complexipy 5.5.0 · mypy-baseline +0.7.4; complexipy's snapshot-write semantics re-measured on the pinned 5.6.0 +on 2026-07-28, and they inverted — see §4.11). **The single reproducible entrypoint — `cf-gate` (`gate_runner.py`).** The battery above is split across many console scripts and external tools so each @@ -135,9 +137,19 @@ become green-forever): to `:0`, so unrelated drift cannot resurrect findings. 6. **Cognitive-complexity watermark:** `complexipy src --snapshot-create` **from the repo root** (the snapshot lands in CWD, not the analyzed path) - → commit `complexipy-snapshot.json`. Observed gotchas: the snapshot does - NOT auto-shrink (the kit owns the re-snapshot on merge, or improvements - are not locked in); piping the gate command masks its exit code. + → commit `complexipy-snapshot.json`. This is the ONLY complexipy command a + consumer ever runs by hand. Measured on the pinned **complexipy 5.6.0** + (2026-07-28): a *passing* plain-run compare REWRITES the snapshot — + `handle_snapshot_watermark` calls `create_snapshot_file` on its + no-violation branch — so the tool will shrink its own floor to `[]` at + exit 0. `cf-gate` therefore measures write-free (`--snapshot-ignore`), AUDITS + the consumer's complexipy config so `snapshot-create` cannot re-open that + write from TOML, and grades in `complexipy_ratchet` (§4.11). The duty that + remains: a shrink is locked in only by a deliberate re-`--snapshot-create`, + because the watermark rule is a `>` bound and a stale watermark still + grandfathers a climb back up to it. Re-boot in the SAME commit as any change + to `max-complexity-allowed`, or the gate refuses the mismatch + (`GATE_COMPLEXIPY_THRESHOLD_RAISED`). 7. **Exemptions:** if the repo carries any gated suppression, register each in `exemptions.json` (five fields: file, symbol_or_line, rule, reason, approver) and set `frozen_count`. @@ -460,11 +472,50 @@ ordinary PR. new. Fixed-only also exits nonzero ("re-sync"), so every shrink is committed — the ratchet direction is enforced by the tool itself. - **complexipy — set-like per function** (new offender fails; baselined - offender worsening fails), but the snapshot does NOT auto-shrink: the - re-snapshot-on-merge step is runbook procedure (§3 step 6), not yet CI - mechanism. The complexipy step IS in `quality-gate.yml` (and `self-ci.yml`) - as of 2026-06-11, with the mypy-style presence rule: a Python repo without - its committed snapshot FAILS; only a Python-free repo skips, visibly. + offender worsening fails), and graded by the KIT, not by the tool. The + 2026-06-10 spike recorded on 5.5.0 that the snapshot never shrinks itself. + Re-measured on the pinned **complexipy 5.6.0** (2026-07-28) that claim is + false and inverted: a *passing* compare rewrites the floor — + `handle_snapshot_watermark` calls `create_snapshot_file` on its no-violation + branch — reproduced at exit 0 shrinking a populated snapshot to `[]` both by + grading a narrower path than the floor and by raising the threshold above + every function. So the vector here was never "the floor cannot shrink", it + was "the floor can be **zeroed** behind a green run", which is laundering by + deletion. That zeroing is CLOSED by construction: `cf-gate` measures with + `--snapshot-ignore`, AUDITS the consumer's own complexipy config so the other + write path cannot be re-opened from TOML (`snapshot-create` resolves + CLI-first/TOML-second and has no negating flag, so a config carrying it made + both measurement runs rewrite the floor — + `GATE_COMPLEXIPY_CONFIG_DEFEATS_MEASUREMENT` now refuses it, naming the key + and the file), and compares in `complexipy_ratchet.grade`, a pure function + of (committed floor, measured census) — nothing the tool does can rewrite + the artifact it is graded against. Residue, and it is the OLD residue in a + narrower form: locking a shrink IN is still runbook procedure (§3 step 6), + not CI mechanism — the watermark is a `>` bound, so an improvement that is + never re-snapshotted leaves a stale watermark a later regression may climb + back to, and a deleted floor file leaves a dead entry that would grandfather + a same-named function if the file returned. Mechanical now, stated exactly + (an earlier pass on this branch wrote that emptying a floor file of functions + FAILS — false, since `--plain` lists every measured function regardless of + threshold): a floor file that still exists but was not **measured**, or that + sits outside the graded root, FAILS + (`COMPLEXIPY_SNAPSHOT_FILE_UNMEASURED` / `COMPLEXIPY_SURFACE_NARROWED`); a + floor **function** missing from a file that WAS measured FAILS + (`COMPLEXIPY_SNAPSHOT_FUNCTION_UNMEASURED` — the shape a + `# complexipy: ignore` comment takes, an unregistered exemption from this + gate); a threshold raised above a committed watermark REFUSES rather than + grading the emptied offender set (`GATE_COMPLEXIPY_THRESHOLD_RAISED`, + threshold-free: a floor entry proves that function was above the boot bar); + and a run that measured zero functions cannot report clean while the floor + names functions or the repo contains Python at all + (`GATE_COMPLEXIPY_MEASURED_NOTHING`, riding the caller's repo-wide + `py_present` so the vacuity leg and the absent-snapshot doctrine cannot + disagree). Not machine-caught: a floor a human zeroes by hand and commits is + green — the ratchet just goes vacuous, and only the + `complexipy-snapshot.json` diff in review shows it. The + complexipy step IS in `quality-gate.yml` (and `self-ci.yml`) as of + 2026-06-11, with the mypy-style presence rule: a Python repo without its + committed snapshot FAILS; only a Python-free repo skips, visibly. - **cf-exemptions — count-based by design**, and honestly so: `frozen_count` refuses silent growth, but a 1-for-1 entry swap at equal count is machine-visible only as an `exemptions.json` diff in review, not diff --git a/README.md b/README.md index 6aad0be..82bce8e 100644 --- a/README.md +++ b/README.md @@ -67,9 +67,14 @@ carries a dated ratchet ticket so green-by-baseline cannot become green-forever) ```bash cf-file-budget init # freeze existing >500-line files at measured size mypy src | mypy-baseline sync -complexipy src # cognitive-complexity snapshot (second metric) +complexipy src --snapshot-create # cognitive-complexity floor (second metric) ``` +`--snapshot-create` is not optional here, and a bare `complexipy src` is never the +right command: with a snapshot present the tool's own *passing* compare REWRITES the +floor (measured on the pinned 5.6.0 — see `configs/BASELINE-CONVENTIONS.md` §1). +Run it from the repo root; the snapshot lands in CWD, not in the analyzed path. + ## Development ```bash diff --git a/configs/BASELINE-CONVENTIONS.md b/configs/BASELINE-CONVENTIONS.md index f488fd4..0deb376 100644 --- a/configs/BASELINE-CONVENTIONS.md +++ b/configs/BASELINE-CONVENTIONS.md @@ -1,9 +1,13 @@ # Baseline conventions — mounting the ratchet on a consumer repo How a repo boots green-by-construction on day one and then only ever shrinks. -Every workflow below is the **observed** behavior of the pinned tools -(complexipy 5.5.0 · mypy-baseline 0.7.4), measured in `docs/tool-spikes.md` — -not read off a README. Every baseline carries a dated ratchet ticket so +Every workflow below is the **observed** behavior of the pinned tools, measured +in `docs/tool-spikes.md` — not read off a README. Version provenance, stated +because a stale label is exactly how a false claim survives here: the spike was +run on **complexipy 5.5.0 · mypy-baseline 0.7.4**; mypy-baseline is still pinned +at 0.7.4, but complexipy is now pinned at **5.6.0** and only its +snapshot-**write** semantics have been re-measured there (2026-07-28 — the first +complexipy gotcha below). Every baseline carries a dated ratchet ticket so green-by-baseline cannot become green-forever (the Law-1 refuter's expiry fix). ## The shared configs in this directory @@ -31,26 +35,90 @@ Exit 0 even when offenders exist — baseline boot is green by construction. The snapshot records only functions over the threshold; an all-clean tree writes a literal `[]`. -**Gate (every CI run, from the repo root):** +**Gate (every CI run, from the repo root) — `cf-gate`, never the tool's own +compare:** ```bash -complexipy src # plain run auto-compares when the snapshot exists +cf-gate # the complexipy stage measures write-free, grades in-kit ``` Fails (exit 1) on any NEW offender, or any baselined offender rising above its -watermark. A function at-or-below its watermark passes. +watermark — complexipy's own rule, applied by `cf_quality.complexipy_ratchet` +over a floor the tool is never allowed to touch. A function at-or-below its +watermark passes. Do **not** mount a bare `complexipy src` as the gate; the first +gotcha is why. **Observed gotchas the mount must respect:** -- The snapshot **does NOT auto-shrink**. After an improvement the plain run - passes but the file still holds the old watermark — a later regression back - up to the stale watermark would pass. The kit owns the re-baseline step: - re-run `--snapshot-create` on merge (or in the shrink ticket) to lock - improvements in. -- Run from the **repo root** so the committed snapshot is the one compared - (snapshot path is CWD-relative). -- Never pipe the gate command (`complexipy … | tail` masks the exit code); - gate on the command's own status, or set `pipefail`. +- **A passing plain-run compare REWRITES the snapshot** — measured on the pinned + **complexipy 5.6.0** (2026-07-28), correcting the 5.5.0 entry that used to + stand here and claim the file never shrinks itself. The destructive call site is + `complexipy/utils/snapshot.py::handle_snapshot_watermark`, which calls + `create_snapshot_file(...)` on its **no-violation** branch — the tool's green + path is its write path. Reproduced twice at exit 0, a populated snapshot + rewritten to `[]`: once by grading a path narrower than the floor describes, + once by raising the threshold above every function. Committed, either one + deletes the watermark forever behind a green run. `cf-gate` therefore measures + write-free with `--plain --color no --snapshot-ignore` (verified on 5.6.0 to + leave the file byte-unchanged) and grades the ratchet itself. +- **Your own `complexipy` config can defeat the gate, so the gate REFUSES instead + of measuring through it.** `--snapshot-ignore` disarms the compare's rewrite, but + `snapshot-create` is a **separate branch** (`main.py:323`) resolved CLI-first, + TOML-second (`utils/toml.py:235-240`) — and `--snapshot-create` has no negating + flag, so `cf-gate` cannot override it from the command line. So a + `complexipy.toml` / `.complexipy.toml` / `[tool.complexipy]` carrying any of + `snapshot-create`, `quiet`, `ratchet`, `failed`, `details = "low"`, + `ignore-complexity`, `report-ignored`, `output`, `output-format`, or the legacy + `output-csv|json|gitlab|sarif` fails the stage typed, naming the key and the file + (`GATE_COMPLEXIPY_CONFIG_DEFEATS_MEASUREMENT`, exit 2). **Remove the key** — there + is no workaround to reach for, because the two you actually want are already + honoured: `max-complexity-allowed` (your threshold — the kit declares none of its + own) and `exclude` (your surface). `no-ignore`, `check-script` and `sort` are + honoured too. An unparseable config is also a refusal + (`GATE_COMPLEXIPY_CONFIG_UNREADABLE`) — unreadable is not absent. +- **Raising `max-complexity-allowed` above a committed watermark fails the gate** + (`GATE_COMPLEXIPY_THRESHOLD_RAISED`, exit 2) rather than quietly grading an empty + offender set. If you mean to raise the bar, raise it and **re-boot the floor at + the new threshold** in the same commit, so the floor and the bar agree. +- **The re-snapshot duty is still yours, in a narrower shape.** With the write + disarmed the floor holds still, which means a shrink is not locked in merely by + passing: the watermark rule is a `>` bound, so a function that got simpler and + later climbs back to its stale watermark passes. Re-run + `complexipy --snapshot-create` from the repo root, deliberately, + and commit it — that is the shrink ticket's job. What is MECHANICAL now, stated + exactly (an earlier draft of this file claimed a floor file emptied of functions + fails — it does **not**: `--plain` lists every measured function regardless of + threshold, so a file whose functions all dropped below the bar is still measured + and still green): + - a floor file that exists but was not **measured** at all — excluded, + ignore-commented, outside the surface, or now functionless — fails + (`COMPLEXIPY_SNAPSHOT_FILE_UNMEASURED`); + - a floor file outside the graded source root fails + (`COMPLEXIPY_SURFACE_NARROWED`); + - a floor **function** missing from a file that WAS measured fails + (`COMPLEXIPY_SNAPSHOT_FUNCTION_UNMEASURED`) — that is the shape a + `# complexipy: ignore` comment takes, and it is an unregistered exemption from + this gate, so it is refused rather than absorbed; + - a run that measured **zero functions** can no longer report clean while the + floor names functions or the repo contains Python at all + (`GATE_COMPLEXIPY_MEASURED_NOTHING`). + + Every one of those messages names the re-boot command as its remedy. Still on you, + not on the gate: a wholly deleted floor file is a legitimate improvement and stays + green, so the snapshot keeps dead entries until a re-boot clears them — and a floor + someone zeroes by hand and commits is green too, visible only as a + `complexipy-snapshot.json` diff in review. +- Run the **boot** from the **repo root** so the snapshot lands beside the code it + describes (its path is CWD-relative, and the gate reads it from the repo root). +- Never pipe a hand-run complexipy command (`complexipy … | tail` masks the exit + code). `cf-gate` runs it with a fixed argv and no shell, and does not *grade* on + its exit code — but it does **cross-check** it: 0 and 1 are the tool's only + verdicts, 1 being the ordinary "some function is over threshold", so exit 1 with an + empty census, or any other non-zero code, is the instrument failing and refuses + distinctly (`GATE_COMPLEXIPY_INSTRUMENT_FAILED`, carrying the exit code and a + stderr excerpt) instead of being charged to your code. A file complexipy cannot + parse likewise refuses (`GATE_COMPLEXIPY_PATHS_UNMEASURABLE`, exit 2, not exit 1): + a narrower surface than the tree makes every other clean reading unsupported. ## 2. mypy-baseline — the 0-new-type-errors gate (multiset set-difference) diff --git a/docs/tool-spikes.md b/docs/tool-spikes.md index 2264749..6c85e6e 100644 --- a/docs/tool-spikes.md +++ b/docs/tool-spikes.md @@ -1,8 +1,21 @@ -# Tool spikes — observed semantics (2026-06-10) +# Tool spikes — observed semantics (2026-06-10; §1 write semantics re-measured 2026-07-28) Real behavior of the two pinned ratchet engines, measured on throwaway fixtures in `/tmp` against the kit's venv. Every claim below was observed, not read off a -README. Versions: **complexipy 5.5.0** · **mypy-baseline 0.7.4** (mypy 1.x). +README. Versions **as measured in the original spike**: **complexipy 5.5.0** · +**mypy-baseline 0.7.4** (mypy 1.x). The pin has since moved — read the skew note. + +> **Version skew — read before trusting §1 (2026-07-28).** `pyproject.toml` pins +> **complexipy 5.6.0**; this spike was run against 5.5.0 and the label was never +> refreshed when the pin moved. Only the snapshot-**write** semantics have been +> re-measured on 5.6.0: the first entry under "Gotchas for the kit", the +> `handle_snapshot_watermark` call site, and the write-free measurement argv. +> The flag table, the snapshot file format and the ratchet exit-code table below +> still carry their original **5.5.0** observation and are unverified against the +> pin — treat each as a claim until re-run. §2's mypy-baseline 0.7.4 is still the +> pinned version. The reason this note exists: a stale version header let a +> flatly false claim about the snapshot survive in this file from 5.5.0 into a +> 5.6.0 world, and the kit's ratchet was designed against it. > **Determinism note (2026-06-16).** The whole `[dev]` gauge battery is now pinned > EXACTLY in `pyproject.toml` (`ruff==0.15.17`, `mypy==2.1.0`, @@ -17,7 +30,7 @@ README. Versions: **complexipy 5.5.0** · **mypy-baseline 0.7.4** (mypy 1.x). --- -## 1. complexipy 5.5.0 — snapshot / watermark ratchet +## 1. complexipy — snapshot / watermark ratchet (5.5.0; writes re-measured on 5.6.0) Fixture: a package with `simple` (CC 1), `branchy` (CC 4), later a deliberately nested `monster` (cognitive complexity 33). Note complexipy measures **cognitive** @@ -62,21 +75,154 @@ Shape: ### Gotchas for the kit -- **The snapshot does NOT auto-shrink.** After an improvement the plain run - passes but `complexipy-snapshot.json` still holds the old watermark; only - `--snapshot-create` rewrites it. The kit's ratchet step must re-create the - snapshot on merge (or a shrink ticket does) or improvements are not locked in - — a later regression back up to the stale watermark would pass. -- **Snapshot lands in CWD** — the CI step must run from the repo root so the - committed snapshot is the one compared. -- **Exit-code caution:** piping output (`complexipy … | tail`) masks the exit - code; gate on the command's own status (or `pipefail`). -- A function exactly **at** its watermark passes — the watermark is a ≤ bound, - shrink-only happens via re-snapshot, not via the compare itself. - -**Verdict:** `--snapshot-create` + plain-run compare gives exactly the -freeze/shrink-only ratchet the plan wanted (plan Q2 answered). The kit must own -the re-baseline step to make shrink sticky. +- **CORRECTED 2026-07-28 — a passing plain-run compare REWRITES the snapshot, so + the tool shrinks its own floor, all the way to `[]` in the reproduced case.** + The 5.5.0 entry that stood here asserted the opposite (that the file never + shrinks itself, and that `--snapshot-create` is the only thing that rewrites + it). Both halves are false on the pinned **complexipy 5.6.0**. The destructive + call site is `complexipy/utils/snapshot.py::handle_snapshot_watermark`, which + calls `create_snapshot_file(...)` on its **no-violation** branch: the tool's + GREEN path *is* its write path. Reproduced twice at **exit 0** against a + populated snapshot, each time leaving `[]` behind — (a) **narrowed surface**: + compare over a path narrower than the floor describes, nothing in that subset + violates, the floor is rewritten to the subset; (b) **raised threshold**: + `-mx 100` puts every function under the bar, same rewrite. Committed, either + one deletes the watermark forever behind a green gate. This is why the kit + never lets the tool near the artifact: `cf-gate` measures with + `--plain --color no --snapshot-ignore` (which makes + `should_run_snapshot_watermark` False — the only other caller of + `create_snapshot_file` is the never-passed `--snapshot-create`) and grades the + ratchet in its own pure function, `cf_quality.complexipy_ratchet.grade`, over + (committed floor, measured census). Also measured on 5.6.0: that argv over a + 142-file tree emitted 681 stdout lines, every one a + ` ` census row, **0 bytes on stderr**, and left + `complexipy-snapshot.json` byte-unchanged. +- **`--snapshot-ignore` alone is NOT enough — the consumer's own config can put the + write back, so the kit AUDITS that config and refuses.** Read in the source, not + inferred: `snapshot_create` resolves CLI-first-then-TOML + (`utils/toml.py:235-240`), the kit passes no CLI value for it, and + `handle_snapshot_file_creation` (`main.py:323`) is an **entirely separate branch** + from the watermark compare that `--snapshot-ignore` disarms. `--snapshot-create` + is declared with no negating secondary name (`main.py:97-102`), so argv cannot + force it off. A consumer's `complexipy.toml` / `.complexipy.toml` / + `[tool.complexipy] snapshot-create = true` therefore made **both** measurement + runs rewrite the floor while the gate graded the pre-write bytes it had already + read — green, artifact silently mutated. Two more keys defeat the run the same + way: `quiet = true` (rejected beside `--plain`, `main.py:748` → exit 2, empty + census, which the gate used to report as `GATE_COMPLEXIPY_MEASURED_NOTHING` and + blame on the repo) and `output-format` / the legacy `output-csv|json|gitlab|sarif` + (a report file written into the consumer's tree, plus `Results saved at …` + (`main.py:510`) and `Deprecated: …` (`main.py:521-536`) printed on **stdout, + unguarded by `plain`, BEFORE the census**). `cf_quality.complexipy_measure` + `audit_config` reads the same file the tool reads, in the tool's own order + (`utils/toml.py:135-148`: `complexipy.toml`, then `.complexipy.toml`, then + `pyproject.toml`, first hit wins — and for the first two an EMPTY document counts + as a hit, so it shadows `pyproject.toml`), and REFUSES with the key and the file + named (`GATE_COMPLEXIPY_CONFIG_DEFEATS_MEASUREMENT`). Neutralising it by moving + the cwd was rejected for cause: `main.py:66-67` resolves the config, + `main.py:321` the snapshot path and `main.py:308-310` every reported path from the + **same `os.getcwd()`**, so a different cwd re-keys the whole census. Deliberately + still honoured: `max-complexity-allowed`, `exclude`, `no-ignore`, `check-script`, + `sort`. +- **A raised threshold empties the offender set, and the ratchet must refuse rather + than grade nothing.** With `max-complexity-allowed` raised (by `-mx` or by + config), the census still lists every function — so every floor file is present + and the surface audit is silent — while `--failed` returns **nothing**, and a + subset check over an empty subset passes trivially. The closure needs no second + threshold authority: a floor entry is proof that function was ABOVE the bar when + the floor was booted (`create_snapshot_file` stores only over-threshold + functions), so a census value at-or-above its watermark that the offender run does + NOT report means `threshold_now >= census >= watermark > threshold_at_boot` + (`GATE_COMPLEXIPY_THRESHOLD_RAISED`). A hand-lowered watermark lands in the same + refusal, correctly — the tool would never have written one at or below its own + threshold. +- **Exit semantics, established from the source** (`main.py:756-767` + `resolve_final_success` reduces to `has_success and valid_paths` with the compare + disarmed; `has_success` is `not failing_functions`, `utils/output.py:44`, which is + filled for every over-threshold function whether or not `--failed` was passed, + `utils/output.py:234-241`). So **0 and 1 are the tool's only verdicts**, 1 being + the legitimate "some function is over threshold" — the normal state of a repo + carrying a floor. Exit 1 with an **empty** census is a contradiction (and is the + offender run's second vacuity leg); any other code is `typer` declining to run + (`BadParameter` → 2 at `main.py:748`, `validate_ratchet` → 2 at + `main.py:726-729`). The kit refuses distinctly on both + (`GATE_COMPLEXIPY_INSTRUMENT_FAILED`) and carries the exit code plus a stderr + excerpt in the refusal context, because a tool-side failure re-attributed to the + repo with its reason deleted is worse than no gate. +- **`--plain` prints EVERY measured function, over threshold or not** + (`utils/output.py:234` drops a row only under `failed_only`; `:243` appends every + other one). That is load-bearing twice over: it is why a function that merely got + simpler is still in the census, and therefore why a floor function absent from a + file that WAS measured means something else — renamed, deleted, or dropped by a + `# complexipy: ignore` comment (`COMPLEXIPY_SNAPSHOT_FUNCTION_UNMEASURED`). The + argv passes neither `--no-ignore` nor `--report-ignored`, so an ignore comment + drops a function from both runs; that is caught structurally now rather than by + flag. **Open for the operator:** adding `--no-ignore` to the gate argv would need + the boot command in `configs/BASELINE-CONVENTIONS.md` to carry the same flag, or + every currently-ignored function floods in as a new offender — a paired change, + not a unilateral one. +- **A path complexipy cannot analyze is a REFUSAL, not a finding** — declared, since + it is a taxonomy change: `print_invalid_paths` → `has_success=False` used to make + it exit 1. `GATE_COMPLEXIPY_PATHS_UNMEASURABLE` exits 2 instead, because an + unparseable file means the graded surface is narrower than the tree, so every + OTHER function's clean reading is unsupported and a findings-level report would + let the rest of the board read green beside a void. A consumer who lands a syntax + error gets a setup-error board; ruff and mypy red independently, so nothing is + hidden. +- **The re-snapshot duty survived the fix — it changed shape, it did not go + away.** With the write disarmed the floor holds still in both directions, so a + shrink is still not locked in merely by passing: the watermark rule is a `>` + bound (`complexipy_ratchet._regressions` mirrors the tool exactly), so a + function that got simpler and later climbs back to its stale watermark passes. + Locking a shrink in is a deliberate, committed + `complexipy --snapshot-create` from the repo root — the shrink + ticket's job. What changed is that forgetting is no longer silent where it used to + be catastrophic. Precisely (corrected 2026-07-28 — an earlier pass on this branch + claimed "an improvement that empties a floor file of functions … FAILS + `COMPLEXIPY_SNAPSHOT_FILE_UNMEASURED`", which is **false**: `--plain` without + `--failed` lists every measured function regardless of threshold, so a file whose + functions all dropped below the bar stays in `census.files` and the gate is + GREEN): + - `COMPLEXIPY_SNAPSHOT_FILE_UNMEASURED` fires when the file was not **measured** + at all — excluded, ignore-commented, outside the measurement surface, or now + containing no function whatsoever — while still existing on disk. + - `COMPLEXIPY_SURFACE_NARROWED` fires when a floor file exists but lies outside + the graded source root. + - `COMPLEXIPY_SNAPSHOT_FUNCTION_UNMEASURED` fires when the file WAS measured and + a floor function inside it was not, which the two rules above cannot see. + - A run that measured **zero functions** can no longer report clean while the + floor still names functions or the repo contains Python at all + (`GATE_COMPLEXIPY_MEASURED_NOTHING`); the second leg is the caller's repo-wide + `py_present`, the same answer the absent-snapshot doctrine rides, so the two + surfaces cannot disagree about whether there was anything to grade. + - Every one of those messages names the re-boot as its remedy, spelled with the + resolved graded root. + + Not caught, honestly: a wholly **deleted** floor file is a legitimate improvement + and stays green, so the snapshot accumulates dead entries until someone re-boots + it, and a floor a human zeroes by hand and commits is green as well — review of + the `complexipy-snapshot.json` diff is the only instrument on that one. +- **Snapshot lands in CWD** — the boot command must run from the repo root, so the + committed snapshot sits where the kit reads it from. +- **Exit-code caution:** piping output (`complexipy … | tail`) masks the exit code; + never pipe a hand-run command. The kit does not *grade* on that exit code — the + grade is `complexipy_ratchet.grade` over (floor, census) — but it does + **cross-check** it, per the exit semantics above: a code that contradicts the + census is the instrument failing, and the kit refuses instead of charging it to + the code. +- A function exactly **at** its watermark passes the watermark rule — a `>` bound, + mirrored exactly; shrink-only happens via re-snapshot, not via the compare itself. + It must however still appear in the `--failed` run, or the bar has moved + (`GATE_COMPLEXIPY_THRESHOLD_RAISED` above) — "at its watermark" and "no longer an + offender at all" are different worlds. + +**Verdict (revised 2026-07-28, measured on the pinned 5.6.0).** +`--snapshot-create` gives the freeze the plan wanted (plan Q2 answered), but the +plain-run **compare** is unusable as a gate: the run that passes is the run that +rewrites the floor it just graded. So the kit demotes complexipy to a measuring +instrument (`--plain … --snapshot-ignore`) and owns both halves of the ratchet — +the comparison, in `complexipy_ratchet.grade`, and the re-baseline step that +makes a shrink sticky. --- diff --git a/src/cf_quality/complexipy_floor.py b/src/cf_quality/complexipy_floor.py new file mode 100644 index 0000000..c8ca1ee --- /dev/null +++ b/src/cf_quality/complexipy_floor.py @@ -0,0 +1,86 @@ +"""The committed cognitive-complexity floor — the ONE place that touches the artifact. + +``complexipy-snapshot.json`` is the per-function watermark no later run may +exceed. The whole point of this rung is that the pinned ``complexipy==5.6.0`` +REWRITES that file on the success path of its own compare +(``complexipy/utils/snapshot.py:85``), so the artifact needs an owner that reads +it and never writes it. That owner is this module, deliberately alone in it: one +importable surface to audit, and a reviewer can prove "the gate never writes the +floor" by reading forty lines instead of grepping a battery. + +Two invariants live here and nowhere else: + +- **The KEY.** :func:`_normalized_path` is a declared mirror of + ``complexipy.utils.output.normalize_path`` (5.6.0, lines 274-280): the snapshot + stores ``path`` and ``file_name`` separately while ``--plain`` prints them + joined. Join them differently and every committed watermark reads as a + brand-new offender. The mirror is pinned by a test calling the INSTALLED + function, not by a declaration file — a 5.7.0 change to that join must fail the + suite, not wait for a reviewer to notice. +- **Unreadable is not empty.** A malformed floor REFUSES. Reading it as ``[]`` + would be green-by-unreadable-file, the same gaming vector as + green-by-missing-file, which the caller already refuses. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +from cf_quality.complexipy_measure import FunctionKey, refusal +from cf_quality.errors import GateError + +#: The committed floor's filename (CWD-relative for complexipy, repo root for us). +SNAPSHOT_FILENAME = "complexipy-snapshot.json" + + +def _normalized_path(path: str, file_name: str) -> str: + """Join a snapshot entry's two path fields the way complexipy's output does.""" + cleaned = path.rstrip("/") + if cleaned.endswith(file_name): + return cleaned + return f"{cleaned}/{file_name}" if cleaned else file_name + + +def _refuse_snapshot(snapshot: Path, reason: str) -> GateError: + return refusal( + "GATE_COMPLEXIPY_SNAPSHOT_UNREADABLE", + f"{SNAPSHOT_FILENAME} is not a readable complexipy snapshot: {reason} — an " + "unreadable floor is not an empty floor, so the gate refuses; re-boot it from " + "the repo root (complexipy --snapshot-create) and commit it", + {"snapshot": str(snapshot), "reason": reason}, + ) + + +def _entry_watermarks(entry: object, snapshot: Path) -> dict[FunctionKey, int]: + """One snapshot entry's watermarks; any other shape REFUSES rather than skips.""" + if not isinstance(entry, dict) or not isinstance(entry.get("functions"), list): + raise _refuse_snapshot(snapshot, f"entry is not {{path, file_name, functions}}: {entry!r}") + path = _normalized_path(str(entry.get("path", "")), str(entry.get("file_name", ""))) + watermarks: dict[FunctionKey, int] = {} + for function in entry["functions"]: + if not isinstance(function, dict) or not isinstance(function.get("complexity"), int): + raise _refuse_snapshot(snapshot, f"function is not {{name, complexity}}: {function!r}") + watermarks[(path, str(function.get("name", "")))] = int(function["complexity"]) + return watermarks + + +def read_snapshot(snapshot: Path) -> dict[FunctionKey, int]: + """The committed floor as ``{(file, function): watermark}`` — read, never written. + + The catch is ``ValueError``, which subsumes both ``json.JSONDecodeError`` and + the ``UnicodeDecodeError`` a non-UTF-8 snapshot raises. The narrower + ``JSONDecodeError`` let that one escape this function, escape + ``gate_runner._run_stage`` (which catches only ``GateError``) and lose the + entire 12-stage board to a traceback instead of one typed refusal. + """ + try: + raw = json.loads(snapshot.read_text(encoding="utf-8")) + except (OSError, ValueError) as exc: + raise _refuse_snapshot(snapshot, str(exc)) from exc + if not isinstance(raw, list): + raise _refuse_snapshot(snapshot, f"top level is {type(raw).__name__}, not a list") + floor: dict[FunctionKey, int] = {} + for entry in raw: + floor.update(_entry_watermarks(entry, snapshot)) + return floor diff --git a/src/cf_quality/complexipy_measure.py b/src/cf_quality/complexipy_measure.py new file mode 100644 index 0000000..7f6fec7 --- /dev/null +++ b/src/cf_quality/complexipy_measure.py @@ -0,0 +1,466 @@ +"""complexipy demoted to a measuring instrument — audited, invoked write-free, read. + +**Why this sits apart from the ratchet.** The ratchet +(:mod:`cf_quality.complexipy_ratchet`) is a pure function of (committed floor, +measured census). Everything that must be TRUE before that function may be +trusted lives here: the consumer's own complexipy configuration, the argv, the +two subprocess runs, and the parse of what came back. One responsibility — +"produce a census the ratchet may reason over, or refuse" — and the ratchet +never touches a subprocess. + +**The write-free claim, and its real precondition.** ``--snapshot-ignore`` alone +does NOT make the run write-free; that was the defect two refuters found in the +first cut of this fix, and it is the same defect the whole rung exists to +remove — a false invariant in a docstring. In the pinned ``complexipy==5.6.0`` +exactly two callers reach ``create_snapshot_file``: + +1. ``complexipy/utils/snapshot.py:85`` — ``handle_snapshot_watermark``'s + no-violation branch, i.e. the compare's SUCCESS path. ``--snapshot-ignore`` + switches that off (``main.py:332``: ``should_run_snapshot_watermark = + snapshot_file_exists and not snapshot_ignore``) and a consumer cannot switch + it back on, because the CLI value wins (``utils/toml.py:179``: + ``get_argument_value`` returns ``arg_value`` whenever it is not None). +2. ``main.py:323`` — ``handle_snapshot_file_creation``, an ENTIRELY separate + branch driven by ``snapshot-create``, which ``--snapshot-ignore`` does not + touch. The kit passes no CLI value for it, so ``utils/toml.py:235-240`` + resolves it from the consumer's ``complexipy.toml`` / ``.complexipy.toml`` / + ``[tool.complexipy]``; and ``--snapshot-create`` is declared with no negating + secondary name (``main.py:97-102``), so argv CANNOT force it off. + +So the argv's guarantee holds only while the consumer's config carries no +``snapshot-create``, and :func:`audit_config` is what makes it hold: it reads +the SAME file the tool will read, in the tool's own search order, and REFUSES +the stage when a key would let the run write the floor, silence the census, or +print anything else onto the census stream. A typed refusal naming the key and +the file cannot be silently wrong. A cwd or env trick could be, and was +rejected for cause: ``main.py:66-67`` resolves the config, ``main.py:321`` the +snapshot path, and ``main.py:308-310`` every reported path from the SAME +``os.getcwd()``, so moving the cwd to hide the config would re-key the entire +census and break the floor comparison — and would silently drop the LEGITIMATE +``max-complexity-allowed`` this design deliberately honours. + +**What is deliberately NOT refused.** ``max-complexity-allowed`` and ``exclude`` +are the consumer's own threshold and surface authorities; the kit declares no +budget of its own here, because a second authority would flag a baselined band +as a new offender. ``no-ignore`` and ``check-script`` only ever make the census +LARGER. ``sort`` reorders rows only, and :func:`parse_census` aggregates +``max()`` per key, so no order can change the result. + +**Reading the instrument.** Three channels come back and all three are evidence: +stdout (the census), the exit code, and stderr. This argv was measured over a +142-file tree emitting 681 census rows and **0 bytes of stderr**, so any stderr +at all rides the refusal context. The exit code is corroboration, never the +grade: with the compare disarmed, ``main.py:756-767 resolve_final_success`` +reduces to ``has_success and valid_paths``, and ``has_success`` is ``not +failing_functions`` (``utils/output.py:44``), which is filled for every function +over the resolved threshold whether or not ``--failed`` was passed +(``utils/output.py:234-241``). Exit 1 is therefore the legitimate "some function +is over threshold"; exit 1 with an EMPTY census is a contradiction; any other +non-zero code is the tool declining to run. +""" + +from __future__ import annotations + +import subprocess +import tomllib +from collections.abc import Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Protocol + +from cf_quality.errors import GateError + +#: ``(repo-relative file, function name)`` — the identity a watermark is keyed by. +#: complexipy keys on ``(path, file_name, name)``; ``path`` already carries the +#: file name in its own output, so the joined form is the same identity. +FunctionKey = tuple[str, str] + +#: The config files complexipy itself consults, in ITS order +#: (``utils/toml.py:135-148`` ``get_complexipy_toml_config``): first hit wins, and +#: a hit is any file that exists and yields a table — so an EMPTY +#: ``complexipy.toml`` shadows a populated ``pyproject.toml``, and the audit must +#: agree with that or it would audit a file the tool never reads. +CONFIG_FILENAMES = ("complexipy.toml", ".complexipy.toml", "pyproject.toml") + +#: Pinned onto every measurement run. ``COLUMNS`` is load-bearing: ``--plain`` +#: prints through rich, which wraps at 80 columns when stdout is not a terminal, +#: and a wrapped census row is an unparseable census row. ``TERM`` is load-bearing +#: for the same reason at one remove — ``rich/console.py:1015-1016`` returns a HARD +#: 80x25 and never reads ``COLUMNS`` at all when ``is_dumb_terminal``, which is +#: ``is_terminal and TERM in ("dumb", "unknown")`` (``rich/console.py:986-988``). +#: ``PYTHONIOENCODING`` pins the child's encoding so the census does not emit by +#: locale. +_MEASURE_ENV = { + "COLUMNS": "10000", + "PYTHONIOENCODING": "utf-8", + "NO_COLOR": "1", + "TERM": "xterm", +} + +#: Cleared for every measurement run: these are the two env names that force +#: rich's ``is_terminal`` True over a pipe (``rich/console.py:955-963``), which is +#: the other half of the dumb-terminal short circuit above. +_MEASURE_ENV_CLEARED = frozenset({"FORCE_COLOR", "TTY_COMPATIBLE"}) + +#: complexipy's own words when it could not analyze a path it was handed +#: (``utils/output.py:297`` ``print_invalid_paths``) — a silently narrower surface. +_UNMEASURABLE_MARKER = "Failed to process" + +#: Every consumer config key that would defeat THIS measurement, with why. Derived +#: by walking the tool's own CLI-first/TOML-second resolver +#: (``utils/toml.py:193-333`` ``get_arguments_value``) for the keys this argv +#: leaves unset — those are exactly the keys a consumer's config can still win. +_DEFEATING_KEYS = { + "snapshot-create": ( + "would make the run WRITE complexipy-snapshot.json from what it just " + "measured (main.py:323, a branch --snapshot-ignore does not disarm)" + ), + "quiet": ( + "is rejected together with --plain, so the run exits 2 with an empty " + "census and the gate would blame the repo (main.py:748)" + ), + "ratchet": ( + "requires --diff, which this argv never passes, so the run exits 2 " + "before measuring anything (main.py:726-729)" + ), + "failed": ( + "would narrow the CENSUS run to offenders too, and the census listing " + "every measured function is what the surface audit reasons over" + ), + "details": ( + 'the legacy spelling of failed — "low" means offenders only (utils/toml.py:259-261)' + ), + "ignore-complexity": ( + "would decouple the exit code from the tool's own threshold verdict, " + "and the exit code is how a contradicting run is caught" + ), + "report-ignored": ( + "prints one ignore-comment location per line onto the census stream " + "(main.py:701-708), which is not a census row" + ), + "output": "redirects machine-readable output into the consumer's tree", + "output-format": ( + "writes a report file and prints 'Results saved at ...' onto stdout " + "BEFORE the census, unguarded by --plain (main.py:510)" + ), + "output-csv": "the legacy output-format spelling — same write, plus a Deprecated line", + "output-json": "the legacy output-format spelling — same write, plus a Deprecated line", + "output-gitlab": "the legacy output-format spelling — same write, plus a Deprecated line", + "output-sarif": "the legacy output-format spelling — same write, plus a Deprecated line", +} + + +class Executor(Protocol): + """The subprocess seam the caller injects (``gate_runner._exec``). + + Structural, not inherited: the runner stays the caller's — one place owns + typed OSError translation and the no-shell fixed-argv discipline — while the + tests keep patching that single seam. + """ + + def __call__( + self, + argv: list[str], + cwd: Path, + env: Mapping[str, str], + *, + stdin: str | None = None, + ) -> subprocess.CompletedProcess[str]: ... + + +@dataclass(frozen=True) +class Census: + """What ONE write-free complexipy run actually measured. + + The census is the gate's independent fact about its own measurement surface: + it is read from the tool's output, never from the snapshot, so "the floor is + empty" and "we graded nothing" can never be the same observation. + + ``functions`` is keyed by :data:`FunctionKey` and aggregated with ``max()``; + ``rows`` counts the census LINES parsed. The two genuinely differ and both + are load-bearing — on a real consumer 681 rows collapse to 680 keys, because + ``@overload`` chains and ``if sys.platform:`` redefinitions repeat a + ``(path, name)`` pair. Counting keys UNDER-reports the measurement, which is + the anti-void evidence this stage exists to produce; and last-writer-wins + would let the HIGHER value of a straddling pair vanish, making our mirror of + complexipy's watermark rule strictly weaker than the rule it mirrors (the + tool iterates its own LIST, not a map) and letting a bogus skew fire when + only one of the two maps kept the high value. + """ + + functions: dict[FunctionKey, int] + rows: int + + @property + def files(self) -> frozenset[str]: + """The distinct files this run measured — the surface audit's evidence.""" + return frozenset(path for path, _ in self.functions) + + +def refusal(code: str, message: str, context: Mapping[str, object]) -> GateError: + """A refusal in the kit's typed vocabulary — the gate could not do its job.""" + return GateError(code=code, message=message, context=dict(context)) + + +def _table(path: Path) -> Mapping[str, object] | None: + """One config file's complexipy table, or None when the tool would not use it. + + Mirrors ``utils/toml.py::load_toml_config`` exactly: ``pyproject.toml`` + contributes only its ``[tool.complexipy]`` sub-table, every other name + contributes the whole document — including an empty one, which is why an + empty ``complexipy.toml`` shadows ``pyproject.toml``. ``ValueError`` is the + catch because it subsumes both ``TOMLDecodeError`` and the + ``UnicodeDecodeError`` a non-UTF-8 config raises; letting either escape would + lose the whole battery to a traceback instead of one typed refusal. + """ + if not path.is_file(): + return None + try: + data = tomllib.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError) as exc: + raise refusal( + "GATE_COMPLEXIPY_CONFIG_UNREADABLE", + f"cannot read {path.name}: {exc} — complexipy resolves its options from " + "this file, so an unreadable one leaves the measurement unaudited; fix " + "the TOML (or delete the file) and re-run", + {"config": str(path), "reason": str(exc)}, + ) from exc + if path.name != "pyproject.toml": + return data + tool = data.get("tool") + if not isinstance(tool, dict): + return None + table = tool.get("complexipy") + return table if isinstance(table, dict) else None + + +def _defeats(key: str, value: object) -> bool: + """True when this key/value pair would defeat the measurement. + + Value-sensitive on purpose: ``snapshot-create = false`` provably writes + nothing (``utils/snapshot.py:17``), so refusing it would be a false refusal, + and a false refusal costs a consumer a red board for nothing. ``details`` is + the one non-boolean-shaped key and only bites at ``"low"``. + """ + if key not in _DEFEATING_KEYS: + return False + if key == "details": + return str(value).strip().lower() == "low" + return bool(value) + + +def audit_config(root: Path) -> str | None: + """The consumer's complexipy config, audited before the instrument is trusted. + + Returns the filename the tool will read (None when it reads none) so a + PASSING verdict can state which config was in force. Raises when that config + carries a key from :data:`_DEFEATING_KEYS` — the mechanism that turns this + module's write-free claim from an aspiration into an enforced precondition. + """ + for name in CONFIG_FILENAMES: + table = _table(root / name) + if table is None: + continue + defeating = sorted(key for key, value in table.items() if _defeats(str(key), value)) + if defeating: + detail = "; ".join(f"{key} ({_DEFEATING_KEYS[key]})" for key in defeating) + raise refusal( + "GATE_COMPLEXIPY_CONFIG_DEFEATS_MEASUREMENT", + f"{name} sets complexipy option(s) this gate cannot measure through: " + f"{detail} — remove them from {name}; max-complexity-allowed and " + "exclude ARE honoured, so a real budget or a real exclusion needs no " + "workaround here", + {"config": name, "keys": defeating}, + ) + return name + return None + + +def measurement_argv(tool: Path, source_root: Path, *, offenders_only: bool) -> list[str]: + """The write-free measurement command — the root of the fix, not a nicety. + + Six tokens plus an optional ``--failed``, and the ABSENCE of a threshold flag + is as load-bearing as the presence of ``--snapshot-ignore``: the kit never + re-declares a budget complexipy already resolves from its own default / CLI / + ``[tool.complexipy]``, because a second authority here would flag a + consumer's baselined band as a new offender. ``--failed`` narrows the census + to the functions the tool's OWN resolved threshold calls offenders. + """ + argv = [str(tool), str(source_root), "--plain", "--color", "no", "--snapshot-ignore"] + if offenders_only: + argv.append("--failed") + return argv + + +def _census_row(line: str) -> tuple[FunctionKey, int] | None: + """`` `` -> the keyed measurement, else None. + + Split from the RIGHT: the complexity and the function name are single tokens + while a path may contain spaces, so ``rsplit`` is the only safe direction. + """ + parts = line.strip().rsplit(maxsplit=2) + if len(parts) != 3 or not parts[0].endswith(".py"): + return None + try: + complexity = int(parts[2]) + except ValueError: + return None + return (parts[0], parts[1]), complexity + + +def parse_census(stdout: str, context: Mapping[str, object] | None = None) -> Census: + """complexipy ``--plain`` stdout -> the measured census. + + ``--plain`` is complexipy's documented scripting form: one + `` `` line per function it measured, over + threshold or NOT — ``utils/output.py:234`` drops a row only when + ``failed_only`` is set and ``:243`` appends every other one. That is why a + function which merely fell below the bar is still HERE, and why its absence + from a measured file means something else entirely (the ratchet's + per-function surface audit reasons on exactly that distinction). + + A non-blank line that is not a census row REFUSES instead of being dropped: + an unreadable census and an empty one look identical from a count, and the + empty one is the exact world this gate exists to catch. + """ + functions: dict[FunctionKey, int] = {} + unreadable: list[str] = [] + rows = 0 + for line in stdout.splitlines(): + row = _census_row(line) + if row is not None: + rows += 1 + functions[row[0]] = max(functions.get(row[0], row[1]), row[1]) + elif line.strip(): + unreadable.append(line.strip()) + if unreadable: + raise refusal( + "GATE_COMPLEXIPY_OUTPUT_UNREADABLE", + f"complexipy --plain emitted {len(unreadable)} line(s) that are not " + "' ' — the census cannot be trusted, so the " + "floor cannot be graded; check the repo root for a complexipy.toml or a " + "[tool.complexipy] table adding output to the run", + {**dict(context or {}), "lines": unreadable[:10], "measured_functions": rows}, + ) + return Census(functions=functions, rows=rows) + + +def _excerpt(text: str) -> list[str]: + """The leading non-blank lines of a stream — evidence, never the whole tail.""" + return [line.strip() for line in text.splitlines() if line.strip()][:5] + + +def _refuse_unmeasurable( + proc: subprocess.CompletedProcess[str], context: Mapping[str, object] +) -> None: + """complexipy's own words for a path it could not analyze — a narrower surface. + + Kept a REFUSAL rather than a finding, deliberately and now declared: an + unparseable file means the graded surface is narrower than the tree, so every + OTHER function's clean reading is unsupported. A findings-level report would + let the rest of the board read green beside a void. Both streams are scanned, + because which one carries it depends on rich's console wiring. + """ + marked = [ + line.strip() + for line in f"{proc.stdout}\n{proc.stderr}".splitlines() + if _UNMEASURABLE_MARKER in line + ] + if not marked: + return + raise refusal( + "GATE_COMPLEXIPY_PATHS_UNMEASURABLE", + f"complexipy could not analyze {len(marked)} path(s) — the graded surface is " + "narrower than the tree, so a clean verdict would be a void; fix the named " + "file(s) (a syntax error also reds ruff and mypy) and re-run the gate", + {**dict(context), "paths": marked[:10]}, + ) + + +def _refuse_instrument_failure( + proc: subprocess.CompletedProcess[str], context: Mapping[str, object] +) -> None: + """Refuse when the exit code contradicts the census, rather than blame the repo. + + Exit semantics taken from the pinned source, not from confidence (see this + module's docstring for the reduction): 0 and 1 are the tool's only verdicts, + 1 means "some function is over threshold", and that cannot be true of a run + which printed no function at all. That second clause is also the OFFENDER + run's vacuity leg — an empty ``--failed`` census beside a non-zero exit is a + contradiction — which is why one rule serves both runs. Any other code is + ``typer`` declining to run: ``BadParameter`` exits 2 (``main.py:748``), as + does ``validate_ratchet`` (``main.py:726-729``). + """ + if proc.returncode == 0: + return + empty = not proc.stdout.strip() + if proc.returncode == 1 and not empty: + return + detail = ( + "exit 1 means 'some function is over threshold', which no run that printed " + "zero functions can be reporting" + if proc.returncode == 1 + else "0 and 1 are the tool's only verdicts; anything else is it declining to run" + ) + raise refusal( + "GATE_COMPLEXIPY_INSTRUMENT_FAILED", + f"complexipy exited {proc.returncode} with " + f"{'an empty' if empty else 'a non-empty'} census — {detail}. This is the " + "instrument failing, not the repo, so the gate refuses instead of charging " + "it to the code; read `stderr` in this context, then check the repo root for " + "a complexipy.toml or a [tool.complexipy] table", + {**dict(context), "stdout_head": _excerpt(proc.stdout)}, + ) + + +def _measure_env(env: Mapping[str, str]) -> dict[str, str]: + """The caller's environment with the census pins ON and the tty forcers OFF.""" + kept = {key: value for key, value in env.items() if key not in _MEASURE_ENV_CLEARED} + return {**kept, **_MEASURE_ENV} + + +def measure( + root: Path, + source_root: Path, + env: Mapping[str, str], + tool: Path, + executor: Executor, + *, + offenders_only: bool, +) -> Census: + """Run one write-free measurement from the repo root and read its census. + + cwd is the repo root deliberately: complexipy resolves its config + (``main.py:66-67``), the snapshot path (``main.py:321``) and every reported + path (``main.py:308-310``) against ``os.getcwd()``, so measuring from + anywhere else would re-key the census and break the comparison. All three + channels are consulted, and the exit code plus a stderr excerpt ride EVERY + refusal raised here — a tool-side failure that reaches the board with its + reason deleted is a failure re-attributed to the repo. + + ``UnicodeDecodeError`` is caught at the seam because ``text=True`` decodes in + the PARENT, by the parent's locale — ``PYTHONIOENCODING`` pins the child + only. That exception is a ``ValueError``, so it would otherwise escape the + executor, escape ``gate_runner._run_stage`` (which catches ``GateError``) and + lose the whole 12-stage board to a traceback. + """ + argv = measurement_argv(tool, source_root, offenders_only=offenders_only) + try: + proc = executor(argv, root, _measure_env(env)) + except UnicodeDecodeError as exc: + raise refusal( + "GATE_COMPLEXIPY_OUTPUT_UNDECODABLE", + f"complexipy's output does not decode in this locale: {exc} — the census " + "is the only instrument the ratchet has, so the gate refuses rather than " + "grade a truncated world", + {"argv": argv, "offenders_only": offenders_only, "reason": str(exc)}, + ) from exc + context: dict[str, object] = { + "argv": argv, + "offenders_only": offenders_only, + "exit_code": proc.returncode, + "stderr": _excerpt(proc.stderr), + } + _refuse_unmeasurable(proc, context) + _refuse_instrument_failure(proc, context) + return parse_census(proc.stdout, context) diff --git a/src/cf_quality/complexipy_ratchet.py b/src/cf_quality/complexipy_ratchet.py new file mode 100644 index 0000000..514aa1c --- /dev/null +++ b/src/cf_quality/complexipy_ratchet.py @@ -0,0 +1,446 @@ +"""The cognitive-complexity ratchet, graded HERE — because the tool eats its own floor. + +**The measured defect.** ``complexipy-snapshot.json`` is the committed floor: the +per-function cognitive-complexity watermark no later run may exceed. In the +pinned ``complexipy==5.6.0`` the tool's OWN snapshot comparison ends, on success, +in a REWRITE of that file — ``complexipy/utils/snapshot.py:85`` +``handle_snapshot_watermark`` returns ``True`` only after calling +``create_snapshot_file(...)`` with the functions IT measured this run. The tool's +green path *is* the destructive path, and two ordinary runs reproduce it at exit +0: a **narrowed surface** (``complexipy `` measures a +subset, finds no violation, rewrites the floor to that subset — a populated +snapshot observed going ``1 entry -> []``) and a **raised threshold** +(``complexipy -mx 100`` puts everything under the bar, same rewrite). +Either one, committed, deletes the floor forever behind a green gate. + +**Why this module rather than a before/after repair.** Detecting the rewrite and +restoring the file would still let it happen, and would only work *because* the +tool destroys the artifact: the moment the write does not occur — a crash between +compare and write, a release that stops rewriting, a run that never reached the +compare — a before/after diff of the file is identical and the narrowed surface +goes invisible again. So the fix sits upstream of the write: + +1. **The gate never lets the tool near the artifact.** The write-free invocation + and the audit that makes it genuinely write-free live in + :mod:`cf_quality.complexipy_measure`; its docstring carries the two + ``create_snapshot_file`` call sites and the config key that would re-open the + second one. The floor itself has exactly one reader, + :mod:`cf_quality.complexipy_floor`, which never writes. +2. **The ratchet is our pure function.** :func:`grade` maps (committed floor, + measured census) to a :class:`~cf_quality.errors.GateVerdict` with no + subprocess and no write in the path — unit-testable, and nothing a green run + can silently destroy. +3. **complexipy is demoted to an instrument.** It answers two questions and + grades nothing: ``--plain`` (every function measured, with its complexity) + and ``--plain --failed`` (the subset ITS OWN threshold calls offenders). The + second run is why the kit never re-declares a budget complexipy already + resolves from its default / CLI / ``[tool.complexipy]`` config. + +**Threshold-neutrality, and the vacuity that hid inside it.** Because the kit +declares no budget, a consumer who RAISES complexipy's threshold empties the +offender set — and the first cut of this fix then graded an empty dict against a +populated floor and reported clean. That is +``gate-that-selects-by-the-value-it-guards-goes-vacuous`` inside the fix for it. +:func:`_bar_moved` closes it without naming a threshold: a floor entry proves +that function was ABOVE the bar when the floor was booted, so if its census value +still sits at-or-above its watermark and yet the offender run does not report it, +then ``threshold_now >= census >= watermark > threshold_at_boot`` — the bar moved, +provable from the census and the floor alone. The offender run's exit code is the +second, independent leg (``complexipy_measure._refuse_instrument_failure``). + +**The taxonomy.** Findings (exit 1, the repo's to fix): ``COMPLEXIPY_NEW_OFFENDER`` +and ``COMPLEXIPY_WATERMARK_REGRESSION`` (complexipy's own watermark rule over data +we own); ``COMPLEXIPY_SURFACE_NARROWED``, ``COMPLEXIPY_SNAPSHOT_FILE_UNMEASURED`` +and ``COMPLEXIPY_SNAPSHOT_FUNCTION_UNMEASURED`` (the floor names a file, or a +function inside a measured file, this run did not grade — the narrowed-surface +trigger, caught structurally, independent of any threshold). Refusals (exit 2, +the gate could not do its job): ``GATE_COMPLEXIPY_SNAPSHOT_UNREADABLE``, +``GATE_COMPLEXIPY_MEASUREMENT_SKEW``, ``GATE_COMPLEXIPY_THRESHOLD_RAISED``, +``GATE_COMPLEXIPY_MEASURED_NOTHING``, plus the instrument-side refusals +:mod:`cf_quality.complexipy_measure` raises. A legitimate improvement — a +function that got simpler, a deleted file — is GREEN; only re-booting the floor +locks it in, which stays the existing runbook duty. + +The absent-watermark doctrine is unchanged and stays in the caller +(``gate_runner._complexipy``): no snapshot + Python present REFUSES +``GATE_COMPLEXIPY_SNAPSHOT_MISSING``; a Python-free repo skips, visibly. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from pathlib import Path +from typing import Protocol + +from cf_quality.complexipy_floor import SNAPSHOT_FILENAME, read_snapshot +from cf_quality.complexipy_measure import ( + Census, + Executor, + FunctionKey, + audit_config, + measure, + refusal, +) +from cf_quality.errors import GateError, GateVerdict, GateViolation + +#: The stage name every verdict from this module carries. +GATE = "complexipy" + + +class _Layout(Protocol): + """The resolved consumer layout this stage reads (``gate_runner.Layout``). + + Structural, and read-only by declaration so a frozen dataclass satisfies it: + importing ``gate_runner.Layout`` would close an import cycle, since + ``gate_runner`` imports this module. ``py_present`` is threaded rather than + re-derived on purpose — the caller already computes the repo-wide answer for + the absent-snapshot doctrine, and a second local derivation is how the two + surfaces came to disagree (a Python-free tree under a present-but-empty + ``src/`` was reported PASS having measured nothing). + """ + + @property + def root(self) -> Path: ... + @property + def source_root(self) -> Path: ... + @property + def py_present(self) -> bool: ... + + +def _reboot(graded: str) -> str: + """The re-boot command, named as the remedy — a condition with no remedy is half a message.""" + return f"re-boot the floor from the repo root: complexipy {graded} --snapshot-create" + + +def _counts( + floor: Mapping[FunctionKey, int], census: Census, offenders: Census +) -> dict[str, object]: + """The measured tally every finding, refusal and PASS carries — never a bare verdict. + + ``measured_functions`` is the ROW count, not the key count: they differ + wherever a ``(path, name)`` pair repeats, and reporting keys would under-state + the very measurement this evidence exists to prove. + """ + return { + "measured_functions": census.rows, + "measured_keys": len(census.functions), + "measured_files": len(census.files), + "measured_offenders": offenders.rows, + "snapshot_functions": len(floor), + "snapshot_files": len({path for path, _ in floor}), + } + + +def _violation( + code: str, message: str, path: str, counts: Mapping[str, object], **detail: object +) -> GateViolation: + return GateViolation(code=code, message=message, path=path, context={**counts, **detail}) + + +def _regressions( + floor: Mapping[FunctionKey, int], + offenders: Census, + counts: Mapping[str, object], + reboot: str, +) -> list[GateViolation]: + """complexipy's own watermark rule, applied to data the tool cannot rewrite. + + Mirrors ``handle_snapshot_watermark`` exactly, including the ``>`` bound — a + function sitting AT its watermark passes; only rising above it fails. + """ + violations: list[GateViolation] = [] + for (path, name), value in sorted(offenders.functions.items()): + watermark = floor.get((path, name)) + if watermark is None: + code = "COMPLEXIPY_NEW_OFFENDER" + message = ( + f"{name} exceeds complexipy's threshold at {value} with no committed " + f"watermark — split it below the threshold, or baseline it deliberately: {reboot}" + ) + elif value > watermark: + code = "COMPLEXIPY_WATERMARK_REGRESSION" + message = ( + f"{name} rose above its committed watermark: {watermark} -> {value} — bring " + f"it back to {watermark} or below; raising the floor instead is a deliberate, " + f"reviewed act: {reboot}" + ) + else: + continue + violations.append(_violation(code, message, path, counts, function=name, measured=value)) + return violations + + +def _graded_prefix(root: Path, source_root: Path) -> str | None: + """The graded root as a repo-relative POSIX prefix (``""`` == the whole repo). + + The ROOTS are resolved, never the file: resolving the file sent a symlinked + module (``src/settings.py -> ../config/settings.py``) outside the graded root + and made the gate accuse a file complexipy really did measure of lying outside + the surface. The floor's own paths are already repo-relative, so containment + is a lexical test on those. ``None`` means the graded root is not inside the + repo at all, and the truthful reading of that is "every floor path is outside + it" — not a crash, and not a pass. + """ + try: + relative = source_root.resolve().relative_to(root.resolve()) + except ValueError: + return None + text = relative.as_posix() + return "" if text == "." else text + + +def _inside(path: str, prefix: str | None) -> bool: + """Is this repo-relative path inside the graded prefix? Lexical, by design.""" + if prefix is None: + return False + return prefix == "" or path == prefix or path.startswith(f"{prefix}/") + + +def _unmeasured_functions( + path: str, + floor: Mapping[FunctionKey, int], + census: Census, + counts: Mapping[str, object], + reboot: str, +) -> list[GateViolation]: + """Floor functions missing from a file this run DID measure. + + The distinction that makes this honest: ``--plain`` without ``--failed`` lists + every measured function regardless of threshold, so a function that merely got + SIMPLER is still in the census. Absence from a measured file therefore means + it was renamed, deleted, or dropped by a ``# complexipy: ignore`` comment — + and that last one is an unregistered, unblessed exemption from the complexity + gate, invisible to ``cf-exemptions`` (whose noqa pattern needs a coded id) and + invisible to a per-FILE audit, because the file is still in ``census.files``. + """ + return [ + _violation( + "COMPLEXIPY_SNAPSHOT_FUNCTION_UNMEASURED", + f"{name} carries a committed watermark of {watermark} but was not measured, " + f"while {path} WAS — a function that merely got simpler would still be in the " + f"census, so this was renamed, deleted, or dropped by a '# complexipy: ignore' " + f"comment; remove the ignore comment, or {reboot}", + path, + counts, + function=name, + watermark=watermark, + ) + for (entry_path, name), watermark in sorted(floor.items()) + if entry_path == path and (entry_path, name) not in census.functions + ] + + +def _surface_violations( + layout: _Layout, + floor: Mapping[FunctionKey, int], + census: Census, + counts: Mapping[str, object], + reboot: str, +) -> list[GateViolation]: + """Every floor entry that still EXISTS must have been measured this run. + + The narrowed-surface trigger caught head-on, independent of any threshold: a + snapshot entry is proof that function HELD an over-threshold complexity, so a + run that produced no measurement for it graded a smaller world than the floor + describes. A file that is GONE is a legitimate improvement and is passed over. + Per-FILE where the whole file went unmeasured, then per-FUNCTION inside the + files that were measured — the second half is what a ``# complexipy: ignore`` + comment used to slip through. + """ + violations: list[GateViolation] = [] + prefix = _graded_prefix(layout.root, layout.source_root) + for path in sorted({path for path, _ in floor}): + if not (layout.root / path).is_file(): + continue + if not _inside(path, prefix): + violations.append( + _violation( + "COMPLEXIPY_SURFACE_NARROWED", + f"the floor covers {path}, which lies OUTSIDE the graded source root " + f"({prefix or '.'}) — widen the declared source_root, or {reboot}", + path, + counts, + source_root=str(layout.source_root), + ) + ) + elif path not in census.files: + violations.append( + _violation( + "COMPLEXIPY_SNAPSHOT_FILE_UNMEASURED", + f"{path} carries a committed watermark but NO function in it was " + f"measured — it is excluded, ignore-commented, or now functionless; " + f"restore it to the measured surface, or {reboot}", + path, + counts, + source_root=str(layout.source_root), + ) + ) + else: + violations.extend(_unmeasured_functions(path, floor, census, counts, reboot)) + return violations + + +def _skew(census: Census, offenders: Census, counts: Mapping[str, object]) -> GateError | None: + """The two write-free runs must describe ONE world. + + The offender set is a filter of the census, so every offender must appear in + the census at the same complexity. A disagreement means the tree changed + between the runs, or a flag moved the measurement surface — either way the + comparison inputs are not a single observation and must not be graded. Both + sides aggregate ``max()`` per key, so a repeated ``(path, name)`` pair can no + longer fake a skew by landing its high value in only one of the two maps. + """ + disagreements = sorted( + f"{path}:{name}" + for (path, name), value in offenders.functions.items() + if census.functions.get((path, name)) != value + ) + if not disagreements: + return None + return refusal( + "GATE_COMPLEXIPY_MEASUREMENT_SKEW", + f"{len(disagreements)} function(s) reported by the offender run are absent from " + "(or disagree with) the census run — the two measurements are not one world, so " + "the gate refuses to compare them; re-run on a quiescent tree, and if it repeats, " + "look for a complexipy.toml moving the surface between the two runs", + {**counts, "functions": disagreements[:10]}, + ) + + +def _bar_moved( + floor: Mapping[FunctionKey, int], + census: Census, + offenders: Census, + counts: Mapping[str, object], + reboot: str, +) -> GateError | None: + """The offender census's vacuity leg — threshold-free, from data already in hand. + + A floor entry is proof that function was ABOVE complexipy's threshold when the + floor was booted (``create_snapshot_file`` stores only over-threshold + functions). So if the census still measures it at-or-above its watermark and + the offender run does NOT report it, then + ``threshold_now >= census >= watermark > threshold_at_boot``: the bar moved + since the floor was booted, and every regression rule downstream is grading a + set the raised bar emptied. Refusing needs no second threshold authority — the + kit still declares no budget of its own, which is the property that made this + hole possible and is worth keeping. + + A hand-lowered watermark lands here too, correctly: complexipy would never + have written a watermark at or below its own threshold, so such an entry is + itself evidence the floor was not produced by the boot command. + """ + stranded = sorted( + f"{path}:{name} (measured {census.functions[(path, name)]} >= watermark {watermark})" + for (path, name), watermark in floor.items() + if (path, name) in census.functions + and census.functions[(path, name)] >= watermark + and (path, name) not in offenders.functions + ) + if not stranded: + return None + return refusal( + "GATE_COMPLEXIPY_THRESHOLD_RAISED", + f"{len(stranded)} committed watermark(s) are at-or-below what complexipy now " + "calls acceptable, so its offender set no longer covers the floor — the ratchet " + "would grade nothing and report clean. complexipy's threshold has been raised " + "since the floor was booted (or a watermark was hand-lowered): restore " + f"max-complexity-allowed, or {reboot} at the threshold you intend", + {**counts, "functions": stranded[:10]}, + ) + + +def _vacuity( + layout: _Layout, + floor: Mapping[FunctionKey, int], + census: Census, + counts: Mapping[str, object], + reboot: str, +) -> GateError | None: + """A run that measured nothing can never report clean. + + Two independent legs, so the refusal survives an ALREADY-emptied floor: a + committed floor that still names functions, or a repo that contains Python at + all. The second leg is the CALLER's repo-wide answer (``layout.py_present``), + the same one the absent-snapshot doctrine rides — not a local walk of + ``source_root``, which was blind exactly when ``source_root`` was the wrong + tree (code in ``app/`` beside an empty-but-present ``src/``, floor ``[]``: + zero functions measured, PASS). Consulting one answer is what stops the two + surfaces from diverging; it is also strictly stricter, since a repo whose only + Python defines no functions now refuses instead of certifying a void. + """ + if census.rows: + return None + if not floor and not layout.py_present: + return None + return refusal( + "GATE_COMPLEXIPY_MEASURED_NOTHING", + "complexipy measured zero functions while there was something to grade — a gate " + "that measured nothing cannot report a clean floor. Check the resolved source " + f"root ({layout.source_root}) actually holds the code, and any complexipy " + f"exclude/ignore configuration; if the tree really did move, {reboot}", + {**counts, "source_root": str(layout.source_root), "python_present": layout.py_present}, + ) + + +def grade( + layout: _Layout, + floor: Mapping[FunctionKey, int], + census: Census, + offenders: Census, + *, + config: str | None = None, +) -> GateVerdict: + """The whole ratchet as a pure function of (committed floor, measured world). + + No subprocess, no write — which is the property the tool's own comparison + cannot have, because its green path IS the rewrite. Every finding and refusal + carries the measured tally, and so does a PASS: the counts ride the verdict's + ``evidence`` and one notice rides its ``notices``, so a machine reading the + aggregated JSON can tell a clean grade from a vacuous one without re-running + anything. Refusal order is deliberate — a skewed pair must not be reasoned + over, and a raised bar must be named before the emptied offender set is + mistaken for a quiet one. + """ + counts = _counts(floor, census, offenders) + reboot = _reboot(_graded_prefix(layout.root, layout.source_root) or ".") + violations = _regressions(floor, offenders, counts, reboot) + violations.extend(_surface_violations(layout, floor, census, counts, reboot)) + error = ( + _skew(census, offenders, counts) + or _bar_moved(floor, census, offenders, counts, reboot) + or _vacuity(layout, floor, census, counts, reboot) + ) + return GateVerdict( + gate=GATE, + violations=violations, + error=error, + notices=[ + f"— measured {census.rows} function(s) in {len(census.files)} file(s) against a " + f"{len(floor)}-function committed floor" + ], + evidence={**counts, "complexipy_config": config}, + ) + + +def complexipy_verdict( + layout: _Layout, + env: Mapping[str, str], + tool: Path, + executor: Executor, +) -> GateVerdict: + """Audit the consumer's config, measure the tree write-free, then grade in our code. + + Order is load-bearing. The config audit runs FIRST because it is what makes + the two measurement runs write-free at all; grading the pre-write bytes of a + floor a later run rewrote is precisely the green-with-the-artifact-mutated + world the refuters found. The caller has already enforced the absent-watermark + doctrine, so the floor exists here. A GateError from the audit or either + measurement propagates as the stage's refusal, which the battery records and + continues past. + """ + config = audit_config(layout.root) + floor = read_snapshot(layout.root / SNAPSHOT_FILENAME) + census = measure(layout.root, layout.source_root, env, tool, executor, offenders_only=False) + offenders = measure(layout.root, layout.source_root, env, tool, executor, offenders_only=True) + return grade(layout, floor, census, offenders, config=config) diff --git a/src/cf_quality/errors.py b/src/cf_quality/errors.py index de263b2..36e3e51 100644 --- a/src/cf_quality/errors.py +++ b/src/cf_quality/errors.py @@ -13,7 +13,9 @@ name, the violations it collected, and the GateError it died on (if any). Its :attr:`~GateVerdict.exit_code` and :attr:`~GateVerdict.passed` derive the kit-wide exit contract (0 clean · 1 violations · 2 the gate could not - run) from those parts, so every gate reports one structured verdict. + run) from those parts, so every gate reports one structured verdict. It also + carries the ``notices`` and ``evidence`` a PASS needs: a verdict with no + measurement behind it cannot be told from one that measured nothing. All three serialize via ``to_dict()`` so the wire form and the in-process form say the same thing. @@ -112,11 +114,22 @@ class GateVerdict: violations: Every finding the gate reported (empty when clean). error: The typed failure the gate raised when it could not run, or None when the gate completed (clean or with findings). + notices: Informational lines a reader needs even on a PASS (default + empty). Appended after the original fields, like GateViolation's + ``severity``/``fixable``, so existing construction is untouched. + evidence: The structured measurement behind the verdict (default + empty) — counts, resolved config, whatever proves the gate looked. + A PASS with no evidence is indistinguishable from a PASS that + measured nothing, and telling those apart is the whole job of a + ratchet; prose in ``notices`` cannot carry that to a machine, so + the numbers ride their own field. """ gate: str violations: list[GateViolation] error: GateError | None = None + notices: list[str] = field(default_factory=list) + evidence: dict[str, Any] = field(default_factory=dict) @property def passed(self) -> bool: @@ -138,4 +151,6 @@ def to_dict(self) -> dict[str, Any]: "exit_code": self.exit_code, "error": self.error.to_dict() if self.error is not None else None, "violations": [violation.to_dict() for violation in self.violations], + "notices": list(self.notices), + "evidence": dict(self.evidence), } diff --git a/src/cf_quality/gate_runner.py b/src/cf_quality/gate_runner.py index 37becc0..30530f0 100644 --- a/src/cf_quality/gate_runner.py +++ b/src/cf_quality/gate_runner.py @@ -37,6 +37,7 @@ from pathlib import Path from typing import Any +from cf_quality.complexipy_ratchet import complexipy_verdict from cf_quality.errors import GateError, GateVerdict, GateViolation from cf_quality.mypy_normalize import normalize_mypy_stdout from cf_quality.repo_config import ( @@ -321,7 +322,7 @@ def _mypy(layout: Layout, env: Mapping[str, str]) -> GateVerdict | None: def _complexipy(layout: Layout, env: Mapping[str, str]) -> GateVerdict | None: - """complexipy through the snapshot ratchet; same refuse/skip doctrine as mypy.""" + """complexipy WRITE-FREE (its compare REWRITES the floor) — refuse/skip as mypy's.""" if not (layout.root / "complexipy-snapshot.json").is_file(): return _ratchet_skip_or_refuse( layout, @@ -329,8 +330,7 @@ def _complexipy(layout: Layout, env: Mapping[str, str]) -> GateVerdict | None: "complexipy-snapshot.json", "boot the snapshot (complexipy --snapshot-create), even when clean", ) - argv = [str(_tool("complexipy")), str(layout.source_root)] - return _run_external("complexipy", argv, cwd=layout.root, env=env) + return complexipy_verdict(layout, env, _tool("complexipy"), _exec) def _ratchet_skip_or_refuse( @@ -449,11 +449,11 @@ def _gate_detail(verdict: GateVerdict) -> list[str]: def _emit_human(agg: _Aggregate) -> None: - """One status line per gate, then full detail for EVERY failing gate.""" + """One status line per gate (with its notices — a PASS must show what it measured).""" failing = [verdict for verdict in agg.verdicts if not verdict.passed] for verdict in agg.verdicts: status = "PASS" if verdict.passed else "FAIL" - print(f"{status} {verdict.gate}") + print(f"{status} {verdict.gate}", *verdict.notices) if not failing: print(f"\n{RUNNER_GATE}: PASS — every gate is clean") return diff --git a/tests/test_complexipy_instrument.py b/tests/test_complexipy_instrument.py new file mode 100644 index 0000000..e45bd8e --- /dev/null +++ b/tests/test_complexipy_instrument.py @@ -0,0 +1,363 @@ +"""complexipy as an instrument — the preconditions that make its census trustworthy. + +``tests/test_complexipy_snapshot.py`` pins the RATCHET: given a floor and a census, +which verdict. This file pins everything that must be true before that census may +be believed at all, which is where two refuters found the first cut of the fix +still defeatable: + +- **the consumer's own complexipy config.** ``--snapshot-ignore`` disarms the + watermark compare's rewrite, but ``handle_snapshot_file_creation`` + (``main.py:323``) is a SEPARATE branch driven by ``snapshot-create``, which the + kit passes no CLI value for — so ``[tool.complexipy] snapshot-create = true`` + wins (``utils/toml.py:235-240``) and both measurement runs rewrite the floor + while the gate grades the pre-write bytes it already read. ``--snapshot-create`` + has no negating secondary name (``main.py:97-102``), so argv cannot override it: + the only honest mechanism is to READ that config and REFUSE. Two more config + keys defeat the run the same way — ``quiet`` (rejected beside ``--plain``, exit + 2, empty census) and ``output-format`` / the legacy ``output-*`` flags (a report + file written into the consumer's tree plus ``Results saved at ...`` and + ``Deprecated: ...`` printed onto stdout BEFORE the census); +- **the exit code and stderr**, which used to be decoration and a discard. Every + tool-side failure was therefore re-attributed to the repo with its reason + deleted; +- **the parse's own hazards** — a duplicated ``(path, name)`` key, a non-UTF-8 + stream the PARENT decodes, and rich's dumb-terminal short circuit that ignores + ``COLUMNS`` outright; +- **the key itself**: ``complexipy_floor._normalized_path`` is a line-for-line copy + of ``complexipy.utils.output.normalize_path``, and the kit ships no + ``MIRRORS.md`` for ``cf-mirror-check`` to gate. The rod here calls the INSTALLED + function, which is a stronger guard than a declaration nobody checks: a 5.7.0 + change to that join fails the suite instead of re-keying every watermark into a + ``NEW_OFFENDER``. +""" + +from __future__ import annotations + +import subprocess +from collections.abc import Mapping +from pathlib import Path + +import pytest +from complexipy.utils.output import normalize_path +from test_complexipy_snapshot import _census, _light, _measured, _snapshot, _stage_verdict +from test_gate_runner import _layout, _write + +from cf_quality import gate_runner +from cf_quality.complexipy_floor import _normalized_path +from cf_quality.complexipy_measure import audit_config, parse_census +from cf_quality.errors import GateError + + +def _config_verdict( + root: Path, monkeypatch: pytest.MonkeyPatch, config: str, body: str +) -> GateError: + """Mount a consumer complexipy config on an otherwise CLEAN repo, return the refusal.""" + _write(root, config, body) + _snapshot(root) + verdict = _stage_verdict(root, monkeypatch, _measured(_census(_light(root)), "")) + assert verdict.error is not None, "a healthy census must not rescue a defeating config" + assert verdict.exit_code == 2 + return verdict.error + + +# --- BLOCKER 1: a consumer config that defeats the measurement ---------------- + + +@pytest.mark.parametrize( + ("config", "body", "key"), + [ + # THE one that re-opens the write. `--snapshot-ignore` does not touch + # handle_snapshot_file_creation, and no CLI flag can negate this key. + ("complexipy.toml", "snapshot-create = true\n", "snapshot-create"), + ("pyproject.toml", "[tool.complexipy]\nsnapshot-create = true\n", "snapshot-create"), + (".complexipy.toml", "snapshot-create = true\n", "snapshot-create"), + # BadParameter beside --plain: exit 2, empty stdout. Un-audited, the gate + # reported GATE_COMPLEXIPY_MEASURED_NOTHING and blamed the repo. + ("complexipy.toml", "quiet = true\n", "quiet"), + # A report file written into the consumer's tree, and two unguarded + # console.print lines landing on stdout BEFORE the census. + ("complexipy.toml", 'output-format = ["json"]\n', "output-format"), + ("complexipy.toml", "output-json = true\n", "output-json"), + ("complexipy.toml", 'output = "reports/"\n', "output"), + # Would narrow the CENSUS run to offenders, blinding the surface audit and + # the per-function audit that reasons on "still present, merely simpler". + ("complexipy.toml", "failed = true\n", "failed"), + ("complexipy.toml", 'details = "low"\n', "details"), + # Decouples the exit code from the threshold verdict, which is the second + # vacuity leg's only witness. + ("complexipy.toml", "ignore-complexity = true\n", "ignore-complexity"), + # Prints ignore-comment locations onto the census stream. + ("complexipy.toml", "report-ignored = true\n", "report-ignored"), + # --ratchet with no --diff exits 2 before measuring anything. + ("complexipy.toml", "ratchet = true\n", "ratchet"), + ], +) +def test_consumer_config_that_defeats_the_measurement_is_refused( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, config: str, body: str, key: str +) -> None: + error = _config_verdict(tmp_path, monkeypatch, config, body) + + assert error.code == "GATE_COMPLEXIPY_CONFIG_DEFEATS_MEASUREMENT" + assert error.context["keys"] == [key], "the refusal names the KEY, not just the file" + assert error.context["config"] == config, "and the file it found it in" + assert key in error.message and config in error.message + + +@pytest.mark.parametrize( + ("config", "body"), + [ + # The consumer's own threshold and surface authorities. The kit declares NO + # budget of its own (measurement_argv carries no -mx), so refusing these + # would break the very neutrality that keeps a baselined band from reading + # as new — and Blocker 2's closure is threshold-free precisely so it can + # stay honoured. + ("complexipy.toml", "max-complexity-allowed = 25\n"), + ("complexipy.toml", 'exclude = ["vendor"]\n'), + # These only ever make the census LARGER, or reorder it (max() per key). + ("complexipy.toml", "no-ignore = true\n"), + ("complexipy.toml", "check-script = true\n"), + ("complexipy.toml", 'sort = "desc"\n'), + # Value-sensitive: a key at its harmless value is not a defeating key, and + # a false refusal costs a consumer a red board for nothing. + ("complexipy.toml", "snapshot-create = false\n"), + ("complexipy.toml", 'details = "high"\n'), + # pyproject.toml with no [tool.complexipy] is not a complexipy config. + ("pyproject.toml", "[tool.other]\nsnapshot-create = true\n"), + ], +) +def test_legitimate_consumer_config_is_honoured_not_refused( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, config: str, body: str +) -> None: + _write(tmp_path, config, body) + _snapshot(tmp_path) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(_census(_light(tmp_path)), "")) + + assert verdict.passed, f"{config} carrying {body!r} must not be refused" + + +def test_config_search_order_mirrors_the_tools_own(tmp_path: Path) -> None: + # utils/toml.py:135-148 stops at the FIRST file that yields a table, and for a + # non-pyproject name that includes an EMPTY document — so an empty + # complexipy.toml shadows a populated pyproject.toml. Audit a different file + # than the tool reads and the audit is theatre. + _write(tmp_path, "pyproject.toml", "[tool.complexipy]\nsnapshot-create = true\n") + _write(tmp_path, ".complexipy.toml", "") + + assert audit_config(tmp_path) == ".complexipy.toml", "the shadowing file is the one in force" + + _write(tmp_path, "complexipy.toml", "") + assert audit_config(tmp_path) == "complexipy.toml", "complexipy.toml outranks both" + + +def test_unreadable_complexipy_config_is_refused_not_ignored( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # An unparseable config is not an absent config: the tool would crash on it, + # and treating it as "no config" would leave the write-free claim unaudited. + error = _config_verdict(tmp_path, monkeypatch, "complexipy.toml", "snapshot-create = tru\n") + + assert error.code == "GATE_COMPLEXIPY_CONFIG_UNREADABLE" + + +def test_non_utf8_complexipy_config_refuses_instead_of_crashing_the_battery( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # UnicodeDecodeError is a ValueError, not a TOMLDecodeError — catching only the + # latter loses the whole 12-stage board to a traceback. + (tmp_path / "complexipy.toml").write_bytes(b'paths = "\xff\xfe"\n') + _snapshot(tmp_path) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(_census(_light(tmp_path)), "")) + + assert verdict.error is not None and verdict.error.code == "GATE_COMPLEXIPY_CONFIG_UNREADABLE" + + +def test_a_clean_repo_reports_which_config_was_in_force( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # A PASS must say what it measured through — "no complexipy config" and "a + # config we never looked at" are the same verdict otherwise. + _write(tmp_path, "complexipy.toml", "max-complexity-allowed = 25\n") + _snapshot(tmp_path) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(_census(_light(tmp_path)), "")) + + assert verdict.passed + assert verdict.evidence["complexipy_config"] == "complexipy.toml" + + +# --- DEFECT 3 / BLOCKER 2(b): the exit code and stderr are evidence ----------- + + +def test_nonzero_exit_with_an_empty_census_refuses_and_carries_the_reason( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # Exit 1 means "some function is over threshold" (utils/output.py:44 + 234-241), + # which no run that printed zero functions can be reporting. This is also the + # OFFENDER run's vacuity leg — one rule, both runs. Before this the gate + # reported GATE_COMPLEXIPY_MEASURED_NOTHING and charged it to the repo, with + # complexipy's stderr thrown away. + _write(tmp_path, "src/heavy.py", "def heavy():\n return 1\n") + _snapshot(tmp_path) + responses = _measured("", "") + responses["complexipy"] = (1, "", "Traceback: the instrument fell over\n") + + verdict = _stage_verdict(tmp_path, monkeypatch, responses) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_INSTRUMENT_FAILED" + assert verdict.error.context["exit_code"] == 1 + assert verdict.error.context["stderr"] == ["Traceback: the instrument fell over"] + + +def test_an_exit_code_that_is_not_a_verdict_refuses( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # 0 and 1 are the tool's only verdicts; 2 is typer declining to run + # (main.py:748 BadParameter, main.py:726-729 validate_ratchet). A census that + # LOOKS parseable beside exit 2 is still a run that did not do its job. + _snapshot(tmp_path) + responses = _measured(_census(_light(tmp_path)), "") + responses["complexipy"] = (2, _census(_light(tmp_path)), "Error: --ratchet requires --diff\n") + + verdict = _stage_verdict(tmp_path, monkeypatch, responses) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_INSTRUMENT_FAILED" + assert verdict.error.context["exit_code"] == 2 + + +def test_the_offender_run_exit_code_is_cross_checked_too( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The census is healthy and the floor is met, so nothing else has anything to + # say: only the offender run's own contradiction is left to catch it. + _write(tmp_path, "src/heavy.py", "def heavy():\n return 1\n") + _snapshot(tmp_path, ("src/heavy.py", "heavy", 33)) + responses = _measured(_census(("src/heavy.py", "heavy", 33)), "") + responses["complexipy-offenders"] = (1, "", "") + + verdict = _stage_verdict(tmp_path, monkeypatch, responses) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_INSTRUMENT_FAILED" + assert verdict.error.context["offenders_only"] is True + + +def test_exit_one_with_a_real_census_is_the_legitimate_offenders_exist( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The control rod on the rod above: exit 1 is the NORMAL state of a repo + # carrying a baselined floor, so the triage must not turn it into a refusal. + _write(tmp_path, "src/heavy.py", "def heavy():\n return 1\n") + _snapshot(tmp_path, ("src/heavy.py", "heavy", 33)) + rows = _census(("src/heavy.py", "heavy", 33)) + responses = _measured(rows, rows) + responses["complexipy"] = (1, rows, "") + + verdict = _stage_verdict(tmp_path, monkeypatch, responses) + + assert verdict.passed, "a baselined floor at its watermark is green, exit 1 or not" + + +def test_output_that_does_not_decode_refuses_instead_of_crashing_the_battery( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # subprocess text=True decodes in the PARENT, by the PARENT's locale — + # PYTHONIOENCODING pins the child only. UnicodeDecodeError is a ValueError, so + # it escaped _exec (OSError only) and _run_stage (GateError only) and took the + # whole board down with a traceback. + _snapshot(tmp_path) + + def exploding_exec( + argv: list[str], cwd: Path, env: Mapping[str, str], *, stdin: str | None = None + ) -> subprocess.CompletedProcess[str]: + raise UnicodeDecodeError("utf-8", b"\xff", 0, 1, "invalid start byte") + + monkeypatch.setattr(gate_runner, "_tool", lambda name: Path("/fake") / name) + monkeypatch.setattr(gate_runner, "_exec", exploding_exec) + stage = gate_runner.Stage("complexipy", gate_runner._complexipy) + + verdict = gate_runner._run_stage(stage, _layout(tmp_path), {}) + + assert verdict is not None and verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_OUTPUT_UNDECODABLE" + assert verdict.exit_code == 2 + + +# --- DEFECT 10: duplicate keys, and the count that must not collapse ---------- + + +def test_duplicate_function_keys_aggregate_max_and_still_count_every_row() -> None: + # Live on a real consumer: 681 census rows collapse to 680 keys, because an + # @overload chain (or an `if sys.platform:` redefinition) repeats a + # (path, name) pair. Last-writer-wins with sort = "desc" would keep the LOW + # value here, dropping a regression complexipy's own compare — which iterates + # its LIST, not a map — would catch. max() is order-independent, which is why + # the fix is aggregation and NOT pinning --sort: a pinned flag would have to + # survive in the argv for the rule to hold. + census = parse_census("src/m.py f 40\nsrc/m.py f 4\nsrc/m.py g 7\n") + + assert census.functions == {("src/m.py", "f"): 40, ("src/m.py", "g"): 7} + assert census.rows == 3, "the count is ROWS: keys under-report what was measured" + + +# --- the declared mirror, pinned against the INSTALLED tool ------------------- + + +@pytest.mark.parametrize( + ("path", "file_name"), + [ + ("src/pkg/mod.py", "mod.py"), + ("src/pkg", "mod.py"), + ("src/pkg/", "mod.py"), + ("", "mod.py"), + ("mod.py", "mod.py"), + ("src/odd dir/mod.py", "mod.py"), + ], +) +def test_snapshot_key_join_mirrors_the_installed_normalize_path(path: str, file_name: str) -> None: + # The kit ships no MIRRORS.md, so cf-mirror-check (skip-if-absent) gates + # nothing here. This rod is the stronger guard: it calls complexipy's OWN + # function, so a 5.7.0 change to the join fails the suite instead of silently + # re-keying every committed watermark into a brand-new offender. + assert _normalized_path(path, file_name) == normalize_path(path, file_name) + + +# --- the env pins (migrated here: they are instrument wiring, not ratchet) ----- + + +def test_measurement_runs_pin_columns_and_term_so_the_census_cannot_wrap( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # rich wraps at 80 columns when stdout is not a terminal, and a wrapped census + # row is an UNPARSEABLE census row — the gate would refuse a healthy repo. But + # COLUMNS is only load-bearing once TERM is: rich/console.py:1015-1016 returns + # a HARD 80x25 and never reads COLUMNS at all when is_dumb_terminal, which is + # `is_terminal and TERM in ("dumb", "unknown")` (rich/console.py:986-988). + # FORCE_COLOR and TTY_COMPATIBLE are the two names that force is_terminal True + # over a pipe (rich/console.py:955-963), so the overlay clears them. + row = _light(tmp_path) + _snapshot(tmp_path) + seen: list[dict[str, str]] = [] + + def recording_exec( + argv: list[str], cwd: Path, env: Mapping[str, str], *, stdin: str | None = None + ) -> subprocess.CompletedProcess[str]: + seen.append(dict(env)) + return subprocess.CompletedProcess(argv, 0, _census(row), "") + + monkeypatch.setattr(gate_runner, "_tool", lambda name: Path("/fake") / name) + monkeypatch.setattr(gate_runner, "_exec", recording_exec) + hostile = {"PATH": "/usr/bin", "TERM": "dumb", "FORCE_COLOR": "1", "TTY_COMPATIBLE": "1"} + + gate_runner._complexipy(_layout(tmp_path), hostile) + + assert len(seen) == 2, "the census run and the offender run" + for env in seen: + assert int(env["COLUMNS"]) >= 1000, "an 80-column wrap would break the census parse" + assert env["PYTHONIOENCODING"] == "utf-8", "the census decodes UTF-8, never by locale" + assert env["TERM"] not in ("dumb", "unknown"), "a dumb TERM ignores COLUMNS outright" + assert "FORCE_COLOR" not in env and "TTY_COMPATIBLE" not in env, "no forced tty" + assert env["PATH"] == "/usr/bin", "the caller's environment survives the overlay" diff --git a/tests/test_complexipy_ratchet_rules.py b/tests/test_complexipy_ratchet_rules.py new file mode 100644 index 0000000..907554c --- /dev/null +++ b/tests/test_complexipy_ratchet_rules.py @@ -0,0 +1,303 @@ +"""The ratchet's rules where they were VACUOUS or WRONG — one rod per refuter finding. + +``tests/test_complexipy_snapshot.py`` pins the worlds the ratchet already graded +correctly. This file pins the worlds it graded green while measuring nothing, or +graded red while nothing was wrong: + +- **the raised bar (the primary rule going vacuous).** ``_vacuity`` guarded only + the CENSUS, and ``_skew`` is a subset check an empty subset satisfies trivially. + So with ``[tool.complexipy] max-complexity-allowed = 100`` the census still + listed every function (every floor file present, surface audit silent), the + ``--failed`` run returned NOTHING, and the stage exited 0 while cheerfully + reporting it had measured 681 functions against a 12-function floor. A function + that went 33 -> 90 was green. That is + ``gate-that-selects-by-the-value-it-guards-goes-vacuous`` inside the fix for it, + and the closure is threshold-free by construction — the kit still declares no + budget of its own; +- **the ignore comment that voided a watermark.** The audit was per-FILE, and an + ignored function leaves its file in ``census.files``, so nothing fired: an + unregistered exemption from the complexity gate, invisible to ``cf-exemptions`` + too; +- **the symlinked module accused of being outside the tree**, because the FILE was + resolved rather than the roots; +- **the duplicate ``(path, name)`` pair** faking a skew, or hiding a regression; +- **the messages**, which stated a condition and named no remedy, while the docs + claimed they named the re-boot; +- **the PASS with no evidence**, which a machine could not tell from a void. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +from test_complexipy_snapshot import A_FUNCTION, _census, _codes, _measured, _snapshot +from test_complexipy_snapshot import _stage_verdict as _verdict +from test_gate_runner import _write + +from cf_quality.errors import GateVerdict +from cf_quality.gate_runner import _Aggregate, _emit_human + +# --- BLOCKER 2: the offender census's threshold-free vacuity closure ----------- + + +def test_a_raised_threshold_empties_the_offender_set_and_is_refused( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # Reproduction (b) from the tool spike, arriving through CONFIG rather than + # through `-mx`: the bar is high enough that a 90-complexity function is not an + # offender. The census is complete, the file is measured, the surface audit has + # nothing to say, the offender set is EMPTY — and the old ratchet iterated that + # empty dict, found no regression, and exited 0 on a 33 -> 90 regression. + # + # The closure names no threshold. A floor entry proves that function was ABOVE + # the bar when the floor was booted, so census 90 >= watermark 33 while the + # offender run stays silent means threshold_now >= 90 >= 33 > threshold_at_boot. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "heavy", 33)) + + verdict = _verdict(tmp_path, monkeypatch, _measured(_census(("src/heavy.py", "heavy", 90)), "")) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_THRESHOLD_RAISED" + assert verdict.exit_code == 2 + assert "src/heavy.py:heavy (measured 90 >= watermark 33)" in verdict.error.context["functions"] + assert "--snapshot-create" in verdict.error.message, "the refusal names its remedy" + + +def test_a_function_exactly_at_its_watermark_still_must_appear_as_an_offender( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The tight edge of the same rule, and the reason it is `>=` and not `>`: at + # census == watermark the function is still above the BOOT threshold, so an + # offender run that omits it can only mean the bar moved. The green counterpart + # (same numbers, offender run reporting it) is pinned in + # test_complexipy_snapshot.test_unchanged_floor_is_clean_at_the_watermark. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "heavy", 33)) + + verdict = _verdict(tmp_path, monkeypatch, _measured(_census(("src/heavy.py", "heavy", 33)), "")) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_THRESHOLD_RAISED" + + +def test_a_hand_lowered_watermark_lands_in_the_same_refusal( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # complexipy stores only OVER-threshold functions, so a watermark of 1 could + # never have come from the boot command. The floor was edited, and the refusal + # is correct rather than a false positive. + _write(tmp_path, "src/light.py", A_FUNCTION) + _snapshot(tmp_path, ("src/light.py", "a_function", 1)) + + census = _census(("src/light.py", "a_function", 4)) + + verdict = _verdict(tmp_path, monkeypatch, _measured(census, "")) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_THRESHOLD_RAISED" + + +def test_a_genuine_shrink_below_its_watermark_is_still_green( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # THE control rod on the closure above: it must fire on a moved bar and NOT on + # the improvement it superficially resembles. 30 -> 4 is under the watermark, so + # census < watermark and the rule does not engage. Without this rod the closure + # could be "refuse whenever the offender set is empty", which would red every + # healthy repo that fixed its last offender. + _write(tmp_path, "src/improved.py", A_FUNCTION) + _snapshot(tmp_path, ("src/improved.py", "improved", 30)) + + verdict = _verdict( + tmp_path, monkeypatch, _measured(_census(("src/improved.py", "improved", 4)), "") + ) + + assert verdict.passed, _codes(verdict) + + +# --- DEFECT 6: the ignore comment that silently voided a watermark ------------ + + +def test_a_floor_function_missing_from_a_MEASURED_file_is_a_violation( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # `# complexipy: ignore` drops a function from BOTH runs while its file stays in + # census.files, so the per-FILE audit saw nothing — an unregistered, unblessed + # exemption from the complexity gate. The distinction that makes this catchable: + # `--plain` without `--failed` lists every measured function regardless of + # threshold, so a function that merely got SIMPLER would still be here. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "ignored", 33), ("src/heavy.py", "kept", 20)) + census = _census(("src/heavy.py", "kept", 20)) + + verdict = _verdict(tmp_path, monkeypatch, _measured(census, census)) + + assert _codes(verdict) == ["COMPLEXIPY_SNAPSHOT_FUNCTION_UNMEASURED"] + assert verdict.exit_code == 1 + violation = verdict.violations[0] + assert violation.path == "src/heavy.py" and violation.context["function"] == "ignored" + assert "complexipy: ignore" in violation.message, "the message names the likely cause" + assert "--snapshot-create" in violation.message, "and the remedy" + + +def test_a_whole_file_going_unmeasured_still_reports_once_not_per_function( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The per-function audit must not turn one narrowed file into N identical + # findings — the file-level reading is the more informative one and comes first. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "one", 33), ("src/heavy.py", "two", 20)) + _write(tmp_path, "src/light.py", A_FUNCTION) + + verdict = _verdict( + tmp_path, monkeypatch, _measured(_census(("src/light.py", "a_function", 1)), "") + ) + + assert _codes(verdict) == ["COMPLEXIPY_SNAPSHOT_FILE_UNMEASURED"] + + +# --- DEFECT 7: the symlinked module accused of being outside the tree --------- + + +def test_a_symlinked_module_inside_the_tree_is_not_called_outside_it( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # `_surface_violations` resolved the FILE, so src/settings.py -> ../vendor/ + # settings.py resolved outside the graded root and the gate reported that the + # floor entry "lies OUTSIDE the graded source root" — flatly false: it is + # inside, and complexipy measured it. Roots are resolved; the floor's own paths + # are repo-relative, so containment is lexical. + _write(tmp_path, "vendor/settings.py", A_FUNCTION) + (tmp_path / "src").mkdir(exist_ok=True) + (tmp_path / "src" / "settings.py").symlink_to(tmp_path / "vendor" / "settings.py") + _snapshot(tmp_path, ("src/settings.py", "settings", 20)) + census = _census(("src/settings.py", "settings", 20)) + + verdict = _verdict(tmp_path, monkeypatch, _measured(census, census)) + + assert verdict.passed, _codes(verdict) + + +def test_a_floor_entry_genuinely_outside_the_graded_root_is_still_reported( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The control rod on the fix above: making the test lexical must not make it + # vacuous. legacy/ is a real sibling of the graded src/, and still fires. + _write(tmp_path, "legacy/heavy.py", A_FUNCTION) + _write(tmp_path, "src/light.py", A_FUNCTION) + _snapshot(tmp_path, ("legacy/heavy.py", "heavy", 33)) + + verdict = _verdict( + tmp_path, monkeypatch, _measured(_census(("src/light.py", "a_function", 1)), "") + ) + + assert _codes(verdict) == ["COMPLEXIPY_SURFACE_NARROWED"] + assert "--snapshot-create" in verdict.violations[0].message, "the finding names its remedy" + + +# --- DEFECT 5: the vacuity leg consults the REPO-WIDE answer ------------------ + + +def test_a_python_free_source_root_beside_real_code_cannot_report_pass( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # `resolve_source_root` returns root/src whenever src/ EXISTS, so a repo whose + # code lives in app/ beside an empty-but-present src/ measured zero functions + # with a floor of [] — and both old vacuity legs were blind, because leg 2 + # walked the same possibly-wrong source_root. The leg now rides the CALLER's + # repo-wide py_present, the same answer the absent-snapshot doctrine uses, so + # the two surfaces cannot disagree. + _write(tmp_path, "app/real.py", A_FUNCTION) + (tmp_path / "src").mkdir() + _snapshot(tmp_path) + + verdict = _verdict(tmp_path, monkeypatch, _measured("", "")) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_MEASURED_NOTHING" + assert verdict.error.context["python_present"] is True + assert verdict.exit_code == 2 + + +# --- DEFECT 10: a duplicate pair must not fake a skew ------------------------- + + +def test_a_duplicated_key_straddling_a_watermark_reports_the_regression_not_a_skew( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # Observed live: one consumer file lists the same (path, name) twice. Under + # last-writer-wins with sort = "desc" the offender map kept 40 while the census + # kept 4, so the runs "disagreed" and the gate raised a bogus + # GATE_COMPLEXIPY_MEASUREMENT_SKEW on a healthy repo — and in the mirror case + # the regression vanished from both maps. max() on both sides gives one world. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "heavy", 20)) + census = _census(("src/heavy.py", "heavy", 40), ("src/heavy.py", "heavy", 4)) + + verdict = _verdict(tmp_path, monkeypatch, _measured(census, census)) + + assert verdict.error is None, "a repeated key is not two worlds" + assert _codes(verdict) == ["COMPLEXIPY_WATERMARK_REGRESSION"] + assert verdict.violations[0].context["measured"] == 40, "max(), so the HIGH value grades" + assert verdict.violations[0].context["measured_functions"] == 2, "rows, not keys" + + +# --- DEFECT 9 + the evidence a PASS must carry ------------------------------- + + +def test_every_regression_message_names_a_remedy( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # A message that states the condition and names nothing to do is half a + # message, and the docs corrected on this branch already claimed these named + # the re-boot. They now do — spelled with the resolved graded root. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "kept", 20)) + census = _census(("src/heavy.py", "kept", 26), ("src/heavy.py", "fresh", 40)) + + verdict = _verdict(tmp_path, monkeypatch, _measured(census, census)) + + assert _codes(verdict) == ["COMPLEXIPY_NEW_OFFENDER", "COMPLEXIPY_WATERMARK_REGRESSION"] + for violation in verdict.violations: + assert "complexipy src --snapshot-create" in violation.message + assert "from the repo root" in violation.message + + +def test_a_passing_stage_carries_its_measurement_in_the_verdict( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The discrimination this whole rung exists to make: a machine reading the + # aggregated JSON must be able to tell a clean grade from a vacuous one. The + # counts used to go only to stderr via a hand-rolled print, so a PASSING stage's + # to_dict() carried no evidence at all — and the workflows pipe only stdout into + # $GITHUB_STEP_SUMMARY, so even the human never saw it on the board. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "heavy", 33)) + rows = _census(("src/heavy.py", "heavy", 33), ("src/other.py", "fine", 2)) + + offenders = _census(("src/heavy.py", "heavy", 33)) + + verdict = _verdict(tmp_path, monkeypatch, _measured(rows, offenders)) + + assert verdict.passed, _codes(verdict) + wire = verdict.to_dict() + assert wire["evidence"]["measured_functions"] == 2 + assert wire["evidence"]["measured_files"] == 2 + assert wire["evidence"]["snapshot_functions"] == 1 + assert wire["notices"] == verdict.notices and len(verdict.notices) == 1 + assert "measured 2 function(s) in 2 file(s)" in verdict.notices[0] + + +def test_the_human_board_shows_the_measurement_beside_a_pass( + capsys: pytest.CaptureFixture[str], +) -> None: + # The notice must ride STDOUT beside the PASS line, because that is the stream + # both workflows tee into $GITHUB_STEP_SUMMARY — a count only on stderr reaches + # the run log and not the board that carries the verdict. + verdict = GateVerdict(gate="complexipy", violations=[], notices=["— m"]) + + _emit_human(_Aggregate(verdicts=[verdict])) + + assert "PASS complexipy — m" in capsys.readouterr().out diff --git a/tests/test_complexipy_snapshot.py b/tests/test_complexipy_snapshot.py new file mode 100644 index 0000000..8770ad6 --- /dev/null +++ b/tests/test_complexipy_snapshot.py @@ -0,0 +1,469 @@ +"""The complexipy snapshot ratchet — the floor the tool used to eat. + +**The reproduced defect.** ``complexipy-snapshot.json`` is the committed +cognitive-complexity floor. In the pinned ``complexipy==5.6.0`` the tool's own +snapshot compare REWRITES that file on its success path +(``complexipy/utils/snapshot.py`` ``handle_snapshot_watermark`` returns True only +after ``create_snapshot_file(...)`` with the functions IT measured), so two +ordinary green runs empty a populated floor at exit 0: measure a narrower surface +(``complexipy src/one_simple_file.py``), or raise the bar (``-mx 100``). Observed +on a real snapshot: ``1 entry -> 0 entries``, ``exit=0``, both ways. Committed, +that deletes the floor forever behind a green gate. + +**What this file pins.** The stage no longer hands its own artifact to the tool: +:mod:`cf_quality.complexipy_ratchet` measures write-free (``--snapshot-ignore``, +never ``--snapshot-create`` — the only two paths to ``create_snapshot_file``) and +grades the ratchet in kit code, where it is a pure function of (committed floor, +measured census) and cannot be silently destroyed. The control rods: + +- RED on the **emptying** world — the floor names a file the run did not measure, + whether it fell outside the graded source root or was skipped inside it; +- RED on a real **complexity regression** — a new offender, and a rise above a + committed watermark; +- RED on a **vacuous** run — a census of zero functions can never report clean, + with a second leg (the caller's repo-wide ``py_present``) so the refusal + survives an already-emptied floor; +- GREEN on an **unchanged** floor, on a **legitimate non-empty shrink**, and on a + genuinely clean repo whose floor is ``[]``; +- the **write-free** proof at the argv altitude, plus the measured COUNT riding + every finding so no verdict can be read without the measurement behind it. + +Two sibling files carry what this one is not about: ``test_complexipy_instrument.py`` +(the consumer config that can defeat the measurement, the exit-code/stderr triage, +the env pins, and the live mirror-pin against the installed ``normalize_path``) and +``test_complexipy_ratchet_rules.py`` (the threshold-free vacuity closures, the +per-function surface audit, and duplicate ``(path, name)`` keys). + +The external tool is faked at the established subprocess seam +(``gate_runner._exec`` / ``gate_runner._tool``, through ``test_gate_runner``'s +helpers) so the REAL grading logic runs against representative ``--plain`` +output, and every stage runs through ``gate_runner._run_stage`` — the altitude the +battery reads, where a raised GateError is already the stage's exit-2 verdict. +:func:`cf_quality.complexipy_ratchet.grade` is additionally exercised with no +subprocess at all, which is the whole point of moving the comparison here. +""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from pathlib import Path + +import pytest +from test_gate_runner import _clean_cf_responses, _install_fakes, _layout, _write + +from cf_quality import complexipy_measure, gate_runner +from cf_quality.complexipy_floor import read_snapshot +from cf_quality.complexipy_measure import Census, measurement_argv +from cf_quality.complexipy_ratchet import grade +from cf_quality.errors import GateError, GateVerdict + +#: Module text that merely EXISTS and defines a function. Every complexity in this +#: file comes from the faked ``--plain`` census, never from this source — the fake +#: is what lets a watermark of 33 be asserted without authoring a monster. +A_FUNCTION = "def a_function():\n return 1\n" + +Responses = Mapping[str, tuple[int, str, str]] + + +def _snapshot(root: Path, *entries: tuple[str, str, int]) -> None: + """Write a committed floor in complexipy's own shape (path, function, watermark). + + Mirrors the observed on-disk form of a real consumer snapshot: ``path`` is + repo-relative and already carries the file name, ``file_name`` is the + basename, and only over-threshold functions are stored (clean tree -> ``[]``). + """ + payload = [ + { + "path": path, + "file_name": Path(path).name, + "functions": [{"name": name, "complexity": watermark}], + } + for path, name, watermark in entries + ] + _write(root, "complexipy-snapshot.json", json.dumps(payload)) + + +def _census(*rows: tuple[str, str, int]) -> str: + """complexipy ``--plain`` stdout: one `` `` line.""" + return "".join(f"{path} {name} {complexity}\n" for path, name, complexity in rows) + + +def _measured(census: str, offenders: str) -> dict[str, tuple[int, str, str]]: + """The clean board with complexipy's two write-free runs answered explicitly. + + Exit 1 whenever a run reports offenders, 0 otherwise — the tool's real + semantics, which the gate now CROSS-CHECKS rather than discards: an empty + ``--failed`` census beside a non-zero exit is a contradiction, and that is the + offender run's vacuity leg (its rod lives in test_complexipy_instrument.py). + """ + responses = _clean_cf_responses() + responses["complexipy"] = (0, census, "") + responses["complexipy-offenders"] = (1 if offenders else 0, offenders, "") + return responses + + +def _stage_verdict( + root: Path, monkeypatch: pytest.MonkeyPatch, responses: Responses +) -> GateVerdict: + """Run the real complexipy stage the way the battery runs it. + + Through ``gate_runner._run_stage`` so a GateError raised by a measurement we + cannot read arrives as the stage's exit-2 verdict, exactly as the board shows + it — never as an exception the caller has to know about. + """ + _install_fakes(monkeypatch, responses) + stage = gate_runner.Stage("complexipy", gate_runner._complexipy) + verdict = gate_runner._run_stage(stage, _layout(root), {}) + assert verdict is not None, "a Python repo carrying a snapshot never skips" + return verdict + + +def _codes(verdict: GateVerdict) -> list[str]: + return [violation.code for violation in verdict.violations] + + +def _light(root: Path, rel: str = "src/light.py") -> tuple[str, str, int]: + """A measured, clean census row for a file that really is on disk.""" + _write(root, rel, A_FUNCTION) + return (rel, "a_function", 1) + + +# --- RED: the emptying world (the reproduced defect) -------------------------- + + +def test_floor_file_outside_the_graded_source_root_is_refused( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The narrowed-surface trigger, structurally: the committed floor holds a + # watermark for a file the gate no longer even looks at. Under the tool's own + # compare this run is exit 0 AND rewrites the floor to the smaller world. + _write(tmp_path, "legacy/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("legacy/heavy.py", "heavy", 33)) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(_census(_light(tmp_path)), "")) + + assert _codes(verdict) == ["COMPLEXIPY_SURFACE_NARROWED"] + assert verdict.exit_code == 1 + assert verdict.violations[0].context["snapshot_functions"] == 1 + + +def test_floor_file_present_but_unmeasured_is_refused( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # Inside the graded tree and still on disk, yet this run produced NO + # measurement for it (excluded, ignore-commented, or emptied). A before/after + # diff of the artifact sees nothing here — there is no rewrite to observe. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "heavy", 33)) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(_census(_light(tmp_path)), "")) + + assert _codes(verdict) == ["COMPLEXIPY_SNAPSHOT_FILE_UNMEASURED"] + assert verdict.exit_code == 1 + assert verdict.violations[0].path == "src/heavy.py" + + +def test_deleted_floor_file_is_a_legitimate_improvement( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The other reading of an unmeasured entry: the file is GONE. That is a real + # shrink, not a narrowed surface, and it must not be charged as a finding. + _snapshot(tmp_path, ("src/removed.py", "heavy", 33)) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(_census(_light(tmp_path)), "")) + + assert verdict.passed, _codes(verdict) + + +# --- RED: real complexity regressions ---------------------------------------- + + +def test_new_offender_without_a_watermark_is_a_violation( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path) + rows = _census(("src/heavy.py", "heavy", 40)) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(rows, rows)) + + assert _codes(verdict) == ["COMPLEXIPY_NEW_OFFENDER"] + assert verdict.exit_code == 1 + assert verdict.violations[0].context["measured"] == 40 + + +def test_rise_above_a_committed_watermark_is_a_violation( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "heavy", 20)) + rows = _census(("src/heavy.py", "heavy", 26)) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(rows, rows)) + + assert _codes(verdict) == ["COMPLEXIPY_WATERMARK_REGRESSION"] + assert verdict.violations[0].context["measured"] == 26 + + +# --- RED: the vacuous run ---------------------------------------------------- + + +def test_vacuous_run_can_never_report_clean( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # THE void rod. The floor is ALREADY `[]` — so the per-file surface audit has + # nothing to say — the tree demonstrably defines a function, and complexipy + # measured NOTHING. An empty census and a clean tree are indistinguishable + # from a count alone, so a run that graded nothing refuses at exit 2. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured("", "")) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_MEASURED_NOTHING" + assert verdict.exit_code == 2 + assert verdict.error.context["measured_functions"] == 0 + + +def test_vacuous_run_against_a_populated_floor_names_the_files_it_lost( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The full emptying world: a populated floor plus a census of nothing. The + # refusal AND the per-file findings both ride the verdict, so the board says + # WHAT went unmeasured, not merely that something did. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "heavy", 33)) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured("", "")) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_MEASURED_NOTHING" + assert _codes(verdict) == ["COMPLEXIPY_SNAPSHOT_FILE_UNMEASURED"] + assert verdict.exit_code == 2 + + +# --- RED: instruments we cannot read ----------------------------------------- + + +def test_unreadable_floor_is_refused_not_read_as_empty( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # green-by-unreadable-file is the same gaming vector as green-by-missing-file, + # which this gate already refuses. + _write(tmp_path, "complexipy-snapshot.json", "{}") + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(_census(_light(tmp_path)), "")) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_SNAPSHOT_UNREADABLE" + assert verdict.exit_code == 2 + + +def test_unparseable_census_line_is_refused( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # A census we cannot parse must not degrade into a SMALLER census: dropping + # the line would be reporting a void as a measurement. + _snapshot(tmp_path) + census = "Analyzing 1 file...\n" + _census(_light(tmp_path)) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(census, "")) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_OUTPUT_UNREADABLE" + assert verdict.exit_code == 2 + + +def test_unmeasurable_path_report_is_refused( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # complexipy telling us it could not analyze a file IS a narrowed surface, + # in the tool's own words. + _snapshot(tmp_path) + census = _census(_light(tmp_path)) + ( + "error: Failed to process src/broken.py - Please check file/folder exists\n" + ) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(census, "")) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_PATHS_UNMEASURABLE" + assert verdict.exit_code == 2 + + +def test_offender_absent_from_the_census_is_measurement_skew( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The offender set is a FILTER of the census, so it cannot hold a function the + # census never saw. If it does, the two runs are not one observation and the + # comparison inputs must not be graded. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "heavy", 33)) + census = _census(("src/heavy.py", "heavy", 33)) + + verdict = _stage_verdict( + tmp_path, monkeypatch, _measured(census, _census(("src/ghost.py", "ghost", 40))) + ) + + assert verdict.error is not None + assert verdict.error.code == "GATE_COMPLEXIPY_MEASUREMENT_SKEW" + assert verdict.exit_code == 2 + + +# --- GREEN: the worlds that must stay green ---------------------------------- + + +def test_unchanged_floor_is_clean_at_the_watermark( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # A function sitting AT its watermark passes — complexipy's own `>` bound, + # preserved exactly (relaxing it to `>=` would be a softening). + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "heavy", 33)) + rows = _census(("src/heavy.py", "heavy", 33)) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(rows, rows)) + + assert verdict.passed, _codes(verdict) + assert verdict.exit_code == 0 + + +def test_legitimate_non_empty_shrink_is_clean( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # One offender simplified below the bar, the other untouched: the floor is + # still populated, the improved file is still MEASURED, and the verdict is + # green. Locking the shrink in stays the runbook's re-boot duty — but the same + # file falling OUT of the census would not pass (the rod above). + _write(tmp_path, "src/improved.py", A_FUNCTION) + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/improved.py", "improved", 30), ("src/heavy.py", "heavy", 20)) + census = _census(("src/improved.py", "improved", 4), ("src/heavy.py", "heavy", 20)) + + verdict = _stage_verdict( + tmp_path, monkeypatch, _measured(census, _census(("src/heavy.py", "heavy", 20))) + ) + + assert verdict.passed, _codes(verdict) + + +def test_clean_repo_with_an_empty_floor_is_clean( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # The boot state a consumer adopts at: `[]` on disk, nothing over the bar, and + # a census that DID measure something. Clean, and provably not a void. + _snapshot(tmp_path) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(_census(_light(tmp_path)), "")) + + assert verdict.passed, _codes(verdict) + assert verdict.exit_code == 0 + + +# --- the write-free invocation (the root of the fix) ------------------------- + + +def test_measurement_argv_can_never_write_the_committed_floor() -> None: + # The pinned tool reaches `create_snapshot_file` from exactly two places: + # `--snapshot-create` (whose TOML twin complexipy_measure.audit_config refuses, + # because argv cannot negate it) and the watermark compare's success path that + # `--snapshot-ignore` switches off. EXHAUSTIVE literal equality, not a flag + # allowlist: the allowlist that stood here bit on REMOVING --snapshot-ignore + # but NOT on ADDING `-mx 100`, which empties the offender set outright. + tool, source_root = Path("/fake/complexipy"), Path("/repo/src") + base = ["/fake/complexipy", "/repo/src", "--plain", "--color", "no", "--snapshot-ignore"] + + assert measurement_argv(tool, source_root, offenders_only=False) == base + assert measurement_argv(tool, source_root, offenders_only=True) == [*base, "--failed"] + + +def test_the_committed_floor_is_only_ever_read( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # Asserted on the artifact itself. With the tool faked this rod can only fail + # if OUR code writes the floor; the tool-side half of the proof is the argv rod + # above plus the two call sites in the pinned package. Under the tool's own + # compare this very green run rewrote these bytes to the measured subset. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + _snapshot(tmp_path, ("src/heavy.py", "heavy", 33)) + snapshot = tmp_path / "complexipy-snapshot.json" + before = snapshot.read_bytes() + rows = _census(("src/heavy.py", "heavy", 33)) + + verdict = _stage_verdict(tmp_path, monkeypatch, _measured(rows, rows)) + + assert verdict.passed, _codes(verdict) + assert snapshot.read_bytes() == before, "the gate graded the floor without touching it" + + +# --- the ratchet as a pure function (no subprocess at all) ------------------- + + +def test_grade_is_a_pure_function_of_the_floor_and_the_census(tmp_path: Path) -> None: + # The property the tool's own compare cannot have: the comparison runs with no + # subprocess, no write and no tool exit code, so it is testable in isolation + # and no green run can destroy it. Every finding carries the measured tally. + _write(tmp_path, "src/heavy.py", A_FUNCTION) + floor = {("src/heavy.py", "heavy"): 20} + census = Census(functions={("src/heavy.py", "heavy"): 31}, rows=1) + + verdict = grade(_layout(tmp_path), floor, census, census) + + assert _codes(verdict) == ["COMPLEXIPY_WATERMARK_REGRESSION"] + assert verdict.violations[0].context["measured_functions"] == 1 + assert verdict.violations[0].context["snapshot_functions"] == 1 + + +def test_read_snapshot_keys_the_observed_on_disk_shape(tmp_path: Path) -> None: + # Keyed the way complexipy's own output joins its two path fields: get that + # join wrong and every committed watermark reads as a brand-new offender. + # Shape and `Class::method` naming taken from a real consumer's snapshot. + _write( + tmp_path, + "complexipy-snapshot.json", + json.dumps( + [ + { + "path": "src/pkg/mod.py", + "file_name": "mod.py", + "functions": [ + {"name": "Engine::_run_inner", "complexity": 18}, + {"name": "Engine::_execute_stage", "complexity": 27}, + ], + } + ] + ), + ) + + floor = read_snapshot(tmp_path / "complexipy-snapshot.json") + + assert floor == { + ("src/pkg/mod.py", "Engine::_run_inner"): 18, + ("src/pkg/mod.py", "Engine::_execute_stage"): 27, + } + + +def test_census_parser_survives_a_path_containing_spaces() -> None: + # `--plain` is space-separated, so the parse must split from the RIGHT: the + # complexity and the function name are single tokens, a path is not. + census = complexipy_measure.parse_census("src/odd dir/mod.py Klass::method 12\n") + + assert census.functions == {("src/odd dir/mod.py", "Klass::method"): 12} + assert census.files == frozenset({"src/odd dir/mod.py"}) + + +# --- the preserved absent-watermark doctrine --------------------------------- + + +def test_absent_floor_doctrine_is_untouched(tmp_path: Path) -> None: + # Unchanged by this rework, and re-pinned here because the rework rewrote the + # stage: Python present with no snapshot REFUSES; a Python-free repo skips. + _write(tmp_path, "src/light.py", A_FUNCTION) + + with pytest.raises(GateError) as excinfo: + gate_runner._complexipy(_layout(tmp_path, py_present=True), {}) + assert excinfo.value.code == "GATE_COMPLEXIPY_SNAPSHOT_MISSING" + + assert gate_runner._complexipy(_layout(tmp_path, py_present=False), {}) is None diff --git a/tests/test_configs.py b/tests/test_configs.py index f2716f1..9bf9977 100644 --- a/tests/test_configs.py +++ b/tests/test_configs.py @@ -166,7 +166,50 @@ def test_baseline_conventions_documents_observed_workflows() -> None: assert "--snapshot-create" in text # complexipy watermark boot assert "mypy-baseline sync" in text # type-debt baseline boot assert "mypy-baseline filter" in text # CI-side set-difference gate - assert "does NOT auto-shrink" in text # the complexipy re-snapshot duty + # 2026-07-28: this line used to pin the substring "does NOT auto-shrink", + # which is FALSE for the pinned complexipy 5.6.0 — a PASSING compare calls + # create_snapshot_file on handle_snapshot_watermark's no-violation branch and + # was observed rewriting a populated snapshot to []. The assertion was + # therefore gating a false sentence into the document. It is replaced, not + # softened: where one substring pinned one (wrong) claim, four now pin the + # whole corrected contract — the measured write behaviour, its evidence, the + # write-free measurement that disarms it, and the operator duty that survived + # it. Delete any one of those teachings from the doc and this test fails. + assert "A passing plain-run compare REWRITES the snapshot" in text # measured on 5.6.0 + assert "handle_snapshot_watermark" in text # the destructive call site, named + assert "--snapshot-ignore" in text # the write-free measurement the gate mounts + assert "The re-snapshot duty is still yours" in text # the surviving shrink duty + + +def test_baseline_conventions_teaches_the_config_that_can_defeat_the_gate() -> None: + # 2026-07-28, second correction: `--snapshot-ignore` alone does NOT make the run + # write-free. `snapshot-create` is a separate branch resolved CLI-first/TOML- + # second with no negating flag, so a consumer's own complexipy config could put + # the rewrite back while the gate graded pre-write bytes. A consumer cannot + # comply with a refusal they were never told about, so the mount doc must carry + # the key list, the refusal, and the two options that ARE honoured. + text = (CONFIGS_DIR / "BASELINE-CONVENTIONS.md").read_text(encoding="utf-8") + assert "GATE_COMPLEXIPY_CONFIG_DEFEATS_MEASUREMENT" in text + assert "snapshot-create" in text and "ignore-complexity" in text and "output-format" in text + assert "max-complexity-allowed" in text, "the threshold the kit deliberately honours" + assert "GATE_COMPLEXIPY_THRESHOLD_RAISED" in text, "raising the bar refuses, not passes" + assert "GATE_COMPLEXIPY_INSTRUMENT_FAILED" in text, "a tool failure is not the repo's" + + +def test_baseline_conventions_cannot_reacquire_the_false_unmeasured_claim() -> None: + # A doc fix pass on this branch asserted that "an improvement that empties a floor + # file of functions … FAILS (COMPLEXIPY_SNAPSHOT_FILE_UNMEASURED)". That is FALSE: + # `--plain` without `--failed` lists every measured function regardless of + # threshold (complexipy/utils/output.py:234,243), so such a file stays in + # census.files and the gate is GREEN. UNMEASURED fires when the file was not + # MEASURED; the per-FUNCTION code is what catches a single lost function. This rod + # bites in both directions: the true rule must be stated, and the false phrasing + # must not come back. + for name in ("BASELINE-CONVENTIONS.md",): + text = (CONFIGS_DIR / name).read_text(encoding="utf-8") + assert "empties a floor file of functions" not in text, f"{name}: the false claim is back" + assert "COMPLEXIPY_SNAPSHOT_FUNCTION_UNMEASURED" in text, "the per-function rule" + assert "was not **measured** at all" in text, "the true trigger, spelled out" def test_ruff_base_gates_blanket_and_unused_noqa() -> None: diff --git a/tests/test_errors.py b/tests/test_errors.py index aa80598..68d997e 100644 --- a/tests/test_errors.py +++ b/tests/test_errors.py @@ -135,6 +135,12 @@ def test_error_dominates_with_exit_two(self) -> None: assert verdict.exit_code == 2 def test_to_dict_composes_the_part_wire_forms(self) -> None: + # `notices` and `evidence` were APPENDED (2026-07-28), the same way + # GateViolation gained severity/fixable: a PASSING gate carried no + # measurement in its wire form, so a machine reading the aggregated JSON + # could not tell a clean grade from one that measured nothing — which is + # the whole discrimination a ratchet exists to make. Both default empty, + # so every existing construction is untouched; the key set grows. violation = GateViolation(code="C", message="m", path="p.py", line=2) error = GateError(code="GATE_X", message="boom", context={"k": "v"}) verdict = GateVerdict(gate="cf-x", violations=[violation], error=error) @@ -144,10 +150,30 @@ def test_to_dict_composes_the_part_wire_forms(self) -> None: "exit_code": 2, "error": error.to_dict(), "violations": [violation.to_dict()], + "notices": [], + "evidence": {}, } def test_clean_to_dict_has_null_error_and_stable_keys(self) -> None: verdict = GateVerdict(gate="cf-x", violations=[]) report = verdict.to_dict() assert report["error"] is None - assert set(report) == {"gate", "passed", "exit_code", "error", "violations"} + assert set(report) == { + "gate", + "passed", + "exit_code", + "error", + "violations", + "notices", + "evidence", + } + + def test_evidence_and_notices_ride_the_wire_form_on_a_pass(self) -> None: + # The rod on the append above: a clean verdict must be able to CARRY its + # measurement, not merely have somewhere to put it. + verdict = GateVerdict( + gate="complexipy", violations=[], notices=["— measured 2"], evidence={"n": 2} + ) + report = verdict.to_dict() + assert report["passed"] is True + assert report["notices"] == ["— measured 2"] and report["evidence"] == {"n": 2} diff --git a/tests/test_gate_runner.py b/tests/test_gate_runner.py index 2e937b2..506dfc6 100644 --- a/tests/test_gate_runner.py +++ b/tests/test_gate_runner.py @@ -31,6 +31,7 @@ import pytest from cf_quality import gate_runner +from cf_quality.complexipy_measure import measurement_argv from cf_quality.errors import GateError, GateVerdict, GateViolation from cf_quality.gate_runner import battery_exit_code, run_battery from cf_quality.import_contract import main as import_contract_main @@ -52,10 +53,16 @@ def _violation(code: str, message: str, path: str) -> GateViolation: def _response_key(argv: Sequence[str]) -> str: - """Map a stage's argv back to its response key (ruff has two stages).""" + """Map a stage's argv back to its response key (ruff and complexipy each run twice). + + complexipy runs write-free TWICE — the ``--plain`` census, then the ``--failed`` + offender subset its OWN threshold selects — keyed apart like ruff's two stages. + """ name = Path(argv[0]).name if name == "ruff": return "ruff-format" if "format" in argv else "ruff-check" + if name == "complexipy": + return "complexipy-offenders" if "--failed" in argv else "complexipy" return name @@ -96,6 +103,10 @@ def _clean_cf_responses() -> dict[str, tuple[int, str, str]]: "ruff-check": (0, "", ""), "ruff-format": (0, "", ""), "pytest": (0, "", ""), + # A census that MEASURED something plus an empty offender subset: the stage + # grades the snapshot ratchet itself now, and a zero census is a void. + "complexipy": (0, "pkg.py greet 1\n", ""), + "complexipy-offenders": (0, "", ""), } for gate in cf_gates: responses[gate] = (0, _verdict_json(gate), "") @@ -206,7 +217,7 @@ def test_one_run_reports_every_failing_gate( # regression, and a failing test. cf-gate must surface ALL four in ONE run. _write(tmp_path, "pkg.py", "x = 1\n") _write(tmp_path, "mypy-baseline.txt", "") - _write(tmp_path, "complexipy-snapshot.json", "{}") + _write(tmp_path, "complexipy-snapshot.json", "[]") responses = _clean_cf_responses() responses["ruff-check"] = (1, "pkg.py:1:1: E501 line too long", "") responses["cf-file-budget"] = ( @@ -219,7 +230,6 @@ def test_one_run_reports_every_failing_gate( ) responses["mypy"] = (1, "pkg.py: error: bad", "") responses["mypy-baseline"] = (1, "pkg.py: error: bad (new over baseline)", "") - responses["complexipy"] = (0, "", "") responses["pytest"] = (1, "1 failed, 0 passed", "") _install_fakes(monkeypatch, responses) @@ -285,11 +295,10 @@ def test_mypy_gates_on_filter_exit_not_mypy_exit( # filter exits 0 (no NEW errors) — the verdict must ride the filter. _write(tmp_path, "pkg.py", "x = 1\n") _write(tmp_path, "mypy-baseline.txt", "") - _write(tmp_path, "complexipy-snapshot.json", "{}") + _write(tmp_path, "complexipy-snapshot.json", "[]") responses = _clean_cf_responses() responses["mypy"] = (1, "pkg.py: error: baselined debt", "") responses["mypy-baseline"] = (0, "", "") - responses["complexipy"] = (0, "", "") _install_fakes(monkeypatch, responses) verdicts = run_battery(tmp_path, {}) @@ -379,7 +388,8 @@ def test_import_contract_aggregates_passes_into_one_verdict( # actually run: ruff rides the derived known-first-party; that first-party set # is the SAME one cf-repo-config resolves; complexipy refuses a Python repo # with no snapshot, skips a Python-free one visibly, and targets the resolved -# source_root in a single unpiped process (a pipe would mask its exit code). +# source_root in unpiped, WRITE-FREE processes (a pipe would mask its exit code; +# an unguarded run rewrites the committed floor — tests/test_complexipy_snapshot.py). def _layout(root: Path, *, first_party: str = "[]", py_present: bool = True) -> gate_runner.Layout: @@ -461,21 +471,29 @@ def test_complexipy_skips_python_free_repo_visibly(tmp_path: Path) -> None: assert gate_runner._complexipy(layout, {}) is None -def test_complexipy_targets_source_root_in_one_unpiped_process( +def test_complexipy_targets_source_root_in_unpiped_write_free_processes( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: - # With the snapshot present, complexipy grades the resolved source_root as a - # SINGLE process — piping it (as the mypy stage pipes through the filter) - # would mask its exit code (the tool-spike defect this rod fences off). - _write(tmp_path, "complexipy-snapshot.json", "{}") + # With the snapshot present, complexipy measures the resolved source_root in + # UNPIPED processes — piping it (as the mypy stage pipes through the filter) + # would mask its exit code (the tool-spike defect this rod fences off). BOTH + # argvs are now asserted EXHAUSTIVELY against measurement_argv, in order: the + # three-flag allowlist that stood here pinned what IS present and nothing + # about what is not, so `-mx 100` could join this argv with every test still + # green — and that flag empties the offender set outright, which is the + # vacuity vector this rung exists to close. + _write(tmp_path, "complexipy-snapshot.json", "[]") calls: list[tuple[list[str], str | None]] = [] _record_calls(monkeypatch, calls) - layout = _layout(tmp_path) - - verdict = gate_runner._complexipy(layout, {}) + tool = Path("/fake") / "complexipy" + # py_present=False is the honest fixture: tmp_path holds no .py at all, so an + # empty census is an empty WORLD, not the void the vacuity leg must refuse. + verdict = gate_runner._complexipy(_layout(tmp_path, py_present=False), {}) assert verdict is not None and verdict.passed - assert len(calls) == 1, "one unpiped process — no stdin handoff like the mypy filter pipe" - argv, stdin = calls[0] - assert argv == [str(Path("/fake") / "complexipy"), str(tmp_path / "src")] - assert stdin is None + assert [argv for argv, _ in calls] == [ + measurement_argv(tool, tmp_path / "src", offenders_only=offenders) + for offenders in (False, True) + ], "census then offenders, each argv exact — an allowlist would let -mx 100 in" + assert not {"-mx", "--max-complexity-allowed"} & set(calls[0][0]), "no kit-side budget" + assert [stdin for _, stdin in calls] == [None, None], "no stdin handoff like mypy's" diff --git a/tests/test_integration_consumer.py b/tests/test_integration_consumer.py index 539dd18..afdc184 100644 --- a/tests/test_integration_consumer.py +++ b/tests/test_integration_consumer.py @@ -107,9 +107,19 @@ def test_injected_type_error_flips_the_battery_red( ) -> None: consumer = _consumer_copy(tmp_path) _stub_pytest(monkeypatch) - # A module-level type error: mypy flags it, ruff does not (it is not a lint - # finding), so the mypy stage is the SOLE failure — the e2e path truly grades. - (consumer / "app" / "widget.py").write_text("bad: int = 'not an int'\n", encoding="utf-8") + # A module-level type error, APPENDED — never written OVER the module. mypy flags + # it; ruff does not (it is no lint finding, and the value is double-quoted so + # `ruff format --check` leaves it alone), so the mypy stage is the SOLE failure and + # the e2e path truly grades. Replacing widget.py would delete the fixture's ONLY + # function, leaving a source root that is Python-but-functionless: the complexipy + # stage then correctly refuses for having measured nothing + # (GATE_COMPLEXIPY_MEASURED_NOTHING — a setup error, exit 2), and the battery goes + # red for a vacuous complexity grade instead of for the injected type error. Red + # for the wrong reason is the failure mode this append exists to prevent. + widget = consumer / "app" / "widget.py" + widget.write_text( + widget.read_text(encoding="utf-8") + 'bad: int = "not an int"\n', encoding="utf-8" + ) verdicts = run_battery(consumer, os.environ) by_gate = {verdict.gate: verdict for verdict in verdicts}