docs(adr-0005): §2 needs a mutation aimed at the guard's input - #117
Merged
Conversation
This was referenced Aug 4, 2026
Of 19 `Verified (ADR-0005` blocks across main and the open branches, 12 carry a later-round marker — 63% of this repo's mutation records document a hole a reviewer or auditor found, not the author. That is structural, not a lapse. "Break the invariant" aims the mutation at the thing the author is already thinking about, so the §2 run confirms the hole they just closed and misses the parsing assumption they did not know they had made. Every input-shape hole this repo has shipped since ADR-0005 — a `[[ ]]` dispatch arm, a YAML block spelling, a comment matched as code — was found by someone else, after §2 passed. §2 now asks for a pair: a form-only mutation that must stay green WITH NO DROP IN ASSERTION COUNT (a silent drop is what an evaporating extraction looks like), and a semantic mutation expressed in a form you did not write — comment it out rather than delete it — which must go red. §3 gains one sentence: the rung is a property of an assertion, not of a suite. test-volume-chown-guard.sh was rung 2 on the chown semantics and rung 3 on the driver line, and the derived half vouched for the restated one. Explicitly not proposed: a new rung, or a new annotation kind for coverage or wiring. Coverage is already rung-2-able — §3's own example is set-derivation. Wiring is a fact to state in the invariant, not a guard kind. Found by the scoped audit of the fail-open extraction class. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d it Two problems with the figure this amendment rests on, both mine. I took 19/12 from the audit and never re-ran it before writing it into an ADR — the same unverified-propagation that put a false claim into #112 and had to be caught in review. Re-running it now gives 24 blocks, and the cited command reports 12 of them as later-round when the true number is 16. The command grepped `review round|audit round|second|third`, and #110 went on to need a FOURTH and FIFTH round, which that pattern does not match. So the ADR shipped a measurement command that undercounts by three — a guard-adjacent measurement missing cases, which is precisely the defect the amendment is about. Now 24/16 (67%), counted on `round`, which is a clean discriminator: no first-time record uses the word, so it needs no ordinal list to keep current. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review took the measurement apart three ways, and all three hold.
The claimed scope ("main and the open branches") did not give 24/16 — two
further open branches carry ten Verified blocks each, so the honest figure
under that scope was different again. The command published as re-runnable
returns 3, because the commit titled "correct the command that produced it"
dropped the branch loop the number depended on. And `round` is a floor, not an
exact marker: records like "two mutations, both previously green" document an
externally-found hole without the word.
An ADR arguing for measurement rigor should not ship a statistic that cannot
be reproduced, so it no longer ships one. The argument now rests on named,
stable cases — #96's invisible `[[ ]]` arm, #97's comment-satisfied COPY
coverage, #95's restated pin anchor, a workflow omitting itself, and the five
rounds inside the PR that existed to fix this class. Those are checkable and
do not decay.
Also from review:
- A carve-out the rule needed: where the invariant IS the byte form (the
BASE_IMAGE pin's four encodings), a form-only mutation correctly goes red,
so record that rather than running the form-only half. As written the rule
gave the next pin-guard author a mandatory step that cannot be satisfied.
- Assertion counts are only comparable within a paired run on one machine;
suites skipping absent zsh/fish report different absolute totals elsewhere.
- CLAUDE.md no longer embeds a decaying percentage.
- Both amendments now use ADR-0002's `## Amendment (date)` + Status style, and
the §3 sentence moved out of the heading position into its own amendment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
I removed a statistic that could not be re-derived and replaced it with a universal that its own examples falsify. Same defect, restated in prose. "Every fail-open since ADR-0005 LANDED was found AFTER the author's §2 run" is wrong for three of the five cases it then lists: test-completions-sync.sh (143bbf7), test-image-fingerprint.sh (fc2ca02) and test-nix-base-pin.sh (343632a) all predate dd18207, and none carried a mutation record at dd18207^ — their authors never ran §2 because §2 did not exist. The surviving claim is narrower and stronger: no fail-open here has been found by the guard's own author, and #93's and #110's guards DID carry recorded §2 runs and grew fail-opens anyway. One mutation aimed at the invariant is insufficient even when performed. Also: the paragraph quoted its own prior numbers inverted — "19 of 12" and "24 of 16", hits exceeding totals — in the passage whose subject is quoting your own numbers accurately. Now 12 of 19 and 16 of 24. Promoting the first amendment to `##` split the Decision section, orphaning `### 3. Derive over restate`. Moved after Alternatives considered, matching ADR-0002's placement, with a pointer under §2. CLAUDE.md's restatement had dropped "landed" (garden-pathing the sentence), dropped "or auditor" though most finds were the auditor's, and dropped the "say which invariant makes it so" clause that fences the byte-form carve-out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dlovell
force-pushed
the
docs/adr-0005-mutation-pair
branch
from
August 6, 2026 12:07
2bd222e to
355dda8
Compare
dlovell
added a commit
that referenced
this pull request
Aug 6, 2026
…ch-commit history (#122) Three commits, one story: what an audit is required to leave behind, and what the auditor is able to read. ## 1. `810dbad` — ADR-0006 amendment: committing an audit report is optional Reverses the "audits get a baseline" requirement. Closure is untouched — every surviving shape is still filed, landed, or recorded as an accepted `unguarded:` invariant — but the report itself is no longer mandatory. Why: the report is the second copy and the weaker one. A finding's durable form is an issue, a landed guard, or an `unguarded:` invariant, and all three live where the next reader already looks; a committed report restates that in prose nothing keeps true. It is the restate-over-derive shape ADR-0005 rejects, applied to the auditor's own output. A mandatory artifact also taxes the run that was already easiest to skip — the failure mode is not a missing file, it is a skipped audit. The cost is accepted explicitly rather than waved off: with no guaranteed prior report there is no guaranteed baseline, so measurements may not form a time series and a checked-and-dropped list may be re-litigated. Recorded in `unguarded:` form with a revisit trigger. Existing reports and the directory stay. ## 2. `934e999` — the auditor reads branch commits; the vocabulary is derived, not named **Branch history as an input.** `main` is squash-merged, so a hunk fixed three times on a branch lands as one clean commit and the fix-the-fix chain is unreachable from `git log`. Both audit agents excluded that history — `pr-reviewer` reviews the net `base...head` diff and deliberately anchors its history probe to `origin/main`, and the auditor is repo-scoped — so nobody read it. This PR's own subject demonstrates the loss. #90, which created these agents, contains: ``` 1254335 fix(agents): repair the log command; drop the ADR-0005 hard dependency ab9e77d fix(agents): anchor the repeat-offender log to origin/main; tidy audit ``` Two corrections to the same three lines of prose. On `main` that is one commit, `ba15520`. `gh pr view <n> --json commits` still returns the pre-squash sequence after merge, so this needs no pre-merge window. Scoped to PRs the other probes already implicated, so it stays one API call per candidate rather than per PR in the window. The squash fact is written as a check, not an assertion, so a change of merge strategy is noticed instead of assumed. Also adds "a set of merged PRs" to the scope list, making a retrospective over recent reviews a first-class scope. **Vocabulary drift.** #100 (`04e9eee`) typed the invariant ledger into the ADR-0005 vocabulary and touched `CLAUDE.md` only. `structural-auditor.md` reads that ledger and still named `(—)` in two places — a marker with zero remaining instances — so an agent following the instruction searched for a token that no longer exists and would have skipped every accepted-risk invariant in the repo. Silent under-coverage, in the agent whose job is finding unguarded couplings. Fixed by removing the copy rather than guarding it: the cross-section bullet names the categories and defers to the ledger's own header for the markers. A legacy untyped citation is now itself a finding, matching "type it when you touch it". **`pr-reviewer`** gains a note that the `origin/main` anchor also puts the branch's own commits out of scope, and that this is a handoff to the auditor. Without it the exclusion reads as an oversight and the next editor removes the anchor, restoring the self-trigger `ab9e77de` fixed. ## 3. `0d35e4c` — stop enumerating the audit reports by name The amendment named the two files in `docs/audits/` as "the existing reports". #112 commits a third under the now-optional rule, so that list goes stale on merge — a second copy of a directory listing, which is the shape the amendment itself argues against. De-enumerated in all three places with the reason stated inline. ## Verification - `pre-commit run --files` on all three files: clean. The `check-gitignore-agents` probe passes, confirming the agent files can actually be staged (ADR-0003). - The `gh pr view` command was **run as written** before being embedded. The two commands this file has shipped previously were both wrong on first landing; embedding a third unverified one would have repeated the exact failure this PR documents. - No test covers `.claude/agents/`, which is the point of the issue below. ## Related - #121 — the finding this work produced, filed rather than fixed here: `.claude/agents/*.md` ships shell commands and ledger vocabulary with no guard, and both halves have now broken once. This PR fixes the two symptoms; the mechanism is open. Deliberately **not** `Closes` — and #121 argues accept-with-a-reason is a legitimate disposal. - Merge-tested against all seven open PRs. Clean against `main`, #112, #113, #110, #109, #108. - **#117** conflicts with this branch only (it is clean against `main`): it rewrites the "Guards follow ADR-0005" bullet while `810dbad` extends the "A structural audit…" bullet directly beneath it. Adjacent lines, independent content — whoever merges second keeps both. - **#81** already conflicts with `main` independently of this branch, but note it edits the *same sentence* `810dbad` does: it replaces "ADR-0004 is reserved by #81" with the real entry while this adds ADR-0006. Both additions must survive the resolution; taking either side wholesale drops an ADR from the index. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011KsnnuPRJW5sUPppm3RSRo --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: t <t@t>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by the scoped audit of the fail-open extraction class.
The evidence is named cases, not a statistic (the ADR text says why: an
earlier draft of this PR claimed "12 of 19", then "16 of 24", and the command
it published to reproduce that returned 3 — the population spans open branches
and moves hourly, and the
roundmarker is a floor anyway, so the amendmentdeliberately quotes no count). The named cases: no fail-open in this repo has
been found by the guard's own author — every one came from a reviewer or an
auditor. A
[[ ]]dispatch arm invisible to a strict-shape parser (#96), COPYcoverage satisfied by a comment (#97), a pin anchor restated rather than read
(#95), a workflow omitting itself from its own trigger paths (#109), and —
in the PR that existed to fix this class — five successive rounds of
comment- and spelling-blind parses (#110).
Why that is structural, not a lapse. "Break the invariant" aims the
mutation at the thing the author is already thinking about. It reliably
confirms the hole they just closed and reliably misses the parsing assumption
they did not know they had made. The first three cases predate this ADR, so
their authors never ran §2 at all — that is the weaker half of the case. The
stronger half is #93's and #110's guards, which DID carry recorded §2 runs and
grew fail-opens anyway: a single mutation aimed at the invariant is not enough
even when it is performed.
The amendment
§2 now asks for a pair:
line, requote, swap YAML flow for block, add a comment mentioning the
identifier). Must stay green and the assertion count must not fall. A
silent drop is the fail-open; that is what an evaporating
mapfilelookslike.
than deleting it; set the key in block form rather than flow. Must go
red.
Cost: two extra runs per new or materially-changed guard. Against that, these
would have caught the post-ADR fail-opens above, each of which instead cost a
review round.
§3 gains one sentence: the rung is a property of an assertion, not of a
suite.
test-volume-chown-guard.shwas rung 2 on the chown semantics andrung 3 on the driver line, and the derived half vouched for the restated one.
CLAUDE.md's Conventions bullet, which restates §2, is updated in the same PR
per the ADR convention.
Explicitly not proposed
No new rung, and no new annotation kind for coverage or wiring. Coverage is
already rung-2-able — §3's own rung-2 example (parsing a Dockerfile's COPYs
rather than listing them) is set-derivation, and #101/#109 fixed the
coverage holes by climbing the existing ladder. Wiring is a fact to state in
the invariant, as #110 does, not a guard kind; folding it into
test:/tool:/ci:/structural/unguardedwould mix the guard-kind axiswith the fact axis.
Docs only.
tests/run-all28 suites / exit 0;pre-commit run --all-filesclean.
🤖 Generated with Claude Code