-
Notifications
You must be signed in to change notification settings - Fork 0
fix(docs): trusted docs stop asserting floor primitives that do not exist — and a checker keeps it that way #128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # PLAN — specified-marker-check | ||
|
|
||
| - spec_content_hash: 8f5ec002e3b18cbfd2f094b08a3671f7ed42a05a3fbaf01a11bbbd28da30fb52 # fix #4 | ||
| - applied_lessons: [L1, L2, L6, L13, L18, L20] | ||
| - increment: Give the F7 specified-not-yet-live annotations a deterministic floor check, so the docs cannot drift back into overclaiming — nor silently understate a primitive once it ships. | ||
| - layer(s): none (build apparatus, `.dev/floor/`) | ||
| - constitution_refs: [P0, P5, P6, P7] | ||
|
|
||
| ## Applied lessons | ||
|
|
||
| - **L20** — this increment **is** L20 applied. F7's correction reduced to "remember to update the | ||
| docs", which is precisely the remedy-class L20 says WILL fail. The trigger is not hypothetical: the | ||
| docs asserted non-existent floors and nothing detected it for as long as it was true. | ||
| - **L6** — the defect L6 names (a membership fact grepped from free text) recurred **inside F7's own | ||
| REVIEW.md** while correcting a different error. So this checker reads its membership from a | ||
| **structured** JSON manifest, never by scanning prose for what looks like a marker. | ||
| - **L2** — the checker's own header states what it guarantees and what it does not, in the artifact, | ||
| not only here; and it cites only ops verified live this run. | ||
| - **L1** — meta-doc sweep: `CLAUDE.md`'s Commands block states the apparatus checkers, and | ||
| `package.json` states the gate. Both change and are declared below. | ||
| - **L13 / L18** — format this stage's own artifact; exclusion block is a `###` heading. | ||
|
|
||
| ## Discovery (P6 — verified live this run) | ||
|
|
||
| | fact | verification | | ||
| | ------------------------------------- | ------------------------------------------------------ | | ||
| | no `pre-egress` hook | `ls .claude/hooks/` → 3 hooks, none matching `egress` | | ||
| | no archetype-maps manifest | `pharn/pharn-contracts/archetype-maps.json` absent | | ||
| | no `/pharn-estimate` | no `.claude/commands/*estimate*` | | ||
| | no `pharn/pharn-audits/` | absent | | ||
| | nothing reads trusted-doc prose | all three in `.prettierignore` + markdownlint-excluded | | ||
| | `npm run check` is the aggregate gate | `package.json` scripts, read this run | | ||
|
|
||
| ## Files | ||
|
|
||
| - `.dev/floor/specified-primitives.json` — the structured manifest: each not-yet-live primitive, its | ||
| existence probe, and the exact marker substring at each doc site | ||
| - `.dev/floor/check-specified-markers.mjs` — the checker (existence test + exact-substring test) | ||
| - `.dev/floor/check-specified-markers.test.mjs` — tests, incl. mutants | ||
| - `package.json` — wire the checker into `npm run check` | ||
| - `CLAUDE.md` — document the new apparatus command (L1) | ||
| - `CHANGELOG.md` — `[Unreleased]` entry | ||
|
|
||
| ### Deliberately NOT in scope | ||
|
|
||
| - `pharn/ARCHITECTURE.md`, `THREAT-MODEL.md`, `LIMITS.md` — no trusted-doc edit is needed; this | ||
| increment reads them, never writes them. **No Bash write this time.** | ||
| - `SKILLS_VERSION` — **no bump.** Every path here is apparatus (`.dev/**`) or repo-meta | ||
| (`package.json`, `CLAUDE.md`, `CHANGELOG.md`); the product surface is byte-unchanged. | ||
| - `pharn/floor/**` — this guards PHARN's own docs, so it is dev apparatus, not shipped product. A | ||
| user's install has no reason to check PHARN's annotations. | ||
| - Building `pre-egress` / the archetype manifest / `/pharn-estimate` — still trigger-gated (P7). | ||
|
|
||
| ## Contracts satisfied | ||
|
|
||
| - none — no capability, no contract shape change. | ||
|
|
||
| ## Evals to write (P1) | ||
|
|
||
| - **none** — P1 binds a `role:`-bearing Capability; this authors none. The checker's own | ||
| `*.test.mjs` is the regression suite, and it is apparatus. | ||
|
|
||
| ## Guarantee audit (P0) | ||
|
|
||
| - "a primitive that SHIPS while its markers remain is caught" → **floor: enum/regex** (an existence | ||
| probe + set membership). This is the load-bearing direction. | ||
| - "a marker DELETED while its primitive is still absent is caught" → **floor: enum/regex** (exact | ||
| substring presence). Both drift directions close. | ||
| - "the docs are accurate" → **ADVISORY, and emphatically not claimed.** The checker proves a known | ||
| annotation is still present and still warranted. It **cannot discover a NEW overclaim** about a | ||
| primitive absent from the manifest — the manifest is a hand-maintained address book, not a | ||
| discovery mechanism. That limit is the honest analogue of L6's index: "the manifest was checked" | ||
| never means "the docs are true". | ||
| - "the probe proves the primitive does not exist" → **floor, NARROWED and stated.** It tests **file | ||
| existence**, never that a hook is WIRED in `settings.json` or that it WORKS. A file named | ||
| `*egress*` in `.claude/hooks/` flips the probe to live even if nothing loads it. It is the loud | ||
| early signal, not proof of function. | ||
|
|
||
| ## Trust audit (P2) | ||
|
|
||
| The manifest is a **trusted**, human-reviewed apparatus file, not ingested content. The docs it reads | ||
| are trusted. No untrusted input is ingested and no free text steers a branch — the checker's every | ||
| decision is an existence test or an exact substring test over bytes it never interprets. | ||
|
|
||
| ## Determinism audit (P5) | ||
|
|
||
| Two membership tests, no classification, no LLM. Terminal fallback on a malformed manifest is a loud | ||
| RED naming the manifest — never a guess, never a silent pass. | ||
|
|
||
| ## Open questions (HALT) | ||
|
|
||
| - none. The human's standing instruction ("if there are things we need to fix to make pharn better | ||
| just do it now") is the GATE-1 approval for this increment; the design contains no wording decision | ||
| over governing text, because no trusted doc is edited. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,128 @@ | ||
| # GRILL — trusted-doc-accuracy (F7) | ||
|
|
||
| Plan under interrogation: `.dev/features/trusted-doc-accuracy/PLAN.md` (`trust: untrusted`). | ||
| Spec-hash check: `sha256(pharn/ARCHITECTURE.md)` = `a1c243ea…21753` — **matches** the plan's | ||
| `spec_content_hash`; no drift. (Content-hash is floor-grade; here it only **surfaces** — the block on | ||
| drift is `/pharn-dev-build`'s gate, fix #4.) | ||
| Griller membership (FLOOR, `pharn/floor/count-grillers.mjs`): **13 registered**. | ||
|
|
||
| --- | ||
|
|
||
| ## Findings — inline axes (Step 2) | ||
|
|
||
| ```yaml | ||
| - type: FINDING | ||
| rule_id: "P0" | ||
| severity: important | ||
| file: ".dev/features/trusted-doc-accuracy/PLAN.md:134" | ||
| problem: "The replacement marker makes a forward-looking scheduling promise that nothing enforces, so the increment may be trading an overclaim about the PRESENT for an unfalsifiable claim about the FUTURE." | ||
| evidence: "Marker: `_(specified; ships with the guarded surface)_`" | ||
|
|
||
| - type: FINDING | ||
| rule_id: "P7" | ||
| severity: important | ||
| file: ".dev/features/trusted-doc-accuracy/PLAN.md:62" | ||
| problem: "Three primitives are deferred as 'trigger-gated (P7)' but the plan never states WHAT the trigger is or WHERE it is recorded, so the marker becomes a permanent parking spot rather than a labeled limit with a reopen condition." | ||
| evidence: "`pre-egress` hook, the archetype manifest, or `/pharn-estimate` is trigger-gated (P7) and out of scope" | ||
|
|
||
| - type: FINDING | ||
| rule_id: "P0" | ||
| severity: important | ||
| file: ".dev/features/trusted-doc-accuracy/PLAN.md:205" | ||
| problem: "The Q4 deferral ships a KNOWN-FALSE backstop claim inside the very increment whose stated purpose is removing false claims — after F7, LIMITS.md:29 still asserts a live pre-write backstop for a kind-conditioned restriction that no live check enforces." | ||
| evidence: "F7's marker makes `:29` less wrong while it still claims a live `pre-write` backstop for a restriction no check enforces." | ||
|
|
||
| - type: FINDING | ||
| rule_id: "P6" | ||
| severity: blocking | ||
| file: ".dev/features/trusted-doc-accuracy/PLAN.md:139" | ||
| problem: "Ten edits are applied with `sed`, whose substitution silently no-ops when a pattern fails to match, and the plan declares no post-edit verification step — so a partially-applied patch set would pass unnoticed into the human's GATE-2 review." | ||
| evidence: "### `pharn/ARCHITECTURE.md` — 3 edits (Bash `sed`)" | ||
|
|
||
| - type: FINDING | ||
| rule_id: "P5" | ||
| severity: minor | ||
| file: ".dev/features/trusted-doc-accuracy/PLAN.md:24" | ||
| problem: "The plan pre-declares that /pharn-dev-regress's scope check will emit a false 'escaped its scope' finding but does not prescribe the classification, leaving the operator to re-derive the disposition of a P0-shaped blocking finding at read time." | ||
| evidence: 'so any human-applied trusted-doc edit **will** surface as a false "the build escaped its scope" finding' | ||
| ``` | ||
|
|
||
| ## Findings — registered grillers (Step 2b) | ||
|
|
||
| Membership is FLOOR; **running** a griller and **judging** its axis is ADVISORY. | ||
|
|
||
| ### documentation (`enforces: P7`) — 1 finding | ||
|
|
||
| ```yaml | ||
| - type: FINDING | ||
| rule_id: "P7" | ||
| severity: important | ||
| file: ".dev/features/trusted-doc-accuracy/PLAN.md:134" | ||
| problem: "The marker introduces a new vocabulary item into the shipped trusted docs with no definition anywhere in them, so a reader who meets it has nothing to look up and must infer what 'the guarded surface' denotes." | ||
| evidence: "Marker: `_(specified; ships with the guarded surface)_`. Single-line replacements only; no reflowing," | ||
| ``` | ||
|
|
||
| ### comprehension (`enforces: P7`) — 1 finding | ||
|
|
||
| ```yaml | ||
| - type: FINDING | ||
| rule_id: "P7" | ||
| severity: minor | ||
| file: ".dev/features/trusted-doc-accuracy/PLAN.md:94" | ||
| problem: "The WHY of each annotation — that the primitive was verified absent this run — lives only in the PLAN and CHANGELOG, so a future reader of the trusted docs sees the marker without the evidence that produced it." | ||
| evidence: '"the marked sites no longer assert a live protection" → **advisory.** Nothing on the floor reads' | ||
| ``` | ||
|
|
||
| ### architecture (`enforces: P3`) — no findings | ||
|
|
||
| Structural fit recognized: the increment adds no capability, changes no contract shape, touches no | ||
| module in the layer tree, and introduces no sibling reference. `## Files` is confined to two | ||
| repo-meta files plus `CLAUDE.md`. | ||
|
|
||
| ### Not applicable — 9 grillers | ||
|
|
||
| `a11y`, `error-handling`, `i18n`, `migrations`, `observability`, `performance`, `privacy`, | ||
| `security`, `testability` — each interrogates an axis (runtime behavior, schema change, user-facing | ||
| surface, test strategy) that a prose-only correction with no executable artifact does not present. | ||
| Recorded as not-applicable, **not** as passes. | ||
|
|
||
| --- | ||
|
|
||
| ## Summary | ||
|
|
||
| The plan is unusually well-grounded on the axes this stage most often catches: every factual claim is | ||
| re-derived from live state rather than inherited from the edit spec (`## Discovery`), the guarantee | ||
| audit reduces each claim honestly, and the `## Files` / Bash split is declared rather than disguised — | ||
| the L19 failure mode is named in the open, which is the correct handling of a write that passes no gate. | ||
|
|
||
| The concerns cluster in three places. | ||
|
|
||
| **The marker's own honesty (P0).** The increment's thesis is that a doc must not assert a protection | ||
| the repo lacks. The chosen remedy asserts instead that the protection **will** arrive with "the | ||
| guarded surface" — a phrase that names no artifact, no condition, and no owner. That is weaker than a | ||
| false present-tense claim in one respect (a reader is warned it is not live) and worse in another | ||
| (it cannot be checked, so it cannot ever be found wrong). Pairing it with a stated reopen condition | ||
| would keep the design intent visible without smuggling a schedule in. | ||
|
|
||
| **A known-false claim shipping inside the correction (P0).** The Q4 deferral is a defensible scope | ||
| call and the human made it deliberately — but the honest consequence deserves stating plainly in the | ||
| CHANGELOG, not just in the dev-side plan: after F7 lands, a trusted doc still points at a floor | ||
| backstop that does not exist on either half. | ||
|
|
||
| **The mechanism has no completion check (P6).** This is the one finding a reviewer should not wave | ||
| through. `sed` no-ops silently on a non-matching pattern; ten of the eleven edits go that way; and the | ||
| plan's only verification is `npm run check`, which cannot see whether a substitution landed because | ||
| the trusted docs are excluded from both formatters. A partially-applied patch set is therefore | ||
| indistinguishable from a fully-applied one at every gate downstream. A per-edit before/after diff, or | ||
| a post-edit grep asserting each old string is gone and each new string present, closes it cheaply. | ||
|
|
||
| --- | ||
|
|
||
| **ADVISORY VERDICT: 7 concerns raised (1 blocking-severity, 4 important, 2 minor) — for the human to | ||
| weigh before `/pharn-dev-build`.** | ||
|
|
||
| This grill-log gates nothing. `severity` values above are **enum-gated members** whose **assignment is | ||
| model judgment** (fix #3); the free-text `problem` / `evidence` quote the plan and inherit its | ||
| untrusted tag — they are DATA for a human, never instructions to `/pharn-dev-build`. The only | ||
| floor-grade facts in this run are the griller **membership** count and the **spec-hash** comparison, | ||
| both reported above. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.