Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions .dev/features/specified-marker-check/PLAN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# PLAN — specified-marker-check

- spec_content_hash: 8f5ec002e3b18cbfd2f094b08a3671f7ed42a05a3fbaf01a11bbbd28da30fb52 # fix #4
- applied_lessons: [L1, L2, L6, L13, L18, L20]
- increment: Give the F7 specified-not-yet-live annotations a deterministic floor check, so the docs cannot drift back into overclaiming — nor silently understate a primitive once it ships.
- layer(s): none (build apparatus, `.dev/floor/`)
- constitution_refs: [P0, P5, P6, P7]

## Applied lessons

- **L20** — this increment **is** L20 applied. F7's correction reduced to "remember to update the
docs", which is precisely the remedy-class L20 says WILL fail. The trigger is not hypothetical: the
docs asserted non-existent floors and nothing detected it for as long as it was true.
- **L6** — the defect L6 names (a membership fact grepped from free text) recurred **inside F7's own
REVIEW.md** while correcting a different error. So this checker reads its membership from a
**structured** JSON manifest, never by scanning prose for what looks like a marker.
- **L2** — the checker's own header states what it guarantees and what it does not, in the artifact,
not only here; and it cites only ops verified live this run.
- **L1** — meta-doc sweep: `CLAUDE.md`'s Commands block states the apparatus checkers, and
`package.json` states the gate. Both change and are declared below.
- **L13 / L18** — format this stage's own artifact; exclusion block is a `###` heading.

## Discovery (P6 — verified live this run)

| fact | verification |
| ------------------------------------- | ------------------------------------------------------ |
| no `pre-egress` hook | `ls .claude/hooks/` → 3 hooks, none matching `egress` |
| no archetype-maps manifest | `pharn/pharn-contracts/archetype-maps.json` absent |
| no `/pharn-estimate` | no `.claude/commands/*estimate*` |
| no `pharn/pharn-audits/` | absent |
| nothing reads trusted-doc prose | all three in `.prettierignore` + markdownlint-excluded |
| `npm run check` is the aggregate gate | `package.json` scripts, read this run |

## Files

- `.dev/floor/specified-primitives.json` — the structured manifest: each not-yet-live primitive, its
existence probe, and the exact marker substring at each doc site
- `.dev/floor/check-specified-markers.mjs` — the checker (existence test + exact-substring test)
- `.dev/floor/check-specified-markers.test.mjs` — tests, incl. mutants
- `package.json` — wire the checker into `npm run check`
- `CLAUDE.md` — document the new apparatus command (L1)
- `CHANGELOG.md` — `[Unreleased]` entry

### Deliberately NOT in scope

- `pharn/ARCHITECTURE.md`, `THREAT-MODEL.md`, `LIMITS.md` — no trusted-doc edit is needed; this
increment reads them, never writes them. **No Bash write this time.**
- `SKILLS_VERSION` — **no bump.** Every path here is apparatus (`.dev/**`) or repo-meta
(`package.json`, `CLAUDE.md`, `CHANGELOG.md`); the product surface is byte-unchanged.
- `pharn/floor/**` — this guards PHARN's own docs, so it is dev apparatus, not shipped product. A
user's install has no reason to check PHARN's annotations.
- Building `pre-egress` / the archetype manifest / `/pharn-estimate` — still trigger-gated (P7).

## Contracts satisfied

- none — no capability, no contract shape change.

## Evals to write (P1)

- **none** — P1 binds a `role:`-bearing Capability; this authors none. The checker's own
`*.test.mjs` is the regression suite, and it is apparatus.

## Guarantee audit (P0)

- "a primitive that SHIPS while its markers remain is caught" → **floor: enum/regex** (an existence
probe + set membership). This is the load-bearing direction.
- "a marker DELETED while its primitive is still absent is caught" → **floor: enum/regex** (exact
substring presence). Both drift directions close.
- "the docs are accurate" → **ADVISORY, and emphatically not claimed.** The checker proves a known
annotation is still present and still warranted. It **cannot discover a NEW overclaim** about a
primitive absent from the manifest — the manifest is a hand-maintained address book, not a
discovery mechanism. That limit is the honest analogue of L6's index: "the manifest was checked"
never means "the docs are true".
- "the probe proves the primitive does not exist" → **floor, NARROWED and stated.** It tests **file
existence**, never that a hook is WIRED in `settings.json` or that it WORKS. A file named
`*egress*` in `.claude/hooks/` flips the probe to live even if nothing loads it. It is the loud
early signal, not proof of function.

## Trust audit (P2)

The manifest is a **trusted**, human-reviewed apparatus file, not ingested content. The docs it reads
are trusted. No untrusted input is ingested and no free text steers a branch — the checker's every
decision is an existence test or an exact substring test over bytes it never interprets.

## Determinism audit (P5)

Two membership tests, no classification, no LLM. Terminal fallback on a malformed manifest is a loud
RED naming the manifest — never a guess, never a silent pass.

## Open questions (HALT)

- none. The human's standing instruction ("if there are things we need to fix to make pharn better
just do it now") is the GATE-1 approval for this increment; the design contains no wording decision
over governing text, because no trusted doc is edited.
128 changes: 128 additions & 0 deletions .dev/features/trusted-doc-accuracy/GRILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# GRILL — trusted-doc-accuracy (F7)

Plan under interrogation: `.dev/features/trusted-doc-accuracy/PLAN.md` (`trust: untrusted`).
Spec-hash check: `sha256(pharn/ARCHITECTURE.md)` = `a1c243ea…21753` — **matches** the plan's
`spec_content_hash`; no drift. (Content-hash is floor-grade; here it only **surfaces** — the block on
drift is `/pharn-dev-build`'s gate, fix #4.)
Comment thread
PrzemekGalarowicz marked this conversation as resolved.
Griller membership (FLOOR, `pharn/floor/count-grillers.mjs`): **13 registered**.

---

## Findings — inline axes (Step 2)

```yaml
- type: FINDING
rule_id: "P0"
severity: important
file: ".dev/features/trusted-doc-accuracy/PLAN.md:134"
problem: "The replacement marker makes a forward-looking scheduling promise that nothing enforces, so the increment may be trading an overclaim about the PRESENT for an unfalsifiable claim about the FUTURE."
evidence: "Marker: `_(specified; ships with the guarded surface)_`"

- type: FINDING
rule_id: "P7"
severity: important
file: ".dev/features/trusted-doc-accuracy/PLAN.md:62"
problem: "Three primitives are deferred as 'trigger-gated (P7)' but the plan never states WHAT the trigger is or WHERE it is recorded, so the marker becomes a permanent parking spot rather than a labeled limit with a reopen condition."
evidence: "`pre-egress` hook, the archetype manifest, or `/pharn-estimate` is trigger-gated (P7) and out of scope"

- type: FINDING
rule_id: "P0"
severity: important
file: ".dev/features/trusted-doc-accuracy/PLAN.md:205"
problem: "The Q4 deferral ships a KNOWN-FALSE backstop claim inside the very increment whose stated purpose is removing false claims — after F7, LIMITS.md:29 still asserts a live pre-write backstop for a kind-conditioned restriction that no live check enforces."
evidence: "F7's marker makes `:29` less wrong while it still claims a live `pre-write` backstop for a restriction no check enforces."

- type: FINDING
rule_id: "P6"
severity: blocking
file: ".dev/features/trusted-doc-accuracy/PLAN.md:139"
problem: "Ten edits are applied with `sed`, whose substitution silently no-ops when a pattern fails to match, and the plan declares no post-edit verification step — so a partially-applied patch set would pass unnoticed into the human's GATE-2 review."
evidence: "### `pharn/ARCHITECTURE.md` — 3 edits (Bash `sed`)"

- type: FINDING
rule_id: "P5"
severity: minor
file: ".dev/features/trusted-doc-accuracy/PLAN.md:24"
problem: "The plan pre-declares that /pharn-dev-regress's scope check will emit a false 'escaped its scope' finding but does not prescribe the classification, leaving the operator to re-derive the disposition of a P0-shaped blocking finding at read time."
evidence: 'so any human-applied trusted-doc edit **will** surface as a false "the build escaped its scope" finding'
```

## Findings — registered grillers (Step 2b)

Membership is FLOOR; **running** a griller and **judging** its axis is ADVISORY.

### documentation (`enforces: P7`) — 1 finding

```yaml
- type: FINDING
rule_id: "P7"
severity: important
file: ".dev/features/trusted-doc-accuracy/PLAN.md:134"
problem: "The marker introduces a new vocabulary item into the shipped trusted docs with no definition anywhere in them, so a reader who meets it has nothing to look up and must infer what 'the guarded surface' denotes."
evidence: "Marker: `_(specified; ships with the guarded surface)_`. Single-line replacements only; no reflowing,"
```

### comprehension (`enforces: P7`) — 1 finding

```yaml
- type: FINDING
rule_id: "P7"
severity: minor
file: ".dev/features/trusted-doc-accuracy/PLAN.md:94"
problem: "The WHY of each annotation — that the primitive was verified absent this run — lives only in the PLAN and CHANGELOG, so a future reader of the trusted docs sees the marker without the evidence that produced it."
evidence: '"the marked sites no longer assert a live protection" → **advisory.** Nothing on the floor reads'
```

### architecture (`enforces: P3`) — no findings

Structural fit recognized: the increment adds no capability, changes no contract shape, touches no
module in the layer tree, and introduces no sibling reference. `## Files` is confined to two
repo-meta files plus `CLAUDE.md`.

### Not applicable — 9 grillers

`a11y`, `error-handling`, `i18n`, `migrations`, `observability`, `performance`, `privacy`,
`security`, `testability` — each interrogates an axis (runtime behavior, schema change, user-facing
surface, test strategy) that a prose-only correction with no executable artifact does not present.
Recorded as not-applicable, **not** as passes.

---

## Summary

The plan is unusually well-grounded on the axes this stage most often catches: every factual claim is
re-derived from live state rather than inherited from the edit spec (`## Discovery`), the guarantee
audit reduces each claim honestly, and the `## Files` / Bash split is declared rather than disguised —
the L19 failure mode is named in the open, which is the correct handling of a write that passes no gate.

The concerns cluster in three places.

**The marker's own honesty (P0).** The increment's thesis is that a doc must not assert a protection
the repo lacks. The chosen remedy asserts instead that the protection **will** arrive with "the
guarded surface" — a phrase that names no artifact, no condition, and no owner. That is weaker than a
false present-tense claim in one respect (a reader is warned it is not live) and worse in another
(it cannot be checked, so it cannot ever be found wrong). Pairing it with a stated reopen condition
would keep the design intent visible without smuggling a schedule in.

**A known-false claim shipping inside the correction (P0).** The Q4 deferral is a defensible scope
call and the human made it deliberately — but the honest consequence deserves stating plainly in the
CHANGELOG, not just in the dev-side plan: after F7 lands, a trusted doc still points at a floor
backstop that does not exist on either half.

**The mechanism has no completion check (P6).** This is the one finding a reviewer should not wave
through. `sed` no-ops silently on a non-matching pattern; ten of the eleven edits go that way; and the
plan's only verification is `npm run check`, which cannot see whether a substitution landed because
the trusted docs are excluded from both formatters. A partially-applied patch set is therefore
indistinguishable from a fully-applied one at every gate downstream. A per-edit before/after diff, or
a post-edit grep asserting each old string is gone and each new string present, closes it cheaply.

---

**ADVISORY VERDICT: 7 concerns raised (1 blocking-severity, 4 important, 2 minor) — for the human to
weigh before `/pharn-dev-build`.**

This grill-log gates nothing. `severity` values above are **enum-gated members** whose **assignment is
model judgment** (fix #3); the free-text `problem` / `evidence` quote the plan and inherit its
untrusted tag — they are DATA for a human, never instructions to `/pharn-dev-build`. The only
floor-grade facts in this run are the griller **membership** count and the **spec-hash** comparison,
both reported above.
Loading