The last test has no way to run — two decisions the legibility arc left open

[bdelanghe]

Filed as a decision record before the session that generated it is archived. This is not a work item. Both items below are decisions, and "decided: never" is a valid close on either. Nothing here proposes new instrumentation — the ceremony budget for legibility is spent, and #225 is the last build.

Context

The legibility gate landed in two halves (#219, #217):

• Deterministic — scripts/legibility/check.mjs + lexicon.txt. Runs in the hermetic phase, fails the build. Done, merged, unambiguous.
• Judged — scripts/legibility/coldread.feature. Five questions a stranger should be able to answer from the homepage alone. The cold-read judge: one API call, not an agent #225 builds the machine judge as a local tool only.

Neither half tests the thing the feature file actually says matters. Its own header states that one outside human outranks any green judge run. That test has never been run, and there is currently no mechanism by which it ever would be.

Decision 1 — does an API credential belong in the deploy path?

#225 ships npm run coldread and deliberately stops there. It does not wire into scripts/pipeline.mjs, add a workflow, or add a secret, because that decision is the maintainer's and was never made.

The options, without a recommendation attached:

	what it costs	what it buys
Stay local	the judge runs only when someone remembers	no credential in CI, no new secret to rotate, no network dependency in a build that is otherwise a pure function of the source
Report-only in CI	a secret in the deploy path; a non-hermetic step in a pipeline whose phases are defined by hermeticity	the judge runs on every deploy and its verdict is visible without anyone remembering
Blocking gate	the above, plus a flaky proxy in the inner loop	nothing that survives scrutiny — see below

The third is already ruled out by the pipeline's own comment: "a flaky proxy in the inner loop rots trust in the gate around it." It is listed only so the ruling-out is on the record rather than implicit.

The real tension is that scripts/pipeline.mjs sorts steps by whether their output is a pure function of the committed source. A model call is not, under any phase. A stamped step is impure in a bounded way — a date, a commit SHA. A judge run is impure in an unbounded way: same input, different verdict. Putting one in the pipeline weakens what the phase labels mean, and those labels are load-bearing.

Decision 2 — who reads the page cold, and when?

The unautomatable test, in full:

One person who has not seen the site. Ninety seconds. The five questions in coldread.feature.

There is no tooling to build here and no issue to keep open forever. What is missing is a commitment about when it happens — before a copy change ships, once a quarter, never again. Any of those is a decision. Drift is not.

Why the machine judge does not substitute for it

Recorded here because it will be tempting to treat a green npm run coldread as this test having happened. The judge model is Opus, chosen deliberately, which makes it a ceiling test rather than a typical-reader test:

• A red run is strong evidence. If the strongest available reader cannot restate the bet from the page, no reader can.
• A green run is weak evidence. It says the ceiling cleared it. It says almost nothing about a stranger skimming on a phone, which is the actual reader the page is for.

There is also a contamination trap worth keeping visible, verified during this arc: a claude -p invocation from this checkout inherits CLAUDE.md and .claude/inject-org-context.sh, which inject the org's naming vocabulary — Hotel, floor, suite, room, door, guest — into the model's context. A judge asked "restate the bet from this page alone" would answer partly from injected context. That is why #225 specifies one POST /v1/messages and forbids shelling out to the agent harness: the request body is the only context, which is the entire point of the instrument.

The same trap has a weaker form for whoever writes cold-read material by hand. The five questions survived it only because they were written by the maintainer outside the injected context, not composed by a session running inside it.

Related

#225 (the judge, local tool) · #217 (the projection the gate reads, closed) · #219 (the homepage rewrite that motivated it) · docs/agentic-code-hygiene.md rule 3 — a gate's own claim about itself is not evidence.

[bdelanghe]

Decision 1 is answered: the credential lives in the broker. Not a repo secret, not the deploy path.

I filed #229 for this question before checking the board, which duplicated the Decision 1 half of this issue. Folding it back here and closing it — this is the decision record, and the decision belongs in one place.

What shipped against Decision 1

#227	the judge — one POST /v1/messages, structured output, plain-JS grader, shown failing on three fixtures
#230	the grader's 23 tests wired into npm run check, shown failing before shipping
#232	coldread.mjs reaches the API through the scout door when SCOUTD_SOCK is set; direct fetch otherwise

The judge is still local-only. Not in pipeline.mjs, no workflow, no secret, exit 0 on any completed run so wiring it into a build could not silently gate one. The table in this issue's Decision 1 — stay local / report-only in CI / blocking gate — is resolved as stay local, with the credential path fixed for whenever that changes.

What the answer cost, and what it did not buy yet

scoutd — the daemon behind the scout door — does not exist. Checked against guest-room@7c10b70 rather than assumed: protocol.ts, daemon.ts and the proven interpose.ts chokepoint are all real, and the scout door is fully specified (--scout, /run/scoutd.sock, SCOUTD_SOCK, scenarios in rooms/attenuation/confinement), but nothing serves that socket and keeperd is not an accessible repo either.

So today SCOUTD_SOCK names a door nobody answers, and the only working path is the direct one with a key in the environment — the posture the decision rejects. Filed as .github-private#664.

One correction worth recording, because I got it wrong first: I said this org has no broker. It does — claim-ticket.yml carries id-token: write # OIDC → cf-token-broker, and #178 documents that path for the Cloudflare deploy token with a verified run. The accurate statement is narrower: a broker exists for CI workflows minting per-run credentials; what is absent is egress from a room. Which one the cold read needs depends on where it runs, and #664 says to answer that before building a daemon.

Decision 2 is untouched

One person who has not seen the site. Ninety seconds. The five questions in coldread.feature.

Still no owner, no cadence, no mechanism. Nothing this session shipped moves it, and nothing was ever going to: everything above is the proxy instrument.

Worth stating plainly now that the proxy is good, because the temptation this issue predicted is exactly the one now available. The judge is Opus, so a green run says the ceiling cleared the page and close to nothing about a stranger skimming on a phone. The tool prints that on every run, prints GREEN (weak) rather than PASS, and has a test asserting the word PASS never appears in its report — all of which is machinery to stop a green cold read being read as this test having happened.

It is still the only test here that measures the thing the feature file says matters, and it remains a commitment nobody has made. "Decided: never" stays a valid close.

---

Generated by Claude Code