FE-1407: Catalogue elicitor failures that published measures miss - #9319
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
github-actions
Bot
added
area/infra
lunelson
changed the title
PR SummaryLow Risk Overview Each mode is documented with operational class, labelled evidence, mechanism, a detection signature (spec for later oracles, not an implemented checker), primary accountable layer (disposition / technique / machinery), a graded prevention claim, and successor evidence pointers to FE-1402, FE-1403, FE-1404, FE-1406, and FE-1431. The doc also defines licensed deferral (durable deposit) versus evasive “later” promises, states explicit limits (no detectors, rates, or proof of prevention), and registers the file in Reviewed by Cursor Bugbot for commit a4dbcf3. Bugbot is set up for automated code reviews on this repo. Configure here. |
4fcb281 to
ec6cdec
Compare
9443650 to
0f619da
Compare
0f619da to
d256616
Compare
ec6cdec to
245487e
Compare
Classify fifteen baseline and published failure modes by typed evidence, mechanism, detection signature, accountable layer, bounded prevention claim, and successor evidence input. Define licensed deferral as durable deposit rather than a promise, while preserving prompt-only and single-run limits. Independent specification and standards reviews passed after evidence typing, disjunctive FM-15 detection, completion-vocabulary restraint, authority-boundary repair, and removal of the non-procedural FE-1404 pointer from the completed baseline protocol. Retain FM-13 and granular successor inputs because they preserve the human-measure mismatch and the evidence needed for ordered successor briefs. Verification: 128 core tests passed; docs index 7/7 passed; git diff --check passed; Linear graph 84 issues/68 hard edges/no cycles/no assignee mismatches. FE-1404 Agent notes now explicitly score applicable catalogue signatures under the approved external-write gate.
6a57484 to
a4dbcf3
Compare
75ccc94 to
2224d62
Compare
github-actions
Bot
added
area/deps
3 participants
🌟 What is the purpose of this PR?
Brunch is an elicitation harness: an AI interviewer that draws a model of a cyber-physical process out of a human expert. The published measures we score interviews against (questionnaire quality, coverage, provenance) miss a class of failures that only show up when a frontier model does the interviewing. This pull request lands a catalogue of fifteen such failure modes, drawn from the two scored baseline runs and the literature, each typed by evidence, mechanism, detection signature, and accountable layer. Under the strategy pivot recorded later in this stack (S-007, ADR-0006), the catalogue's role is an oracle list: each signature is a check the walking-skeleton run is tested against.
🔗 Related links
libs/@hashintel/brunch-agent/docs/reference/research/elicitation/frontier-model-elicitor-failure-catalogue.mddocs/control/STEERING.md(Proof 1) and the FE-1404 issue, both in the top branch of this stack🚫 Blocked by
🔍 What does this change?
Adds one reference document and its index row. Fifteen failure modes (FM-01 to FM-15) are classified by the evidence that shows them, the mechanism behind them, a detection signature that a transcript or model can be tested for, the layer accountable for preventing them (interviewer, harness, plugin, or human measure), a bounded prevention claim, and the successor evidence each one needs. Licensed deferral is defined as a durable deposit rather than a promise. FM-13 is retained deliberately because it records where human measures and machine measures disagree.
🏗️ Agent notes
Stacked as
#9311→#9312→#9313→ this branch → #9320 → #9321 → #9322.Commit
944365009eCatalogue frontier-model elicitor failuresStatus under S-007
The design-convergence queue this issue opened (FE-1407 → FE-1402 → FE-1403 → FE-1404) was closed by strategy entry S-007 and ADR-0006 in the FE-1404 branch of this stack. The catalogue survives as test-bed material and as the oracle list for Proof 1; its former hard-blocker relation to FE-1404 was removed on 2026-08-25 because Linear blockers state current prerequisites, not history. The FE-1404 skeleton run records a verdict per applicable signature: observed, not observed, or not observable.
Verification at authoring
128 core tests, docs-index 7/7,
git diff --check, Linear graph audit (no cycles, no assignee mismatches). Re-verified aftergt syncat the stack tip: 182/182 core tests, 0 Oxlint errors.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
The catalogue's prose predates ADR-0006 and still calls
docs/specs/plugin-contract.md"provisional"; that remains true, so it was left unchanged.🐾 Next steps
FE-1404 (top of this stack, then a later run branch) tests the skeleton run against each signature.
🛡 What tests cover this?
packages/core/test/architecture/docs-index.test.ts— the INDEX row resolves and the document's relative links check.❓ How to test this?
docs/reference/research/elicitation/frontier-model-elicitor-failure-catalogue.md.npx vitest runinlibs/@hashintel/brunch-agent/packages/coreand confirm the docs-index gate passes.📹 Demo
Documentation only.