Skip to content

docs(FAFF-269): infosec threat-prior design-settle spike — ADR-0046 + refutation log + eval fixtures#282

Merged
alechill merged 4 commits into
mainfrom
faff-269-infosec-threat-prior-spike
Jul 6, 2026
Merged

docs(FAFF-269): infosec threat-prior design-settle spike — ADR-0046 + refutation log + eval fixtures#282
alechill merged 4 commits into
mainfrom
faff-269-infosec-threat-prior-spike

Conversation

@alechill

@alechill alechill commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Design-settle spike for FAFF-269 (design register E; the FAFF-9 §A stopped branch). Deliverable is a refutation log + ADR + reproducible eval fixtures — no shippable code (per the spike anti-pattern rules).

What this settles

  • SQ1 — producer: a human-curated, git-committed threat-model doc. Autonomous mining refuted — faff's threats are semantic (don't fit the deterministic faff profile mine), and a persisted LLM-mined .faff/ prior is build-lane-forgeable (ADR-0039) → a new attack surface on the lens meant to catch attacks. Autonomy relocates to consumption (the lens reads the doc every review, no human in the loop).
  • SQ2 — storage/consumption: version control (the ADR-0013 human-override lane), consumed as one --context file via the FAFF-267-reserved seam — no aggregation or verdict-contract change. Forge concern dodged by construction.
  • SQ3 — value: empirical, GO-conditional on a measured primed-vs-generic catch-rate run against committed fixtures refutation-spec-007..010 (the human-supervised step — the frontier eval driver can't nest in a session). GO-narrow on a real lift; NULL closes design-E's infosec sub-question if ≈0.

Re-casts the ticket's misspecified code-blindness open question to the real forge/integrity concern (the spec reviewer is not the code-blind lane; the evaluator is).

Deliverables

  • docs/adr/0046-…md — the decision (Nygard; faff adr validate passes).
  • docs/spikes/2026-07-05-FAFF-269-…-refutation-log.md — SQ1/SQ2/SQ3, ≥2 refutation attempts each, candidates A/B/C refuted.
  • eval/cases/refutation-spec-007..010.json — 3 faff-specific threat catches + 1 clean near-miss; reuse the existing refutation-spec KIND + seam-registry row (no new KIND).

Acceptance Criteria (the spike's DONE)

  • ADR accepted; GO-conditional narrow recorded (not a forced binary) — docs/adr/0046-…md, faff adr validate exit 0
  • ADR records the code-blindness-question re-cast (reviewer ≠ code-blind; forge/integrity concern, ADR-0039/FAFF-325)
  • SQ1 / SQ2 / SQ3 each closed with rationale — ADR Decision bullets + refutation-log closes
  • Refutation log committed, ≥2 attempts/question, A/B/C each refuted
  • Fixture set (≥3 threat + ≥1 near-miss) under eval/cases/ as reproducible refutation-spec cases — 007..010
  • Catch-rate note on FAFF-283; disposition note on FAFF-13
    Verified: comments posted (f13e2587 / 029e6c6e)
  • PR is docs + eval fixtures only — no shippable miner/prompt/schema/grounding file, no accepted baseline
    Verified: git diff main...HEAD --stat → only docs/ + eval/cases/ (7 files, +418)
  • faff adr validate passes
    Verified: exit 0, "46 ADR(s) valid"
  • The measured primed-vs-generic run (GO/NULL) — the named human-supervised follow-up, not this PR

Review (senior-engineer pass): pass, zero findings — fixtures verified as genuine generic-misses + a real near-miss, ADR coherent, measurement deferral a legitimate narrow, revert-clean.

alechill added 3 commits July 6, 2026 03:42
…g + eval fixtures

Design-settle spike. SQ1: producer is a human-curated committed threat-model
doc (autonomous mining refuted — semantic content, and a persisted .faff/ prior
is build-lane-forgeable). SQ2: version-control storage + FAFF-267 --context
consumption, no aggregation change. SQ3: value is empirical, GO/NO-GO gated on a
measured primed-vs-generic run against committed fixtures 007-010 (the
human-supervised step; frontier eval can't nest in-session). Re-casts the
ticket's misspecified code-blindness question to the forge/integrity concern.
No shippable miner/prompt/schema/grounding artifact.
…or fixtures

Adding refutation-spec-007..010 (the FAFF-269 primed-vs-generic threat-prior
corpus) requires the lockstep golden-count update in the 'all eval/cases load
and validate' test. No behaviour change; the fixtures reuse the existing
refutation-spec KIND + seam-registry row.
@alechill alechill force-pushed the faff-269-infosec-threat-prior-spike branch from 7806441 to bdb177d Compare July 6, 2026 03:43
#283 (FAFF-346) landed its own ADR-0046 while this branch sat — the concurrent-graft
allocation collision (faff adr new allocates at branch-base time). Renumbered to the
next free number; faff adr validate passes (contiguous 0001..0047).
@alechill alechill merged commit 6876978 into main Jul 6, 2026
2 checks passed
@alechill alechill deleted the faff-269-infosec-threat-prior-spike branch July 6, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant