Add /spec-loop skill: five-phase spec loop with cross-model review

[yulonglin]

SPEC-A "Spec-Driven Agent Workflow" deliverables, built and reviewed via the five-phase loop the spec itself describes (doer per phase, codex-companion adversarial review after each phase, findings injected forward).

What's in here (two commits, ordering is deliberate)

1. 7b35e05 — the /spec-loop skill (claude/skills/spec-loop/SKILL.md, a Workflow template driving plan → implement → test → run → analyze with cross-model review), three spec-refinement prompts carrying the five-section template, the detect-only warn_spec_outside_vault.sh hook + settings wiring, spec-conventions.md updated to the five-section template, interview-conventions.md, and three test suites.
2. 8f7097c — the CLAUDE.md /spec-loop row alone. Lands second because the row must never exist without the skill (amendment A2 from the e2e run); verified git cat-file -e HEAD:claude/skills/spec-loop/SKILL.md before committing.

Merge note: use a merge commit (not squash) to preserve the skill-before-row commit ordering on main.

Verification

Suite	Result
tests/test_spec_loop_template.mjs	53/53
tests/test_spec_loop_prompts.sh	28/28
tests/test_spec_outside_vault_hook.sh	15/15

The template was validated by a real end-to-end run (wf_d66350d0-07d, 13 agents, status ok, kill+resume exercised twice) which caught two template defects live — Workflow args arriving as a JSON-encoded string, and codex-companion's terminal status being "completed" not "success" — both fixed and pinned by tests T8/T4.

Full run report: ~/vault/tooling/dotfiles/docs/2026-08-09-spec-loop-spec-a-report.md (vault; not on this public branch).

https://claude.ai/code/session_01Mm1WU9eHyEYQWVUPtq46Um

[yulonglin]

Shall we standardise/merge?

[yulonglin]

Merged — interview-conventions.md is deleted and its content now lives as an "Interview conventions" section in claude/rules/spec-conventions.md, so one file owns spec + interview standards. Note /spec-interview and /spec-interview-research are plugin skills outside this repo, so the rule documents the convention they should follow rather than editing them directly.

[yulonglin]

A large batched AskUserQuestion call with 10-20 is fine too?

[yulonglin]

Yes — encoded in the merged section: AskUserQuestion caps a single call at 4 questions, so 10-20 questions run as a series of successive batched calls (explicitly blessed as fine), or as a single markdown answer file when questions need long what-I-found preambles or async answering. The anti-pattern is a drip of one-question prompts; /grilling stays the deliberate one-at-a-time exception.

[yulonglin]

critique and improve on this. also some things like audit trails aren't always needed. and signposting/bullet points are better than long walls of text!

[yulonglin]

Rewritten. clean-spec now says: remove unnecessary detail — process artifacts (audit trails, change logs, interview transcripts) are opt-in, never default; prefer signposting and bullet points over walls of text; cut repetition. The five-section template mandate stays. Body is itself bullets now, practicing what it preaches.

[yulonglin]

it's weird for us to sha256 guard these things?? I should be able to manually edit the prompts, or maybe with new coding agents

[yulonglin]

Agreed, dropped. The sha256 pinning is gone — all three prompts are now plain editable text headed "Edit freely". What replaced it: tests/test_spec_loop_prompts.sh pins only the structure the skill relies on (five-section template mandate, the six improve-spec dimensions, /grill-me pointers, clean-spec cleaning guidance) — so you or any agent can reword freely, and the test only fails if a load-bearing piece disappears.

[yulonglin]

is this standard for claude code workflows? if it is, and is structured, good. otherwise seems a bit odd to me cf just writing down what to do

[yulonglin]

Yes, this is standard — Workflow is Claude Code's native orchestration tool and these scripts are its documented format (the meta block, agent(), phase(), JSON schemas for structured output). What it buys over "just writing down what to do": deterministic control flow (the loop/retry logic runs as code, not model judgment), parallel fan-out with structured results, and resumability — a killed run resumes with completed phases served from cache. Prose instructions get re-interpreted each session; the script executes the same way every time.

[yulonglin]

should this skill be shorter? for memory purposes

[yulonglin]

Memory cost is smaller than it looks: only the frontmatter name + description load per session; the body loads on invocation only. So length costs nothing until the skill is actually used. Still trimmed where prose wasn't load-bearing (args paragraph, refinement section), and the embedded template is the irreducible core — it IS the deliverable the skill pastes into Workflow.

[yulonglin]

we should have /simplify, prune and refactoring for readability, maintainability, extensibility at the end?

cos there might be repetitive work, extraneous tests (e.g. for the sake of it)

fewer lines of code (but readable/understandable) is good

[yulonglin]

also, a few other things:

Legibility over cleverness. When an agent can generate 500 lines in a minute, the scarce resource is your ability to check that those lines do what you think. Boring, conventional, explicit code wins. Flat structure, obvious naming, minimal indirection, no clever abstractions that save keystrokes at the cost of auditability. This was always good advice for research code; agents make it non-negotiable, because you're now reviewing far more code than you write.

Fast, cheap feedback loops. Agents are dramatically better when they can run something and see whether it worked. Small units that execute in seconds, smoke tests on tiny models/datasets, a --debug mode that runs the full pipeline in 30 seconds. If your only feedback signal is "the 6-hour training run finished and the numbers look plausible", both you and the agent are flying blind.

Aggressive defence against silent wrongness. The classic research-code failure mode isn't crashing — it's running fine and producing subtly wrong results: leakage across a train/test split, a shape bug that broadcasting papers over, off-by-one in an index. Agents amplify this because they produce highly plausible code at speed, and plausible-but-wrong is exactly what slips through review. So: assertions on shapes and value ranges everywhere, sanity checks baked into the pipeline (does the random baseline get chance accuracy? does the metric hit 1.0 on a known-perfect input?), and plots of intermediate quantities, not just final numbers.

Protect the trusted core. Split your codebase mentally into two tiers. The load-bearing parts — metric definitions, eval harness, data splits, the statistical analysis — should be small, hand-reviewed line by line, tested, and rarely touched. The experimental scaffolding around it — sweep scripts, plotting, one-off analyses — can be agent-churned freely, because errors there are visible or cheap. Most agent-induced research disasters come from letting the agent casually "refactor" the thing that defines whether your result is real. (For your line of work specifically: never let an agent modify the eval while it's also trying to make the eval pass. That's a conflict of interest you of all people should recognise.)

Provenance and reproducibility, ratcheted up. Agents let you run ten variants where you'd previously run two, so "which code state produced this number" becomes the question you'll ask constantly. Everything in git, small commits, config files rather than edited constants, seeds logged, and every result file stamped with the commit hash and config that produced it. The cost is low and it's exactly the discipline agents are happy to maintain for you if you ask.

[yulonglin]

Added a Simplify phase — a /simplify-style pass over everything produced so far (implementation, tests, AND analysis scripts): prune repetition and extraneous tests-for-their-own-sake, fewer lines while staying readable, full suite green afterwards, behavior unchanged. One deliberate deviation from "at the end": it runs between Test and Run, not after Analyze — so the run's numbers come from the final code state (your provenance principle: every result stamped with the commit that produced it). Analysis scripts written during Analyze get the same simplify treatment inside that phase. Flag if you want a second full pass at the very end too.

[yulonglin]

All five encoded as a PRINCIPLES constant injected into every doer brief in the template: legibility over cleverness; fast cheap feedback loops (smoke tests on tiny models, --debug mode); defence against silent wrongness (assert shapes/ranges, random-baseline-hits-chance and metric-hits-1.0-on-perfect-input sanity checks, plot intermediates); protect the trusted core (metric definitions, eval harness, splits, analysis stay small and hand-reviewed — NEVER modify the eval while also trying to make it pass); provenance (config files over edited constants, seeds logged, results stamped with commit + config). T9 in tests/test_spec_loop_template.mjs pins that all six doer briefs carry them.

[yulonglin]

Even for grilling, we shouldn't have only one qn per call. That's annoying. Just think for longer to come up with a good list of questions!

[yulonglin]

Done in e5e10c2 — the exception is gone at both ends: /grilling's own skill now says think longer upfront, map the decision tree, assemble a good question list, and deliver it in batched AskUserQuestion calls (four per call, the tool's cap); a question is held back only when it genuinely depends on an earlier answer. spec-conventions.md now reads "never a drip of one-question prompts, /grilling included."

[yulonglin]

Not necessarily what you found, but just an explanation of what's happening, rather than jumping in without context

[yulonglin]

Reframed in e5e10c2: the preamble is now "an explanation of what's happening and why the question arises, so it never lands cold" — findings go in when relevant, but context is the point. (Was "what-I-found preamble", which over-indexed on findings.)