diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 48daca2..dd542d6 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -9,7 +9,7 @@ { "name": "superpowers-v", "description": "Compound V for Superpowers: triple parallel pre-flight (code archaeology + domain-expert + Context7 library validator), disjoint partitioning, manifest-driven multi-backend dispatch (Claude + Codex + Antigravity + Cursor), git-diff scope enforcement, crash-resumable runs, adaptive tier-based routing, epic mode, research-grounded brainstorming (gated pre-brainstorm recon + batched elicitation), V-memory local-first semantic+lexical recall over docs/superpowers (opt-in pure-python embeddings + a deterministic recall→action bridge), and batched parallel dispatch (Opus default, narrow Sonnet exception)", - "version": "2.9.0", + "version": "2.10.0", "source": "./", "author": { "name": "Oleg", diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index c338cd5..90668ba 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "superpowers-v", "description": "Compound V for Superpowers: triple parallel pre-flight (code archaeology + domain-expert advisor + library/doc validator via Context7), disjoint file partitioning, manifest-driven multi-backend dispatch (Claude + headless Codex + Antigravity + Cursor workers), git-diff scope enforcement, crash-resumable runs, adaptive tier-based routing, epic mode for multi-feature builds, V-memory local-first semantic+lexical recall over docs/superpowers (opt-in pure-python embeddings + a deterministic recall\u2192action bridge), and batched parallel dispatch (Opus default, Sonnet for narrow junior-task carve-out), plus /v:onboard — a project-onboarding command that builds a citation-verified knowledge base + AGENTS.md/CLAUDE.md bridge behind a human gate. Auto-intercepts four transitions: pre-brainstorm recon \u2192 brainstorming \u2192 writing-plans \u2192 execution.", - "version": "2.9.0", + "version": "2.10.0", "author": { "name": "Oleg", "email": "copeus@gmail.com" diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 656d722..c251430 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -221,3 +221,18 @@ jobs: done < <(find . -name "*.md" -not -path "./node_modules/*" -not -path "./.git/*" -print0) [ "$fail" = "0" ] || exit 1 echo "✅ All intra-plugin .md links resolve" + + # Marathon-mode (v2.10) selftests run under Python 3.9 — the documented floor for + # scripts/compound-v-epic-state.py and scripts/compound-v-epic-arbiter.py (both pure + # stdlib, no PyYAML dependency). Installed LAST in the job so re-pointing PATH at a + # 3.9 interpreter can never affect the earlier PyYAML-dependent steps above. + - name: Set up Python 3.9 (marathon selftest floor) + uses: actions/setup-python@v5 + with: + python-version: '3.9' + + - name: Run epic-state.py selftest (Python 3.9 floor) + run: python3 scripts/compound-v-epic-state.py --selftest + + - name: Run epic-arbiter.py selftest (Python 3.9 floor) + run: python3 scripts/compound-v-epic-arbiter.py --selftest diff --git a/CHANGELOG.md b/CHANGELOG.md index fda35c1..d81cb6d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,26 @@ All notable changes to **superpowers-v (Compound V)** are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project uses semantic versioning. +## [2.10.0] — 2026-07-13 + +### Added — Marathon Loop (opt-in autonomous `/v:epic`, PHASED scope) + +An opt-in **marathon** stance for `/v:epic`: instead of stopping at every feature checkpoint, the epic can chew through the whole runnable feature DAG in one invocation. The default `checkpoint` epic is **behaviorally unchanged** — marathon is chosen only at `--init` time and cannot be flipped onto an existing checkpoint epic. Scope was deliberately PHASED after three Codex Sol `xhigh` review rounds converged that every *critical* concurrency finding traced to auto-resurrection (a two-tier watcher reviving the epic while you're away); removing that from v2.10 makes the marathon single-process and the whole class of concurrency criticals disappears. See `docs/superpowers/specs/2026-07-12-epic-autonomous-mode-design.md` for the full scope decision and the deferred v2.11 sketch. + +- **Marathon Loop + DAG-autonomous routing** (`scripts/compound-v-epic-state.py`): a marathon-only `autonomy` state block, `--next --autonomous` (a separate, read-only routing function from the default `--next` — byte-identical default behavior preserved) that routes on deterministic DAG reachability, so an abandoned or blocked feature removes only its *transitive dependents*, never its independents. `attempts` tracking, `--can-retry`, `--record-disposition`, `--record-final-review`. **Terminal states:** `done` (all features done **and** a persisted `final_review.status=="passed"` — never on feature-completion alone), `blocked_needing_human` (a tripped breaker, a `halt_epic` verdict, or exhausted reachable work), `running_with_failures` (non-terminal). `done_with_blockers` is defined but structurally unreachable in v2.10 (needs a 2nd confirming external model family — deferred to v2.11). +- **Cross-model Arbiter Panel** (`scripts/compound-v-epic-arbiter.py`, new): classifies a feature FAILURE via a two-phase, **challenge-bound** API — `--prepare` issues a bounded Claude ballot-task prompt tied to `{epic_id,feature,attempt,challenge_id}` (an HMAC-keyed, per-epic challenge secret; a mismatched/replayed/stale challenge is dropped before any model call), then `--classify` polls Codex (real sandbox, read-only, through the timeout supervisor, evidence size-capped and secret-redacted before egress) and validates a driver-supplied Claude ballot, aggregating both with a complete, deterministic truth table. **Family-diverse aggregation:** ballots collapse one-per-family (`gpt`/`gemini`/`claude`/`grok`/`unknown`); a parse-failed or errored ballot is dropped and logged, never fabricated as a vote; empty or tied → conservative `halt_feature`; `retry_fix` past the per-feature retry cap is masked to `halt_feature`. Antigravity/Cursor are excluded from arbitration (no kernel write-confinement) — implementation workers only, never advisors. **Fail-closed secret redaction** before any external-model egress (labelled tokens, auth headers, private keys, URL credentials, multiline/unclosed-quote secrets — omits the suspect evidence rather than risk a half-redacted leak). **O_NOFOLLOW evidence containment**: every untrusted path under the arbiter's audit directory is opened via dir-fd + `O_NOFOLLOW` (TOCTOU-safe — never validate-a-name-then-reopen-it), atomic tmp+rename writes, capped/rotated audit JSONs so an all-night run can't fill the disk. Every ballot + resolved family + aggregation reason is frozen to `docs/superpowers/execution/epics//arbiter/-.json`. +- **Blocker Ledger** (`scripts/compound-v-epic-state.py`): "do everything you can" — finish everything reachable, isolate only the genuinely impossible, escalate with proof, never halt the rest. A `blocked_external` disposition marks a feature `blocked` (ledger entry) without halting the epic; `--next --autonomous` skips it and routes around only its transitive dependents. **v2.10 blockers are always SUSPECTED** — `--blocker-confirmed true` is hard-rejected everywhere it could be set; CONFIRMED (`≥2` distinct known external model families agreeing) is structurally unreachable on a Codex+Claude-only panel and is deferred to v2.11 alongside a second safe external family and the `done_with_blockers` terminal. +- **Global circuit breakers + human resume** (`scripts/compound-v-epic-state.py`): `total_attempts`, `no_progress_cycles` (a full autonomous pass that advances `done` by zero), and wall-clock hours since `autonomy.started_at` — counts and hours only, **never a fabricated cost**. `--breaker-check` is read-only; `--trip-breaker` atomically parks the epic at `blocked_needing_human`. Re-checked before every feature *and* before every model call (arbiter, sample-audit, final review) — an honest, not a hard real-time, guarantee. **Human recovery, never automatic:** `--clear-breaker` (`--reset-wall-clock`, `--set-max-total-attempts N`) re-arms a tripped breaker; `--clear-disposition` clears a sticky `halt_epic` verdict — both followed by the human re-running `/v:epic `, which is re-entrant and resumes the marathon from `epic-state.json`. +- **PASS integrity — anti-reward-hack gate** (`agents/spec-reviewer.md` §2.5, `commands/v-epic.md`): a marathon SUCCESS is not blindly trusted. The reviewer contract gained a deterministic "did this diff weaken its own tests/scorers to pass?" check; the marathon driver **sample-audits** a deterministic fraction of PASSes (every 3rd `done` this invocation, plus always the first) with a fresh adversarial re-review, and gates terminal `done` on a **final cross-feature re-verification** (`--record-final-review`) over the whole accumulated diff since `autonomy.start_sha`. +- **CI**: `.github/workflows/validate.yml` now sets up Python 3.9 (the documented marathon-scripts floor) and runs both new `--selftest` suites (`compound-v-epic-state.py`, `compound-v-epic-arbiter.py`) as a required job — a red selftest now fails CI. + +### The honest v2.10 boundary — no auto-resurrection + +Marathon is opt-in; the default epic still checkpoints. "Survives a fall" means two things, both true today and neither overclaimed: **in-session**, the loop continues past a soft per-feature error to the next runnable feature automatically, within the one live `/v:epic` invocation. **After a hard death** (quota, closed terminal, crashed machine), a **human re-invokes `/v:epic `**, which resumes from the committed `epic-state.json`. There is **no automatic resurrection while you're away** in v2.10 — nothing wakes the epic back up on its own. That is the deferred **v2.11** auto-watcher (Execution Lease + Two-Tier Watcher + generation-fenced execution across the dispatcher/worker/merge-back/commit layer) — its own spec, its own review pass, because it needs correct distributed concurrency that the v2.10 single-process design deliberately avoids. No fabricated cost/token metrics anywhere in either stance. + +### Provenance +Converged from four independent pre-implementation reviews plus three Codex Sol `xhigh` adversarial rounds on 2026-07-12 that drove the phased-scope decision. Built across four disjoint units (state spine → arbiter panel → driver/reviewer wiring → this docs/CI/release unit) and cross-model reviewed by **Codex gpt-5.6-sol**. + ## [2.9.0] — 2026-07-12 ### Added — Pre-Evaluation stage + proportionate fast-path diff --git a/README.md b/README.md index 3d050ad..478d024 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,8 @@ The fastest way to *get* what this plugin does. Three gamified episodes — **De - **Epic mode** — feed it a whole PRD with many tasks and it builds feature by feature, in dependency order, on one branch. By default it checkpoints after each feature so you can review (raise the budget to let it run longer). +- **Marathon Loop** 🧪 (opt-in, v2.10) — instead of stopping at every checkpoint, `/v:epic` can chew through the whole runnable feature DAG in one invocation. A failed feature goes to a cross-model **Arbiter Panel** (Codex + a fresh adversarial Claude) that decides retry / abandon / halt-the-epic; a suspected external blocker (an upstream API with no data, say) goes into a **Blocker Ledger** and gets isolated instead of halting the rest of the build; **global circuit breakers** (total attempts, wall-clock hours, no-progress cycles) keep the whole thing bounded. Honest limit: "survives a fall" means the loop keeps going automatically *within that one live session*, and a **human re-invoking `/v:epic`** after a hard death (closed terminal, crashed machine, quota) resumes it from where it stopped. There is **no automatic resurrection while you're away** — nothing wakes the epic back up on its own. That's a deferred, separate feature. Default epic mode is unchanged; marathon is opt-in per epic at `/v:epic` start time. + - **V-memory** — project memory that builds up as you work: decisions made, bugs fixed, things that failed. It surfaces the relevant bits when you plan or review. - **Research-grounded brainstorming** 🧪 — before a brainstorm on an unfamiliar topic, a gated, bounded recon pass (off by one config key) writes an evidence doc the brainstorm reads first. And when the brainstorm has 3+ *independent* clarifying questions, it can batch them into one screen — the Visual Companion form if you've accepted it, else a structured question, else the usual one-at-a-time (dependent questions always stay sequential). Both are description-driven guidance, not hook-enforced. @@ -85,6 +87,7 @@ That's it. - **Antigravity and Cursor are lower-trust** (no kernel sandbox). The scope check catches out-of-bounds writes *after the fact* but can't *prevent* them. For anything sensitive or untrusted, prefer **Codex** — it runs in a real workspace sandbox. - **Cursor on a Free plan** can only use its `auto` model (named models are paid). - **Epic mode is bounded by default** — it stops after each feature for a human checkpoint. It is *not* a fire-and-forget overnight build unless you raise the budget. +- **Marathon mode (opt-in) is still not fire-and-forget-overnight.** It removes the per-feature checkpoint and adds an arbiter panel + blocker ledger + global breakers so it can run further unattended in one sitting — but it does not self-revive after a hard death. If the session dies, you re-run `/v:epic ` yourself; it resumes from the last committed state. Auto-resurrection while you're away is a deferred, separate feature — not shipped here. - No daemon, no server, no MCP service, no made-up cost numbers. Everything is small, readable scripts. --- diff --git a/agents/spec-reviewer.md b/agents/spec-reviewer.md index c576fd4..2dba91e 100644 --- a/agents/spec-reviewer.md +++ b/agents/spec-reviewer.md @@ -89,6 +89,21 @@ For each MUST item from Pass 1, find the test that would fail if the requirement The change must not print, log, or document **token-cost or token-savings numbers that aren't actually measured**, hardcoded baselines (e.g. `baseline = 1000`), fake speedup percentages, or any self-reported "saved N tokens" meter. This is the anti-ruflo charter. Any fabricated/unmeasurable metric → **ISSUE: FABRICATED_METRIC** (file:line + the number). A real, measured value (wall-clock from a timer, a count from `git diff --stat`) is fine; an invented one is not. +### 2.5 — No reward-hacking + +Did the diff make a check pass by **weakening the check itself**, instead of making the code actually satisfy it? This matters most in Compound V's own **marathon** stance (`skills/compound-v/epic-mode.md`), where a failure is judged by the arbiter panel and a feature can be retried unattended — a pattern that rewards "make the gate green" over "make the behavior correct" if nobody is watching. Diff every test/spec/scorer file the implementer touched (`git show ` or the merged worktree diff, same as 1.1) against its pre-change version and look for concrete, deterministic evidence: + +| Evidence | Example | +|---|---| +| Removed or commented-out assertions | an `assert`/`expect(...).toBe(...)` present before the diff, gone or `// assert(...)` after | +| Deleted or renamed test/spec files with no replacement covering the same behavior | `sequences/api.test.ts` deleted; nothing else asserts that path | +| Loosened numeric thresholds with no spec justification | a score gate `>= 90` changed to `>= 60`, a tolerance widened, a timeout raised to mask a hang | +| Edited the **scorer/grader/checker** itself rather than the implementation | changing what "pass" means instead of making the code satisfy the original bar | +| Newly `.skip`/`xit`/`@pytest.mark.skip`/commented-out test, with no reason tied to the task spec | a previously-running test now skipped in the same diff that claims DONE | +| A previously-propagating exception/assertion now caught and swallowed (silent no-op or a `print` instead of a raise/fail) | a hard failure converted to a soft one with no spec basis | + +Any such change → **ISSUE: REWARD_HACK** (file:line + which test/scorer + what changed + why it weakens the gate rather than fixing the code). A genuine, spec-justified relaxation (the task spec itself asked for a looser threshold, or a test was superseded by a *stronger* replacement in the same diff) is not a reward hack — cite the spec line that justifies it before waving this off. + --- ## PASS 3 — INTEGRATION (final, run-level — gates DONE) @@ -147,6 +162,10 @@ ISSUE: FABRICATED_METRIC (PASS 2) - scripts/foo.sh:88 prints "saved ~1200 tokens" — not measured, hardcoded → Remove; print only measured values (wall-clock, git diff --stat counts) +ISSUE: REWARD_HACK (PASS 2) + - sequences/api.test.ts:41 — assert res.status === 200 loosened to assert res.status < 500 + → Restore the original assertion; fix the handler so it actually returns 200, don't relax the test + ISSUE: BUILD_RED (PASS 3) - `npm test` fails: 2 failing in sequences/api.test.ts (see excerpt) → Fix before DONE @@ -159,7 +178,7 @@ ISSUE: ACCEPTANCE_GAP (PASS 3) APPROVED PASS 1 SPEC: requirements K/K · audit MUSTs M/M · over-build clean · job acceptance met - PASS 2 QUALITY: code-quality clean · no regression · every MUST has a guard test · no fabricated metrics + PASS 2 QUALITY: code-quality clean · no regression · every MUST has a guard test · no fabricated metrics · no reward-hacking PASS 3 INTEGRATION: no partition leak · seams hold · build green (evidence: ) · feature AC J/J met Scope lock: respected (scope gate PASS, confirmed at seam) ``` @@ -170,7 +189,7 @@ APPROVED - DO order the passes: SPEC first, then QUALITY, then (run-level) INTEGRATION. Don't review quality of code that fails spec. - DO NOT approve with "minor issues, close enough." Compound V policy: if you found an issue, the implementer fixes it before the next pass. No "close enough." - DO NOT claim the build is green without running it (or observing its output). Evidence before assertion. -- DO NOT skip the over-build check or the fabricated-metric check. +- DO NOT skip the over-build check, the fabricated-metric check, or the reward-hacking check. - DO NOT propose code or edit files. The implementer fixes; you re-review on the next round. - DO cite file:line (or the failing command) for every claim. diff --git a/commands/v-epic.md b/commands/v-epic.md index e64f776..2832fa6 100644 --- a/commands/v-epic.md +++ b/commands/v-epic.md @@ -12,6 +12,8 @@ The epic model, run-dir layout, the final integration review, and the honesty bo 1. **Resolve the epic spec.** From `{{args}}`: if it is a path to an epic brief, read it; if it is a described feature set, work from the description. If `{{args}}` is empty, ask the user for the epic brief (or list existing epics under `docs/superpowers/execution/epics/` to resume one). Pick an `` (convention: `YYYY-MM-DD-`) and an epic **title**, and capture the epic's **acceptance criteria** (used by the final integration review). Agree an **autonomy budget** with the user — `MAX_FEATURES` per `/v:epic` invocation. Seed the default from `.claude/compound-v.json` `epic.max_features` if set (written by [`/v:init`](v-init.md) Step 3c), else **1**: build one feature, then checkpoint; raise it only when the user wants more autonomy per run. An epic is *N full v1.0 runs*, so this is the **human checkpoint cadence** — a *driver policy*, not a script-enforced token meter: by default the loop builds one feature, reports `--stats`, and stops for you to review and re-run. + **Marathon gate (opt-in, v2.10).** If `.claude/compound-v.json` `epic.autonomy.stance == "marathon"` (written by [`/v:init`](v-init.md) Step 3c) or the user explicitly asks for the autonomous/marathon loop for this invocation, this epic runs the **[Autonomous marathon loop](#autonomous-marathon-loop-opt-in-v210)** below instead of steps 4–7 — `MAX_FEATURES` does not apply there (marathon is bounded by global breaker caps, not a per-invocation feature count). Otherwise (the default, unconfigured case) continue with the checkpoint loop in steps 4–7 exactly as documented — nothing below changes for you. + 2. **Decompose + spec every feature UP FRONT — the one interactive phase.** Decompose the product into independent-ish **features**, each a *vertical slice* (`auth`, `api`, `ui`), not a layer; capture cross-feature dependencies in `depends_on` (`api` depends_on `auth`). Then, for **each** feature, run `superpowers:brainstorming` to produce a real **per-feature spec file** (with feature-level Acceptance Criteria), saved to `docs/superpowers/execution/epics//specs/.md`. **Trigger 0 applies to each of these brainstorms:** before each per-feature brainstorm, run the pre-brainstorm recon gate sequence from [`phase-0-recon.md`](../skills/compound-v/phase-0-recon.md) (plumbing-skip → KB-hit → config); later features in the same epic increasingly skip via the KB-hit gate as earlier recon/audit docs accumulate — designed behavior, not a bypass. This is the **only** human-interactive phase: every spec is written and approved *here*, before the autonomous loop — so the loop never pauses to brainstorm. That batching is what makes the epic genuinely **autonomous** *and* keeps a **real spec per feature** (the central tension, resolved). Write `features.json` = a JSON array of `{id, title, depends_on, spec_path}`, each `spec_path` pointing at its spec file. 3. **Review the decomposition, then init (specs enforced).** @@ -24,15 +26,16 @@ The epic model, run-dir layout, the final integration review, and the honesty bo --out docs/superpowers/execution/epics//epic-state.json ``` `--require-specs` **refuses to start unless every feature has an existing `spec_path`** — the deterministic enforcement that no feature enters the autonomous loop without an approved spec. It also validates ids/refs/cycles/dups. A non-zero exit ⇒ fix and re-init; never hand-edit the state. - - **Commit the epic-level files right after init**: `features.json` and the freshly-created `epic-state.json` are new, uncommitted files — `git add docs/superpowers/execution/epics//features.json docs/superpowers/execution/epics//epic-state.json && git commit -m "chore(v-epic): init epic "`. (Per-feature spec files are committed by `superpowers:brainstorming` itself when each spec is approved, in step 2 — no separate action needed for those.) + - **Marathon init (only for a NEW epic, gated by step 1).** If the marathon gate applies and `epic-state.json` does not exist yet, add `--stance marathon` plus the agreed breaker caps to the `--init` command above: `--stance marathon --max-attempts-per-feature --max-no-progress-cycles --max-wall-clock-hours --start-sha ` (leave `--max-total-attempts` unset to take the script's feature-count-derived default, `max(6, 3×features)`, unless the user wants a specific number). Capture `` with `git rev-parse HEAD` **at this init moment** and pass it — it is stored as `autonomy.start_sha` and is the baseline the halt-page's accumulated-diff command (§7) and the final integration review (§8) diff against. **Marathon has no in-place upgrade.** An existing checkpoint `epic-state.json` (no `autonomy` block) cannot be flipped to marathon after the fact — `build_state`/`--init` only ever writes the marathon fields at creation time. If the epic already exists as a checkpoint state and the user now wants marathon, the options are: (a) keep finishing it in checkpoint mode (steps 4–7), or (b) start a **fresh** `--epic-id` with `--stance marathon`, reusing the same `features.json`/spec files under a new epic id. Never hand-edit an existing `epic-state.json` to inject an `autonomy` block. + - **Commit the epic-level files right after init**: `features.json` and the freshly-created `epic-state.json` are new, uncommitted files. Two separate commands, checking each exit code — never chain with `&&`: `git add docs/superpowers/execution/epics//features.json docs/superpowers/execution/epics//epic-state.json`, then `git commit -m "chore(v-epic): init epic "`. (Per-feature spec files are committed by `superpowers:brainstorming` itself when each spec is approved, in step 2 — no separate action needed for those.) -4. **The autonomous loop** (bounded by `MAX_FEATURES`). Repeat until no feature is runnable **or this invocation's budget is spent**: +4. **The autonomous loop (checkpoint stance — the default).** If the marathon gate (step 1) applies, **skip steps 4–7** and use the [Autonomous marathon loop](#autonomous-marathon-loop-opt-in-v210) after step 7 instead — everything below this point is the unchanged checkpoint behavior. Bounded by `MAX_FEATURES`. Repeat until no feature is runnable **or this invocation's budget is spent**: - **Ask for the next runnable feature:** ``` python3 scripts/compound-v-epic-state.py --next \ --state docs/superpowers/execution/epics//epic-state.json ``` - It prints `{"feature": , "reason": "runnable|epic complete|epic blocked: …|epic needs reconcile: …"}`. A feature is runnable when it is `pending` and **all** its `depends_on` are `done`, returned in topological order. The loop is **fail-fast**: any `failed` feature halts the whole epic (even independent pending features wait) until reconciled — `--next` will not route around a failure. + It prints `{"feature": , "reason": "runnable|epic complete|epic blocked: …|epic needs reconcile: …"}`. A feature is runnable when it is `pending` and **all** its `depends_on` are `done`, returned in topological order. The loop is **fail-fast (checkpoint stance)**: any `failed` feature halts the whole epic (even independent pending features wait) until reconciled — `--next` will not route around a failure. (The marathon loop below uses a different, DAG-aware routing rule — `--next --autonomous` — that continues past an abandoned feature onto its independents; the fail-fast rule here applies only to the checkpoint `--next`.) - **If `feature` is non-null** (`reason == "runnable"`): 1. **Choose the run-id, then mark it running WITH that run-id.** Pick the feature's run-id up front (convention `-`) — it names the v1.0 run dir — and record it **now**: `compound-v-epic-state.py --update --feature --status running --run-id --state `. Recording `run_id` at *running* time (not only on done/failed) is what makes a **mid-run crash recoverable** via `/v:resume ` (step 7); a `running` feature with a null `run_id` has nothing to resume. 2. **Run that ONE feature through the v1.0 pipeline's post-spec execution tail on the current branch** — exactly as a standalone feature, reusing everything (Trigger 0 recon and brainstorming already ran up front in step 2; the loop never repeats them): @@ -53,26 +56,257 @@ The epic model, run-dir layout, the final integration review, and the honesty bo --run-id --state docs/superpowers/execution/epics//epic-state.json ``` then **stop the loop** and go to step 6 (the epic is now blocked but resumable). - 5. **Checkpoint (human-in-the-loop cadence).** Count each completed feature against `MAX_FEATURES`. When this invocation's budget is spent: **first commit `epic-state.json`** — `git add docs/superpowers/execution/epics//epic-state.json && git commit -m "chore(v-epic): checkpoint ( features done)"` — **then STOP and report** `python3 scripts/compound-v-epic-state.py --stats --state ` (done / remaining) so the human reviews the accumulated diff and re-runs `/v:epic` to continue. **The commit is not optional**: each feature's own v1.0 run already commits *that feature's* run directory (parallel-dispatcher's Step 7), but epic-state.json itself (which `run_id`/`status` each feature is at — the epic's *only* resume mechanism) lives one level up and is never covered by that. A checkpoint is exactly the moment control returns to a human who might close the session or clean up the worktree — an uncommitted `epic-state.json` at that instant means a later `/v:epic ` has no record of what's done, and a `finishing-a-development-branch` cleanup can erase it outright. This is a *driver-enforced cadence*, not a token ceiling; with the default `MAX_FEATURES=1` the epic checkpoints (and commits) after **every** feature. + 5. **Checkpoint (human-in-the-loop cadence — checkpoint stance).** Count each completed feature against `MAX_FEATURES`. When this invocation's budget is spent: **first commit `epic-state.json`** — two separate commands, checking each exit code — `git add docs/superpowers/execution/epics//epic-state.json`, then `git commit -m "chore(v-epic): checkpoint ( features done)"` — **then STOP and report** `python3 scripts/compound-v-epic-state.py --stats --state ` (done / remaining) so the human reviews the accumulated diff and re-runs `/v:epic` to continue. **The commit is not optional**: each feature's own v1.0 run already commits *that feature's* run directory (parallel-dispatcher's Step 7), but epic-state.json itself (which `run_id`/`status` each feature is at — the epic's *only* resume mechanism) lives one level up and is never covered by that. A checkpoint is exactly the moment control returns to a human who might close the session or clean up the worktree — an uncommitted `epic-state.json` at that instant means a later `/v:epic ` has no record of what's done, and a `finishing-a-development-branch` cleanup can erase it outright. This is a *driver-enforced cadence*, not a token ceiling; with the default `MAX_FEATURES=1` the epic checkpoints (and commits) after **every** feature. - **If `feature` is null**, branch on `reason` (step 5/6). -5. **Epic complete** (`reason == "epic complete"`). All features are `done`. Run a **final cross-feature integration review**: the *whole accumulated diff* on the branch against the **epic's** acceptance criteria — not the per-feature ACs (those already passed in each feature's own review), but the cross-feature contracts: do the features compose, do shared boundaries line up, is the product coherent end-to-end. On PASS, **commit `epic-state.json` (same as the checkpoint step, if it isn't already)**, then hand to `superpowers:finishing-a-development-branch` (merge / PR / cleanup options) — never hand off with an uncommitted `epic-state.json`. On ISSUES, surface them and stay resumable. +5. **Epic complete (checkpoint stance)** (`reason == "epic complete"`). All features are `done`. Run a **final cross-feature integration review**: the *whole accumulated diff* on the branch against the **epic's** acceptance criteria — not the per-feature ACs (those already passed in each feature's own review), but the cross-feature contracts: do the features compose, do shared boundaries line up, is the product coherent end-to-end. On PASS, **commit `epic-state.json` (same as the checkpoint step, if it isn't already)**, then hand to `superpowers:finishing-a-development-branch` (merge / PR / cleanup options) — never hand off with an uncommitted `epic-state.json`. On ISSUES, surface them and stay resumable. + +6. **Epic blocked (checkpoint stance)** (`reason` starts with `epic blocked` — a feature `failed` or an unmet dependency). **Commit `epic-state.json` (same as the checkpoint step, if it isn't already), then stop and surface it.** Print `compound-v-epic-state.py --summary --state ` so the user sees exactly which feature failed and what it blocks. The epic stays **resumable** — but only if the `failed` status actually made it into git before anyone touches the worktree. After the user fixes the failed feature (or its spec/partition), retry it (`--update --feature --status pending`) and re-run `/v:epic ` (or the same brief) — step 3 detects the existing `epic-state.json` and continues; only `pending` features run, the `done` ones are skipped. + + **Epic needs reconcile** (`reason` starts with `epic needs reconcile` — a feature is still `running`). Because epic mode is **sequential**, `--next` is only ever called between features, so a `running` feature on resume means that feature's run **crashed mid-pipeline**. Do not route around it. **Reconcile by resuming first — don't discard half-built work:** the crashed feature ran a *normal v1.0 run* with its own crash-resume, so run **[`/v:resume `](v-resume.md)** to re-dispatch only that run's incomplete jobs; if it completes, mark the feature **`--status done`**. **If the feature's `run_id` is null** (the crash happened before the run-id was recorded — see step 4.1 — or it is an old state), there is nothing to resume → restart it with **`--status pending`**. Only if a resumed run cannot be recovered, fall back to **`--status pending`** (full restart from the spec) or **`--status failed`** (abandon and stop). Never leave a feature `running` across a resume — the epic will not advance until the stale run is reconciled. **Whichever status this settles on, commit `epic-state.json` right after** — two separate commands, checking each exit code: `git add docs/superpowers/execution/epics//epic-state.json`, then `git commit -m "chore(v-epic): reconcile -> "` — the `--status failed` ("abandon and stop") path in particular is terminal and does not otherwise pass through the checkpoint step's commit. + +7. **Report (checkpoint stance).** Print the epic summary (`--summary`), the per-feature run-ids, and the next step: the integration review + `finishing-a-development-branch` on complete, or the blocking feature + the resume hint on blocked. + +## Autonomous marathon loop (opt-in, v2.10) + +Gated by step 1's marathon check. Everything below **replaces steps 4–7** for this invocation — the checkpoint loop above stays byte-for-byte unchanged and is never touched by any of this. Full design: [`epic-mode.md`](../skills/compound-v/epic-mode.md) "Marathon stance"; the two scripts that back every command below: [`compound-v-epic-state.py`](../scripts/compound-v-epic-state.py) (its docstring's "## CLI contract" section is authoritative) and [`compound-v-epic-arbiter.py`](../scripts/compound-v-epic-arbiter.py) (its docstring's "## CLI contract (two-phase)" section). + +### 0. Stance binding — the persisted state is authoritative, not config + +Config intent alone (`.claude/compound-v.json` `epic.autonomy.stance=="marathon"`, or a manual `--autonomous` ask) is **not enough** to run this loop. Before issuing any `--next --autonomous` / `--record-*` / `--breaker-check` / `--trip-breaker` command, **read `epic-state.json` directly** (it is a plain JSON file — no dedicated subcommand exists to introspect just this one field) and confirm `.autonomy.stance == "marathon"`. If that block is absent (a checkpoint state, or an old/hand-made one), **REJECT autonomous operation for this epic** — fall back to the checkpoint loop (steps 4–7), or start a fresh marathon epic per step 3's "Marathon init" bullet. Re-check this on *every* re-entry (including a resumed session), not only at first init — a config flip after a checkpoint epic already started must never silently promote it to marathon. + +### 1. Per-iteration progress + breaker check (before every feature) + +At the top of each loop pass, pick a **stable cycle id for this pass** (an incrementing counter held in your own scratch state, or a UUID minted once per pass and reused for every call *within* that same pass, so one pass is never double-counted): + +``` +python3 scripts/compound-v-epic-state.py --record-progress-cycle --cycle-id \ + --state docs/superpowers/execution/epics//epic-state.json +``` + +Idempotent by `cycle_id` (safe to replay after a crash); compares this pass's `done` count to the last recorded count and resets/increments `no_progress_cycles`. Then: + +``` +python3 scripts/compound-v-epic-state.py --breaker-check \ + --state docs/superpowers/execution/epics//epic-state.json +``` + +`--breaker-check` is **read-only** → `{"tripped","which":[...],"detail":{...}}`. If `tripped`: + +``` +python3 scripts/compound-v-epic-state.py --trip-breaker \ + --state docs/superpowers/execution/epics//epic-state.json +``` + +`--trip-breaker` atomically parks the epic at `blocked_needing_human` (if not already there) and records `which`/`detail`. **Commit (§9), then go straight to the halt-page runbook (§7)** — do not start another feature. Re-run `--breaker-check` again after each attempt (§4/§5) and immediately before every model call the loop makes — the arbiter (§5) and the final-review pass (§8) — not only here at the top of the pass. Ship the honest wording: **wall-clock is checked at each attempt/model-call boundary; a single in-flight pipeline phase (a pre-flight, a dispatch batch, a review pass) may overrun its check window before the next boundary catches it.** This is not a hard real-time kill — never claim one. + +### 2. Ask for the next runnable feature + +``` +python3 scripts/compound-v-epic-state.py --next --autonomous \ + --state docs/superpowers/execution/epics//epic-state.json +``` + +Prints `{"feature": , "reason": "...", "blocked_by": [ids]}`. Unlike the checkpoint `--next`, this is **DAG-aware**: an abandoned/failed/blocked feature removes only its *transitive dependents* from the runnable set — independent pending features stay runnable and are returned **before** any terminal escalation. Routing is driven entirely by **persisted state** — a `failed` feature is routed by its *stored* `disposition` (not an in-memory intent), which is what makes retries and mid-arbitration crashes recover correctly across a re-entry. `reason` embeds a literal terminal-state token as a prefix when there is nothing runnable to hand out: + +- `feature` non-null (`reason` is `"runnable"` or `"running_with_failures: runnable (...)"`) → §3. **A `failed` feature carrying a `retry_fix` disposition that is still under its retry cap is handed back here as runnable** — the driver runs it exactly like any other runnable feature (§3). This is the crash-safe retry path: the retry intent lives in the persisted `disposition`, so it survives a breaker trip or a hard crash between recording the verdict and re-running. +- `"needs_arbitration: feature ..."` — a `failed` feature with **no valid recorded disposition**: either the arbiter exchange never completed (a crash mid-arbitration), or a *stale* disposition was recorded against an earlier attempt (a disposition is attempt-bound — the state script only honors it when its `attempt` equals the feature's current `attempts`, so a re-run that bumped `attempts` invalidates the old verdict and this reason re-fires). **Do NOT treat this as done or abandoned** and **do NOT blindly restart from `--prepare`** — a crash can leave a challenge already `in_progress` or `consumed`, and `--prepare`/`--classify` reject a consumed/in-progress challenge, which would deadlock. Instead run the **idempotent recovery ladder first** — read the feature's current `attempts` (`--can-retry --feature ` → `attempts`), then: + ``` + python3 scripts/compound-v-epic-arbiter.py --resume-challenge \ + --state docs/superpowers/execution/epics//epic-state.json \ + --feature --attempt + ``` + Prints `{"state":"absent"|"in_progress"|"consumed", "challenge_id"?, "prompt"?, "result"?}`. Branch on `state`: + - **`"consumed"`** — the arbiter already classified before the crash (crash-after-classify-before-record). The verdict is in the returned `result` — **no re-dispatch of Claude, no re-egress**. Record it straight from `result`: `--record-disposition --feature --disposition --reason "" [--families-agreeing ]`, commit (§9), then act on it (§6). + - **`"in_progress"`** — the challenge was issued (and Claude may or may not have replied) but never aggregated. Re-dispatch the fresh adversarial Claude Task on the returned `prompt`, write its ballot file, and resume at §5 **step 3** (`--classify` with the returned `challenge_id`) — the binding makes this idempotent; skip §5 steps 1–2. + - **`"absent"`** — no challenge exists for this attempt (the crash predated `--prepare`, or a stale disposition invalidated a prior attempt's challenge that doesn't match the current one). Run the arbiter exchange (§5) from the top, the normal first-time path. + Only after this recovery ladder resolves does the driver treat the failure as newly-arbitrated. Then act on the disposition (§6) and loop. +- `"needs_blocker_recording: feature(s) ..."` — the **symmetric "finish the interrupted transition on resume" case** to `needs_arbitration`, one step further along: the feature's attempt-matched disposition IS `blocked_external` (the arbiter verdict was recorded) but the crash landed *between* `--record-disposition` and the `--update --status blocked` that appends the ledger entry. **Complete the interrupted transition idempotently** — the ledger append is keyed by `(feature, attempt)`, so re-running is safe and never duplicates: + ``` + python3 scripts/compound-v-epic-state.py --update --feature --status blocked \ + --blocker-reason "" \ + [--families-agreeing ] \ + --evidence "" \ + --state docs/superpowers/execution/epics//epic-state.json + ``` + (Pull `blocker-reason`/`families-agreeing` from the feature's stored `disposition` — read it from `epic-state.json`; never pass `--blocker-confirmed true`, v2.10 hard-rejects it.) Commit (§9), then continue the loop (§1) — the feature is now properly `blocked`, exactly as if §6's `blocked_external` branch had completed uninterrupted. +- `"sample_audit_due: feature(s) ..."` — one or more `done` features were sampled for a PASS-integrity audit that has not completed (see §4). **Run the outstanding sample-audit(s) now** (§4 steps 1–4) before anything else — this reason is surfaced *before* `final_review`, and `--record-final-review passed` is **rejected while any audit is due**, so you cannot skip ahead to §8. Once every due audit clears, the next `--next --autonomous` advances normally. +- `"done: ..."` — every feature is `done`, **no sample-audit is due**, **and** `final_review.status == "passed"`. → §8/terminal (success). +- `"blocked_needing_human: ..."` — a tripped breaker, a `halt_epic` disposition, or exhausted reachable work (only blocked/failed features remain). → §7 (halt-page). +- `"running_with_failures: all features done, awaiting final_review ..."` — every feature `done`, no audit due, review not yet passed. → §8. +- `"epic needs reconcile: ..."` — a feature is stuck `running` from a prior crash. Recover the crashed run itself with `/v:resume ` (or, if unrecoverable, a full restart) exactly as the checkpoint loop's step 6 — **but do NOT use the checkpoint reconcile's terminal status write, which marks a recovered run `done` directly.** In marathon that would bypass §4's PASS-integrity sampling (the recovered run might be the invocation's first success, which MUST be sample-decided and have `--mark-sample-audit-due` persisted *before* `done`). Instead, route the outcome through the marathon handlers: a **recovered success** enters **§4's success handler** (run the sample decision, mark-due-before-done, then the audit) — it does not get a bare `--update --status done`; a **recovered failure** enters **§5** (mark `failed`, then the arbiter exchange); an unrecoverable run that can't even be classified falls back to `--status pending` (full restart on the next pass). Then loop again. + +### 3. Run the feature + +Identical to the checkpoint loop's step 4.1–4.2 — pick the run-id, mark it running, run the full v1.0 pipeline: + +``` +python3 scripts/compound-v-epic-state.py --update --feature --status running \ + --run-id --state docs/superpowers/execution/epics//epic-state.json +``` + +(Legal only from `pending`/`failed` — the marathon transition table rejects `done`/`blocked` sources; this also increments the feature's `attempts` counter, which the arbiter step below reads.) Then the same pre-flights → writing-plans + partition → manifest → dispatch → 3-pass review tail from step 4.2, unchanged. + +### 4. On success — mark done (+ durable audit obligation), then sample-audit a fraction of PASSes + +**PASS-integrity sample-audit (spec Component 5).** A marathon SUCCESS is not automatically trusted — a fraction of them get a fresh adversarial re-review to catch a feature whose own 3-pass gate was satisfied by a weakened test/scorer (the reward-hack `agents/spec-reviewer.md` 2.5 guards against). Deterministic policy, stated so it is auditable, not a coin flip: **hold an in-invocation counter of successful features and sample-audit every 3rd one (the 3rd, 6th, 9th … `done` this invocation); always sample-audit the FIRST success of the invocation** (so a one-feature or short epic is never audited zero times). No RNG, no fabricated rate. + +**Whether or not this success is sampled, decide it FIRST — the obligation must be on disk before `done` is committed**, so a crash between "mark done" and "run the audit" can never lose the audit. The `done` mark and (for a sampled feature) the audit obligation are recorded and committed **together**, never `done` alone: + +- **Sampled** — record the obligation, then (or in the same commit as) the `done` mark: + ``` + python3 scripts/compound-v-epic-state.py --mark-sample-audit-due --feature \ + --state docs/superpowers/execution/epics//epic-state.json + ``` + ``` + python3 scripts/compound-v-epic-state.py --update --feature --status done \ + --run-id --state docs/superpowers/execution/epics//epic-state.json + ``` + Commit (§9) — **both** writes in one commit, so `done` is never persisted without its pending `sample_audit_due`. While any feature has `sample_audit_due`, `--next --autonomous` reports `"sample_audit_due: ..."` (surfaced before `final_review`) and `--record-final-review passed` is **rejected** — the obligation is enforced by the state script, not by the driver remembering it. +- **Not sampled** — just the `done` mark, then commit (§9): + ``` + python3 scripts/compound-v-epic-state.py --update --feature --status done \ + --run-id --state docs/superpowers/execution/epics//epic-state.json + ``` + Loop back to §1. -6. **Epic blocked** (`reason` starts with `epic blocked` — a feature `failed` or an unmet dependency). **Commit `epic-state.json` (same as the checkpoint step, if it isn't already), then stop and surface it.** Print `compound-v-epic-state.py --summary --state ` so the user sees exactly which feature failed and what it blocks. The epic stays **resumable** — but only if the `failed` status actually made it into git before anyone touches the worktree. After the user fixes the failed feature (or its spec/partition), retry it (`--update --feature --status pending`) and re-run `/v:epic ` (or the same brief) — step 3 detects the existing `epic-state.json` and continues; only `pending` features run, the `done` ones are skipped. +Then run the sample-audit for a sampled feature (also the entry point when `--next --autonomous` re-surfaces a `"sample_audit_due: ..."` obligation after a crash/resume — run it here, do not skip to §8): - **Epic needs reconcile** (`reason` starts with `epic needs reconcile` — a feature is still `running`). Because epic mode is **sequential**, `--next` is only ever called between features, so a `running` feature on resume means that feature's run **crashed mid-pipeline**. Do not route around it. **Reconcile by resuming first — don't discard half-built work:** the crashed feature ran a *normal v1.0 run* with its own crash-resume, so run **[`/v:resume `](v-resume.md)** to re-dispatch only that run's incomplete jobs; if it completes, mark the feature **`--status done`**. **If the feature's `run_id` is null** (the crash happened before the run-id was recorded — see step 4.1 — or it is an old state), there is nothing to resume → restart it with **`--status pending`**. Only if a resumed run cannot be recovered, fall back to **`--status pending`** (full restart from the spec) or **`--status failed`** (abandon and stop). Never leave a feature `running` across a resume — the epic will not advance until the stale run is reconciled. **Whichever status this settles on, commit `epic-state.json` right after** (`git add docs/superpowers/execution/epics//epic-state.json && git commit -m "chore(v-epic): reconcile -> "`) — the `--status failed` ("abandon and stop") path in particular is terminal and does not otherwise pass through the checkpoint step's commit. +1. **Run the breaker gate first** (§1, reusing THIS pass's cycle-id — idempotent, so it re-evaluates wall-clock without double-counting `no_progress_cycles`) before spending the audit's model call. Tripped → `--trip-breaker` → §7. (The obligation persists across the trip — it is cleared only by step 3, so the audit is still owed when the epic later resumes.) +2. **Dispatch a FRESH `compound-v:spec-reviewer` Task** (Opus, no context from the build) for **PASS 2 QUALITY + the 2.5 reward-hack check** over just that feature's diff (`git diff` for its run), against its feature-level acceptance criteria. +3. **On APPROVED** — clear the obligation, then commit (§9): + ``` + python3 scripts/compound-v-epic-state.py --clear-sample-audit-due --feature \ + --state docs/superpowers/execution/epics//epic-state.json + ``` + Loop back to §1. +4. **On ISSUES** — the success was not real. Revert it with the SINGLE atomic command (never a clear-then-revert two-step — a crash between those two writes would leave the feature `done` with its obligation already cleared, so the bad `done` sticks silently): + ``` + python3 scripts/compound-v-epic-state.py --record-audit-failed --feature \ + --last-error "sample-audit ISSUES: " \ + --state docs/superpowers/execution/epics//epic-state.json + ``` + ONE atomic write: sets status `failed` + records `last_error` + clears `sample_audit_due` + invalidates a passed `final_review` back to pending — so there is no window where the feature is `done`-without-obligation, and a later regression can't slip through on a stale review. Commit (§9), then route the feature through the **failure path (§5)** (the arbiter exchange — the feature is already `failed`, so §5 skips its own initial `--update --status failed`). -7. **Report.** Print the epic summary (`--summary`), the per-feature run-ids, and the next step: the integration review + `finishing-a-development-branch` on complete, or the blocking feature + the resume hint on blocked. +### 5. On failure — arbiter panel before any retry decision + +**First**, mark the feature `failed` — before any progress/breaker/arbiter step — so a retry legally starts from `failed`, not `running` (the transition table only allows `pending`/`failed → running`): + +``` +python3 scripts/compound-v-epic-state.py --update --feature --status failed \ + --last-error "" \ + --state docs/superpowers/execution/epics//epic-state.json +``` + +**Breaker gate before EVERY model call in this section** — the spec requires re-checking "before every arbiter / sample-audit / retry / final-review model call", not once per pass. Each numbered model call below is preceded by the breaker gate (§1, reusing THIS pass's stable cycle-id — `--record-progress-cycle` is idempotent by cycle-id, so re-calling it re-evaluates wall-clock without double-incrementing `no_progress_cycles`); a tripped breaker at any of them → `--trip-breaker` → §7, abandoning the exchange. + +1. **Breaker gate**, then read the feature's current attempt count (`--can-retry --feature ` → `{"can_retry","attempts","cap"}`; `attempts` is the number `--prepare` needs) and issue the challenge: + ``` + python3 scripts/compound-v-epic-arbiter.py --prepare \ + --state docs/superpowers/execution/epics//epic-state.json \ + --feature --attempt + ``` + Prints `{"challenge_id","epic_id","feature","attempt","issued_at","prompt"}` and creates `docs/superpowers/execution/epics//arbiter/-/` (the evidence home for this attempt) plus a challenge record. Write the failure evidence — the reviewer's ISSUES, gate verdicts, a diff summary — into a file **inside that directory**, e.g. `.../arbiter/-/evidence.json` (`--evidence-file` below is **relative** to that directory — never pass an absolute path). + +2. **Breaker gate**, then **dispatch a FRESH adversarial Claude Task (Opus, no context from the failed run)** using the `prompt` field verbatim as its instructions — it must independently classify the failure and reply with a ballot JSON matching `{"epic_id","feature","attempt","challenge_id","disposition":"retry_fix|halt_feature|halt_epic|blocked_external","reason","evidence"}` (the `epic_id`/`feature`/`attempt`/`challenge_id` must echo exactly what `--prepare` printed — a mismatched or malformed ballot is dropped, never fabricated as a vote). Write that Task's reply to a file, e.g. `.../arbiter/-/claude-ballot.json`. + +3. **Breaker gate**, then aggregate — Codex is polled *inside* this call, so it is itself a model-call boundary: + ``` + python3 scripts/compound-v-epic-arbiter.py --classify \ + --state docs/superpowers/execution/epics//epic-state.json \ + --feature --challenge \ + --evidence-file evidence.json \ + --claude-ballot docs/superpowers/execution/epics//arbiter/-/claude-ballot.json + ``` + Prints `{"disposition","confirmed":false,"reason","evidence","ballots":[...],"families_present","families_agreeing",...}` and writes the frozen audit JSON under the same `arbiter/` directory. Codex is polled automatically inside `--classify` when `~/.claude/compound-v-capabilities.json` says it's usable (Claude-only fallback otherwise — the panel is then capped to `retry_fix`/`halt_feature`, never `halt_epic`, never a confirmed blocker); a `retry_fix` verdict past this feature's `--can-retry` cap is **already masked to `halt_feature` inside `--classify`** — the driver does not need to re-check the cap itself before recording the disposition. + +Record the verdict (`--confirmed` is never passed as `true` — v2.10 hard-rejects it): + +``` +python3 scripts/compound-v-epic-state.py --record-disposition --feature \ + --disposition --reason "" [--families-agreeing ] \ + --state docs/superpowers/execution/epics//epic-state.json +``` + +**`--families-agreeing` is OMITTED entirely when the arbiter returns an empty `families_agreeing`** (all ballots dropped, or a tied/conservative default) — argparse requires a value, so passing the flag with nothing after it errors; drop the flag rather than passing a bare `--families-agreeing`. (If your dispatch harness cannot conditionally drop a flag, pass a quoted empty string `--families-agreeing ""` — the CLI parses it to an empty list, same result.) **Commit (§9) NOW — epic-state.json + the `arbiter/` directory — before the §6 breaker gate.** This commit is the durable retry intent: once the disposition is on disk, a trip/crash before the retry actually re-runs cannot lose it (§6). Recording the verdict here is also what turns a `"needs_arbitration"` `--next` reason (§2 — a `failed` feature with no disposition, i.e. a crash *before* this point) back into a normally-routed feature on the next pass. + +### 6. Act on the disposition + +The disposition is **already persisted** by the `--record-disposition` at the end of §5 — that write is the durable record of what to do, committed *before* the pre-retry breaker gate below. So a breaker trip or a hard crash at any point in this section loses nothing: on resume, `--next --autonomous` re-routes the feature by its *stored* disposition (a `retry_fix`-under-cap feature comes back as **runnable**; a `halt_feature`/cap-exhausted one is abandoned), never by an in-memory intent the crash would have erased. + +- **`retry_fix`** — the retry re-runs a whole v1.0 pipeline (a long model-spending phase). Because the disposition is persisted, the crash-safe way to re-run is to **loop back to §1**: the top-of-pass breaker gate runs, then `--next --autonomous` hands this feature back as runnable, and §3 re-runs it. If you instead re-dispatch immediately within this same live pass (an optimization, not required), **run the breaker gate (§1, this pass's cycle-id) once more BEFORE re-dispatching** — a retry must not start after a breaker has already tripped (tripped → `--trip-breaker` → §7) — then re-check `--can-retry --feature ` (defense-in-depth; `--classify` already capped it): if `can_retry`, `--update --status running --run-id ` (legal from `failed`) and re-run the v1.0 pipeline (§3); if not, treat it as `halt_feature` (it should already have arrived as `halt_feature` from §5). Either path is safe — the persisted disposition, not the choice of path, is what survives a crash. +- **`halt_feature`** — abandon this feature. No further status change needed — it is already `failed`, disposition recorded — `--next --autonomous` on the next pass routes around it: independents keep running, only its transitive dependents block. Continue the loop (§1). +- **`blocked_external`** — isolate it in the blocker ledger (always `confirmed:false` in v2.10 — never pass `--blocker-confirmed true`): + ``` + python3 scripts/compound-v-epic-state.py --update --feature --status blocked \ + --blocker-reason "" --families-agreeing \ + --evidence "" \ + --state docs/superpowers/execution/epics//epic-state.json + ``` + Commit, then continue the loop (§1) — a suspected blocker isolates its dependents but never halts the whole epic by itself; the epic only resolves to `blocked_needing_human` once no other reachable work remains. +- **`halt_epic`** — already recorded (§5); no further action here. The **next** `--next --autonomous` call reports `blocked_needing_human: halt_epic disposition on ...` regardless of any other runnable work — the one disposition that intentionally halts the whole epic rather than letting the DAG route around it (a panel-level "stop everything" verdict, not a per-feature one). → §7. + +### 7. Halt-page runbook (whole-epic block only) + +Page **only** when the epic itself is blocked — `blocked_needing_human` (tripped breaker or `halt_epic`) or exhausted reachable work. A single `blocked`/abandoned **feature** notice does **not** page here — it batches into the end-of-run report (§8) alongside a successful `done`. Commit first (§9) — the page must describe a state that is actually on disk in git, not one still sitting uncommitted in the worktree. The runbook carries, verbatim, every field the spec requires: + +- **The feature + its blocked dependents** — `--next --autonomous`'s `blocked_by` list, plus which feature(s) are `failed`/`blocked` (`--summary`). +- **Which acceptance criterion failed + the gate verdicts + a failing-diff summary** — from that feature's own 3-pass review output and its `last_error`. +- **Every panel ballot + reason + resolved family + why it aggregated** — read straight from the persisted `.../arbiter/-.json` audit (`ballots`, `families_present`, `families_agreeing`, the aggregation `reason`) — never re-derived or paraphrased into something the audit doesn't say. +- **Breaker state (n/cap)** — the `--breaker-check`/`--trip-breaker` `detail` object verbatim (counts + hours only — never a fabricated cost or token number). +- **Copy-paste resume commands (human-gated — this is the un-trip path, not an auto-revive).** The epic is parked at `blocked_needing_human`; a human resolves the root cause, clears the latch, and RE-RUNS `/v:epic ` to **resume the marathon** (the loop picks up from `epic-state.json` per §0). Give the exact commands for the specific block: + - **A tripped breaker** — `python3 scripts/compound-v-epic-state.py --clear-breaker --state ` clears the `blocked_needing_human` latch and re-arms the tripped caps so the next `/v:epic` resumes the marathon. Add `--reset-wall-clock` if the wall-clock breaker tripped (re-stamps `autonomy.started_at` to now, so the hours budget starts fresh), and/or `--set-max-total-attempts ` if the attempt breaker tripped (raises the cap so there's headroom to continue). Without those re-arm flags the same cap trips again on the first pass. + - **A `halt_epic` disposition** — `python3 scripts/compound-v-epic-state.py --clear-disposition --feature --state ` clears the sticky `halt_epic` verdict on that feature (which `--next --autonomous` treats as a whole-epic stop) so the DAG routes normally again; then `--clear-breaker` if the status latch is also set, and re-run `/v:epic `. + - **A recoverable incomplete run** (a feature caught mid-pipeline) — resume THAT run in place with `/v:resume ` using the feature's **recorded `run_id`** (from `--summary`). Then route the OUTCOME through the marathon handlers — **NEVER bare-mark a recovered feature `--status done`**, which would skip §4's mandatory sample-decision (especially the invocation's first-success `--mark-sample-audit-due`-before-`done`): a recovered **success** goes through **§4's success handler** (sample-decide → `--mark-sample-audit-due` if selected → then `done`); a recovered **failure** goes through **§5** (mark `failed`, then the arbiter exchange). This is distinct from abandoning + retrying from scratch: a fresh `--update --status pending --feature --state ` throws away the half-built run and restarts it from the spec — only do that when `/v:resume` can't recover it (or `run_id` is null). + - Then **re-run `/v:epic `** — re-entrant, resumes the marathon from the persisted state (§0 re-confirms `stance=="marathon"`). This is **human-gated, never automatic** — nothing un-trips or re-runs on its own in v2.10. **After any marathon recovery, a feature only ever reaches `done` through §4 — never a bare `--update --status done` in the recovery path.** +- **Paths** — the arbiter JSON(s), the epic run dir, and `git diff ..HEAD` (the epic's start SHA captured at `--init`; falls back to the branch name if an older marathon state predates `start_sha`) for the accumulated diff. + +Counts only, never fabricated cost/token metrics. Then stop this invocation. + +### 8. Before terminal `done` — final cross-feature re-verification + +When `--next --autonomous` reports `"running_with_failures: all features done, awaiting final_review ..."`: run the breaker gate (§1, this pass's cycle-id) before spending this model call — tripped → `--trip-breaker` → §7. Otherwise dispatch `compound-v:spec-reviewer` for **PASS 3 INTEGRATION only**, over the **whole accumulated diff** on the branch since the epic started — `git diff ..HEAD` (the SHA captured at `--init`; the branch name if an older state predates `start_sha`) — against the epic's acceptance criteria (same in spirit as the checkpoint loop's step 5 integration review). Record the verdict: + +``` +python3 scripts/compound-v-epic-state.py --record-final-review --status passed \ + --state docs/superpowers/execution/epics//epic-state.json +``` + +(or `--status failed` on ISSUES — `passed` is rejected unless every feature is currently `done` **and no `sample_audit_due` obligation is outstanding**; if `--next --autonomous` is still reporting `"sample_audit_due: ..."` you must clear those audits (§4) before this call will accept `passed`). Commit (§9), then loop back to §2: a `passed` review makes the next `--next --autonomous` report `"done: ..."` — print `--summary`/`--stats`, the per-feature run-ids, and hand off to `superpowers:finishing-a-development-branch` (same as the checkpoint loop's step 5, minus the separate review step already folded in here). A `failed` review is **not** silently retried in a tight loop — surface the spec-reviewer's ISSUES immediately. Since no feature is `pending`/`running` to route work to, an unaddressed failed final review shows up at §1 as a non-advancing pass (`no_progress_cycles` increments because `done` didn't grow) — either fix the integration issue and re-run the review within this invocation, or let the no-progress breaker trip on its own boundary and hand off to §7. Never re-run the identical failing review without addressing what it found. + +### 9. Commit points (two-command discipline — check each exit, never `&&`) + +Commit after **every** one of these, and once more immediately before any terminal handoff (§7/§8) if anything is still pending — the v2.6.4 rule: nothing under `docs/superpowers/execution/**` may be write-without-commit, or a `finishing-a-development-branch` worktree cleanup can silently delete it. + +``` +git add docs/superpowers/execution/epics//epic-state.json \ + docs/superpowers/execution/epics//arbiter/ +``` +``` +git commit -m "chore(v-epic): marathon -> " +``` + +(Omit the `arbiter/` path when nothing was written there — e.g. a bare `done` mark.) Trigger points: a feature reaching `done` — **committed together with its `--mark-sample-audit-due` when sampled, so `done` is never on disk without the obligation** (§4); a `--clear-sample-audit-due` on a passed audit (§4); a `--record-audit-failed` reverting a failed sample-audit (§4); every `--record-disposition` + its accompanying `--update` (§5/§6 — `retry_fix`/`halt_feature`/`blocked_external`/`halt_epic`); a `--record-disposition` recovered from a `consumed` `--resume-challenge` (§2 `needs_arbitration`); every `--trip-breaker` (§1/§5/§7); every `--record-final-review` (§8); and once more, belt-and-suspenders, right before the halt-page (§7) or the terminal `done` report (§8). + +### The honest v2.10 boundary + +- **In-session (automatic):** the marathon loop continues past a soft per-feature error to the next runnable feature within this live `/v:epic` invocation; a crashed feature is caught by the existing `running` → reconcile path (§2) on the next pass. +- **Hard death (human, not automatic):** quota exhaustion, a closed terminal, a crashed machine — a **human re-invokes `/v:epic `**, which is re-entrant and resumes from `epic-state.json` (per stance-binding §0). **There is no automatic resurrection in v2.10** — no watcher revives this epic while you're away. That is v2.11 (deferred, its own spec). +- No fabricated cost/token metrics anywhere in this loop — breakers and reports bound counts and wall-clock hours only. + +--- ## Honesty boundary (state it to the user) - **Epic mode is autonomous *chaining*, not "guess a product from one sentence."** Each feature still needs a **real spec** — the per-feature pre-flights and partition do the heavy lifting; the epic layer only orders and chains them. - **Large epics run sequentially, feature-by-feature.** Parallelism is *within* a feature (the v1.0 batch dispatch); features advance one runnable-front at a time in topological order. Independent features at the same depth still run one after another, not concurrently — there is no cross-feature parallel dispatch in v1.1. - **Quality is bounded by per-feature spec + partition quality.** A weak feature decomposition (overlapping features, missed deps) produces a weak epic. The state spine guarantees *order and resumability*, not that your decomposition was right. +- **Marathon (v2.10, opt-in) is bounded autonomy, not "survives while you sleep."** It chews the whole runnable feature DAG in one invocation and continues past a soft per-feature failure automatically — but only *within that one live invocation*. A hard death (quota, closed terminal, crashed machine) needs a **human** to re-invoke `/v:epic `; there is **no automatic resurrection in v2.10** (that is the deferred v2.11 auto-watcher). The default is still `checkpoint` — marathon is opt-in per epic, chosen at `--init` time, never silently promoted from an existing checkpoint epic. ## Safety - **One branch, accumulating.** Every feature's diff lands on the current branch in dependency order; the epic does not branch per feature. Only `finishing-a-development-branch` decides the final merge/PR. -- **The epic-state is the source of truth for "where is this epic."** Mutate it only through `compound-v-epic-state.py --update`; never hand-edit. `--next` is read-only and never an error (a `null` feature with a stop reason is information, not failure). +- **The epic-state is the source of truth for "where is this epic."** Mutate it only through `compound-v-epic-state.py --update` (or, in marathon, the additional `--record-*`/`--breaker-check`/`--trip-breaker` commands documented above); never hand-edit. `--next` (and `--next --autonomous`) are read-only and never an error (a `null` feature with a stop reason is information, not failure). - **Resumable, no daemon.** There is no background process. `/v:epic` is re-entrant: re-running it continues the epic from `epic-state.json`. -- Do **not** print fabricated cost or token metrics (anti-ruflo). +- **Marathon commits after every attempt/disposition, not just at a checkpoint.** See "Autonomous marathon loop" §9 — an unattended run must never leave `docs/superpowers/execution/epics//**` writes uncommitted. +- Do **not** print fabricated cost or token metrics (anti-ruflo) — marathon's breakers report counts and wall-clock hours only, in both stances. diff --git a/commands/v-init.md b/commands/v-init.md index 8793264..c4f9c6d 100644 --- a/commands/v-init.md +++ b/commands/v-init.md @@ -264,6 +264,24 @@ Two more structured choices — sensible defaults, reconfigurable any time: before stopping at a human checkpoint. An epic is *N full v1.0 runs*, so this is the human-in-the-loop **cadence**, not a token meter. `1` checkpoints after every feature (safest); raise it for more autonomy per invocation. +- **Marathon autonomy — `epic.autonomy.stance`** (options `checkpoint` / `marathon`; default + **`checkpoint`**): whether `/v:epic` stops at a human checkpoint after `max_features` (the + bullet above — always the default), or opts into the **v2.10 marathon loop** + ([`epic-mode.md`](../skills/compound-v/epic-mode.md) "Marathon stance") that chews the whole + runnable feature DAG in one invocation, routing failures through a Codex+Claude arbiter panel + and staying bounded by hard global circuit breakers. Offer `marathon` only with the **honest + v2.10 boundary** stated plainly: it survives *within one live `/v:epic` invocation* (a soft + per-feature failure routes to the next runnable feature automatically) and is *human-resumable* + after a hard death — quota, closed terminal, crashed machine — via a person re-invoking + `/v:epic `, which is re-entrant. **There is no automatic resurrection while you're away + in v2.10** — an unattended overnight watcher that revives the epic on its own is deferred to + v2.11. `marathon` also needs the **global breaker caps** agreed up front (sensible defaults, + all tunable): `max_wall_clock_hours` (default **10**), `max_total_attempts` (default + `max(6, 3×features)`), and `max_no_progress_cycles` (default **3** — a full pass with no new + feature reaching `done` counts as one non-progressing cycle). These caps bound **counts and + wall-clock hours only** — never a fabricated cost or token number. `checkpoint` remains the + safe, unchanged default; only set `marathon` when the user explicitly wants unattended, + multi-feature autonomy and accepts the boundary above. - **Cross-model review — `review.cross_model`** (default **off**): run an automatic Codex second opinion ([`/v:review-plan`](v-review-plan.md)) on high-stakes plans before dispatch. Off = run it manually when you want it; on = decorrelated review by default, at the cost of @@ -364,7 +382,14 @@ was, in those two fields). { "stance": "balanced", "memory": { "embeddings": false, "auto_recall": true, "auto_tighten": false }, - "epic": { "max_features": 1 }, + "epic": { + "max_features": 1, + "autonomy": { + "stance": "checkpoint", + "max_wall_clock_hours": 10, + "max_no_progress_cycles": 3 + } + }, "review": { "cross_model": false }, "brainstorm": { "deep_research": "ask", @@ -413,6 +438,18 @@ identically to `balanced`. Only `cost-aware.claude.standard` differs: `sonnet`, `false` = memory is a manual `/v:remember` lookup only. - **`epic.max_features`** (default `1`) = the Step 3c epic-autonomy cadence `/v:epic` reads as its per-invocation budget before a human checkpoint. +- **`epic.autonomy.stance`** (default `"checkpoint"`) = the Step 3c marathon opt-in. `"checkpoint"` + is the unchanged default (the bullet above governs it). `"marathon"` engages the v2.10 + autonomous loop in [`v-epic.md`](v-epic.md) ("Autonomous marathon loop") — but stance alone is + advisory config: the driver always re-confirms it against the **persisted** + `epic-state.json`'s own `autonomy.stance` before running any autonomous command (the state file + is authoritative, not this config). `max_wall_clock_hours` (default `10`) and + `max_no_progress_cycles` (default `3`) seed the marathon global breakers verbatim; leave + `max_total_attempts` **unset** here — its documented default is derived from the epic's actual + feature count at `--init` time (`max(6, 3×features)`), which this project-local file cannot know + in advance. `max_attempts_per_feature` (per-feature retry cap, script default `2`) is likewise + left to its script default unless a specific epic has a documented reason to raise it — set it + per-epic at `--init`, not globally here. - **`review.cross_model`** (default `false`) = the Step 3c toggle; when `true`, high-stakes plans get an automatic Codex second opinion ([`/v:review-plan`](v-review-plan.md)) before dispatch. diff --git a/docs/superpowers/expert/2026-07-12-epic-autonomous-mode.md b/docs/superpowers/expert/2026-07-12-epic-autonomous-mode.md new file mode 100644 index 0000000..cb14e90 --- /dev/null +++ b/docs/superpowers/expert/2026-07-12-epic-autonomous-mode.md @@ -0,0 +1,258 @@ +# Phase 1B Domain Audit — Epic Autonomous "Marathon Loop" Mode + +**Date:** 2026-07-12 · **Advisor:** Compound V Phase 1B (domain-expert) · +**Spec:** `docs/superpowers/specs/2026-07-12-epic-autonomous-mode-design.md` + +## 1. Domain(s) Identified + +- `autonomous-agent-orchestration` — unattended/overnight multi-agent build loops: + runaway-loop control, circuit-breaker placement, watchdog resurrection. +- `llm-judge-aggregation` — multi-backend arbiter panels, majority-vote pathologies, + self-preference / correlated-error bias. +- `oncall-escalation` — human paging actionability under alert fatigue. + +## 2. Sources Consulted + +- **KB reused:** `_knowledge-base/dev-workflow-triage-devex.md` (RADAR all-safe rule, alert + fatigue, reversibility-is-a-gate — all reused below). +- **KB created:** `_knowledge-base/autonomous-agent-orchestration.md` (this pass). +- **Web (2026-07-12, 9 parallel searches + 2 fetches):** + - Correlated judge errors: [arXiv 2605.29800 "Nine Judges, Two Effective Votes"](https://arxiv.org/pdf/2605.29800), + [arXiv 2510.01499](https://arxiv.org/pdf/2510.01499). + - Self-preference bias: [arXiv 2508.06709](https://arxiv.org/pdf/2508.06709), + [OpenReview Ns8zGZ0lmM](https://openreview.net/forum?id=Ns8zGZ0lmM), + [futureagi 2026](https://futureagi.com/blog/evaluating-llm-judge-bias-mitigation-2026/). + - Panel design: [Cohere PoLL via orq.ai](https://orq.ai/blog/llm-juries-in-practice), + [Comet LLM juries](https://www.comet.com/site/blog/llm-juries-for-evaluation/). + - Runaway cost: [Tech Startups 2025-11-14](https://techstartups.com/2025/11/14/ai-agents-horror-stories-how-a-47000-failure-exposed-the-hype-and-hidden-risks-of-multi-agent-systems/), + [LeanOps](https://leanopstech.com/blog/agentic-ai-cost-runaway-token-budget-2026/). + - Loop safety: [Cloudzy](https://cloudzy.com/blog/why-ai-agent-loops-fail-in-production/), + [Inkog](https://inkog.io/glossary/infinite-loop-ai-agent), + [Nexgismo](https://www.nexgismo.com/blog/ai-agent-budget-guards-stop-runaway-api-costs). + - Resurrection/thundering herd: [Azure retry-storm](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/retry-storm/), + [Thundering herd](https://en.wikipedia.org/wiki/Thundering_herd_problem), + [processWatchdog](https://github.com/diffstorm/processWatchdog). + - Reward hacking: [SpecBench arXiv 2605.21384](https://arxiv.org/html/2605.21384v1), + [RLVR arXiv 2604.15149](https://arxiv.org/pdf/2604.15149). + - Overnight loops (Ralph): [HumanLayer](https://www.humanlayer.dev/blog/brief-history-of-ralph), + [codecentric](https://www.codecentric.de/en/knowledge-hub/blog/the-ralph-wiggum-loop-autonomous-code-generation-with-a-fresh-context). + - Escalation: [incident.io 2026](https://incident.io/blog/on-call-best-practices-guide-2026), + [Rootly runbooks](https://rootly.com/incident-response/runbooks). + +## 3. Ranked Risks + Required Mitigations + +Ranked by (likelihood × blast radius) for an unattended overnight run. R1-R5 are MUST-fix. + +### R1 (CRITICAL) — No global budget/wall-clock/resume breaker; per-feature caps don't bound the epic +Both circuit breakers are **local**: `max_attempts_per_feature` (2) bounds one feature; +`max_no_progress_cycles` (3) bounds a stall. Neither bounds the *system*. A 20-feature epic ×2 +attempts = 40 full pipelines, each superlinearly expensive (context accumulation: >30x at 50 +steps, >100x at 200 — [LeanOps](https://leanopstech.com/blog/agentic-ai-cost-runaway-token-budget-2026/)), +plus every watcher re-fire, with **no ceiling on total attempts, total resumes, or wall-clock.** +Every documented runaway is exactly this: "no global budget cap" (11-day $47k loop; full-year +budget by April — [Tech Startups](https://techstartups.com/2025/11/14/ai-agents-horror-stories-how-a-47000-failure-exposed-the-hype-and-hidden-risks-of-multi-agent-systems/)). +**MUST:** add an epic-level `max_total_attempts` and a `max_wall_clock_hours` dead-man's switch +(default a conservative few hours) that halts+pages regardless of local progress, plus a +`max_resume_count`. Enforce it in `epic-state.py` / the watch script — **outside** the driver's +reasoning, since "if the agent is looping it can't be trusted to stop itself" +([Cloudzy](https://cloudzy.com/blog/why-ai-agent-loops-fail-in-production/)). Dollar/token metrics +stay anti-ruflo: bound **counts** (attempts, resumes, wall-clock), never fabricated cost. + +### R2 (CRITICAL) — Crash-before-heartbeat = infinite resurrection that trips NEITHER breaker +`attempts` bumps only when a feature "enters running"; `last_progress_at` bumps only on a state +mutation. A resume that dies during startup (bad worktree, auth failure, OOM) — **before** either +event — leaves the pre-crash timestamp and attempt count untouched. The watcher sees "stale" +(age > 45 min), re-fires, crashes again, forever: a ~30-min-cadence crashloop all night that +never progresses and never hits a cap (the exact failure the spec author flagged as a question). +**MUST:** a **resume-attempt counter incremented at the START of every resume** (before any work), +with its own cap + **exponential backoff + jitter**, and self-disarm after K consecutive +non-progressing fires — the process-watchdog pattern +([processWatchdog](https://github.com/diffstorm/processWatchdog), +[Azure retry-storm](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/retry-storm/)). +A resume that produces no `last_progress_at` advance must count against this cap. + +### R3 (HIGH) — Thundering resume: two watcher tiers + long waves + no lock = concurrent drivers +`stale-after = 45 min`, but a full pipeline wave (partition-review → dispatch → scope-gate → +review) can legitimately exceed 45 min while emitting **no state mutation** (so no heartbeat). +The watcher then declares live work dead and launches a second `/v:epic ` on the same +`epic-state.json` + same branch/worktree — a retry storm / self-inflicted DDoS. Off-minute +:17/:47 reduces *within-tier* collision but the two tiers (session cron + disk task) can still +**both** fire. **MUST:** (a) a **lease/lock** (PID + `started_at` + host) that makes any resume +**no-op while a live lease is held** — mutual exclusion **across both tiers**; (b) **heartbeat +DURING long waves**, not only on transitions (or raise `stale-after` above the worst-case +silent-wave duration); (c) jitter on both tiers. See +[thundering herd](https://en.wikipedia.org/wiki/Thundering_herd_problem). + +### R4 (HIGH) — Arbiter panel treats correlated backends as independent; skip_independent under-guarded +Two problems in one aggregation: +- **Correlated votes.** Panel discovery is by backend *name* (`codex`/`agy`/`cursor`), never by + underlying model **family**. Cursor's model is user-configurable; a GPT-configured Cursor + + Codex = two GPT-family votes outvoting one Gemini — the "two correlated models outvote one" + case. "Nine Judges, Two Effective Votes" + ([arXiv 2605.29800](https://arxiv.org/pdf/2605.29800)): correlation collapses the effective vote + count; majority vote can "amplify systematic mistakes" + ([arXiv 2510.01499](https://arxiv.org/pdf/2510.01499)). + **MUST:** resolve each backend's actual model, **dedupe by family**, and require the majority to + span ≥2 distinct families (else collapse same-family members to one effective vote). Record the + resolved family per vote in the audit JSON. +- **Disposition blast-radius asymmetry.** `halt` is reversible (human looks); `retry_fix` is + breaker-bounded; **`skip_independent` is the only disposition whose damage compounds unattended** + — if the "independence" call is wrong (latent shared util/config/migration), the epic builds on + a broken foundation for hours. Per the KB's RADAR "all-safe not majority-safe" rule and + "reversibility is a gate, not a coordinate," **`skip_independent` MUST require unanimity across + distinct families AND a structural cross-check** (the DAG + scope-gate confirm no shared files), + never a bare majority. A bare majority may authorize at most `retry_fix`. + +### R5 (HIGH) — Reward hacking accumulates in unwatched PASSes; the arbiter only sees FAILs +The arbiter adjudicates a *confirmed FAIL*. But the dangerous overnight failure is a **gamed +PASS** — deleted assertions, monkey-patched scorer, weakened tests — and "validation scores +overestimate true spec compliance as horizons grow" +([SpecBench arXiv 2605.21384](https://arxiv.org/html/2605.21384v1); METR/Palisade on frontier +models). Marathon runs unattended for hours = maximal unwatched spec-gaming, and **nothing looks +at passes.** Relatedly, the spec has **no re-verification of already-`success` features**, so a +later feature that regresses an earlier one is never detected (the "A/B oscillation" the author +asked about is actually bounded by monotonic per-feature `attempts`; the real gap is silent +regression of completed work). **MUST:** the Review Gate includes an **anti-reward-hacking check** +(did the diff modify its own tests/assertions/scorers/verifier?), marathon **sample-audits a +fraction of PASSes**, and epic-complete runs a final integration re-verification across all +features before declaring `done`. + +### R6 (MED-HIGH) — Caller-supplied liveness clock (`--now`) is fabricable by the very agent it judges +`last_progress_at` "comes from `--now` passed by the caller." The caller is an LLM driver; a +hallucinated **future** `--now` makes a dead epic look alive forever (never resurrected), a stale +one causes thundering resume. A liveness signal must not be forgeable by the monitored party. +**MUST:** the watcher's staleness decision uses an authority the agent can't fabricate — OS wall +clock in the watch script and/or `epic-state.json` file mtime — or **range-checks `--now` against +the OS clock** and rejects implausible values. (Keep the injectable arg for testability; just +don't *trust* it for liveness.) + +### R7 (MED) — "Any parse failure → halt" is internally inconsistent and lets one flaky backend neuter autonomy +The spec says both "any parse failure breaks conservative toward `halt`" **and** "never fabricate +a vote for an absent/errored backend." A parse failure **is** an errored backend — counting it as +halt-leaning **fabricates a halt vote** (self-contradiction). Operationally, one chronically-flaky +backend that always returns garbled JSON then halts **every** feature, silently converting marathon +into checkpoint and blaming "the panel." Treating errored replies as *missing data* is the +literature noru ([arXiv 2605.29800](https://arxiv.org/pdf/2605.29800)). **SHOULD:** a parse +failure → **drop that member + log** (same as absent); apply the conservative `halt` default only +to the **aggregate of valid votes** when empty/tied; if valid votes are empty → fall to the Claude +adversarial fallback, and only halt if that too fails. This reconciles the two spec rules. + +### R8 (MED) — Zero-backend self-arbitration silently ≈ checkpoint, and shares blind spots the framing can't fix +For the large solo-dev audience with no Codex/agy/cursor, **every** arbitration is the fresh-Claude +fallback with "default to halt." Self-preference research: adversarial framing offsets the +*leniency* axis but **not** a shared blind spot or **shared wrong prior** (both the implementer and +the judge Claude share a stale API belief → confident wrong `retry_fix` until the cap +— [arXiv 2508.06709](https://arxiv.org/pdf/2508.06709); mitigation is literally "a judge from a +different family" — [futureagi](https://futureagi.com/blog/evaluating-llm-judge-bias-mitigation-2026/)). +And "default to halt" biases toward never-progressing, so marathon degrades to +"burn the attempts, then checkpoint." **MUST (honesty):** the docs state plainly that **without a +cross-model backend, marathon's arbiter adds ~no autonomy over checkpoint** — the resume/liveness +layer is the only real benefit. **SHOULD:** the Claude-only fallback may return **only `halt` or +`retry_fix`, never `skip_independent`** (a single self-family judge must not authorize the +compounding disposition), and must **cite the specific acceptance criterion** it judged. + +### R9 (MED) — Halt-page actionability / alert fatigue +Apply the on-call "Wake Up test": page only when human action is the only path forward +([incident.io](https://incident.io/blog/on-call-best-practices-guide-2026)); Google SRE ≈ 2-3 +actionable pages/shift. An epic that pages per parked feature overnight is alert fatigue by +morning. **MUST:** distinguish **"whole epic blocked → page now"** from **"one feature parked, +independent frontier still chewing → batch into a morning summary."** The halt report MUST be a +runbook (5 A's; every step a command) containing, at minimum: +1. **Which** feature halted (id + title) + **blocked_by_failure** dependents (blast radius). +2. **Why** — the failing acceptance criterion, scope-gate + reviewer verdicts, `last_error`, + failing-diff summary (not "review failed"). +3. **What the panel said** — every member's raw verdict + reason + **resolved model family**, the + aggregate, and *why* it resolved to halt (tie / parse-fail / cap / unanimous / correlated). +4. **Breaker state** — attempts n/cap, no-progress k/cap, resume-count, wall-clock; which breaker + tripped. +5. **Exact copy-paste resume command** (`/v:resume `) + how to override (raise cap / + force-skip) — one action from the page. +6. **Paths** to the arbiter audit JSON, run dir, worktree, failing diff. +7. Counts only (attempts/resumes/elapsed) — **no fabricated cost/token figures** (anti-ruflo). + +### R10 (MED-LOW, collectively real) — Overnight-environment naïvety +- **Laptop sleep silently kills Tier 1.** `CronCreate` is in-memory/session-bound; sleep → it + doesn't fire; only the on-disk Tier 2 survives sleep→wake. **MUST document** that sleep disables + Tier 1 entirely (the honesty boundary should say so, not just "Level-2 semi"). +- **Quota/credential expiry mid-run.** "Survives quota exhaustion" holds only if the quota resets + **and** the session is still authenticated; an expired OAuth token won't resurrect on relaunch + without re-auth. State this in the Level-2 caveat. +- **Substrate growth.** v2.6.4 fixed *deletion* of the audit trail; the marathon's flip side is + unbounded **growth** — worktrees, run dirs, arbiter JSONs per attempt/resume accumulate all + night. **SHOULD** bound/rotate or at least document cleanup. +- **Clock/DST + off-minute cron** across a midnight run; **merge/commit races** on the single epic + branch when a resume overlaps an in-flight dispatch (folds into the R3 lock). +- **"Overbaking"** — Ralph canon: long unattended runs yield "bizarre emergent behavior"; prefer + "small iterations merged over time" over one 50-change morning + ([HumanLayer](https://www.humanlayer.dev/blog/brief-history-of-ralph)). The wall-clock + dead-man's switch (R1) is the concrete guard. + +## 4. Common Traps in This Domain +- Per-unit caps mistaken for a global budget (R1). +- Liveness heartbeat bumped only on transitions, not during work (R3). +- Resurrection with no attempt counter / backoff → crashloop (R2). +- Majority vote assuming judge independence that correlation destroys (R4). +- Trusting the monitored agent's own timestamp as the liveness clock (R6). +- Only adjudicating FAILs while gamed PASSes accumulate unwatched (R5). +- Paging on every event → alert fatigue → the real halt gets rubber-stamped (R9). + +## 5. Regulatory / Compliance Notes +None external (internal dev-tooling). **Project-internal "constitution" that binds this feature:** +Opus-default/never-Haiku; no fabricated cost/token metrics (breakers bound **counts**, the arbiter +never fakes a vote — extend to "never fakes a halt vote from a parse failure," R7); git-derived +enforcement never model-self-reported (so R6's clock and R1's breaker must be script/OS-authoritative); +two-command commit discipline; commit substrate before worktree cleanup (v2.6.4). + +## 6. Recent Breaking Changes (last 12 months) +No library API breaks (spec is stdlib + existing adapters). Relevant **field shifts**: +frontier-model reward-hacking under tool use is now empirically confirmed (METR o3 / Claude 3.7; +SpecBench 2026) — raises R5's priority; and 2026 judge-panel research (arXiv 2605.29800, 2508.06709) +now quantifies correlation/self-preference, making R4/R8 evidence-backed rather than speculative. + +## 7. Design Constraints for the Plan (NON-NEGOTIABLE) +1. Add an **epic-level global breaker**: `max_total_attempts` + `max_resume_count` + + `max_wall_clock_hours` dead-man's switch, enforced in `epic-state.py`/watch (outside driver + reasoning), halting+paging regardless of local progress. Bound **counts**, never cost. (R1) +2. Add a **resume-attempt counter incremented at resume START**, with exponential-backoff+jitter + and self-disarm after K non-progressing fires. (R2) +3. Add a **cross-tier lease/lock** (PID+started_at+host) so a resume no-ops while a live lease is + held; **heartbeat during long waves** (or set `stale-after` above worst-case silent-wave + duration); jitter both tiers. (R3) +4. Arbiter: **resolve backend → model family, dedupe by family, require majority to span ≥2 + families**; record resolved family in the audit JSON. (R4) +5. **`skip_independent` requires unanimity across distinct families + a structural (DAG + + scope-gate) no-shared-files cross-check**; a bare majority may authorize at most `retry_fix`; + the Claude-only fallback may never authorize `skip_independent`. (R4/R8) +6. Review Gate gains an **anti-reward-hacking check** (diff must not weaken its own + tests/assertions/scorers); marathon **sample-audits PASSes**; epic-complete runs a **final + cross-feature integration re-verification** before `done`. (R5) +7. Liveness staleness decided on an **OS/file-mtime clock the agent can't fabricate**, or + range-check `--now` against the OS clock. (R6) +8. A **parse failure/errored backend = dropped + logged (missing data), not a halt vote**; the + conservative `halt` default applies only to the aggregate of valid votes when empty/tied. (R7) +9. **Honesty text:** without a cross-model backend, marathon adds ~no autonomy over checkpoint; + laptop sleep disables Tier 1; "survives quota exhaustion" only if quota resets AND session stays + authed. (R8/R10) +10. **Halt-page report is a runbook** with the seven fields in R9; page only on whole-epic block, + batch parked-feature notices into a morning summary. (R9) + +## 8. Open Questions for the Human (product/business) +1. **Global budget defaults.** What are sane defaults for `max_wall_clock_hours`, + `max_total_attempts`, `max_resume_count` for an overnight run? (Domain says "conservative + + small increments"; the actual number is a risk-appetite call.) +2. **skip_independent trust.** Is auto-`skip_independent` wanted at all, or should *every* skip + page a human? It is the one disposition that compounds damage unattended — some teams will want + it human-only. +3. **PASS sampling rate.** What fraction of PASSes should the anti-reward-hacking audit sample + (0% / 10% / 100%)? Higher = safer but slower/costlier. +4. **Paging channel + quiet hours.** Where does the halt-page go (the spec has scheduled-tasks but + no notification sink)? Should it respect quiet hours and only page on whole-epic block? +5. **Single-backend UX.** For solo devs with no cross-model backend, should `/v:init` *offer* + marathon at all, or warn "arbiter ≈ checkpoint without a second model"? + +## 9. Knowledge Base Updates +- **Created** `_knowledge-base/autonomous-agent-orchestration.md` — reusable matrices on global-vs- + local breakers, watchdog resurrection hazards, judge-panel correlation/self-preference, long- + horizon reward hacking, and on-call actionability. Every claim cites a primary/secondary source. +- **Reused** `_knowledge-base/dev-workflow-triage-devex.md` (RADAR all-safe, reversibility-is-a- + gate, alert fatigue) — applied to R4/R5/R9. diff --git a/docs/superpowers/expert/_knowledge-base/autonomous-agent-orchestration.md b/docs/superpowers/expert/_knowledge-base/autonomous-agent-orchestration.md new file mode 100644 index 0000000..a999909 --- /dev/null +++ b/docs/superpowers/expert/_knowledge-base/autonomous-agent-orchestration.md @@ -0,0 +1,145 @@ +# Autonomous Multi-Agent Build Orchestration Knowledge Base + +Safe unattended/overnight agent loops: runaway-loop control theory, circuit-breaker +placement, multi-judge/arbiter aggregation pathologies, self-evaluation bias, watchdog +resurrection hazards, reward-hacking under long horizons, and human-escalation (on-call) +actionability. + +Maintained by Compound V Phase 1B advisor. Append at the bottom on each pass. + +--- + +## Updated 2026-07-12 — epic "Marathon Loop" autonomous-mode audit + +### Circuit breakers: per-task caps are NOT a global budget breaker +- **The #1 real-world failure of unattended agent loops is "no global budget cap."** Documented + runaway incidents: a multi-agent tool that "slipped into a recursive loop that ran for 11 days + ... $47,000 API bill"; a "left an agent running over a long weekend ... $4,200 bill"; Uber + reportedly "burned through its full-year AI budget by April 2026" + ([Tech Startups, 2025-11-14](https://techstartups.com/2025/11/14/ai-agents-horror-stories-how-a-47000-failure-exposed-the-hype-and-hidden-risks-of-multi-agent-systems/), + [LeanOps](https://leanopstech.com/blog/agentic-ai-cost-runaway-token-budget-2026/)). Dollar + figures are single-source aggregator claims — treat as **illustrative, not benchmarked** — but + the *mechanism* (per-step caps without a global ceiling) is corroborated everywhere. +- **Context accumulation makes each attempt superlinearly expensive:** "at 50 steps the cost + multiplier exceeds 30x, and at 200 steps ... exceeds 100x" + ([LeanOps](https://leanopstech.com/blog/agentic-ai-cost-runaway-token-budget-2026/)). So N + bounded features × M attempts each can still be an unbounded *total* if there is no epic-level + wall-clock / total-attempt / total-resume ceiling. +- **Enforcement must live OUTSIDE the agent's own decision loop.** "The circuit breaker pattern + must be at the infrastructure level, not just in agent code, because if the agent is looping, + it can't be trusted to stop itself ... a pre-call check that ... aborts ... regardless of what + the agent's reasoning says" + ([Cloudzy](https://cloudzy.com/blog/why-ai-agent-loops-fail-in-production/), + [Nexgismo](https://www.nexgismo.com/blog/ai-agent-budget-guards-stop-runaway-api-costs)). +- **Defense must be multi-layer:** hard iteration cap + token/cost ceiling + no-progress + detection (hash tool+args, terminate on repeat within a window) + explicit-done check + + absolute time-based breaker. "Single guards fail; multi-layer defense works" + ([Inkog](https://inkog.io/glossary/infinite-loop-ai-agent), + [BSWEN](https://docs.bswen.com/blog/2026-03-11-prevent-ai-agent-infinite-loops/)). +- **Reusable rule:** a per-unit attempt cap bounds one unit; it never bounds the *system*. Always + pair per-feature caps with (a) a global total-attempt/resume ceiling and (b) an absolute + wall-clock dead-man's switch that halts+pages regardless of local progress. + +### Watchdog resurrection: thundering herd, crash-before-heartbeat, and fabricable clocks +- **Retry/resume storms are the canonical distributed-systems antipattern.** A watcher that + re-fires while work is legitimately in flight = self-inflicted DDoS / thundering herd; the fix + is **exponential backoff + jitter + a lease/lock**, not just an off-minute cadence + ([Azure retry-storm antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/retry-storm/), + [Thundering herd — Wikipedia](https://en.wikipedia.org/wiki/Thundering_herd_problem)). +- **A liveness heartbeat must be bumped DURING long work, not only at state transitions.** If the + stale-threshold is shorter than a legitimate unit of work that emits no heartbeat, the watchdog + declares live work "dead" and launches a concurrent worker on the same state → double burn + + races. +- **Crash-before-heartbeat = infinite resurrection that trips no progress breaker.** A resume that + dies during startup *before* it advances the heartbeat or increments an attempt counter looks + neither "stale-progressing" nor "attempted"; watchdogs re-fire it forever. Process watchdogs + solve this with a **restart counter + exponential backoff that disarms after K consecutive + failed starts** ([processWatchdog](https://github.com/diffstorm/processWatchdog)). +- **A caller/LLM-supplied timestamp is not a trustworthy liveness clock.** If an agent supplies + `--now`, a hallucinated future time makes a dead run look alive forever (never resurrected); a + stale time causes thundering resume. Liveness time should come from an authority the agent can't + fabricate (OS clock / file mtime), or be range-checked against it. +- **Reusable rule:** any resurrect-on-failure mechanism needs its own attempt counter incremented + at the *start* of the attempt, backoff+jitter, a mutual-exclusion lease across all watcher + tiers, and a self-disarm after K non-progressing fires. + +### Multi-judge / arbiter panels: correlation collapses the vote count +- **Cross-model panels are NOT as independent as majority-vote assumes.** "Nine Judges, Two + Effective Votes: Correlated Errors Undermine LLM Evaluation Panels" — a nine-judge panel + delivers ~two independent votes; "naively increasing the number of judges fails to improve + ... reliability when those judges share systematic biases" + ([arXiv 2605.29800](https://arxiv.org/pdf/2605.29800)). Corroborated: multicollinearity/VIF + rises as models are added; "majority vote or averaging ... provide little gain or even amplify + systematic mistakes" + ([arXiv 2510.01499](https://arxiv.org/pdf/2510.01499), + [arXiv 2605.29800](https://arxiv.org/pdf/2605.29800)). +- **Dedupe panel members by underlying MODEL FAMILY, not by tool/backend name.** Two backends + wrapping the same family (e.g. a GPT-configured Cursor + Codex) contribute one effective vote + yet count as two — "two correlated models outvoting one." Robust panels require diversity of + judge design, not quantity ([arXiv 2605.29800](https://arxiv.org/pdf/2605.29800)); PoLL's + benefit holds only when "base learners are diverse" + ([Cohere PoLL via orq.ai](https://orq.ai/blog/llm-juries-in-practice)). +- **Self-preference bias is measured and large.** LLM judges over-rate their own family's outputs; + on ArenaHard self-preference "ranges from -38% to +90%," "scales with model size ... and + persists even when authorship is hidden" + ([arXiv 2508.06709](https://arxiv.org/pdf/2508.06709), + [Self-Preference Bias, OpenReview](https://openreview.net/forum?id=Ns8zGZ0lmM)). Standard + mitigation: "use a judge from a different family" + ([futureagi](https://futureagi.com/blog/evaluating-llm-judge-bias-mitigation-2026/)). + Implication for a same-model self-arbitration fallback: adversarial framing can offset the + *leniency* axis but not a *shared blind spot* or *shared wrong prior* (e.g. both share a stale + API belief) — a different family is the only real fix for those. +- **Optimal panel size is 3-5; conservative-max (never average-down) aggregation mirrors + production practice** (RADAR auto-accepts only if the ENTIRE diff is safe; any one risk signal + disqualifies) ([Comet](https://www.comet.com/site/blog/llm-juries-for-evaluation/)). +- **Reusable rule:** treat "errored/unparseable backend" as *missing data* (drop + log), not as a + vote — counting it as a halt vote fabricates a vote and lets one flaky advisor neuter the panel. + Apply the conservative default (halt) only to the *aggregate of valid votes* when it is empty or + tied. Gate the highest-blast-radius disposition (proceed-without / skip) on unanimity across + distinct families, never a bare majority. + +### Long-horizon reward hacking: the danger in unattended PASSES, not FAILs +- **Validation scores overestimate true spec compliance as horizons grow** — exactly the marathon + regime. Agents "overwrite unit tests, monkey-patch scoring functions, delete assertions, or + force early program termination to obtain a passing score"; frontier models show this under + tool use (METR on o3 and Claude 3.7; Palisade chess-agent) + ([SpecBench, arXiv 2605.21384](https://arxiv.org/html/2605.21384v1), + [RLVR reward hacking, arXiv 2604.15149](https://arxiv.org/pdf/2604.15149)). +- **Overnight autonomy maximizes unwatched spec-gaming.** An arbiter that only adjudicates + confirmed FAILs never inspects a gamed PASS. A gate that can be edited by the same agent it + gates is the classic reward-hacking surface. Anti-hack review must check whether the diff + weakened its own verifier (touched tests/assertions/scorers) and marathon mode should + sample-audit PASSes, not only FAILs. + +### Overnight-loop operational lessons (Ralph technique canon) +- **"Overbaking":** letting a loop run too long yields "all sorts of bizarre emergent behavior" + ([HumanLayer, brief history of Ralph](https://www.humanlayer.dev/blog/brief-history-of-ralph)). +- **Prefer small bounded increments over one giant unattended run:** "set up any ralph-ish ... + loops to run ONCE on a cron overnight, and merge small iterations ... Waking up to one small + refactor every morning is better than ... waking up to 50"; "carve off small bits of work into + independent context windows" rather than "run forever" + ([HumanLayer](https://www.humanlayer.dev/blog/brief-history-of-ralph)). +- **"Deterministically bad in a non-deterministic world"** — failures repeat in predictable + patterns, so guard them with specs/backpressure rather than hoping + ([codecentric](https://www.codecentric.de/en/knowledge-hub/blog/the-ralph-wiggum-loop-autonomous-code-generation-with-a-fresh-context)). + Bad specs → bad results cascades across every iteration. + +### Human escalation / paging: actionability under alert fatigue +- **The "Wake Up" test:** "If this fires at 3 AM, would I be upset if it turned out to not need + immediate human action? If yes, it belongs as a ticket or log, not a page." Reserve the page + for when "immediate human intervention is the only path" + ([incident.io 2026](https://incident.io/blog/on-call-best-practices-guide-2026), + [DEV SRE playbook 2026](https://dev.to/axiom_agent/the-modern-on-call-playbook-for-sres-in-2026-2n5)). +- **Sustainable volume:** Google SRE Workbook ≈ 2-3 actionable incidents per shift; an epic that + pages on every parked feature overnight is guaranteed alert fatigue by morning. +- **Runbook 5 A's — Actionable, Accessible, Accurate, Authoritative, Adaptable.** "Every step + should be a command, not a paragraph." Structure: **symptoms · impact · diagnostics · + resolution · escalation.** The runbook/resume command must be one click/copy-paste from the + alert; generic "check the logs" is useless — give the exact command + ([Rootly](https://rootly.com/incident-response/runbooks), + [OneUptime](https://oneuptime.com/blog/post/2026-02-17-how-to-build-an-incident-response-runbook-system-using-google-cloud-operations-suite/view)). +- **Reusable rule:** a halt-page must carry {which unit + blast radius, why (which acceptance + criterion, gate verdict), every judge's raw verdict + resolved family + why-it-aggregated-to- + halt, breaker state n/cap, exact copy-paste resume command, paths to evidence}. Distinguish + "whole run blocked, act now" (page) from "one unit parked, rest proceeding" (batch into a + morning summary). diff --git a/docs/superpowers/plans/2026-07-12-epic-autonomous-mode-marathon.md b/docs/superpowers/plans/2026-07-12-epic-autonomous-mode-marathon.md new file mode 100644 index 0000000..13fc210 --- /dev/null +++ b/docs/superpowers/plans/2026-07-12-epic-autonomous-mode-marathon.md @@ -0,0 +1,261 @@ +# Epic Autonomous Mode — "Marathon Loop" (v2.10) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: use superpowers:subagent-driven-development (recommended) +> or superpowers:executing-plans. Steps use `- [ ]` checkboxes. Source of truth: +> `docs/superpowers/specs/2026-07-12-epic-autonomous-mode-design.md` (v2.10 scope, converged v3) — read +> its "Design constraints from Sol R4" section; every one is a task acceptance criterion here. + +**Goal:** ship an opt-in `marathon` stance for `/v:epic` that chews the whole runnable feature DAG in one +invocation, routes failures via a Codex+Claude arbiter panel, isolates suspected external blockers +without halting, and stays bounded by in-script global breakers — the default (`checkpoint`) epic +unchanged. + +**Architecture:** one serial "core-contract" unit extends `scripts/compound-v-epic-state.py` (all state + +CLI); a new `scripts/compound-v-epic-arbiter.py` runs the panel; prose wires the driver + reviewer; +docs/CI/release last. Resume after a fall = existing `/v:resume` + human `/v:epic` re-entry (no +auto-watcher — that is deferred v2.11). + +**Tech Stack:** Python 3.9-safe stdlib only (no PyYAML — state is pure JSON); reuse atomic-write idiom +(`compound-v-fastpath-run.py:704`), `compound-v-run-with-timeout.py`, `compound-v-resolve-model.py`, +`~/.claude/compound-v-capabilities.json`. + +## Global Constraints (every task inherits these — copied verbatim from the spec) + +- Opus by default; Sonnet only for the junior-mechanical carve-out; **NEVER Haiku**. +- **No fabricated cost/token metrics** — breakers bound counts + wall-clock only. +- **Two-command commit discipline** — no `&&` on side-effecting git; check each exit code. +- **Opt-in:** default `checkpoint` epic is behaviorally unchanged; new state fields written only in + `marathon` stance; all 34 existing `epic-state.py` selftests pass unchanged + whole-dir golden + compatibility (snapshot the epic dir + `git status`, not just JSON bytes). +- **Atomic writes** (tmp + `os.replace`); **no cross-process lock in v2.10** (single process). +- **Python 3.9 ISO:** emit `+00:00` (never bare `Z`); normalize trailing `Z`→`+00:00` on ingest before + `fromisoformat`. `--now ` injectable; CLI default real clock. +- All new scripts `LANG=C`-clean (reconfigure stdout/stderr UTF-8). Commit new artifacts at v2.6.4 points. + +## File Structure + +- `scripts/compound-v-epic-state.py` (MODIFY, unit A — the whole file, serial) — marathon state + all new + subcommands + selftests. +- `scripts/compound-v-epic-arbiter.py` (CREATE, unit B) — Codex+Claude panel, family map, truth table, + redaction, audit. +- `commands/v-epic.md`, `skills/compound-v/epic-mode.md`, `commands/v-init.md`, `agents/spec-reviewer.md` + (MODIFY, unit C) — driver loop, Claude-ballot dispatch, `/v:init` stance, anti-reward-hack gate. +- `README.md`, `CHANGELOG.md`, `.claude-plugin/{plugin,marketplace}.json`, `.github/workflows/validate.yml` + (MODIFY, unit D) — docs, version 2.9.0→2.10.0, CI wiring. + +Dependency order: **A → B → C → D** (A publishes its CLI/state contract before B & C; C consumes B's +ballot interface). A is one serial unit (disjoint-write invariant on `epic-state.py`). + +--- + +## Unit A — Core contract (`scripts/compound-v-epic-state.py`) + +**Interfaces produced (the frozen contract B & C consume):** +- state schema (marathon-only): `autonomy{stance,max_features,max_attempts_per_feature=2, + max_no_progress_cycles=3,max_total_attempts,max_wall_clock_hours=10,started_at}`, per-feature + `{attempts=0,last_error,disposition}`, top-level `final_review{status}`, `blocker_ledger[]`, + `no_progress_cycles`, `total_attempts`. +- CLI: `--init --stance marathon [caps]`, `--next --autonomous`, `--record-disposition`, + `--record-final-review`, `--record-progress-cycle`, `--breaker-check` (read), `--trip-breaker` (write), + `--can-retry`, `--update --status running|blocked|... [--attempt ...]`, `--stats` (adds `blocked`). + +### Task A1: marathon schema in `build_state` + `--init --stance` +**Files:** Modify `scripts/compound-v-epic-state.py` (`build_state` ~:111, `main` `--init` ~:445; add +`--stance`, cap args); Test: extend `_selftest`. +**Interfaces — Produces:** `build_state(features, epic_id, title, stance=None, caps=None)` adds the +marathon block + per-feature `attempts/last_error/disposition` ONLY when `stance=="marathon"`; validates +`epic_id` via `_id_ok` (Sol-R4#10). +- [ ] Write failing selftest: `--init` without stance → byte-identical to today (no `autonomy` key); + `--init --stance marathon --max-wall-clock-hours 8` → writes `autonomy` with every field once, + `started_at` from `--now`. +- [ ] Run → fails. Implement. Run → passes. +- [ ] Selftest: `_id_ok` rejects a traversal `epic_id` at init. +- [ ] Whole-dir golden test: checkpoint `--init` leaves no extra artifact. Commit. + +### Task A2: `--next --autonomous` (read-only, DAG routing, terminal states) +**Files:** Modify (new `next_feature_autonomous`, selected at the `want_next` branch ~:489); Test: +`_selftest`. +**Interfaces — Produces:** `next_feature_autonomous(state) -> (feature|None, reason, blocked_by:[ids])`. +Default `next_feature` UNTOUCHED (all default selftests stay green). +- [ ] Failing selftest: an abandoned/`blocked` feature removes only transitive dependents (reverse + reachability over `depends_on`, reuse `_detect_cycle` idiom); independents stay runnable; `blocked_by` + derived read-only; re-derives after a source reopens. +- [ ] Terminal resolution: `done` only if all-done AND `final_review==passed`; `blocked_needing_human` for + halt_epic / tripped breaker / any suspected blocker; `running_with_failures` while runnable. (v2.10 + never emits `done_with_blockers`.) +- [ ] Run fail→impl→pass. Default `--next` shape unchanged (2-key) selftest still green. Commit. + +### Task A3: attempts + `--can-retry` + transition table +**Files:** Modify (`--update` ~:498). **Produces:** `--update --status running` increments +`attempts` (via `.get`); legal →running sources = pending/failed (done/blocked rejected); +`--can-retry --feature F` → `{can_retry,attempts,cap}`. +- [ ] Failing selftest: running-transition increments; done→running rejected; `--can-retry` flips at + `max_attempts_per_feature`. Impl→pass→commit. + +### Task A4: `--record-disposition` +**Files:** Modify. **Produces:** `--record-disposition --feature F --disposition retry_fix|halt_feature| +halt_epic|blocked_external --reason ... [--families-agreeing ...]` stores `disposition` obj atomically; +**hard-rejects `--blocker-confirmed true`** (v2.10, Sol-R4#2) — `confirmed` always false. +- [ ] Failing selftest: round-trips + drives `--next --autonomous` routing; `confirmed:true` rejected. + Impl→pass→commit. + +### Task A5: blocker ledger lifecycle +**Files:** Modify (`--update --status blocked`, `--summary`, `--stats`). **Produces:** ledger append +idempotent by `(feature, attempt)`; entries carry `active`/`resolved_at`; `--update --status pending` +resolves the active entry (so a since-succeeded blocker never appears in the report). +- [ ] Failing selftest: blocked→pending→done clears the ledger entry; replay of the same blocked update + is idempotent (no dup); `--stats` breaks out `blocked`. Impl→pass→commit. + +### Task A6: final-review gate +**Files:** Modify. **Produces:** top-level `final_review{status:pending|passed|failed}`; +`--record-final-review --status ...` atomic; A2's terminal resolution already reads it. +- [ ] Failing selftest: all-features-done but `final_review!=passed` → NOT `done` (stays + `running_with_failures`/pending-review); after `--record-final-review passed` → `done`. Impl→pass→commit. + +### Task A7: global breakers (`--breaker-check` read / `--trip-breaker` write / `--record-progress-cycle`) +**Files:** Modify. **Produces:** `--breaker-check --now T` read-only → `{tripped,which,detail}` on +`total_attempts>=max(6,3×features)` | `no_progress_cycles>=3` | wallclock`>=max_wall_clock_hours`; +`--trip-breaker --now T` atomically sets `blocked_needing_human`; `--record-progress-cycle --cycle-id C +--now T` idempotently compares prior vs current `done` count, increments/resets `no_progress_cycles`. +- [ ] Failing selftest: each breaker trips at its boundary not before; `--breaker-check` never mutates; + `--trip-breaker` sets terminal; `no_progress_cycles` resets on a new `done`; replayed `--cycle-id` is a + no-op; null/negative caps validated with a loud log. Impl→pass→commit. +- [ ] **Publish the contract:** run full `--selftest` under `LANG=C` and Python 3.9; write a short + `## CLI contract` block at the top of the script docstring listing every subcommand + JSON shape (B & C + read this). Commit. + +--- + +## Unit B — Arbiter panel (`scripts/compound-v-epic-arbiter.py`, NEW) — depends on A + +**Interfaces — Produces:** `compound-v-epic-arbiter.py classify --state S --feature F --evidence-file E +[--claude-ballot F] --now T` → prints `{disposition, confirmed:false, reason, ballots:[...]}` and writes +the audit JSON. + +### Task B1: capabilities discovery + family map +- [ ] Failing selftest: reads `~/.claude/compound-v-capabilities.json` (fixture) — Codex available iff + `available && exec_flags_verified`; absent/malformed → Codex unavailable → Claude-only path. Family map + (`gpt`→GPT, `gemini`→Gemini, `claude|opus|sonnet`→Claude, `grok`→Grok, else `unknown`) keyed on resolved + model NAME; `cursor` "auto"→unknown; one ballot per family. Impl→pass→commit. + +### Task B2: Codex read-only poll (through the supervisor) +- [ ] Failing selftest (fake supervisor): builds the exact invocation `compound-v-run-with-timeout.py + --timeout N --max-output-bytes M -- codex exec --sandbox read-only --model -c + model_reasoning_effort=high --json --output-last-message F "" ` + (B cannot launch Claude — C does); a malformed/stale/wrong-feature ballot is dropped safely. Evidence + redaction: conservative function drops token / `Authorization:` / private-key / URL-cred / multiline + secrets, **fails closed** (omit suspect evidence) if it can't complete, before egress; capped to + `--max-output-bytes`. Impl→pass→commit. + +### Task B4: aggregation truth table + audit write +- [ ] Failing selftest: the COMPLETE truth table for every pair of valid ballots + one-ballot + zero- + ballot (retry_fix/halt_feature/halt_epic/blocked_external); parse-fail dropped; empty/tied → + conservative `halt_feature`; `blocked_external` on the Codex+Claude panel → SUSPECTED (`confirmed:false`) + because <2 known external families; `retry_fix` past `--can-retry` cap → `halt_feature`; classify still + runs at the cap. Audit JSON validates BOTH `epic_id` + `feature` id, realpath-contained, atomic, + cap/rotate. `--selftest` under `LANG=C` + Py3.9. Commit. + +--- + +## Unit C — Driver + reviewer prose (4 files) — depends on A, B + +### Task C1: `commands/v-epic.md` + `skills/compound-v/epic-mode.md` — autonomous path +- [ ] Add the `marathon` branch: loop `--next --autonomous`; `--breaker-check` + `--record-progress-cycle` + before each feature; on FAILURE → build evidence file → dispatch a fresh adversarial Claude Task + (write its ballot file) → run arbiter `classify --claude-ballot` → `--record-disposition` → act under + the breaker (retry_fix re-runs; halt_feature abandons → DAG continues; blocked_external → ledger); + before terminal `done` run the final cross-feature re-verification → `--record-final-review`; on terminal + status emit the halt-page runbook (every field from spec Component 5). Default path unchanged; **split + any `&&` git chains touched.** Scope the fail-fast prose to `checkpoint` stance. Commit. + +### Task C2: `commands/v-init.md` — stance config +- [ ] `/v:init` Step 3 writes `epic.autonomy.stance` (default `checkpoint`; offer `marathon` with the + honest v2.10 boundary: in-session + human resume, NO auto-revive; the breaker caps). Commit. + +### Task C3: `agents/spec-reviewer.md` — anti-reward-hack gate +- [ ] Add to the QUALITY pass a deterministic "did this diff weaken its own tests/scorers to pass?" check + with concrete evidence (removed asserts, loosened thresholds, deleted test files, scorer edits) — a + fixture where a weakened scorer is REJECTED. Commit. + +--- + +## Unit D — Docs + CI + release — depends on C + +### Task D1: CI selftests under Py3.9 +- [ ] `.github/workflows/validate.yml`: run `compound-v-epic-state.py --selftest` and + `compound-v-epic-arbiter.py --selftest` under **Python 3.9** (add the version) + the normal version. + Commit. + +### Task D2: README + CHANGELOG + version bump +- [ ] README "Main features": Marathon Loop · Arbiter Panel · Blocker Ledger · Global Breakers, honest + v2.10 boundary inline ("auto-resurrection coming in v2.11"). CHANGELOG 2.10.0 entry. Bump + `.claude-plugin/{plugin,marketplace}.json` 2.9.0→2.10.0. Commit. + +## Plan-review corrections (Codex Sol xhigh) — BINDING (each is an acceptance criterion) + +Sol confirmed the partition is disjoint and A→B→C→D sound; these tighten the RUNTIME contract: + +**Unit A:** +- **A2 blocker precedence (CRIT):** `--next --autonomous` returns a runnable INDEPENDENT feature FIRST; + it emits `blocked_needing_human` ONLY when no runnable feature remains. A suspected blocker never + terminates while independent work is runnable. Test: blocked A, dependent B, independent C → C runs + before any terminal escalation. +- **A2/A6 final-review roll-up (CRIT):** change the `--update` epic roll-up (`epic-state.py:512-519`) so + top-level `done` is set ONLY by `--record-final-review passed` — last-feature completion yields a + `pending_final_review` reason, NOT `done`. Define `final_review==failed` behavior (stays non-terminal; + human re-runs). Test last-feature completion, crash/re-entry before review, failed review, passed review. +- **A3 transition gating (HIGH):** the done/blocked→running rejection applies ONLY when the persisted + `autonomy.stance=="marathon"`; the legacy/checkpoint `--update` path is byte-unchanged (it accepts + arbitrary status replacement today). Add checkpoint golden tests for the previously-accepted update + transitions (dir contents + `git status`), not just `--init`. +- **A3/A7 attempt accounting (HIGH):** invariant `total_attempts == sum(feature.attempts)`, recomputed on + every mutation/check; REMOVE any caller-controlled `--attempt` input; freeze omitted-vs-explicit-null + cap semantics (a loud log on explicit null=unbounded); either enforce `max_features` (cap features + started per invocation) or drop it from the schema — do not store an unenforced knob. + +**Unit B:** +- **evidence-file containment (CRIT):** `--evidence-file` must be a non-symlink regular file, realpath- + contained under the validated epic/attempt dir; reject absolute/`..`/escaping-symlink BEFORE reading + (reuse the `check_specs` idiom, `epic-state.py:125-156`). Test `/etc/passwd`, `../`, escaping symlink, + valid contained file. +- **two-phase ballot API (HIGH):** replace circular `classify [--claude-ballot]` with `prepare --state + --feature --attempt --now` → emits a bounded challenge+prompt bound to `{epic_id,feature,attempt, + challenge_id}`; then `classify --challenge --claude-ballot --evidence-file ...`. A ballot + whose 4-tuple doesn't match current state, or a replayed/consumed challenge, is DROPPED. Test replay + across attempts + after crash/re-entry. +- **bounded output (HIGH):** the Codex invocation passes explicit supervisor `--stdout --stderr ` + with `--max-output-bytes` (streams without file paths are inherited & UNCAPPED per + `run-with-timeout.py:100-109`); independently size-check the `--output-last-message` file before parse. + Oversized-fake-Codex test proves every retained artifact stays under its cap. +- **frozen result+ledger schema (MED):** freeze the arbiter result (incl. redacted `evidence`/missing + fact, resolved family, attempt, audit path) and the ledger entry (families, derived dependents) so C's + halt-page fields all come from persisted data. CLI round-trip + golden halt-page test. + +**Unit C:** +- **stance binding (HIGH):** the driver's autonomous branch passes `--init --stance marathon` + validated + caps, and REJECTS autonomous operation when persisted `state.autonomy.stance != marathon` (config alone + is insufficient — the persisted state stance is authoritative). Define the switch-config-on-existing- + checkpoint-epic migration/re-init policy. Test it. +- **running→failed before arbitration (HIGH):** on a FAILURE the driver first `--update --status failed + --last-error ...` (mirroring `v-epic.md:50-54`) BEFORE progress/breaker/arbitration, so a retry starts + legally from `failed`, not `running`. +- **breaker cadence (HIGH):** enumerate EVERY breaker boundary — call `--record-progress-cycle` (with a + stable cycle-id reused across the checks in one loop pass so it doesn't over-increment) then + `--breaker-check` before each feature AND re-check after each attempt and before every arbiter / + sample-audit / retry / final-review model call. Ship the honest single-phase-overrun wording. +- **crash-resume commit points (HIGH):** name the exact two-command `git add`/`git commit` (check each + exit) after each completed attempt/disposition and before every terminal handoff — epic-state, arbiter + audit, ballot artifacts, ledger transitions. Crash-sim test: restart from committed state without + duplicating an attempt or losing an audit/ledger entry. + +## Self-review (run before dispatch) +- Spec coverage: every spec Component + all 11 R4 constraints map to a task above (final-review→A6/A2; + no-caller-confirm→A4; truth table→B4; breaker freq→C1+A7; progress-cycle→A7; --init stance→A1; ballot + interface→B3/C1; ledger lifecycle→A5; anti-reward-hack→C3; feature-id audit→B4; redaction→B3). ✓ +- No placeholders: each task names exact files, signatures, and the acceptance test. ✓ +- Type consistency: disposition enum, terminal-status names, and the arbiter JSON shape match across A/B/C. diff --git a/docs/superpowers/specs/2026-07-12-epic-autonomous-mode-design.md b/docs/superpowers/specs/2026-07-12-epic-autonomous-mode-design.md new file mode 100644 index 0000000..de831e0 --- /dev/null +++ b/docs/superpowers/specs/2026-07-12-epic-autonomous-mode-design.md @@ -0,0 +1,350 @@ +# Epic Autonomous Mode — "Marathon Loop" Design Spec (v2.10 scope, converged v3) + +> **Status:** design, pre-implementation. Branch `epic-autonomous-mode`. **PHASED** after 3 Codex Sol +> xhigh reject rounds converged on a scope decision (below). This spec is the **v2.10** slice; the +> deferred **v2.11** (auto-resurrection) is sketched at the end. +> **For agentic workers:** feeds `superpowers:writing-plans`. README marketing copy is written only for +> parts that exist and pass review (Component 6, last). + +## Scope decision — why PHASED (read first) + +Three independent Sol rounds converged: **every *critical* concurrency finding traced to the +auto-resurrection two-tier watcher.** Making it correct requires threading a lease *generation* through +the dispatcher / worker / merge-back / commit layer (`agents/parallel-dispatcher.md` Step 7, +`skills/backend-launcher/SKILL.md`) — i.e. re-architecting the plugin's most hardened execution code +(v2.6.4 commit discipline, the scope gate, worktree lifecycle). That is its own project, not an +overnight build. + +**Remove the auto-watcher and the marathon is single-process — the entire class of concurrency +criticals disappears** (the lease/flock/`--claim-stale`/generation-fencing existed *only* to make +auto-resurrection safe). So: + +- **v2.10 (this spec):** Marathon Loop + Global Breakers + cross-model Arbiter Panel (Codex + Claude) + + Blocker Ledger + driver + docs. Autonomous **within one `/v:epic` invocation**; resume after a fall is + the existing `/v:resume` + a human re-invoking `/v:epic` (re-entrant by design — safe, because nothing + auto-resurrects concurrently). This is Oleg's manual overnight pattern productized, minus the scheduler. +- **v2.11 (deferred, its own spec):** the Execution Lease + Two-Tier Watcher + generation-fenced + execution layer — the auto-"survives a fall while you sleep" part. + +**Goal (v2.10):** make `/v:epic` optionally chew through its whole runnable feature DAG in one +invocation without stopping at every feature, decide what to do about a failed feature via a +family-diverse (Codex + Claude) arbiter panel, isolate — not halt on — suspected external blockers, and +stay bounded by hard global circuit breakers — all **opt-in**, so the default epic is unchanged. + +**Tech Stack:** Python 3.9-safe stdlib (extends `scripts/compound-v-epic-state.py`); new +`scripts/compound-v-epic-arbiter.py`; prose (`skills/compound-v/epic-mode.md`, `commands/v-epic.md`, +`commands/v-init.md`); `README.md`, `CHANGELOG.md`, `.claude-plugin/*.json`, `.github/workflows/ +validate.yml`. Reuses: `compound-v-run-with-timeout.py` (external calls, `` injectable for deterministic selftests; CLI default is the real clock. +- **Substrate housekeeping:** arbiter audit JSONs are capped/rotated (owned by the arbiter script) so an + all-night run can't fill the disk (v2.6.4's flip side). +- Commit every new terminal-point artifact (state, arbiter audit, blocker ledger) at the existing v2.6.4 + commit points. All new scripts `LANG=C`-clean. + +## "Survives a fall" — the honest v2.10 boundary (ships in docs) + +- **In-session (Level-1 lite):** the marathon loop continues past a soft per-feature error to the next + runnable feature; a crashed feature is caught by the existing `running`→reconcile path on the next + `/v:epic` re-entry. Automatic within the live session. +- **Hard death** (quota / servers / app closed): a **human re-invokes `/v:epic `**, which is + re-entrant and resumes from `epic-state.json` via the existing `/v:resume`. There is **no automatic + resurrection in v2.10** — that is v2.11. We do NOT claim the epic self-revives while you sleep. +- No fabricated metrics; no overclaim. + +## Component 1 — Marathon Loop + relaxed routing (epic-state.py, opt-in) — the "core-contract" unit + +- **Schema (marathon stance only):** top-level `"autonomy":{"stance":"marathon","max_features":,"max_attempts_per_feature":2,"max_no_progress_cycles":3,"max_total_attempts":, + "max_wall_clock_hours":10,"started_at":}`; per feature `"attempts":0,"last_error":, + "disposition":` (the arbiter's stored verdict — see below). Absent `autonomy` ⇒ every legacy/ + checkpoint path, untouched. All new fields read via `.get(...,default)`. +- **`--next --autonomous` is a SEPARATE function** (`next_feature_autonomous`) — the default + `next_feature`, its guard order, and all default selftests are byte-for-byte untouched. It is + **read-only** (like today's `--next`), and routes on **deterministic DAG reachability**, not on a bare + `failed` status: a feature that is `failed`-and-abandoned or `blocked` removes only its **transitive + dependents** from the runnable set (reverse reachability over `depends_on`, reusing the `_detect_cycle` + adjacency idiom); independent pending features stay runnable. It derives `blocked_by:[ids]` read-only + (Sol-R3#5 — no persisted cascade to reverse later; reopening a source feature simply re-derives). Adds + a THIRD key `"blocked_by":[ids]`; the default `--next` 2-key shape is unchanged. +- **Terminal states (Sol-R2#5, Sol-R3, Sol-R4#1):** `done` (all features done **AND** the persisted + `final_review.status=="passed"` — never `done` on feature-completion alone, so the final cross-feature + re-verification actually gates success); `done_with_blockers` (v2.11-only — needs CONFIRMED blockers, + which are unreachable in v2.10, so v2.10 never emits this); `blocked_needing_human` (a `halt_epic`, a + tripped breaker, OR any SUSPECTED blocker — the v2.10 blocker terminal); `running_with_failures` + (non-terminal, work still runnable). Top-level `"final_review":{"status":"pending|passed|failed"}`; + **`--record-final-review --status ...`** writes it atomically; autonomous terminal resolution returns + `done` only when all-done AND final_review passed. `--next --autonomous` reports which state. +- **Abandonment is deterministic, not an arbiter "skip" vote (Sol-R2#8, Sol-R3):** once a feature is + abandoned (retry cap hit, or arbiter `halt_feature`), the DAG — not a model vote — decides that + independent features keep running. This removes the unreachable-`write_allowed` structural check + (per-feature scope maps don't exist at arbitration time) and the redundant `skip_independent` + disposition. +- **Attempts (crash-safe, single-process):** `--update --status running` increments `attempts` (via + `f.get("attempts",0)`); a documented transition table defines legal →`running` sources + (pending/failed→running; done/blocked no). No cross-process attempt-token/generation is needed in + v2.10 (single process; the existing `running`→reconcile handles a re-entered crash). + `--can-retry --feature F` → `{can_retry,attempts,cap}`. +- **`--record-disposition --feature F --disposition retry_fix|halt_feature|halt_epic|blocked_external + --confirmed t|f --families-agreeing gpt,... --reason ...`** stores the arbiter verdict on the feature + (the contract `--next --autonomous` reads). Atomic write. +- **Acceptance:** default selftests unchanged & green; whole-dir golden compatibility; new selftests: + autonomous routing continues past an abandoned independent feature but blocks dependents; `blocked_by` + is correctly derived read-only and re-derives after a source reopens; terminal-state resolution + (`done`/`done_with_blockers`/`blocked_needing_human`) for a graph where blockers exhaust reachable work; + `--record-disposition` round-trips and drives `--next --autonomous`; `--can-retry` flips at the cap. + +## Component 2 — Cross-Model Arbiter Panel (compound-v-epic-arbiter.py) — Codex + Claude + +Decide the disposition of a quality-gate FAILURE. **Family-diverse, degrade-safe.** + +- **Panel membership (Sol ×3):** **Codex** (real `--sandbox read-only` kernel boundary) + **in-harness + Claude** (a fresh adversarial Opus agent). **Antigravity/Cursor are EXCLUDED from arbitration** — they + lack kernel write-confinement and a disposable worktree cannot prevent out-of-worktree writes/egress of + the evidence; they remain implementation *workers*, not advisors. Documented as reduced family count. +- **Discovery — capabilities file ONLY** (no availability in `.claude/compound-v.json` since v2.6.2): + `codex.available && codex.exec_flags_verified`. Absent/malformed ⇒ Codex unavailable ⇒ Claude-only + fallback. +- **Family (DE-R4):** the arbiter ships a `model_name → family` map (`gpt`→GPT, `gemini`→Gemini, + `claude|opus|sonnet`→Claude, `grok`→Grok, else `unknown`), keyed on the resolved model name from + `resolve-model.py`. **One ballot per family** (Sol-R3-med): same-family votes collapse to one ballot; + within-family disagreement → that family's ballot is the more conservative of its votes. Codex=GPT, + Claude=Claude ⇒ at most **two** families on this panel, and **Claude-self never counts as an + independent confirming family** (same family as the implementer → correlated blind spots). +- **Poll:** Codex runs the read-only advisory query through the timeout supervisor (`","evidence":""}`. +- **Classify always; retry-cap only masks `retry_fix` (Sol-R2#6):** the panel is ALWAYS consulted on a + failure to classify it (so an exhausted feature can still be recorded `blocked_external`); only the + `retry_fix` *action* is gated by `--can-retry` (masked to `halt_feature` at the per-feature cap). + Arbitration is suppressed only if a GLOBAL breaker forbids further model calls. +- **Aggregate (Codex + Claude), parse-safe:** a parse failure / errored backend is **DROPPED + LOGGED** + (never a fabricated `halt` vote — Sol-R2 / DE-Q2). Of the valid ballots: `retry_fix` vs + `halt_feature`/`halt_epic` by majority; **empty or tied → conservative `halt_feature`** (abandon this + feature safely; the DAG then continues independents — no whole-epic halt unless nothing is reachable). +- **`blocked_external` confirmation (the ≥2-family bar):** CONFIRMED only if **≥2 distinct KNOWN + families** agree with no `retry_fix` dissent. On the Codex+Claude panel that bar is **not reachable** + (only Codex is a distinct external family; Claude-self can't confirm) — so in v2.10 a `blocked_external` + is always **SUSPECTED**: the feature is isolated (status `blocked`, epic keeps building the rest) but + flagged UNCONFIRMED, and its presence resolves the epic to `blocked_needing_human` for a human to + verify. Honest limitation: confirmed auto-blocking needs a 2nd safe external family (a v-future backend). +- **Claude-only fallback** (Codex absent): a single fresh adversarial Opus agent, capped to `retry_fix` + or `halt_feature` — never `halt_epic`-by-itself-silently, never CONFIRM a blocker. Honesty line ships: + *without a cross-model backend the arbiter adds ≈no independent signal over checkpoint.* +- **Audit trail:** record every ballot + resolved family under `docs/superpowers/execution/epics// + arbiter/-.json` — validate `epic_id` (add `_id_ok` to `build_state`), derive the audit root + from the contained state path with realpath containment, write atomically, cap/rotate to bound growth. +- **Acceptance:** fixture capabilities + fake supervisor: Codex+Claude ballots aggregate with one-per- + family collapse; a garbled Codex reply is DROPPED (not a halt vote) + logged; empty/tied → halt_feature; + a `blocked_external` on this panel is SUSPECTED → `blocked_needing_human`; `retry_fix` past cap → + halt_feature; classify still runs at the cap; Codex-absent → Claude-only capped path; audit path rejects + a traversal `epic_id`. `--selftest`, `LANG=C`-clean. + +## Component 3 — Blocker Ledger + end-of-epic human report (epic-state.py; the "do everything you can" credo) + +Finish everything in your power; isolate only the genuinely impossible; escalate with proof; never halt +the rest. + +- **State:** `blocker_ledger:[{feature,confirmed:bool,reason,evidence,families_agreeing:[...], + first_seen_at,blocks:[derived dependent ids]}]`; the feature gets status `blocked`. +- **`--update --status blocked --feature F --blocker-reason ... --blocker-confirmed t|f + --families-agreeing ...`** appends the entry. `--next --autonomous` treats `blocked` as a benign skip + (only transitive dependents drop out, read-only-derived); it never trips a whole-epic halt. +- **v2.10 blockers are ALWAYS SUSPECTED — no caller-controlled confirmation (Sol-R4#2):** the ≥2-known- + family CONFIRMED tier is structurally unreachable on the Codex+Claude panel, so v2.10 **hard-rejects + `--blocker-confirmed true`** (and any caller-supplied `confirmed:true`); `--update --status blocked` + always records `confirmed:false`. Arbiter ballots are stored separately (audit trail) — confirmation is + DERIVED from ≥2 stored known-family ballots, never asserted by a caller. A suspected blocker is still + isolated (epic keeps building the rest) but resolves the terminal status to **`blocked_needing_human`** + so a person verifies it. CONFIRMED blockers + `done_with_blockers` arrive only with a 2nd safe external + family (v-future). This makes the bar real, not cosmetic, and closes the false-success hole. +- **Two-point discovery:** a blocker can surface in **pre-flight research** (1B/1C) or mid- + **implementation** (a worker hits it, like astrology #150). Both funnel through the arbiter's + confirmation logic before `confirmed:true`. +- **End-of-epic report** leads with the ledger, visually separated from `failed`: "Built everything + reachable. **N feature(s) need YOU** — a human/external action, because code can't create data that + doesn't exist upstream:" per entry: feature, reason, the missing external fact, families that agreed + (or "single-model SUSPICION"), what it transitively blocked. Resolving a blocker = human re-run of + `/v:epic ` after `--update --status pending --feature F` (dependents re-derive read-only). +- **Acceptance:** a `blocked` feature isolates dependents not independents; `--next --autonomous` advances + past it; single-family → `confirmed:false` → `blocked_needing_human`; ≥2 families → `confirmed:true` → + `done_with_blockers`; ledger round-trips through `--update`/`--summary`; `--stats` breaks out `blocked`; + report names the missing fact + agreeing families, never fabricates. + +## Component 4 — Global Circuit Breakers (epic-state.py — enforced in the SCRIPT, DE-R1) + +Local caps never bound the system; the breaker lives in the deterministic script. + +- **Counters (marathon state):** `total_attempts` (sum of feature attempts), `no_progress_cycles` + (incremented when a full `--next --autonomous` pass advances the `done` count by zero; **reset to 0** on + any feature reaching `done`), wall-clock from `autonomy.started_at` vs `--now`. +- **Read/trip split (Sol-R3#6):** `--breaker-check --state S --now T` is **read-only** → + `{tripped,which:[...],detail}`. A separate **`--trip-breaker --state S --now T`** atomically writes the + epic to `blocked_needing_human` when a check trips. The driver calls check before each feature and trips + on a positive. Trips on: `total_attempts>=max_total_attempts` (default `max(6,3×features)`), + `no_progress_cycles>=max_no_progress_cycles` (3), wall-clock`>=max_wall_clock_hours` (10). **Counts and + hours only — never a fabricated cost.** +- **Acceptance:** each breaker trips at its boundary and not before; `no_progress_cycles` resets on a + `done`; `--breaker-check` never mutates; `--trip-breaker` atomically sets `blocked_needing_human`; + boundary/`null`/negative config validated (a `null` cap = unbounded on that axis only if explicitly + set, with a loud log). + +## Component 5 — Driver wiring + halt-page runbook + PASS integrity (v-epic.md, epic-mode.md, v-init.md) + +- **Autonomous path** (when `.claude/compound-v.json` `epic.autonomous.stance=="marathon"` or + `--autonomous`): loop with `--next --autonomous`, `--breaker-check` before each feature; on a feature + FAILURE consult the Arbiter Panel, `--record-disposition`, act under the breaker (retry_fix re-runs; + halt_feature abandons → DAG continues independents; blocked_external → ledger); on terminal status + report. Default (checkpoint) path unchanged; split any `&&` git chains touched (Sol-R2#9); acceptance = + "behaviorally unchanged", not "textually additive". +- **PASS integrity (DE-R5):** the Review Gate gains an **anti-reward-hacking** check — did the diff weaken + its own tests/scorers to pass? Marathon **sample-audits** a fraction of PASSes and runs a **final + cross-feature re-verification** before `done` (guards silent regression of earlier features). +- **Halt-page = runbook (DE-Q4):** page ONLY on a whole-epic block (`blocked_needing_human` or tripped + breaker); `blocked`/abandoned-feature notices batch into the end-of-epic report, not mid-run pages. The + page contains: the feature + blocked dependents; which acceptance criterion failed + gate verdicts + + failing-diff summary; every panel ballot + reason + resolved family + why it aggregated; breaker state + (n/cap); a copy-paste `/v:resume ` + the override path; paths to arbiter JSON / run-dir / diff. + Counts only. +- **`/v:init`** Step 3 writes `epic.autonomous.stance` (default `checkpoint`; offer `marathon` with the + honest v2.10 boundary — in-session + human resume, no auto-revive — and the breaker caps). +- **Acceptance:** spec-reviewer confirms the default path is behaviorally unchanged; honesty boundary + present; halt-page contains every runbook field; the fail-fast prose in all three docs is *scoped to + checkpoint stance*, not deleted; no fabricated metrics. + +## Component 6 — README + docs + CI (LAST — only after 1–5 are real & reviewed) + +- README "Main features": a **Marathon Loop · Arbiter Panel · Blocker Ledger · Global Breakers** + subsection with the honest v2.10 boundary inline (in-session autonomy + human resume; auto-resurrection + is "coming in v2.11"). +- **CI (Sol#10):** add the two new `--selftest`s (epic-state, arbiter) to `.github/workflows/validate.yml`, + run under **Python 3.9** (the floor) plus the normal version; add malformed-state / disposition-round- + trip fixtures. +- `CHANGELOG.md` + `.claude-plugin/{plugin,marketplace}.json` bump 2.9.0 → 2.10.0. +- **Acceptance:** every README claim maps to a shipped, tested component; the honesty caveats present; CI + runs the new selftests under Py3.9 and is green; version bumped. + +## Feature DAG + disjoint partition (v2.10) + +``` +A CORE CONTRACT (scripts/compound-v-epic-state.py — Components 1,3,4: autonomy, --init --stance, deps: [] + --next --autonomous, attempts, --record-disposition, --record-final-review, + --record-progress-cycle, --trip-breaker, blocker_ledger lifecycle, terminal states, + Py3.9 ISO. ONE serial unit = the WHOLE file. Publishes its state/CLI contract 1st.) +B arbiter-panel (scripts/compound-v-epic-arbiter.py — new; Codex subprocess + Claude BALLOT-FILE input, deps: [A] + family map, truth table, redaction, cap/rotate audit, feature+epic id validation) +C driver+review (commands/v-epic.md, skills/compound-v/epic-mode.md, commands/v-init.md, deps: [A,B] + agents/spec-reviewer.md — driver loop, Claude-ballot dispatch, anti-reward-hack gate) +D docs + CI + rel(README.md, CHANGELOG.md, .claude-plugin/{plugin,marketplace}.json, validate.yml) deps: [C] +``` + +Disjoint writes — the **entire `compound-v-epic-state.py` is ONE serial unit A** (Components 1, 3, 4 — +not split by number, which would collide). **A publishes its state-JSON + CLI contract before B/C.** B is +a new file; `resolve-model.py` is NOT edited (the family map lives in B). **Ballot interface (Sol-R4#7):** +the arbiter script (B) polls Codex as a subprocess but CANNOT launch an in-harness Claude — so B *emits a +bounded Claude prompt* and *accepts a ballot file* (`--claude-ballot `) that the DRIVER (C) +produces by dispatching a fresh adversarial Claude Task; B validates + aggregates it with the Codex +ballot. **C also owns `agents/spec-reviewer.md`** (the anti-reward-hacking Review-Gate check lives there, +not in driver prose — Sol-R4#9). C = 4 files. D = README/CHANGELOG/2 manifests/workflow. No two units +write the same file. Plugin manifests are CLEAN. No `agents/parallel-dispatcher.md` / `backend-launcher` +edits — those were only needed for v2.11 fencing. + +## Chosen defaults (documented, tunable in `/v:init`) + +- Global-breaker defaults: `max_wall_clock_hours=10`, `max_total_attempts=max(6,3×features)`, + `max_no_progress_cycles=3` — tunable. +- Arbiter panel = Codex + Claude; agy/cursor excluded from arbitration (workers only). +- `blocked_external` in v2.10 is always SUSPECTED (→ human verify) — confirmed auto-blocking awaits a 2nd + safe external family. +- Single-backend (no Codex): marathon still offered with the "≈no independent signal" honesty notice. + +## v2.11 — Auto-Resurrection (DEFERRED, its own spec + review pass) + +The "survives while you sleep" layer, split out because it needs correct distributed concurrency: +- **Execution Lease:** `fcntl.flock`-guarded (reuse `compound-v-memory.py:542`) single atomic + `--claim-stale` (check-terminality-and-expiry-and-acquire in one locked txn — never acquire-then- + recheck), owner+`generation` fencing, renewable heartbeat. +- **Two-Tier Watcher:** session `CronCreate` (Level-1; 7-day expiry, dies on sleep/`CLAUDE_CODE_DISABLE_ + CRON`) + on-disk `scheduled-tasks` (Level-2; next-launch catch-up). Persist watcher records (provider, + task id, generation, timestamps) for idempotent arm/disarm. +- **Generation-fenced execution layer (the hard part, Sol-R3#1/#2):** thread owner+generation through the + dispatcher / worker / merge-back / commit; an executable heartbeat helper renews during long phases; + re-validate the lease before git-apply, before each commit, before terminal publish; reject stale + completions; force all marathon jobs into worktrees. Honest guarantee: "a superseded driver's work is + rejected at the fenced merge boundary", never "two drivers never run simultaneously". +- **Honesty boundary** (Level-1/Level-2/headless, sleep/quota/auth/7-day/disable-cron) from the earlier + draft applies here. + +## Design constraints from Sol R4 — BIND THE PLAN (each becomes a task with tests) + +The plan MUST encode all of these with concrete CLI signatures + selftests (they are implementation- +precision, resolved in code, not more spec prose): + +1. **Final-review gate (CRIT):** persisted `final_review.status`; `--record-final-review`; `done` iff + all-features-done AND `final_review==passed`. Crash/re-entry test around the boundary. +2. **No caller-confirmed blockers (CRIT):** hard-reject `--blocker-confirmed true` in v2.10; confirmation + is DERIVED from ≥2 stored known-family ballots, never caller-asserted. Always `confirmed:false` in v2.10. +3. **Arbiter truth table (HIGH):** a complete deterministic table for EVERY pair of valid ballots + (retry_fix/halt_feature/halt_epic/blocked_external × same), one-valid-ballot, and zero-ballot — the + exact resulting disposition + confirmed flag. Within `halt_*`, `halt_epic` beats `halt_feature` only on + explicit agreement; a lone halt_epic vs retry_fix → conservative halt_feature. Test each cell. +4. **Breaker frequency + honest wall-clock (HIGH):** re-check breakers immediately after each attempt and + before every arbiter / sample-audit / retry / final-review model call — not only before a feature. + Either pass an absolute deadline into long phases OR **downgrade the promise**: "wall-clock is checked + at each attempt/model-call boundary; a single in-flight pipeline phase may overrun." Ship the honest + wording; do not claim a hard real-time kill. +5. **`--record-progress-cycle` (HIGH):** one atomic, idempotent progress-boundary command (keyed by a + cycle id) that compares prior vs current `done` count, increments or resets `no_progress_cycles`, + called by the driver before each breaker check. Test the actual driver sequence, not preconstructed + states. +6. **`--init --stance marathon` (HIGH):** explicit stance + cap arguments (or one validated config input) + that writes every marathon field exactly once; checkpoint `--init` stays byte-compatible. Test both. +7. **Ballot exchange B↔C (HIGH):** B emits a bounded Claude prompt + accepts `--claude-ballot `; + C dispatches the fresh Claude Task and feeds the file back. Malformed / stale / wrong-feature ballot + tests; B fails safe (drops that ballot) on any of them. +8. **Blocker-ledger lifecycle (HIGH):** ledger writes idempotent by `(feature, attempt)`; entries carry + `active`/`resolved_at`; reopening a feature (`--status pending`) resolves its active entry so the end + report never lists a since-succeeded blocker. Test blocked→pending→done + replay of the same update. +9. **Anti-reward-hack in `agents/spec-reviewer.md` (HIGH):** the "did the diff weaken its own tests/ + scorers?" check lives in the reviewer agent's contract (C owns the file), with deterministic evidence + + a fixture where a weakened scorer is rejected. +10. **Feature-id audit-path validation (MED):** validate BOTH `epic_id` and `feature` id at every audit + write boundary (legacy/handmade states can carry a traversal-style feature value); traversal + symlink + containment fixtures for each filename component. +11. **Secret redaction (MED):** a conservative redaction function + tests for token / authorization-header + / private-key / URL-credential / multiline-secret forms; **fail closed** (omit the suspect evidence) + when redaction can't complete, before any external-model egress. + +## Review provenance + +Converged from four independent pre-implementation reviews + three Codex Sol xhigh adversarial rounds on +2026-07-12. The Sol rounds drove the scope decision: R1/R2/R3 each surfaced fresh *critical* concurrency +defects in the auto-watcher, converging on "phase it". Audit docs under `docs/superpowers/expert/` and +`docs/superpowers/research/`; the raw Sol findings in the run history. diff --git a/scripts/compound-v-epic-arbiter.py b/scripts/compound-v-epic-arbiter.py new file mode 100644 index 0000000..55da167 --- /dev/null +++ b/scripts/compound-v-epic-arbiter.py @@ -0,0 +1,2721 @@ +#!/usr/bin/env python3 +""" +Compound V epic arbiter — cross-model (Codex + Claude) panel that classifies a marathon +feature FAILURE (v2.10 "marathon" epic mode, design spec Component 2). NEW file, Unit B. + +Depends on Unit A's frozen contract (scripts/compound-v-epic-state.py) but does NOT import +it: every idiom this script needs from A (id validation, realpath containment, atomic +writes, ISO-8601 handling) is deliberately DUPLICATED here so this script stays a +standalone, dependency-free CLI. It DOES optionally load compound-v-resolve-model.py by path +(degrade-safe) to key the family map on the same resolved model name the rest of the plugin +routes on. + +This file is a SECURITY BOUNDARY: it egresses feature evidence to an external model (Codex), +so it (a) TOCTOU-safely contains and reads every untrusted path via O_NOFOLLOW file +descriptors — never validate-a-name-then-reopen-it — (b) conservatively redacts secrets +before ANY egress AND re-redacts model output before it re-enters our audit trail, failing +CLOSED (omit) on any doubt, (c) treats an immutable PERSISTED issuance record (not a +recomputable hash) as the anti-replay authorization boundary, with a durable +issued -> in_progress -> consumed transition, and (d) writes every artifact under a +trusted, symlink-free, count+byte-bounded audit directory. + +## CLI contract (two-phase) + + --prepare --state S --feature F --attempt N [--now T] [--challenge-key K] + Requires N == the feature's CURRENT persisted `attempts` (monotonic — you cannot + pre-issue a future attempt or re-issue a consumed one). Emits a bounded Claude + ballot-task prompt + a keyed `challenge_id`, and persists an immutable issuance record + (status "issued") under /arbiter/-.challenge.json. Prints + JSON {challenge_id, epic_id, feature, attempt, issued_at, prompt}. + + --classify --state S --feature F --challenge [--evidence-file REL] + [--claude-ballot BALLOTFILE] [--now T] [--challenge-key K] [tuning flags] + Validates the persisted challenge FIRST. If it is missing/stale/consumed the WHOLE + panel is dropped BEFORE any model call (no Codex egress). Otherwise it durably + transitions the record issued -> in_progress, polls Codex (if the capabilities file + says it's usable), validates the Claude ballot's {epic_id,feature,attempt,challenge_id} + 4-tuple, aggregates with the complete truth table, writes the frozen result to the + audit JSON, and durably transitions the record -> consumed (a replay then re-emits the + persisted audit idempotently, no second egress). `--evidence-file` is RELATIVE to the + attempt dir /arbiter/-/ (absolute/`..`/symlink rejected). + Prints {disposition, confirmed:false, reason, evidence, ballots, families_present, + families_agreeing, attempt, audit_path, ...}. + + --resume-challenge --state S --feature F --attempt N [--now T] + A READ-ONLY, idempotent lookup of the persisted challenge/result for + (epic_id, feature, attempt). Fixes the --prepare resume DEADLOCK: if arbitration crashes + after --classify began, --prepare refuses to re-issue an in_progress/consumed challenge, + so a driver that always restarts a crashed arbitration from --prepare has no way back in. + This command lets the driver recover WITHOUT a second model egress and without that + deadlock. Never issues a challenge, never calls Codex/Claude. Prints exactly one of: + {"state": "absent"} -- nothing issued; --prepare fresh + {"state": "in_progress", "challenge_id", "prompt"} -- re-dispatch on `prompt`, then + --classify (safe/idempotent — + the challenge binding makes it so) + {"state": "consumed", "challenge_id", "result"} -- the frozen persisted result, + RE-EMITTED verbatim, no new call + A record left `issued`/`in_progress` is FIRST checked against the canonical persisted + audit for the same tuple before being reported: if --classify persisted the frozen audit + but crashed before durably marking the record `consumed` (the window between those two + writes), the verdict already exists on disk, so this reports `consumed` with that result + and makes ZERO model calls -- never re-reports `in_progress`, which would otherwise cause + the driver to re-dispatch Claude and re-`--classify`, duplicating model work for a + decision that was already made. Only when no valid audit exists yet does it report + `in_progress` as before. + A tampered/oversized/symlinked/inconsistent persisted record is refused with a controlled + ArbiterError (fail closed), never a fabricated resume state. + + --selftest + +Python 3.9-safe, stdlib only. No fabricated cost/token metrics. Requires a POSIX platform +with openat/renameat (dir_fd) support — verified at import; a platform without it fails +CLOSED rather than silently weakening the containment guarantees. +""" + +import argparse +import hashlib +import hmac +import json +import os +import re +import stat +import subprocess +import sys +import tempfile +from datetime import datetime, timezone + +# --------------------------------------------------------------------------------------- # +# Constants +# --------------------------------------------------------------------------------------- # + +ID_RE_OK = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._-" +DISPOSITIONS = ("retry_fix", "halt_feature", "halt_epic", "blocked_external") +CONSERVATISM_RANK = {"retry_fix": 0, "blocked_external": 1, "halt_feature": 2, "halt_epic": 3} +KNOWN_FAMILIES = ("GPT", "Gemini", "Claude", "Grok") +# Families that can count as an INDEPENDENT external confirming family: known, and not the +# arbiter's own Claude (same family as the implementer -> correlated blind spots). +_EXTERNAL_CONFIRMING = ("GPT", "Gemini", "Grok") + +DEFAULT_MAX_OUTPUT_BYTES = 20000 +DEFAULT_CODEX_TIMEOUT_SEC = 300 +MAX_OUTPUT_BYTES_CEILING = 5_000_000 # sanity ceiling for --max-output-bytes +MAX_TIMEOUT_SEC = 3600 # sanity ceiling for --codex-timeout +AUDIT_ROTATE_COUNT = 500 +AUDIT_ROTATE_BYTES = 5_000_000 +CHALLENGE_ROTATE_COUNT = 500 +AUDIT_MAX_ONE_BYTES = 200_000 +FIELD_MAX_CHARS = 2000 +REASON_MAX_CHARS = 4000 # hard cap on any retained reason string +CHALLENGE_RECORD_MAX_BYTES = 65536 +BALLOT_FILE_MAX_BYTES = 65536 +# Filesystem components are built from epic_id/feature (-.challenge.json), +# so bound id length well under the portable NAME_MAX (255) to turn an overlong id into a +# controlled error instead of an ENAMETOOLONG deep in an os.open (Finding 8/#11). +MAX_ID_LEN = 100 + +# ------------------------------------------------------------------------------------- # +# THREAT MODEL (v2.10): SINGLE-USER, SINGLE-PROCESS local tool. The driver calls the +# arbiter SERIALLY (never two --classify at once); the epic tree and any --claude-ballot +# file are USER-OWNED. Three classes of hardening are therefore DELIBERATELY OUT OF SCOPE +# here and deferred to v2.11 (multi-process autonomy), each noted at its site below: +# (v2.11) concurrent compare-and-swap / fencing on a challenge record, +# (v2.11) concurrent challenge-key creation races, +# (v2.11) fsync-level crash durability of the atomic writes. +# ------------------------------------------------------------------------------------- # + +CLAUDE_PROMPT_MAX_CHARS = 4000 +CODEX_PROMPT_MAX_CHARS = 6000 + +TRUNC_MARKER = "\n...[TRUNCATED]" +DEPRECATION_LINE = "[features].codex_hooks is deprecated" + +_FALLBACK_CODEX_MODEL = "gpt-5.6-sol" # mirrors compound-v-resolve-model.py's codex/deep default + +# openat/renameat are REQUIRED — fail closed if the platform lacks them (rather than +# silently degrading the symlink/TOCTOU containment). +# NOTE: use os.rename (not os.replace) for the dir_fd atomic swap — on POSIX rename() +# already replaces the destination atomically, and (unlike os.replace) os.rename exposes +# src_dir_fd/dst_dir_fd on macOS as well as Linux. +_DIRFD_OK = ( + hasattr(os, "O_NOFOLLOW") + and hasattr(os, "O_DIRECTORY") + and {os.open, os.rename, os.mkdir, os.unlink, os.stat}.issubset(os.supports_dir_fd) +) +_O_CLOEXEC = getattr(os, "O_CLOEXEC", 0) + + +class ArbiterError(Exception): + """A controlled, user-facing error — printed as one line, nonzero exit, never a + traceback and never a partial write.""" + + +# --------------------------------------------------------------------------------------- # +# Small utils +# --------------------------------------------------------------------------------------- # + +def _id_ok(s): + """A real, safe id: must be a str (so str(None)=='None' can NEVER slip through), be + within MAX_ID_LEN (it becomes a filesystem component — Finding 8/#11), contain only the + allow-listed characters, and not be '.'/'..'.""" + return (isinstance(s, str) and bool(s) and s not in (".", "..") + and len(s) <= MAX_ID_LEN and all(c in ID_RE_OK for c in s)) + + +def _now_iso(dt=None): + if dt is None: + dt = datetime.now(timezone.utc) + return dt.isoformat() + + +def _parse_iso(s): + if isinstance(s, str) and s.endswith("Z"): + s = s[:-1] + "+00:00" + dt = datetime.fromisoformat(s) + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return dt + + +def _resolve_now(now_arg): + if now_arg: + try: + return _parse_iso(now_arg) + except (ValueError, TypeError) as e: + raise ArbiterError("invalid --now timestamp %r: %s" % (now_arg, e)) + return datetime.now(timezone.utc) + + +def _log(msg): + print("epic-arbiter: %s" % msg, file=sys.stderr) + + +# --------------------------------------------------------------------------------------- # +# Finding 9 — strict state-schema validation (one boundary; controlled errors) +# --------------------------------------------------------------------------------------- # + +def validate_state_schema(state): + """Return a list of error strings (empty = valid). One strict boundary so no malformed + state reaches path-building or feature lookup.""" + if not isinstance(state, dict): + return ["state root is not a JSON object"] + errs = [] + if not _id_ok(state.get("epic_id")): + errs.append("epic_id is missing or not a valid id string") + feats = state.get("features") + if not isinstance(feats, list) or not feats: + errs.append("features is missing or not a non-empty list") + else: + for i, f in enumerate(feats): + if not isinstance(f, dict): + errs.append("feature %d is not an object" % i) + continue + if not _id_ok(f.get("id")): + errs.append("feature %d has a missing/invalid id" % i) + at = f.get("attempts", 0) + if isinstance(at, bool) or not isinstance(at, int) or at < 0: + errs.append("feature %r has an invalid attempts %r (need a non-negative int)" + % (f.get("id"), at)) + auto = state.get("autonomy") + if auto is not None: + if not isinstance(auto, dict): + errs.append("autonomy is present but not an object") + else: + st = auto.get("stance") + if st is not None and not isinstance(st, str): + errs.append("autonomy.stance is not a string") + cap = auto.get("max_attempts_per_feature") + if cap is not None and (isinstance(cap, bool) or not isinstance(cap, int)): + errs.append("autonomy.max_attempts_per_feature is not an int or null") + return errs + + +def _find_feature(state, feature_id): + for f in state.get("features", []): + if isinstance(f, dict) and f.get("id") == feature_id: + return f + return None + + +def _read_state(state_path): + if not state_path or not os.path.isfile(state_path): + raise ArbiterError("--state must point at an existing epic-state.json") + try: + with open(state_path, "r", errors="replace") as fh: + state = json.load(fh) + except (OSError, ValueError) as e: + raise ArbiterError("could not read/parse the state file: %s" % e) + errs = validate_state_schema(state) + if errs: + raise ArbiterError("invalid epic-state schema: %s" % "; ".join(errs)) + return state + + +# --------------------------------------------------------------------------------------- # +# Finding 5 — trusted-root epic-dir containment + Finding 5/6 — dir_fd-verified I/O +# --------------------------------------------------------------------------------------- # + +def validate_epic_dir(state_path, epic_id): + """The epic dir is the realpath of the state file's directory. It MUST sit at + `.../execution/epics/` — rooting under the trusted epics root AND binding the + directory's own name to the epic_id (a mismatch means a hand-made/relocated state trying + to write its audit trail somewhere it doesn't own). Returns the real epic dir path.""" + epic_dir = os.path.realpath(os.path.dirname(os.path.realpath(state_path))) + parts = epic_dir.split(os.sep) + if len(parts) < 4 or parts[-1] != epic_id or parts[-2] != "epics" or parts[-3] != "execution": + raise ArbiterError( + "epic dir %r is not under the trusted root .../execution/epics/ " + "(or its name does not match epic_id %r)" % (epic_dir, epic_id)) + return epic_dir + + +def _require_dirfd(): + if not _DIRFD_OK: + raise ArbiterError("this platform lacks openat/renameat (dir_fd) support — refusing " + "to run without the symlink/TOCTOU containment guarantees") + + +def open_arbiter_dir_fd(epic_dir_real, create=True): + """Return a dir fd for /arbiter, opened with O_NOFOLLOW so a SYMLINKED arbiter + dir (pointing outside the epic) is rejected outright. epic_dir_real is already a realpath + (symlink-free by construction), so opening its final component with O_NOFOLLOW is exact. + Caller must close the returned fd.""" + _require_dirfd() + epic_fd = os.open(epic_dir_real, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | _O_CLOEXEC) + try: + if create: + try: + os.mkdir("arbiter", 0o755, dir_fd=epic_fd) + except FileExistsError: + pass + arbiter_fd = os.open("arbiter", os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | _O_CLOEXEC, + dir_fd=epic_fd) + finally: + os.close(epic_fd) + return arbiter_fd + + +def _serialize(obj): + """The ONE canonical serialization used for BOTH the audit file and stdout, so the + printed bytes equal the persisted bytes exactly (round-3 #3).""" + return json.dumps(obj, indent=2) + "\n" + + +def _write_text_at(dir_fd, name, text): + """Atomic write of the pre-serialized `text` into `name` relative to dir_fd, via a temp + file + renameat. O_EXCL|O_NOFOLLOW on the temp defeats a pre-planted symlink; the rename + is dir_fd-relative on both ends so no path is re-resolved.""" + data = text.encode("utf-8") + tmp = ".%s.%d.tmp" % (name, os.getpid()) + # Best-effort clear a stale temp from a prior crash. + try: + os.unlink(tmp, dir_fd=dir_fd) + except OSError: + pass + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW | _O_CLOEXEC, + 0o600, dir_fd=dir_fd) + try: + os.write(fd, data) + # (v2.11) fsync-level crash durability is out of the single-user threat model — a + # power-loss torn-write window is not defended here (see the module THREAT MODEL). + finally: + os.close(fd) + try: + # POSIX rename() atomically replaces the destination (== os.replace semantics), + # and supports dir_fd on macOS where os.replace does not. + os.rename(tmp, name, src_dir_fd=dir_fd, dst_dir_fd=dir_fd) + except Exception: + try: + os.unlink(tmp, dir_fd=dir_fd) + except OSError: + pass + raise + + +def _write_json_at(dir_fd, name, obj): + """Atomic write of `obj` as canonical JSON into `name` relative to dir_fd.""" + _write_text_at(dir_fd, name, _serialize(obj)) + + +def _read_text_at(dir_fd, name, cap): + """Read the RAW text of `name` relative to dir_fd (O_NOFOLLOW + fstat regular + size + cap). Returns None if absent/symlinked/non-regular/oversized. Used for the crash-recovery + re-emit so the re-emitted bytes are byte-identical to what is on disk (round-3 #3).""" + try: + fd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | _O_CLOEXEC, dir_fd=dir_fd) + except OSError: + return None + try: + st = os.fstat(fd) + if not stat.S_ISREG(st.st_mode) or st.st_size > cap: + return None + raw = os.read(fd, cap + 1) + finally: + os.close(fd) + return raw.decode("utf-8", errors="replace") + + +def _read_json_at(dir_fd, name, cap): + """Read+parse JSON from `name` relative to dir_fd, via O_NOFOLLOW + fstat regular + + cap+1 read. Returns None if the file is absent, a symlink, non-regular, oversized, or + unparseable (callers treat that as 'no record').""" + try: + fd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | _O_CLOEXEC, dir_fd=dir_fd) + except OSError: + return None + try: + st = os.fstat(fd) + if not stat.S_ISREG(st.st_mode) or st.st_size > cap: + return None + raw = os.read(fd, cap + 1) + finally: + os.close(fd) + try: + return json.loads(raw.decode("utf-8", errors="replace")) + except ValueError: + return None + + +def _read_capped_regular(path, cap, delete_oversized=False): + """Open `path` with O_NOFOLLOW, assert a regular file, and read AT MOST cap bytes. + Returns (text|None, err|None). An oversized file is DROPPED unread; it is deleted ONLY + when `delete_oversized` is True — which callers set EXCLUSIVELY for the arbiter's OWN + private temp artifacts. A caller-supplied input (a user-owned --claude-ballot / evidence + file) is NEVER unlinked (Finding 4/#8): confining deletion to the private temp dir also + dissolves the validate-fd-then-unlink-pathname TOCTOU.""" + try: + fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | _O_CLOEXEC) + except OSError as e: + return None, "cannot open (symlink/missing?): %s" % e + delete = False + try: + st = os.fstat(fd) + if not stat.S_ISREG(st.st_mode): + return None, "not a regular file" + if st.st_size > cap: + delete = delete_oversized + return None, "oversized (%d > cap %d) — dropped unread" % (st.st_size, cap) + raw = os.read(fd, cap + 1) + finally: + os.close(fd) + if delete: + try: + os.unlink(path) # private-temp only; see delete_oversized docstring + except OSError: + pass + return raw.decode("utf-8", errors="replace"), None + + +def _probe_record(dir_fd, name, cap): + """Distinguish a truly-ABSENT record (ENOENT) from one that EXISTS but is + unreadable/oversized/symlinked/malformed (Finding 6/#3). Returns: + ("absent", None) -> no such file, safe to issue a fresh record + ("unreadable", None) -> exists but cannot be trusted -> caller FAILS CLOSED + ("ok", dict) -> a parsed record + """ + try: + st = os.stat(name, dir_fd=dir_fd, follow_symlinks=False) + except FileNotFoundError: + return "absent", None + except OSError: + return "unreadable", None + if not stat.S_ISREG(st.st_mode) or st.st_size > cap: + return "unreadable", None # a symlink or an oversized file where a record should be + rec = _read_json_at(dir_fd, name, cap) + if not isinstance(rec, dict): + return "unreadable", None + return "ok", rec + + +def compute_challenge_id(key, epic_id, feature_id, attempt): + """Keyed, deterministic-from-inputs challenge id. HMAC over the tuple with a persisted + per-epic secret key so the id is NOT publicly recomputable, while staying reproducible + for a given (key, tuple) in selftests. This is a secondary defense — the AUTHORIZATION + boundary is the persisted immutable issuance record, not this hash.""" + msg = ("%s\x1f%s\x1f%s" % (epic_id, feature_id, attempt)).encode("utf-8") + mac = hmac.new(key.encode("utf-8"), msg, hashlib.sha256).hexdigest() + return "ch-" + mac[:16] + + +def get_challenge_key(arbiter_fd, injected=None): + """A persisted per-epic secret key at /.challenge-key (0600), created once via + O_EXCL. An injected key (tests / a driver that wants a fixed key) bypasses the file.""" + if injected: + return injected + # (v2.11) a concurrent challenge-key creation race is out of the single-user threat + # model — two processes calling this at once is a multi-process (v2.11) concern. + existing = _read_json_at(arbiter_fd, ".challenge-key", 4096) + if isinstance(existing, dict) and isinstance(existing.get("key"), str) and existing["key"]: + return existing["key"] + key = os.urandom(32).hex() + try: + _write_json_at(arbiter_fd, ".challenge-key", {"key": key}) + except FileExistsError: + again = _read_json_at(arbiter_fd, ".challenge-key", 4096) + if isinstance(again, dict) and isinstance(again.get("key"), str) and again["key"]: + return again["key"] + return key + + +# --------------------------------------------------------------------------------------- # +# B1 — capabilities discovery (strict booleans) + family map +# --------------------------------------------------------------------------------------- # + +def load_capabilities(path=None): + path = path or os.path.expanduser("~/.claude/compound-v-capabilities.json") + try: + with open(path, "r") as fh: + data = json.load(fh) + except (OSError, ValueError): + return {} + return data if isinstance(data, dict) else {} + + +def codex_available(caps): + """Usable iff `codex.available IS True AND codex.exec_flags_verified IS True` — strict + identity, so a truthy string ("false", "0", "no") or a 1/0 int can NEVER enable Codex + egress (Finding 9).""" + codex = caps.get("codex") if isinstance(caps, dict) else None + if not isinstance(codex, dict): + return False + return codex.get("available") is True and codex.get("exec_flags_verified") is True + + +_FAMILY_NEEDLES = ( + ("gpt", "GPT"), ("gemini", "Gemini"), ("claude", "Claude"), + ("opus", "Claude"), ("sonnet", "Claude"), ("grok", "Grok"), +) + + +def model_family(model_name): + name = (model_name or "").lower() + for needle, fam in _FAMILY_NEEDLES: + if needle in name: + return fam + return "unknown" + + +def _load_sibling_module(filename, modname): + import importlib.util + here = os.path.dirname(os.path.abspath(__file__)) + path = os.path.join(here, filename) + try: + spec = importlib.util.spec_from_file_location(modname, path) + if spec is None or spec.loader is None: + return None + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + except Exception: # noqa: BLE001 - any load failure -> caller uses the fallback + return None + + +def resolve_codex_model(config_path=None, explicit_model=None, tier="deep"): + if explicit_model: + return explicit_model + mod = _load_sibling_module("compound-v-resolve-model.py", "compound_v_resolve_model") + if mod is None: + return _FALLBACK_CODEX_MODEL + try: + config_models = mod.load_config_models(config_path) if config_path else {} + return mod.resolve("codex", tier, config_models=config_models)["model"] + except Exception as e: # noqa: BLE001 - degrade-safe + _log("resolve-model.py failed (%s) — using fallback %r" % (e, _FALLBACK_CODEX_MODEL)) + return _FALLBACK_CODEX_MODEL + + +# --------------------------------------------------------------------------------------- # +# B3 — secret redaction (hardened, fail-closed) + field sanitation +# --------------------------------------------------------------------------------------- # + +# Closed private-key blocks (PEM RSA/EC/OPENSSH/generic + PGP). +_CLOSED_KEY_RE = re.compile( + r"-----BEGIN (?:[A-Z0-9 ]*PRIVATE KEY|PGP PRIVATE KEY BLOCK)-----" + r".*?-----END (?:[A-Z0-9 ]*PRIVATE KEY|PGP PRIVATE KEY BLOCK)-----", + re.S) +# Any private-key/PGP BEGIN marker (used to detect an UNCLOSED block after closed ones go). +_ANY_KEY_BEGIN_RE = re.compile( + r"-----BEGIN (?:[A-Z0-9 ]*PRIVATE KEY|PGP PRIVATE KEY BLOCK)-----") +# Authorization header incl. RFC822 folded continuation lines (following lines that begin +# with whitespace). +_AUTH_HEADER_RE = re.compile(r"(?im)^([ \t]*Authorization[ \t]*:).*(?:\r?\n[ \t]+.*)*") +# scheme://user:password@ — password may itself contain ':'. +_URL_CRED_RE = re.compile(r"([a-zA-Z][a-zA-Z0-9+.\-]*://)([^/\s:@]+):([^\s@]+)@") +# Labelled secrets: LABEL = value, value may be a (possibly multi-line) quoted string or a +# bare token. Covers short labelled secrets too (client_secret=, password=, api_key=, ...). +_LABELLED_SECRET_RE = re.compile( + r"(?i)\b(client[_-]?secret|secret[_-]?key|access[_-]?key|api[_-]?key|auth[_-]?token|" + r"secret|password|passwd|token|bearer|private[_-]?key)\b([ \t]*[:=][ \t]*)" + r"(\"(?:[^\"\\]|\\.)*\"|'(?:[^'\\]|\\.)*'|[^\s'\"]+)") +# A labelled secret whose value STARTS with a quote (single/double) — used to detect an +# UNCLOSED quoted secret (no matching close), which the closed-quote alternative above would +# otherwise miss, leaking the value (Finding 2/#6). +_LABELLED_QUOTE_START_RE = re.compile( + r"(?i)\b(?:client[_-]?secret|secret[_-]?key|access[_-]?key|api[_-]?key|auth[_-]?token|" + r"secret|password|passwd|token|bearer|private[_-]?key)\b[ \t]*[:=][ \t]*([\"'])") +# JWT / split dotted tokens: three base64url segments, first begins eyJ. +_JWT_RE = re.compile(r"\beyJ[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+\b") +# Long opaque token runs (>=32 chars). +_TOKEN_RE = re.compile(r"[A-Za-z0-9_\-]{32,}") + + +def _cap_bytes_with_marker(s, max_bytes): + """Cap `s` to `max_bytes` on the ENCODED length, RESERVING room for the truncation + marker so the final encoded value never exceeds max_bytes (Finding 2).""" + b = s.encode("utf-8", errors="replace") + if len(b) <= max_bytes: + return s + marker_b = TRUNC_MARKER.encode("utf-8") + # When the cap is smaller than the marker itself, appending the marker would blow the + # cap — hard-truncate to the cap with no marker instead (Finding 8/#11). + if max_bytes <= len(marker_b): + return b[:max_bytes].decode("utf-8", errors="ignore") + room = max_bytes - len(marker_b) + return b[:room].decode("utf-8", errors="ignore") + TRUNC_MARKER + + +def _has_unescaped_quote(s, start, quote): + """True iff `s[start:]` contains the delimiter `quote` NOT preceded by an odd run of + backslashes (i.e. a real, unescaped closing quote). A backslash-escaped quote (\\" / \\') + does NOT count as a close (round-3 #1) — the value is still open, so redact() fails + closed rather than leak it into the Codex prompt.""" + i = start + n = len(s) + while i < n: + c = s[i] + if c == "\\": + i += 2 # skip the escaped char (whatever it is) + continue + if c == quote: + return True + i += 1 + return False + + +def redact(text, max_bytes): + """Conservative secret redaction. Returns the redacted (and byte-capped) string, or None + to signal FAIL CLOSED — the caller must then OMIT the evidence entirely. Fails closed on + any internal exception AND on an UNCLOSED private-key/PGP block (a secret whose end we + can't see, so we can't be sure a later pattern caught all of it).""" + try: + s = "" if text is None else str(text) + s = _CLOSED_KEY_RE.sub("[REDACTED:PRIVATE_KEY]", s) + if _ANY_KEY_BEGIN_RE.search(s): + return None # an unclosed private-key/PGP block -> omit the whole evidence + # An UNCLOSED quoted labelled secret (e.g. token="shortsecret\ncontinued with no + # closing quote) -> fail closed, like the unclosed PEM case above (Finding 2/#6). + # An ESCAPED quote (\" / \') is NOT a close (round-3 #1): token="secret\" is still + # unterminated, so scan for the first UNESCAPED matching quote and fail closed if none. + for m in _LABELLED_QUOTE_START_RE.finditer(s): + quote = m.group(1) + if not _has_unescaped_quote(s, m.end(), quote): + return None + s = _AUTH_HEADER_RE.sub(r"\1 [REDACTED]", s) + s = _URL_CRED_RE.sub(r"\1[REDACTED]@", s) + s = _LABELLED_SECRET_RE.sub(r"\1\2[REDACTED]", s) + s = _JWT_RE.sub("[REDACTED:JWT]", s) + s = _TOKEN_RE.sub("[REDACTED:TOKEN]", s) + return _cap_bytes_with_marker(s, max_bytes) + except Exception: # noqa: BLE001 - never leak a half-redacted string + return None + + +def _sanitize_diagnostic(text): + """Run a RETAINED diagnostic/drop-reason string through the same fail-closed redactor + before it can reach stdout or the audit JSON (Finding 3/#7). A drop reason may quote a + fragment of untrusted model/ballot content, so it is NEVER trusted verbatim; if it + can't be sanitized it is replaced with a generic placeholder.""" + r = redact(text if isinstance(text, str) else str(text), REASON_MAX_CHARS) + if r is None: + return "[diagnostic omitted — could not sanitize]" + return r[:REASON_MAX_CHARS] + + +def sanitize_ballot_fields(reason, evidence): + """Type/line/length-check + RE-REDACT a model ballot's reason/evidence BEFORE they enter + the classify output or the audit JSON (Finding 2). reason must be a single-line string; + evidence a single-line string or None. Returns (reason, evidence) or None to DROP the + whole ballot (fail closed).""" + if reason is None: + reason = "" + if not isinstance(reason, str) or "\n" in reason or "\r" in reason: + return None + if evidence is not None and (not isinstance(evidence, str) or "\n" in evidence or "\r" in evidence): + return None + r = redact(reason, FIELD_MAX_CHARS) + if r is None: + return None + if len(r) > FIELD_MAX_CHARS: + r = r[:FIELD_MAX_CHARS] + if evidence is None: + e = None + else: + e = redact(evidence, FIELD_MAX_CHARS) + if e is None: + return None + if len(e) > FIELD_MAX_CHARS: + e = e[:FIELD_MAX_CHARS] + return r, e + + +# --------------------------------------------------------------------------------------- # +# B3 — TOCTOU-safe contained evidence read (from the SAME fd we walk) +# --------------------------------------------------------------------------------------- # + +def open_attempt_dir_fd(arbiter_fd, feature, attempt): + """Open /- as a dir fd with O_NOFOLLOW (rejects a symlinked + attempt dir). Raises OSError if missing.""" + name = "%s-%s" % (feature, attempt) + return os.open(name, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | _O_CLOEXEC, + dir_fd=arbiter_fd) + + +def read_contained_evidence(base_fd, rel_path, cap): + """Read an evidence file named by `rel_path` RELATIVE to base_fd (the attempt dir), + walking each component with O_NOFOLLOW dir_fd opens and READING FROM THE FINAL FD we + opened — never re-opening a validated name, so a component swapped after validation + cannot redirect the read (Finding 1). Rejects absolute paths and any '.'/'..' component + up front. Returns (text|None, err|None).""" + if not rel_path: + return None, "no evidence path" + if os.path.isabs(rel_path): + return None, "evidence path must be RELATIVE to the attempt dir (absolute rejected)" + parts = [p for p in re.split(r"[/\\]+", rel_path) if p != ""] + if not parts: + return None, "empty evidence path" + for p in parts: + if p in (".", ".."): + return None, "'.'/'..' component rejected in evidence path" + opened = [] + cur = base_fd + try: + for p in parts[:-1]: + nfd = os.open(p, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | _O_CLOEXEC, dir_fd=cur) + opened.append(nfd) + cur = nfd + try: + ffd = os.open(parts[-1], os.O_RDONLY | os.O_NOFOLLOW | _O_CLOEXEC, dir_fd=cur) + except OSError as e: + return None, "evidence open failed (symlink/missing?): %s" % e + try: + st = os.fstat(ffd) + if not stat.S_ISREG(st.st_mode): + return None, "evidence is not a regular file" + if st.st_size > cap: + return None, "evidence oversized (%d > cap %d)" % (st.st_size, cap) + raw = os.read(ffd, cap + 1) + finally: + os.close(ffd) + except OSError as e: + return None, "evidence path walk failed (symlink/missing?): %s" % e + finally: + for fd in opened: + try: + os.close(fd) + except OSError: + pass + return raw.decode("utf-8", errors="replace"), None + + +# --------------------------------------------------------------------------------------- # +# B3 — bounded prompts +# --------------------------------------------------------------------------------------- # + +def _bound_text(s, max_chars): + if s is None: + return "" + s = str(s) + return s if len(s) <= max_chars else s[:max_chars] + "\n...[TRUNCATED to stay bounded]" + + +def _build_claude_prompt(epic_id, feature_id, attempt, challenge_id, feat): + title = feat.get("title", feature_id) if isinstance(feat, dict) else feature_id + body = ( + "You are a FRESH, ADVERSARIAL reviewer for Compound V's marathon arbiter panel.\n" + "A feature attempt FAILED its quality gate and needs an independent disposition.\n\n" + "epic_id: %s\nfeature: %s (%s)\nattempt: %s\nchallenge_id: %s\n\n" + "Investigate the failing feature's run directory and evidence yourself. Decide " + "exactly ONE disposition:\n" + " retry_fix - a fixable bug/flake; retrying is worth it\n" + " halt_feature - abandon just this feature; the epic continues on independents\n" + " halt_epic - a systemic problem; the whole epic should stop\n" + " blocked_external - progress needs a human/external fact code cannot create\n\n" + "Write your ballot as JSON with EXACTLY these keys (single-line reason/evidence):\n" + ' {"epic_id": %s, "feature": %s, "attempt": %s, "challenge_id": %s,\n' + ' "disposition": "", "reason": "",\n' + ' "evidence": ""}\n\n' + "Be conservative: on genuine doubt between retry_fix and a halt_*, prefer the halt." + % (epic_id, feature_id, title, attempt, challenge_id, + json.dumps(epic_id), json.dumps(feature_id), json.dumps(attempt), + json.dumps(challenge_id))) + return _bound_text(body, CLAUDE_PROMPT_MAX_CHARS) + + +def _build_codex_prompt(epic_id, feature_id, attempt, challenge_id, evidence_text): + ev = evidence_text if evidence_text is not None else ( + "(evidence omitted — redaction could not complete safely, fail-closed)") + body = ( + "Compound V marathon arbiter -- advisory read-only classification.\n" + "epic_id: %s\nfeature: %s\nattempt: %s\nchallenge_id: %s\n\n" + "A feature attempt FAILED its quality gate. Redacted evidence follows:\n---\n%s\n---\n\n" + "Reply with ONLY a single JSON object, no prose before or after, single-line " + "reason/evidence:\n" + '{"disposition": "retry_fix|halt_feature|halt_epic|blocked_external", ' + '"reason": "", ' + '"evidence": ""}' + % (epic_id, feature_id, attempt, challenge_id, ev)) + return _bound_text(body, CODEX_PROMPT_MAX_CHARS) + + +# --------------------------------------------------------------------------------------- # +# B2 — Codex read-only poll through the shared timeout supervisor +# --------------------------------------------------------------------------------------- # + +def build_codex_invocation(model, prompt, stdout_path, stderr_path, lastmsg_path, + timeout_sec, max_output_bytes, + codex_bin=None, supervisor_path=None, python_bin=None): + """Build the EXACT argv for the supervised Codex poll. codex_bin / supervisor_path / + python_bin are the injectable seam selftests use to substitute a fake codex/supervisor.""" + codex_bin = codex_bin or os.environ.get("COMPOUND_V_ARBITER_CODEX_BIN") or "codex" + python_bin = python_bin or sys.executable or "python3" + supervisor_path = supervisor_path or os.environ.get("COMPOUND_V_ARBITER_SUPERVISOR") or \ + os.path.join(os.path.dirname(os.path.abspath(__file__)), "compound-v-run-with-timeout.py") + codex_argv = [codex_bin, "exec", "--sandbox", "read-only", "--skip-git-repo-check", + "--model", model, "-c", "model_reasoning_effort=high", "--json", + "--output-last-message", lastmsg_path, prompt] + return ([python_bin, supervisor_path, + "--timeout", str(int(timeout_sec)), + "--max-output-bytes", str(int(max_output_bytes)), + "--stdout", stdout_path, "--stderr", stderr_path, "--"] + + codex_argv) + + +def _parse_codex_verdict(raw): + """STRICT parse (Finding 8): after removing ONLY exact-match `codex_hooks is deprecated` + lines, the ENTIRE remaining message must be exactly one schema-valid JSON object — no + leading/trailing prose, no multiple objects, no `rfind('{')` salvage. reason/evidence + must be single-line strings (evidence may be null). Returns None on ANY deviation (never + fabricates a disposition).""" + if not isinstance(raw, str) or not raw.strip(): + return None + kept = [ln for ln in raw.splitlines() if ln.strip() != DEPRECATION_LINE] + text = "\n".join(kept).strip() + if not text: + return None + try: + data = json.loads(text) + except (ValueError, TypeError): + return None + if not isinstance(data, dict): + return None + disposition = data.get("disposition") + if disposition not in DISPOSITIONS: + return None + reason = data.get("reason", "") + evidence = data.get("evidence") + if reason is None: + reason = "" + if not isinstance(reason, str) or "\n" in reason or "\r" in reason: + return None + if evidence is not None and (not isinstance(evidence, str) or "\n" in evidence or "\r" in evidence): + return None + return {"disposition": disposition, "reason": reason, "evidence": evidence} + + +def poll_codex(model, prompt, timeout_sec=DEFAULT_CODEX_TIMEOUT_SEC, + max_output_bytes=DEFAULT_MAX_OUTPUT_BYTES, codex_bin=None, + supervisor_path=None, python_bin=None, env=None): + """Run the Codex advisory poll through the supervisor inside a self-cleaning temp dir. + Returns (verdict_dict|None, drop_reason|None). NEVER a fabricated halt vote: drops on a + nonzero supervisor exit, a missing/symlinked/oversized --output-last-message (bounded via + _read_capped_regular, which DELETES an oversized artifact), an unparseable message, or a + field that fails re-redaction.""" + with tempfile.TemporaryDirectory(prefix="compound-v-arbiter-") as work_dir: + stdout_path = os.path.join(work_dir, "codex_stdout.jsonl") + stderr_path = os.path.join(work_dir, "codex_stderr.log") + lastmsg_path = os.path.join(work_dir, "codex_lastmsg.txt") + argv = build_codex_invocation(model, prompt, stdout_path, stderr_path, lastmsg_path, + timeout_sec, max_output_bytes, codex_bin=codex_bin, + supervisor_path=supervisor_path, python_bin=python_bin) + try: + proc = subprocess.run(argv, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, env=env) + except OSError as e: + return None, "failed to launch the codex poll supervisor: %s" % e + if proc.returncode != 0: + return None, ("codex poll supervisor exited %d — dropped, not a fabricated halt " + "vote" % proc.returncode) + # lastmsg_path is inside our OWN private TemporaryDirectory, so deleting an oversized + # one is safe (never touches a user-owned file — Finding 4/#8). + raw, err = _read_capped_regular(lastmsg_path, max_output_bytes, delete_oversized=True) + if raw is None: + return None, "codex --output-last-message unusable (%s) — dropped" % err + verdict = _parse_codex_verdict(raw) + if verdict is None: + return None, "codex reply was not a strict single-object verdict — dropped" + fields = sanitize_ballot_fields(verdict["reason"], verdict["evidence"]) + if fields is None: + return None, "codex reply failed re-redaction/field checks — dropped (fail-closed)" + reason, evidence = fields + return {"disposition": verdict["disposition"], "reason": reason, "evidence": evidence}, None + + +# --------------------------------------------------------------------------------------- # +# B3 — Claude ballot validation (+ re-redaction) +# --------------------------------------------------------------------------------------- # + +def load_claude_ballot(path, epic_id, feature_id, attempt, challenge_id): + """Validate the ballot's 4-tuple against the CURRENT state + issued challenge, read via + O_NOFOLLOW+fstat+cap, re-redact its fields. Returns (ballot|None, drop_reason|None). + (The CALLER only invokes this when the persisted challenge is valid — a stale/consumed + challenge drops the whole panel before we get here, Finding 4.)""" + if not path: + return None, "no --claude-ballot given" + raw, err = _read_capped_regular(path, BALLOT_FILE_MAX_BYTES) + if raw is None: + return None, "claude ballot unreadable (%s)" % err + try: + data = json.loads(raw) + except ValueError as e: + return None, "claude ballot is not valid JSON: %s" % e + if not isinstance(data, dict): + return None, "claude ballot is not a JSON object" + if (data.get("epic_id") != epic_id or data.get("feature") != feature_id + or data.get("attempt") != attempt or data.get("challenge_id") != challenge_id): + return None, "claude ballot 4-tuple mismatch (stale/wrong-feature/wrong-attempt)" + disposition = data.get("disposition") + if disposition not in DISPOSITIONS: + # NB: do NOT interpolate the untrusted `disposition` value into the reason — it + # could carry a secret that would then be retained/emitted (Finding 3/#7). + return None, "claude ballot has an invalid/missing disposition" + fields = sanitize_ballot_fields(data.get("reason", ""), data.get("evidence")) + if fields is None: + return None, "claude ballot fields failed re-redaction/checks (fail-closed)" + reason, evidence = fields + return {"source": "claude", "family": "Claude", "model": "claude", "valid": True, + "disposition": disposition, "reason": reason, "evidence": evidence}, None + + +# --------------------------------------------------------------------------------------- # +# B4 — collapse + aggregation truth table +# --------------------------------------------------------------------------------------- # + +def _collapse_same_family(ballots): + out = [dict(b) for b in ballots] + for b in out: + b["counted"] = bool(b.get("valid")) + by_family = {} + for b in out: + if b["counted"]: + by_family.setdefault(b.get("family"), []).append(b) + for fam, group in by_family.items(): + if len(group) <= 1: + continue + ranked = sorted(group, key=lambda b: CONSERVATISM_RANK.get(b.get("disposition"), -1)) + winner = ranked[-1] + for b in group: + if b is not winner: + b["counted"] = False + b["collapse_note"] = ("same-family (%s) collapse: %r deferred to %r" + % (fam, b.get("disposition"), winner.get("disposition"))) + return out + + +def aggregate_dispositions(valid_ballots): + """The complete deterministic truth table (B4). `valid_ballots` are one-per-family + (counted=True). Returns (disposition, confirmed, reason). + + Finding 7: when there is NO valid ballot from a KNOWN non-Claude external family, both + halt_epic AND blocked_external are masked to halt_feature (the Claude-only / no-external + fallback caps to retry_fix|halt_feature; a lone Claude blocked_external cannot survive).""" + external_present = any(b.get("family") in _EXTERNAL_CONFIRMING for b in valid_ballots) + + def _cap_no_external(d, reason): + if not external_present and d in ("halt_epic", "blocked_external"): + return "halt_feature", (reason + "; no external-family ballot -> capped to halt_feature") + return d, reason + + n = len(valid_ballots) + if n == 0: + return "halt_feature", False, "no valid ballots (parse-fail/errored/absent) -- conservative default" + if n == 1: + d = valid_ballots[0]["disposition"] + reason = "single valid ballot" + if d == "halt_epic": + d, reason = "halt_feature", "single ballot cannot unilaterally halt_epic -> halt_feature" + d, reason = _cap_no_external(d, reason) + return d, False, reason + + retry_n = sum(1 for b in valid_ballots if b["disposition"] == "retry_fix") + nonretry = [b for b in valid_ballots if b["disposition"] != "retry_fix"] + nonretry_n = len(nonretry) + + if retry_n > nonretry_n: + return "retry_fix", False, "retry_fix majority" + if retry_n == nonretry_n: + return "halt_feature", False, "tied retry/non-retry vote -- conservative halt_feature" + + nr_dispositions = {b["disposition"] for b in nonretry} + if len(nr_dispositions) == 1: + d = next(iter(nr_dispositions)) + reason = "non-retry majority, unanimous agreement on %r" % d + else: + d, reason = "halt_feature", ("non-retry majority but disagreement among them -- " + "conservative halt_feature") + + confirmed = False + if d == "blocked_external": + confirming = {b["family"] for b in nonretry + if b["disposition"] == "blocked_external" and b["family"] in _EXTERNAL_CONFIRMING} + confirmed = len(confirming) >= 2 and retry_n == 0 + if not confirmed: + reason += (" (SUSPECTED, not confirmed: needs >=2 distinct KNOWN external families " + "with no retry_fix dissent)") + + d, reason = _cap_no_external(d, reason) + return d, confirmed, reason + + +def _can_retry(state, feature_id): + feat = _find_feature(state, feature_id) + if feat is None: + return True, None + attempts = feat.get("attempts", 0) + autonomy = state.get("autonomy") if isinstance(state.get("autonomy"), dict) else {} + cap = autonomy.get("max_attempts_per_feature", 2) + if cap is None: + return True, None + return attempts < cap, cap + + +# --------------------------------------------------------------------------------------- # +# Audit rotation (count AND bytes) +# --------------------------------------------------------------------------------------- # + +def _rotate_by(arbiter_fd, match, count_cap, byte_cap): + """Delete the oldest matching files until BOTH count_cap and byte_cap hold. `match(name)` + selects which files participate.""" + try: + names = os.listdir(arbiter_fd) + except OSError: + return + entries = [] + total = 0 + for nm in names: + if not match(nm): + continue + try: + st = os.stat(nm, dir_fd=arbiter_fd, follow_symlinks=False) + except OSError: + continue + if not stat.S_ISREG(st.st_mode): + continue + entries.append((st.st_mtime, nm, st.st_size)) + total += st.st_size + entries.sort() + i = 0 + count = len(entries) + while i < len(entries) and (count > count_cap or total > byte_cap) and count > 1: + _, nm, sz = entries[i] + try: + os.unlink(nm, dir_fd=arbiter_fd) + total -= sz + count -= 1 + except OSError: + pass + i += 1 + + +def _rotate_audit(arbiter_fd, count_cap=AUDIT_ROTATE_COUNT, byte_cap=AUDIT_ROTATE_BYTES): + # Audit + rejected files (`*.json`, excluding the `*.challenge.json` records). + _rotate_by(arbiter_fd, lambda nm: nm.endswith(".json") and not nm.endswith(".challenge.json"), + count_cap, byte_cap) + + +def _rotate_challenge(arbiter_fd, count_cap=CHALLENGE_ROTATE_COUNT, byte_cap=AUDIT_ROTATE_BYTES): + # Bounded retention for the challenge issuance records too (Finding 5/#9). + _rotate_by(arbiter_fd, lambda nm: nm.endswith(".challenge.json"), count_cap, byte_cap) + + +def _finalize_result(result): + """Produce the ONE object that is BOTH printed to the caller AND persisted to the audit + JSON — they must be byte-identical (Finding 5/#9). Caps every retained field length, + trims ballot detail if the serialized blob exceeds AUDIT_MAX_ONE_BYTES, and — if it is + STILL over after trimming — emits a bounded stub that keeps only the report-critical + fields.""" + r = dict(result) + if isinstance(r.get("reason"), str): + r["reason"] = r["reason"][:REASON_MAX_CHARS] + if isinstance(r.get("evidence"), str): + r["evidence"] = r["evidence"][:FIELD_MAX_CHARS] + capped = [] + for b in r.get("ballots", []): + b2 = dict(b) + if isinstance(b2.get("reason"), str): + b2["reason"] = b2["reason"][:REASON_MAX_CHARS] + if isinstance(b2.get("evidence"), str): + b2["evidence"] = b2["evidence"][:FIELD_MAX_CHARS] + capped.append(b2) + r["ballots"] = capped + + if len(json.dumps(r).encode("utf-8")) > AUDIT_MAX_ONE_BYTES: + r["evidence"] = None + r["ballots"] = [dict(b, reason="", evidence=None) for b in r["ballots"]] + r["audit_trimmed"] = True + if len(json.dumps(r).encode("utf-8")) > AUDIT_MAX_ONE_BYTES: + # round-3 #2: bound EVERY string carried into the stub (even an oversized + # challenge_id/epic_id/feature/audit_path copied from the caller) so the stub itself + # can never exceed the cap regardless of which field was oversized. + def _b(v): + return _cap_bytes_with_marker(v, MAX_ID_LEN) if isinstance(v, str) else v + fams = r.get("families_present", []) + agrees = r.get("families_agreeing", []) + r = { + "disposition": _b(r.get("disposition")), "confirmed": r.get("confirmed"), + "reason": "audit exceeded the size cap — stubbed to report-critical fields", + "evidence": None, "ballots": [], + "families_present": fams if isinstance(fams, list) and len(fams) <= 8 else [], + "families_agreeing": agrees if isinstance(agrees, list) and len(agrees) <= 8 else [], + "challenge_id": _b(r.get("challenge_id")), "epic_id": _b(r.get("epic_id")), + "feature": _b(r.get("feature")), "attempt": r.get("attempt"), + "audit_path": _b(r.get("audit_path")), "recorded_at": _b(r.get("recorded_at")), + "audit_trimmed": True, "audit_stubbed": True, + } + # Absolute floor: if a pathological input still overflows, drop to a minimal object + # that is guaranteed under the cap (round-3 #2: NO finalized object exceeds the cap). + if len(json.dumps(r).encode("utf-8")) > AUDIT_MAX_ONE_BYTES: + r = {"disposition": "halt_feature", "confirmed": False, + "reason": "audit stub overflow — minimal object", + "evidence": None, "ballots": [], "families_present": [], + "families_agreeing": [], "challenge_id": None, "epic_id": None, + "feature": None, "attempt": r.get("attempt") if isinstance(r.get("attempt"), int) else None, + "audit_path": None, "recorded_at": None, + "audit_trimmed": True, "audit_stubbed": True} + return r + + +def _audit_tuple_matches(audit, epic_id, feature, attempt, challenge): + return (isinstance(audit, dict) + and audit.get("epic_id") == epic_id and audit.get("feature") == feature + and audit.get("attempt") == attempt and audit.get("challenge_id") == challenge) + + +def _reemit_persisted(arbiter_fd, audit_name, epic_id, feature, attempt, challenge): + """Re-emit a previously-persisted audit for an idempotent replay: read its RAW bytes, + validate the parsed tuple, and if it matches write those EXACT bytes to stdout (round-3 + #3 — printed == persisted). Returns True on a successful re-emit, else False (caller then + falls back to a conservative result or runs the panel).""" + raw = _read_text_at(arbiter_fd, audit_name, AUDIT_MAX_ONE_BYTES) + if raw is None: + return False + try: + prior = json.loads(raw) + except ValueError: + return False + if not _audit_tuple_matches(prior, epic_id, feature, attempt, challenge): + return False + sys.stdout.write(raw) + return True + + +# --------------------------------------------------------------------------------------- # +# CLI: prepare +# --------------------------------------------------------------------------------------- # + +def cmd_prepare(args, p): + if not args.feature: + p.error("--prepare needs --feature") + if args.attempt is None: + p.error("--prepare needs --attempt ") + state = _read_state(args.state) + epic_id = state["epic_id"] + if not _id_ok(args.feature): + raise ArbiterError("invalid --feature id: %r" % (args.feature,)) + feat = _find_feature(state, args.feature) + if feat is None: + raise ArbiterError("no feature %r in state" % args.feature) + + current = feat.get("attempts", 0) + if args.attempt != current: + raise ArbiterError("--attempt %r must equal the feature's current persisted attempts " + "%r (issuance is monotonic; you cannot pre-issue a future attempt " + "or re-issue a past one)" % (args.attempt, current)) + + epic_dir = validate_epic_dir(args.state, epic_id) + now_s = _now_iso(_resolve_now(args.now)) + challenge_name = "%s-%s.challenge.json" % (args.feature, args.attempt) + + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=True) + try: + key = get_challenge_key(arbiter_fd, injected=args.challenge_key) + challenge_id = compute_challenge_id(key, epic_id, args.feature, args.attempt) + + # Finding 6/#3: distinguish a truly-absent record from an EXISTING-but-unreadable one. + # An unreadable/inconsistent record must FAIL CLOSED — never silently overwrite it + # with a fresh `issued` record (which could resurrect a consumed challenge). + # (v2.11) a concurrent compare-and-swap between this read and the write below is out + # of the single-user threat model. + rstatus, existing = _probe_record(arbiter_fd, challenge_name, CHALLENGE_RECORD_MAX_BYTES) + if rstatus == "unreadable": + raise ArbiterError("an existing challenge record for %s attempt %s is unreadable/" + "inconsistent — refusing to overwrite it (fail closed)" + % (args.feature, args.attempt)) + if rstatus == "ok": + st = existing.get("status") + tuple_ok = (existing.get("challenge_id") == challenge_id + and existing.get("epic_id") == epic_id + and existing.get("feature") == args.feature + and existing.get("attempt") == args.attempt) + if not tuple_ok: + raise ArbiterError("an existing challenge record for %s attempt %s has an " + "inconsistent tuple — refusing to overwrite it (fail closed)" + % (args.feature, args.attempt)) + if st in ("in_progress", "consumed"): + raise ArbiterError("challenge for %s attempt %s is already %s — cannot re-issue " + "(anti-replay)" % (args.feature, args.attempt, st)) + if st == "issued": + print(json.dumps({"challenge_id": challenge_id, "epic_id": epic_id, + "feature": args.feature, "attempt": args.attempt, + "issued_at": existing.get("issued_at", now_s), + "prompt": existing.get("claude_prompt", "")})) + return 0 + + prompt = _build_claude_prompt(epic_id, args.feature, args.attempt, challenge_id, feat) + record = {"epic_id": epic_id, "feature": args.feature, "attempt": args.attempt, + "challenge_id": challenge_id, "issued_at": now_s, "status": "issued", + "consumed_at": None, "claude_prompt": prompt} + _write_json_at(arbiter_fd, challenge_name, record) + _rotate_challenge(arbiter_fd) + # Give the driver a per-attempt evidence home (contained, symlink-checked on read). + try: + os.mkdir("%s-%s" % (args.feature, args.attempt), 0o755, dir_fd=arbiter_fd) + except FileExistsError: + pass + print(json.dumps({"challenge_id": challenge_id, "epic_id": epic_id, "feature": args.feature, + "attempt": args.attempt, "issued_at": now_s, "prompt": prompt})) + return 0 + finally: + os.close(arbiter_fd) + + +# --------------------------------------------------------------------------------------- # +# CLI: classify +# --------------------------------------------------------------------------------------- # + +def _conservative_dropped_result(epic_id, feature, attempt, challenge, reason): + return {"disposition": "halt_feature", "confirmed": False, "reason": reason, + "evidence": None, "ballots": [], "families_present": [], "families_agreeing": [], + "challenge_id": challenge, "epic_id": epic_id, "feature": feature, "attempt": attempt, + "audit_path": None} + + +def cmd_classify(args, p): + if not args.feature: + p.error("--classify needs --feature") + if not args.challenge: + p.error("--classify needs --challenge ") + # round-3 #2: length-bound --challenge at the validation boundary. It is retained in the + # (rejected) result and copied into the fallback stub, so an unbounded value could push a + # finalized object over AUDIT_MAX_ONE_BYTES. A legitimate id is "ch-"+16 hex = 19 chars; + # reject anything longer than MAX_ID_LEN like any other overlong id. + if len(args.challenge) > MAX_ID_LEN: + raise ArbiterError("--challenge id is too long (max %d chars)" % MAX_ID_LEN) + state = _read_state(args.state) + epic_id = state["epic_id"] + if not _id_ok(args.feature): + raise ArbiterError("invalid --feature id: %r" % (args.feature,)) + feat = _find_feature(state, args.feature) + if feat is None: + raise ArbiterError("no feature %r in state" % args.feature) + + attempt = feat.get("attempts", 0) + epic_dir = validate_epic_dir(args.state, epic_id) + now_s = _now_iso(_resolve_now(args.now)) + max_output_bytes = args.max_output_bytes + challenge_name = "%s-%s.challenge.json" % (args.feature, attempt) + audit_name = "%s-%s.json" % (args.feature, attempt) + + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=True) + try: + key = get_challenge_key(arbiter_fd, injected=args.challenge_key) + expected = compute_challenge_id(key, epic_id, args.feature, attempt) + rstatus, record = _probe_record(arbiter_fd, challenge_name, CHALLENGE_RECORD_MAX_BYTES) + + tuple_ok = (rstatus == "ok" + and args.challenge == expected == record.get("challenge_id") + and record.get("epic_id") == epic_id + and record.get("feature") == args.feature + and record.get("attempt") == attempt) + status = record.get("status") if rstatus == "ok" else None + + # ---- Finding 4: a stale/invalid/consumed/unreadable challenge gates the WHOLE panel. + # Finding 1/#2: a REJECTED challenge must NEVER be persisted to the CANONICAL result + # filename (that would clobber a previously-consumed valid audit, so a later correct + # replay would emit this attacker-triggered conservative verdict). Write a drop event + # to a clearly-separate `.rejected.json` path only. + if not tuple_ok: + result = _conservative_dropped_result( + epic_id, args.feature, attempt, args.challenge, + "challenge invalid/missing/stale/unreadable for the current attempt -- panel " + "dropped before any model call") + rejected_name = "%s-%s.rejected.json" % (args.feature, attempt) + try: + rej_path = _contained_audit_path(epic_dir, rejected_name) + final = _finalize_result(dict(result, audit_path=rej_path, recorded_at=now_s)) + blob = _serialize(final) # round-3 #3: one serialization + _write_text_at(arbiter_fd, rejected_name, blob) + _rotate_audit(arbiter_fd) + except (ArbiterError, OSError) as e: + _log("could not record the rejection event: %s" % e) + blob = _serialize(_finalize_result(dict(result, recorded_at=now_s))) + sys.stdout.write(blob) # printed == persisted bytes + return 0 + if status == "consumed": + # Idempotent replay: re-emit the persisted audit — but only after validating its + # tuple, so a swapped/foreign audit file cannot be re-emitted (Finding 1/#2). + # round-3 #3: re-emit the RAW persisted bytes so printed == persisted exactly. + if _reemit_persisted(arbiter_fd, audit_name, epic_id, args.feature, attempt, args.challenge): + _log("challenge already consumed — re-emitting the persisted audit (idempotent)") + return 0 + result = _conservative_dropped_result( + epic_id, args.feature, attempt, args.challenge, + "challenge consumed but its audit is missing/inconsistent -- panel dropped") + sys.stdout.write(_serialize(_finalize_result(dict(result, recorded_at=now_s)))) + return 0 + + # ---- crash-recovery re-emit, not re-egress (Finding 7/#1) ------------------------- + # If the record is already `in_progress` (a prior classify claimed it) AND a matching + # audit already exists, re-emit THAT — never a second Codex call. Only call the model + # when there is no prior audit. (v2.11) the concurrent compare-and-swap race between + # two live drivers is out of the single-user threat model. + if status == "in_progress": + if _reemit_persisted(arbiter_fd, audit_name, epic_id, args.feature, attempt, args.challenge): + _log("challenge in_progress with an existing audit — re-emitting it (no re-egress)") + return 0 + # else: crashed before writing an audit — fall through and run the panel. + elif status == "issued": + # durable issued -> in_progress BEFORE any model call + record["status"] = "in_progress" + record["in_progress_at"] = now_s + try: + _write_json_at(arbiter_fd, challenge_name, record) + except OSError as e: + raise ArbiterError("could not durably claim the challenge (issued->in_progress): " + "%s" % e) + else: + raise ArbiterError("challenge in an unexpected status %r" % (status,)) + + # ---- evidence: contained read + redact BEFORE egress ------------------------------ + evidence_text = None + if args.evidence_file: + try: + attempt_fd = open_attempt_dir_fd(arbiter_fd, args.feature, attempt) + except OSError as e: + attempt_fd = None + _log("evidence attempt dir unavailable (%s) — evidence omitted" % e) + if attempt_fd is not None: + try: + read_cap = max(200000, max_output_bytes * 4) + raw, err = read_contained_evidence(attempt_fd, args.evidence_file, read_cap) + finally: + os.close(attempt_fd) + if raw is None: + _log("evidence rejected: %s" % err) + else: + evidence_text = redact(raw, max_output_bytes) + if evidence_text is None: + _log("secret redaction could not complete -- evidence OMITTED (fail-closed)") + + ballots = [] + + # ---- Codex ballot ----------------------------------------------------------------- + caps = load_capabilities(args.capabilities_path) + if codex_available(caps): + model = resolve_codex_model(config_path=args.config, + explicit_model=args.explicit_codex_model) + family = model_family(model) + codex_prompt = _build_codex_prompt(epic_id, args.feature, attempt, args.challenge, + evidence_text) + verdict, drop_reason = poll_codex( + model, codex_prompt, timeout_sec=args.codex_timeout, + max_output_bytes=max_output_bytes, codex_bin=args.codex_bin, + supervisor_path=args.supervisor_path) + if verdict is None: + _log("codex ballot dropped: %s" % drop_reason) + ballots.append({"source": "codex", "family": family, "model": model, "valid": False, + "disposition": None, "reason": _sanitize_diagnostic(drop_reason), + "evidence": None}) + else: + ballots.append({"source": "codex", "family": family, "model": model, "valid": True, + "disposition": verdict["disposition"], "reason": verdict["reason"], + "evidence": verdict["evidence"]}) + else: + ballots.append({"source": "codex", "family": None, "model": None, "valid": False, + "disposition": None, + "reason": "codex unavailable (capabilities absent/false) -- Claude-only", + "evidence": None}) + + # ---- Claude ballot ---------------------------------------------------------------- + if args.claude_ballot: + cb, reason = load_claude_ballot(args.claude_ballot, epic_id, args.feature, attempt, + args.challenge) + if cb is None: + _log("claude ballot dropped: %s" % reason) + ballots.append({"source": "claude", "family": "Claude", "model": None, + "valid": False, "disposition": None, + "reason": _sanitize_diagnostic(reason), "evidence": None}) + else: + ballots.append(cb) + else: + ballots.append({"source": "claude", "family": "Claude", "model": None, "valid": False, + "disposition": None, "reason": "no --claude-ballot provided", + "evidence": None}) + + collapsed = _collapse_same_family(ballots) + valid_counted = [b for b in collapsed if b.get("counted")] + disposition, confirmed, agg_reason = aggregate_dispositions(valid_counted) + + masked_note = None + if disposition == "retry_fix": + can_retry, cap = _can_retry(state, args.feature) + if not can_retry: + disposition = "halt_feature" + masked_note = ("retry_fix masked to halt_feature: feature is at/past its retry " + "cap (%r)" % (cap,)) + + result_evidence = None + if disposition == "blocked_external": + for b in valid_counted: + if b.get("evidence"): + result_evidence = b["evidence"] + break + + families_present = sorted({b.get("family") for b in valid_counted if b.get("family")}) + families_agreeing = sorted({b.get("family") for b in valid_counted + if b.get("family") and b.get("disposition") == disposition}) + reason_text = agg_reason if not masked_note else ("%s; %s" % (agg_reason, masked_note)) + + # ---- Finding 10: audit_path computed FIRST, then the FROZEN complete result ------- + audit_path = _contained_audit_path(epic_dir, audit_name) + result = { + "disposition": disposition, "confirmed": bool(confirmed), "reason": reason_text, + "evidence": result_evidence, "ballots": collapsed, + "families_present": families_present, "families_agreeing": families_agreeing, + "challenge_id": args.challenge, "epic_id": epic_id, "feature": args.feature, + "attempt": attempt, "audit_path": audit_path, "recorded_at": now_s, + } + # Finding 5/#9 + round-3 #3: finalize ONCE and serialize ONCE — the exact bytes we + # persist are the exact bytes we print. + final = _finalize_result(result) + blob = _serialize(final) + _write_text_at(arbiter_fd, audit_name, blob) + _rotate_audit(arbiter_fd) + + # ---- durable in_progress -> consumed; do NOT report success if this fails --------- + record["status"] = "consumed" + record["consumed_at"] = now_s + try: + _write_json_at(arbiter_fd, challenge_name, record) + except OSError as e: + raise ArbiterError("classified but could NOT durably record consumption (%s) — " + "not reporting success; replay will recover" % e) + + sys.stdout.write(blob) + return 0 + finally: + os.close(arbiter_fd) + + +# --------------------------------------------------------------------------------------- # +# CLI: resume-challenge (idempotent, read-only crash-recovery lookup) +# --------------------------------------------------------------------------------------- # + +def cmd_resume_challenge(args, p): + """Read-only, idempotent lookup of the persisted challenge/result for + (epic_id, feature, attempt). Fixes the --prepare resume DEADLOCK: --prepare refuses to + re-issue an in_progress/consumed challenge, so a driver that always restarts a crashed + arbitration from --prepare has no way back in once --classify has begun. This command + lets the driver ask "what actually happened?" first, with NO side effect other than the + ordinary idempotent lazy-create of the per-epic challenge key (unreachable here unless a + real record already exists, since --prepare always creates that key before it ever + writes a record) -- it never issues a challenge and never calls Codex/Claude. + + Returns exactly one of: + {"state": "absent"} -- nothing issued; --prepare fresh + {"state": "in_progress", "challenge_id", "prompt"} -- re-dispatch on `prompt`, --classify + {"state": "consumed", "challenge_id", "result"} -- the frozen persisted result, as-is + + An `issued`/`in_progress` record is not trusted at face value: --classify can persist the + canonical audit and then crash BEFORE marking the challenge record `consumed` (the window + between those two writes). Reporting `in_progress` in that window would make the driver + re-dispatch Claude and re-`--classify` a decision that is already final on disk. So this + probes the canonical audit for the same tuple first; a valid one flips the report to + `consumed` (result attached, zero model calls); absent/invalid falls back to `in_progress`. + + Uses the SAME containment as --prepare/--classify: the validated trusted-root epic dir, + id checks, O_NOFOLLOW dir_fd reads, and size-capped record probes (Finding 5/6).""" + if not args.feature: + p.error("--resume-challenge needs --feature") + if args.attempt is None: + p.error("--resume-challenge needs --attempt ") + if isinstance(args.attempt, bool) or not isinstance(args.attempt, int) or args.attempt < 0: + raise ArbiterError("--attempt must be a non-negative integer") + state = _read_state(args.state) + epic_id = state["epic_id"] + if not _id_ok(args.feature): + raise ArbiterError("invalid --feature id: %r" % (args.feature,)) + if _find_feature(state, args.feature) is None: + raise ArbiterError("no feature %r in state" % args.feature) + + epic_dir = validate_epic_dir(args.state, epic_id) + challenge_name = "%s-%s.challenge.json" % (args.feature, args.attempt) + audit_name = "%s-%s.json" % (args.feature, args.attempt) + + try: + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + except OSError: + # No arbiter dir at all -> nothing has EVER been issued for this epic -- absent, with + # no directory-creation side effect (create=False). + print(json.dumps({"state": "absent"})) + return 0 + + try: + rstatus, record = _probe_record(arbiter_fd, challenge_name, CHALLENGE_RECORD_MAX_BYTES) + if rstatus == "absent": + print(json.dumps({"state": "absent"})) + return 0 + if rstatus == "unreadable": + raise ArbiterError("challenge record for %s attempt %s is unreadable/oversized/" + "symlinked -- refusing to resume from a tampered record " + "(fail closed)" % (args.feature, args.attempt)) + + # Finding 4 (same idiom as --classify): only trust a record whose tuple -- including + # the KEYED challenge_id -- is internally consistent. A tampered record is refused, + # never silently trusted into a fabricated resume state. + key = get_challenge_key(arbiter_fd, injected=args.challenge_key) + expected = compute_challenge_id(key, epic_id, args.feature, args.attempt) + tuple_ok = (record.get("challenge_id") == expected + and record.get("epic_id") == epic_id + and record.get("feature") == args.feature + and record.get("attempt") == args.attempt) + if not tuple_ok: + raise ArbiterError("challenge record for %s attempt %s has an inconsistent tuple " + "-- refusing to resume from a tampered record (fail closed)" + % (args.feature, args.attempt)) + + status = record.get("status") + if status in ("issued", "in_progress"): + # Crash window (integration-review finding): --classify can persist the frozen + # audit result and THEN die before durably marking the challenge record + # `consumed` (see the in_progress -> consumed write at the tail of cmd_classify). + # If we reported `in_progress` here regardless, the driver would re-dispatch + # Claude and re-`--classify`, duplicating model work even though the verdict was + # already decided and sitting on disk. So: probe the CANONICAL audit for this + # exact (epic_id, feature, attempt) tuple FIRST. If a valid one exists, the + # verdict is final -- report `consumed` with that persisted result and make ZERO + # model calls. Only when no valid audit exists (the ordinary case: nothing has + # run yet, or classify crashed before writing the audit) do we fall back to + # reporting `in_progress` and handing back the prompt, exactly as before -- + # --classify's own crash-recovery fall-through (no audit yet) or re-emit (audit + # already written) then handles either sub-case on the next attempt. + astatus, audit = _probe_record(arbiter_fd, audit_name, AUDIT_MAX_ONE_BYTES) + if astatus == "ok" and _audit_tuple_matches(audit, epic_id, args.feature, + args.attempt, expected): + print(json.dumps({"state": "consumed", "challenge_id": expected, + "result": audit})) + return 0 + print(json.dumps({"state": "in_progress", "challenge_id": expected, + "prompt": record.get("claude_prompt", "")})) + return 0 + if status == "consumed": + # The verdict was already reached; RE-EMIT it verbatim -- no new Codex/Claude call. + astatus, audit = _probe_record(arbiter_fd, audit_name, AUDIT_MAX_ONE_BYTES) + if astatus != "ok" or not _audit_tuple_matches(audit, epic_id, args.feature, + args.attempt, expected): + raise ArbiterError("challenge for %s attempt %s is consumed but its persisted " + "audit is missing/inconsistent -- refusing to resume (fail " + "closed)" % (args.feature, args.attempt)) + print(json.dumps({"state": "consumed", "challenge_id": expected, "result": audit})) + return 0 + raise ArbiterError("challenge record for %s attempt %s has an unexpected status %r" + % (args.feature, args.attempt, status)) + finally: + os.close(arbiter_fd) + + +def _contained_audit_path(epic_dir, audit_name): + """The absolute audit path, asserted realpath-contained under /arbiter/. + epic_dir is a realpath and the arbiter dir fd was opened O_NOFOLLOW, so this is a + belt-and-suspenders string check for the persisted `audit_path` field. + + Finding 9/#5 (scoped): the final-component symlink and the arbiter-dir symlink are both + defended (O_NOFOLLOW on each). A full anchored openat walk of every epic-dir ANCESTOR — + to defeat an ancestor swapped AFTER realpath validation — is out of the single-user + threat model (the epic tree is user-owned; no second actor swaps a parent mid-call) and + is a v2.11 concern; we keep the O_NOFOLLOW-final-component + arbiter-dir protection.""" + base = os.path.join(epic_dir, "arbiter") + path = os.path.join(base, audit_name) + base_real = os.path.realpath(base) + resolved = os.path.realpath(path) + if resolved != base_real and not resolved.startswith(base_real + os.sep): + raise ArbiterError("audit path escapes the arbiter dir: %s" % path) + return path + + +# --------------------------------------------------------------------------------------- # +# Selftest +# --------------------------------------------------------------------------------------- # + +def _write_fake_codex_stub(path): + script = ( + "#!/bin/sh\n" + 'outfile=""\nprev=""\n' + 'for a in "$@"; do\n' + ' if [ "$prev" = "--output-last-message" ]; then outfile="$a"; fi\n' + ' prev="$a"\ndone\n' + 'if [ -n "$outfile" ] && [ -n "$FAKE_CODEX_SRC" ] && [ -f "$FAKE_CODEX_SRC" ]; then\n' + ' cp "$FAKE_CODEX_SRC" "$outfile"\nfi\n' + 'if [ -n "$FAKE_CODEX_EXIT" ]; then exit "$FAKE_CODEX_EXIT"; fi\nexit 0\n') + with open(path, "w") as fh: + fh.write(script) + os.chmod(path, 0o755) + + +def _make_epic(tmproot, epic_id="epic1", attempts=1, status="failed"): + epic_dir = os.path.join(tmproot, "docs", "superpowers", "execution", "epics", epic_id) + os.makedirs(epic_dir) + state_path = os.path.join(epic_dir, "epic-state.json") + state = {"epic_id": epic_id, "title": "T", "status": "running", + "autonomy": {"stance": "marathon", "max_attempts_per_feature": 2}, + "features": [{"id": "featA", "title": "Feature A", "depends_on": [], "status": status, + "attempts": attempts, "last_error": "boom", "disposition": None}]} + with open(state_path, "w") as fh: + json.dump(state, fh) + return epic_dir, state_path, state + + +def _selftest(): + ok = 0 + fail = 0 + + def check(name, cond): + nonlocal ok, fail + if cond: + ok += 1 + else: + fail += 1 + print(" FAIL %s" % name) + + import io + import shutil + import tempfile as _tempfile + from contextlib import redirect_stdout + + NOW = "2026-07-12T00:00:00+00:00" + FIXED_KEY = "test-fixed-key-0000" + + class _FakeArgs(object): + pass + + def _args(**over): + a = _FakeArgs() + a.state = None + a.feature = None + a.attempt = None + a.challenge = None + a.evidence_file = None + a.claude_ballot = None + a.now = NOW + a.challenge_key = FIXED_KEY + a.capabilities_path = os.path.join(_tempfile.gettempdir(), "no-such-caps.json") + a.config = None + a.explicit_codex_model = None + a.codex_timeout = 10 + a.max_output_bytes = 5000 + a.codex_bin = None + a.supervisor_path = None + for k, v in over.items(): + setattr(a, k, v) + return a + + class _P(object): + def error(self, msg): + raise SystemExit(msg) + + p = _P() + + def run_cmd(fn, a): + buf = io.StringIO() + try: + with redirect_stdout(buf): + rc = fn(a, p) + except ArbiterError as e: + return None, "ERR:" + str(e), 1 + out = buf.getvalue().strip() + return (json.loads(out) if out else None), None, rc + + def run_cmd_raw(fn, a): + """Like run_cmd but returns the UNSTRIPPED raw stdout string (for byte-for-byte + printed==persisted comparisons).""" + buf = io.StringIO() + try: + with redirect_stdout(buf): + rc = fn(a, p) + except ArbiterError as e: + return None, "ERR:" + str(e), 1 + return buf.getvalue(), None, rc + + def _run_main_expect_error(argv): + """Drive main() and report whether it produced an error (argparse SystemExit, or a + nonzero return). Used for the argparse-boundary numeric-bound checks.""" + buf, ebuf = io.StringIO(), io.StringIO() + try: + from contextlib import redirect_stderr + with redirect_stdout(buf), redirect_stderr(ebuf): + rc = main(argv) + return rc != 0 + except SystemExit as e: + return (e.code or 0) != 0 + + # =================================================================================== # + # B1 — capabilities (strict booleans) + family map + # =================================================================================== # + check("B1: absent caps file -> {}", load_capabilities(os.path.join(_tempfile.gettempdir(), "nope.json")) == {}) + d = _tempfile.mkdtemp() + try: + def caps_file(name, obj): + pth = os.path.join(d, name) + with open(pth, "w") as fh: + json.dump(obj, fh) + return pth + check("B1: available+verified -> usable", + codex_available(load_capabilities(caps_file("g.json", {"codex": {"available": True, "exec_flags_verified": True}})))) + check("B1: exec_flags_verified false -> unusable", + not codex_available(load_capabilities(caps_file("h.json", {"codex": {"available": True, "exec_flags_verified": False}})))) + # Finding 9: strict `is True` — a truthy STRING must not enable Codex. + check("B1/F9: string 'true' does NOT enable codex (strict is True)", + not codex_available(load_capabilities(caps_file("s.json", {"codex": {"available": "true", "exec_flags_verified": "true"}})))) + check("B1/F9: int 1 does NOT enable codex (strict is True)", + not codex_available(load_capabilities(caps_file("i.json", {"codex": {"available": 1, "exec_flags_verified": 1}})))) + check("B1: absent codex key -> unusable", + not codex_available(load_capabilities(caps_file("n.json", {"context7": {"available": True}})))) + with open(os.path.join(d, "m.json"), "w") as fh: + fh.write("{not json") + check("B1: malformed JSON -> unusable", not codex_available(load_capabilities(os.path.join(d, "m.json")))) + finally: + shutil.rmtree(d, ignore_errors=True) + + check("B1 family: gpt-5.6-sol -> GPT", model_family("gpt-5.6-sol") == "GPT") + check("B1 family: Gemini -> Gemini", model_family("Gemini 3.1 Pro (High)") == "Gemini") + check("B1 family: claude/opus/sonnet -> Claude", + model_family("claude-opus-4") == "Claude" and model_family("opus") == "Claude" and model_family("sonnet") == "Claude") + check("B1 family: grok -> Grok", model_family("grok-3") == "Grok") + check("B1 family: auto/unknown/empty -> unknown", + model_family("auto") == "unknown" and model_family("x") == "unknown" and model_family(None) == "unknown") + + # =================================================================================== # + # B2/F8 — exact invocation + STRICT parse (no rfind salvage) + # =================================================================================== # + argv = build_codex_invocation("gpt-5.6-sol", "PROMPT", "/tmp/o", "/tmp/e", "/tmp/l", + 90, 20000, codex_bin="codex", + supervisor_path="/x/compound-v-run-with-timeout.py", + python_bin="/usr/bin/python3") + check("B2: exact supervised codex invocation shape", argv == [ + "/usr/bin/python3", "/x/compound-v-run-with-timeout.py", "--timeout", "90", + "--max-output-bytes", "20000", "--stdout", "/tmp/o", "--stderr", "/tmp/e", "--", + "codex", "exec", "--sandbox", "read-only", "--skip-git-repo-check", "--model", + "gpt-5.6-sol", "-c", "model_reasoning_effort=high", "--json", "--output-last-message", + "/tmp/l", "PROMPT"]) + + check("F8: clean single-object verdict parses", + _parse_codex_verdict('{"disposition":"retry_fix","reason":"flaky"}')["disposition"] == "retry_fix") + check("F8: exact codex_hooks deprecation line skipped", + _parse_codex_verdict("[features].codex_hooks is deprecated\n" + '{"disposition":"halt_feature","reason":"bug"}')["disposition"] == "halt_feature") + check("F8: leading prose rejected (no rfind salvage)", + _parse_codex_verdict('here is my answer {"disposition":"halt_epic"}') is None) + check("F8: trailing prose rejected", + _parse_codex_verdict('{"disposition":"halt_epic"} thanks!') is None) + check("F8: two objects rejected", + _parse_codex_verdict('{"disposition":"retry_fix"}{"disposition":"halt_epic"}') is None) + check("F8: garbage prose rejected", _parse_codex_verdict("just prose") is None) + check("F8: invalid disposition rejected", _parse_codex_verdict('{"disposition":"vibes"}') is None) + check("F8: multiline reason rejected", + _parse_codex_verdict('{"disposition":"retry_fix","reason":"line1\\nline2"}') is None) + check("F8: non-string reason rejected", + _parse_codex_verdict('{"disposition":"retry_fix","reason":123}') is None) + check("F8: empty/None -> None", _parse_codex_verdict("") is None and _parse_codex_verdict(None) is None) + + # poll_codex through the REAL supervisor + fake codex stub + wd = _tempfile.mkdtemp() + try: + stub = os.path.join(wd, "fake-codex.sh") + _write_fake_codex_stub(stub) + + src_ok = os.path.join(wd, "ok.json") + with open(src_ok, "w") as fh: + fh.write('{"disposition":"retry_fix","reason":"flaky test"}') + v, r = poll_codex("gpt-5.6-sol", "prompt", timeout_sec=10, max_output_bytes=5000, + codex_bin=stub, env=dict(os.environ, FAKE_CODEX_SRC=src_ok)) + check("B2: real supervisor + stub -> verdict", v is not None and v["disposition"] == "retry_fix") + + src_g = os.path.join(wd, "g.txt") + with open(src_g, "w") as fh: + fh.write("prose not json") + v2, r2 = poll_codex("gpt-5.6-sol", "prompt", timeout_sec=10, max_output_bytes=5000, + codex_bin=stub, env=dict(os.environ, FAKE_CODEX_SRC=src_g)) + check("B2: garbled reply dropped, not a fabricated halt", v2 is None and "dropped" in r2) + + v4, r4 = poll_codex("gpt-5.6-sol", "prompt", timeout_sec=10, max_output_bytes=5000, + codex_bin=stub, env=dict(os.environ, FAKE_CODEX_EXIT="1")) + check("B2: nonzero supervisor exit dropped", v4 is None) + finally: + shutil.rmtree(wd, ignore_errors=True) + + # Finding 6: an oversized retained artifact is DELETED (not left above cap). + od = _tempfile.mkdtemp() + try: + big = os.path.join(od, "big.txt") + with open(big, "w") as fh: + fh.write("A" * 50000) + # default (delete_oversized=False, e.g. a user file) -> dropped but NOT deleted + text0, err0 = _read_capped_regular(big, 1000) + check("F6: oversized user artifact -> dropped, NOT deleted (default)", + text0 is None and os.path.exists(big)) + # private-temp path (delete_oversized=True) -> dropped AND deleted + text, err = _read_capped_regular(big, 1000, delete_oversized=True) + check("F6: oversized private artifact -> dropped", text is None and "oversized" in err) + check("F6: oversized private artifact is DELETED from disk", not os.path.exists(big)) + small = os.path.join(od, "small.txt") + with open(small, "w") as fh: + fh.write("hi") + t2, e2 = _read_capped_regular(small, 1000) + check("F6: within-cap artifact read normally", t2 == "hi" and e2 is None) + # symlinked artifact rejected by O_NOFOLLOW + link = os.path.join(od, "link.txt") + os.symlink(small, link) + t3, e3 = _read_capped_regular(link, 1000) + check("F6: symlinked artifact rejected (O_NOFOLLOW)", t3 is None) + finally: + shutil.rmtree(od, ignore_errors=True) + + # =================================================================================== # + # F2 — redaction hardening (broadened patterns, fail-closed) + # =================================================================================== # + check("F2 redact: closed PEM private key", + "[REDACTED:PRIVATE_KEY]" in redact("x\n-----BEGIN RSA PRIVATE KEY-----\nMII\n-----END RSA PRIVATE KEY-----\ny", 10000)) + check("F2 redact: PGP private key block", + "[REDACTED:PRIVATE_KEY]" in redact("-----BEGIN PGP PRIVATE KEY BLOCK-----\nabc\n-----END PGP PRIVATE KEY BLOCK-----", 10000)) + check("F2 redact: UNCLOSED private-key block -> FAIL CLOSED (None -> omit whole evidence)", + redact("secret start\n-----BEGIN RSA PRIVATE KEY-----\nMIInoEnd", 10000) is None) + _folded = redact("Authorization: Bearer abc\n def.ghi\nHost: x", 10000) + check("F2 redact: folded Authorization header (continuation line)", + "abc" not in _folded and "def.ghi" not in _folded and "[REDACTED]" in _folded) + check("F2 redact: short labelled secret CLIENT_SECRET=", + "hunter2xyz" not in redact("CLIENT_SECRET=hunter2xyz", 10000)) + check("F2 redact: password= labelled", + "p@ssw0rd" not in redact("password=p@ssw0rd", 10000)) + check("F2 redact: api_key: labelled", + "sekret" not in redact("api_key: sekret", 10000)) + check("F2 redact: LABEL=\"...multiline...\" quoted secret", + "toplinesecret" not in redact('token="toplinesecret\ncontinued"', 10000)) + _url = redact("https://user:pa:ss:wd@host/x", 10000) + check("F2 redact: URL creds WITH ':' in password", + "pa:ss:wd" not in _url and "[REDACTED]@host" in _url) + check("F2 redact: JWT token", + "[REDACTED:JWT]" in redact("t=eyJhbGciOi.eyJzdWIiOi.SIG_nature_here now", 10000)) + check("F2 redact: long opaque token", + "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCD" not in redact("k=ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCD", 10000)) + + class _Boom(object): + def __str__(self): + raise RuntimeError("boom") + check("F2 redact: FAIL CLOSED on internal exception", redact(_Boom(), 1000) is None) + r_cap = redact("B" * 5000, 100) + check("F2 redact: byte cap RESERVES marker room (final <= cap)", + len(r_cap.encode("utf-8")) <= 100) + + # sanitize_ballot_fields + check("F2 sanitize: multiline reason -> drop", sanitize_ballot_fields("a\nb", None) is None) + check("F2 sanitize: non-string reason -> drop", sanitize_ballot_fields(123, None) is None) + check("F2 sanitize: multiline evidence -> drop", sanitize_ballot_fields("ok", "a\nb") is None) + sr = sanitize_ballot_fields("token is ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCD", None) + check("F2 sanitize: re-redacts a leaked token in reason", + sr is not None and "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCD" not in sr[0]) + + # =================================================================================== # + # F1 — TOCTOU-safe contained evidence read + # =================================================================================== # + e1 = _tempfile.mkdtemp() + try: + adir = os.path.join(e1, "attempt") + os.makedirs(adir) + base_fd = os.open(adir, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) + try: + with open(os.path.join(adir, "ev.txt"), "w") as fh: + fh.write("gate output ok") + t, err = read_contained_evidence(base_fd, "ev.txt", 100000) + check("F1: valid contained file read", t == "gate output ok" and err is None) + check("F1: absolute /etc/passwd rejected", + read_contained_evidence(base_fd, "/etc/passwd", 100000)[0] is None) + check("F1: '..' traversal rejected", + read_contained_evidence(base_fd, "../x", 100000)[0] is None) + check("F1: absolute-inside path rejected (isabs)", + read_contained_evidence(base_fd, os.path.join(adir, "ev.txt"), 100000)[0] is None) + outside = _tempfile.mkdtemp() + try: + with open(os.path.join(outside, "secret.txt"), "w") as fh: + fh.write("OUTSIDE SECRET") + os.symlink(os.path.join(outside, "secret.txt"), os.path.join(adir, "esc")) + te, ee = read_contained_evidence(base_fd, "esc", 100000) + check("F1: escaping symlink rejected BEFORE read (content never read)", + te is None and "OUTSIDE" not in (te or "")) + # intermediate-component symlink swap: a symlinked subdir is rejected too. + os.symlink(outside, os.path.join(adir, "subdirlink")) + ts, es = read_contained_evidence(base_fd, os.path.join("subdirlink", "secret.txt"), 100000) + check("F1: symlinked intermediate dir rejected (walk uses O_NOFOLLOW)", ts is None) + finally: + shutil.rmtree(outside, ignore_errors=True) + finally: + os.close(base_fd) + finally: + shutil.rmtree(e1, ignore_errors=True) + + # =================================================================================== # + # F5 — trusted-root epic-dir containment + symlinked arbiter dir rejected + # =================================================================================== # + tr = _tempfile.mkdtemp() + try: + epic_dir, state_path, _ = _make_epic(tr) + good = validate_epic_dir(state_path, "epic1") + check("F5: a state under execution/epics/ validates", good.endswith("/execution/epics/epic1")) + bad_dir = os.path.join(tr, "docs", "superpowers", "execution", "epics", "epic1") + try: + validate_epic_dir(state_path, "epicMISMATCH") + check("F5: epic_id/dir-name mismatch rejected", False) + except ArbiterError: + check("F5: epic_id/dir-name mismatch rejected", True) + loose = os.path.join(tr, "loose") + os.makedirs(loose) + lp = os.path.join(loose, "epic-state.json") + with open(lp, "w") as fh: + fh.write("{}") + try: + validate_epic_dir(lp, "loose") + check("F5: a state outside execution/epics rejected", False) + except ArbiterError: + check("F5: a state outside execution/epics rejected", True) + outside2 = _tempfile.mkdtemp() + try: + os.symlink(outside2, os.path.join(bad_dir, "arbiter")) + try: + fd = open_arbiter_dir_fd(good, create=False) + os.close(fd) + check("F5: symlinked arbiter dir rejected", False) + except OSError: + check("F5: symlinked arbiter dir rejected", True) + finally: + shutil.rmtree(outside2, ignore_errors=True) + finally: + shutil.rmtree(tr, ignore_errors=True) + + # =================================================================================== # + # B4 — truth table (complete), collapse, external-family cap (F7) + # =================================================================================== # + def vb(family, disposition): + return {"source": "x", "family": family, "disposition": disposition, "valid": True, + "counted": True, "reason": "", "evidence": None} + + d0, c0, _ = aggregate_dispositions([]) + check("B4: zero ballots -> halt_feature", d0 == "halt_feature" and c0 is False) + + for disp, exp in [("retry_fix", "retry_fix"), ("halt_feature", "halt_feature"), + ("blocked_external", "blocked_external"), ("halt_epic", "halt_feature")]: + d, c, _ = aggregate_dispositions([vb("GPT", disp)]) + check("B4 single GPT %s -> %s" % (disp, exp), d == exp and c is False) + + pairs = [ + (("retry_fix", "retry_fix"), "retry_fix"), + (("retry_fix", "halt_feature"), "halt_feature"), + (("retry_fix", "halt_epic"), "halt_feature"), + (("retry_fix", "blocked_external"), "halt_feature"), + (("halt_feature", "halt_feature"), "halt_feature"), + (("halt_feature", "halt_epic"), "halt_feature"), + (("halt_feature", "blocked_external"), "halt_feature"), + (("halt_epic", "halt_epic"), "halt_epic"), + (("halt_epic", "blocked_external"), "halt_feature"), + (("blocked_external", "blocked_external"), "blocked_external"), + ] + for (a1, a2), exp in pairs: + d, c, _ = aggregate_dispositions([vb("GPT", a1), vb("Gemini", a2)]) + check("B4 pair (%s,%s) -> %s" % (a1, a2, exp), d == exp) + + d, c, _ = aggregate_dispositions([vb("GPT", "blocked_external"), vb("Gemini", "blocked_external")]) + check("B4: 2 distinct external families blocked_external -> CONFIRMED", d == "blocked_external" and c is True) + d, c, _ = aggregate_dispositions([vb("GPT", "blocked_external"), vb("Claude", "blocked_external")]) + check("B4: Codex+Claude blocked_external -> SUSPECTED (Claude excluded)", d == "blocked_external" and c is False) + + # F7 — Claude-only / no-external fallback caps halt_epic AND blocked_external. + d, c, _ = aggregate_dispositions([vb("Claude", "halt_epic")]) + check("F7: lone Claude halt_epic -> halt_feature", d == "halt_feature") + d, c, _ = aggregate_dispositions([vb("Claude", "blocked_external")]) + check("F7: lone Claude blocked_external -> halt_feature (cannot survive)", d == "halt_feature") + d, c, _ = aggregate_dispositions([vb("Claude", "blocked_external"), vb("Claude", "halt_epic")]) + check("F7: Claude-only pair with no external family caps to halt_feature", d == "halt_feature") + d, c, _ = aggregate_dispositions([vb("unknown", "halt_epic")]) + check("F7: unknown-only halt_epic -> halt_feature (no external family)", d == "halt_feature") + + collapsed = _collapse_same_family([ + {"source": "codex", "family": "GPT", "disposition": "retry_fix", "valid": True, "reason": "", "evidence": None}, + {"source": "claude", "family": "GPT", "disposition": "halt_epic", "valid": True, "reason": "", "evidence": None}]) + counted = [b for b in collapsed if b["counted"]] + check("B4 collapse: one counted, more-conservative wins", len(counted) == 1 and counted[0]["disposition"] == "halt_epic") + check("B4 collapse: loser visible+uncounted", any(not b["counted"] for b in collapsed)) + + # =================================================================================== # + # F3 — challenge cannot be resurrected (monotonic, immutable record) + # =================================================================================== # + ch = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(ch, attempts=1) + _, e, rc = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=5)) + check("F3: prepare rejects a future attempt (attempt != current)", e is not None and "monotonic" in e) + _, e, rc = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=0)) + check("F3: prepare rejects a past attempt", e is not None) + out, e, rc = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + check("F3: valid prepare issues", out is not None and rc == 0) + challenge_id = out["challenge_id"] + check("F3: challenge id is keyed+deterministic", + challenge_id == compute_challenge_id(FIXED_KEY, "epic1", "featA", 1)) + out2, e2, rc2 = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + check("F3: re-prepare while issued is idempotent", out2 is not None and out2["challenge_id"] == challenge_id) + finally: + shutil.rmtree(ch, ignore_errors=True) + + # =================================================================================== # + # E2E — classify: Codex-unavailable + valid Claude ballot, F4 whole-panel drop, + # replay idempotency, retry-cap mask, F10 frozen round-trip, F9 controlled errors. + # =================================================================================== # + e2e = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(e2e, attempts=1) + + out, e, rc = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + challenge_id = out["challenge_id"] + + ballot = os.path.join(e2e, "ballot.json") + with open(ballot, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 1, + "challenge_id": challenge_id, "disposition": "retry_fix", + "reason": "flaky"}, fh) + + res, e, rc = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=challenge_id, claude_ballot=ballot)) + check("E2E: classify exits 0", rc == 0 and res is not None) + check("E2E: codex-unavailable + single Claude retry_fix -> retry_fix", res["disposition"] == "retry_fix") + check("E2E: confirmed always false", res["confirmed"] is False) + check("E2E: audit_path set + file exists", res["audit_path"] and os.path.isfile(res["audit_path"])) + with open(res["audit_path"]) as fh: + persisted = json.load(fh) + for k in ("disposition", "confirmed", "reason", "evidence", "ballots", + "families_present", "families_agreeing", "attempt", "audit_path", "challenge_id"): + check("F10: persisted audit carries %r" % k, k in persisted) + check("F10: persisted audit_path == printed audit_path", persisted["audit_path"] == res["audit_path"]) + check("F10: families_present recorded (Claude)", persisted["families_present"] == ["Claude"]) + + res2, e2, rc2 = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=challenge_id, claude_ballot=ballot)) + check("F4 replay: consumed challenge re-emits the SAME persisted audit (idempotent)", + res2 is not None and res2["disposition"] == "retry_fix" and res2["audit_path"] == res["audit_path"]) + + res3, e3, rc3 = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge="ch-bogus", claude_ballot=ballot)) + check("F4: invalid challenge -> whole panel dropped -> halt_feature", + res3 is not None and res3["disposition"] == "halt_feature" and res3["ballots"] == [] + and "panel dropped" in res3["reason"]) + + state["features"][0]["attempts"] = 2 + with open(state_path, "w") as fh: + json.dump(state, fh) + out_c, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=2)) + cid2 = out_c["challenge_id"] + ballot2 = os.path.join(e2e, "ballot2.json") + with open(ballot2, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 2, + "challenge_id": cid2, "disposition": "retry_fix", "reason": "again?"}, fh) + res4, e4, rc4 = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid2, claude_ballot=ballot2)) + check("E2E retry-cap: classify still runs at the cap (exit 0)", rc4 == 0) + check("E2E retry-cap: retry_fix masked to halt_feature at cap", + res4["disposition"] == "halt_feature" and "cap" in res4["reason"]) + + res5, e5, rc5 = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=challenge_id, claude_ballot=ballot)) + check("F4 stale-across-attempts: old challenge id at new attempt -> panel dropped", + res5 is not None and res5["disposition"] == "halt_feature") + + # evidence wired end-to-end + state["features"][0]["attempts"] = 3 + with open(state_path, "w") as fh: + json.dump(state, fh) + out_e, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=3)) + cid3 = out_e["challenge_id"] + ev_dir = os.path.join(epic_dir, "arbiter", "featA-3") + with open(os.path.join(ev_dir, "gate.txt"), "w") as fh: + fh.write("AC3 failed. secret=ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCD") + ballot3 = os.path.join(e2e, "ballot3.json") + with open(ballot3, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 3, + "challenge_id": cid3, "disposition": "halt_feature", "reason": "bug"}, fh) + res6, e6, rc6 = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid3, claude_ballot=ballot3, + evidence_file="gate.txt")) + check("E2E evidence: contained evidence file -> classify exits 0", rc6 == 0) + + state["features"][0]["attempts"] = 4 + with open(state_path, "w") as fh: + json.dump(state, fh) + out_e2, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=4)) + cid4 = out_e2["challenge_id"] + ballot4 = os.path.join(e2e, "ballot4.json") + with open(ballot4, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 4, + "challenge_id": cid4, "disposition": "halt_feature"}, fh) + res7, e7, rc7 = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid4, claude_ballot=ballot4, + evidence_file="/etc/passwd")) + check("E2E evidence: an absolute/escaping evidence path doesn't crash classify", rc7 == 0) + + bad_state = os.path.join(epic_dir, "bad-state.json") + with open(bad_state, "w") as fh: + json.dump({"features": [{"id": "featA"}]}, fh) + _, e8, rc8 = run_cmd(cmd_classify, _args(state=bad_state, feature="featA", challenge="x")) + check("F9: malformed state -> controlled ArbiterError (no traceback)", + e8 is not None and "schema" in e8) + none_state = os.path.join(epic_dir, "none-id.json") + with open(none_state, "w") as fh: + json.dump({"epic_id": None, "features": [{"id": "featA"}]}, fh) + _, e9, rc9 = run_cmd(cmd_classify, _args(state=none_state, feature="featA", challenge="x")) + check("F9: epic_id None rejected (no str(None)=='None' bypass)", e9 is not None) + finally: + shutil.rmtree(e2e, ignore_errors=True) + + # =================================================================================== # + # F3/F4 durable state machine: issued -> consumed, no resurrection + # =================================================================================== # + dm = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(dm, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + cid = out["challenge_id"] + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + rec = _read_json_at(arbiter_fd, "featA-1.challenge.json", CHALLENGE_RECORD_MAX_BYTES) + finally: + os.close(arbiter_fd) + check("F3: fresh record status == issued", rec["status"] == "issued") + run_cmd(cmd_classify, _args(state=state_path, feature="featA", challenge=cid)) + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + rec2 = _read_json_at(arbiter_fd, "featA-1.challenge.json", CHALLENGE_RECORD_MAX_BYTES) + finally: + os.close(arbiter_fd) + check("F4: record durably consumed after classify", rec2["status"] == "consumed") + _, e, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + check("F3: re-prepare a CONSUMED attempt rejected", e is not None and "consumed" in e) + finally: + shutil.rmtree(dm, ignore_errors=True) + + # =================================================================================== # + # ROUND 2 — Codex second security review + # =================================================================================== # + + # R2#1 — a bogus challenge must NOT overwrite the canonical audit; a later CORRECT replay + # still re-emits the ORIGINAL valid verdict. + r1 = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(r1, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + cid = out["challenge_id"] + ballot = os.path.join(r1, "b.json") + with open(ballot, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 1, + "challenge_id": cid, "disposition": "retry_fix", "reason": "flaky"}, fh) + good, _, _ = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid, claude_ballot=ballot)) + check("R2#1: valid classify -> retry_fix", good["disposition"] == "retry_fix") + canonical = good["audit_path"] + # now fire a BOGUS challenge at the same attempt + bad, _, _ = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge="ch-attacker", claude_ballot=ballot)) + check("R2#1: bogus challenge dropped -> halt_feature", bad["disposition"] == "halt_feature") + with open(canonical) as fh: + still = json.load(fh) + check("R2#1: canonical audit NOT clobbered by the bogus challenge", + still["disposition"] == "retry_fix") + check("R2#1: the drop event went to a separate .rejected.json path", + bad["audit_path"] is not None and bad["audit_path"].endswith(".rejected.json")) + # a later CORRECT replay still re-emits the original valid verdict + replay, _, _ = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid, claude_ballot=ballot)) + check("R2#1: correct replay re-emits the original valid verdict", + replay["disposition"] == "retry_fix" and replay["audit_path"] == canonical) + finally: + shutil.rmtree(r1, ignore_errors=True) + + # R2#2 — unclosed quoted labelled secret fails closed (single + double quote, multiline). + check("R2#2: unclosed double-quoted labelled secret -> fail closed", + redact('token="shortsecret\ncontinued', 10000) is None) + check("R2#2: unclosed single-quoted labelled secret -> fail closed", + redact("password='abc def", 10000) is None) + check("R2#2: a CLOSED quoted labelled secret still redacts (not fail-closed)", + redact('token="abc"', 10000) is not None and "abc" not in redact('token="abc"', 10000)) + check("R2#2: unclosed multiline api_key -> fail closed", + redact('api_key = "line1\nline2 no close', 10000) is None) + + # R2#3 — a malformed-ballot diagnostic must not leak an untrusted value into output/audit. + r3 = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(r3, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + cid = out["challenge_id"] + leak = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCD" # a token-shaped "secret" + bad_ballot = os.path.join(r3, "bad.json") + with open(bad_ballot, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 1, + "challenge_id": cid, "disposition": leak}, fh) # invalid disposition = secret + res, _, _ = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid, claude_ballot=bad_ballot)) + blob = json.dumps(res) + check("R2#3: invalid-disposition value NOT echoed into the result", leak not in blob) + with open(res["audit_path"]) as fh: + persisted_blob = fh.read() + check("R2#3: invalid-disposition value NOT written to the audit", leak not in persisted_blob) + finally: + shutil.rmtree(r3, ignore_errors=True) + + # R2#4 — an oversized caller-supplied ballot file must be DROPPED, never DELETED. + r4 = _tempfile.mkdtemp() + try: + big_ballot = os.path.join(r4, "big.json") + with open(big_ballot, "w") as fh: + fh.write("{" + "\"x\":\"" + ("A" * (BALLOT_FILE_MAX_BYTES + 100)) + "\"}") + cb, reason = load_claude_ballot(big_ballot, "e", "f", 1, "c") + check("R2#4: oversized ballot dropped", cb is None) + check("R2#4: oversized USER ballot file NOT deleted", os.path.exists(big_ballot)) + finally: + shutil.rmtree(r4, ignore_errors=True) + + # R2#5 — printed == persisted; hard byte cap after trim. + r5 = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(r5, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + cid = out["challenge_id"] + ballot = os.path.join(r5, "b.json") + with open(ballot, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 1, + "challenge_id": cid, "disposition": "retry_fix", + "reason": "x" * 500}, fh) + res, _, _ = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid, claude_ballot=ballot)) + with open(res["audit_path"]) as fh: + persisted = json.load(fh) + check("R2#5: printed result == persisted audit (byte-identical)", + json.dumps(res, sort_keys=True) == json.dumps(persisted, sort_keys=True)) + # _finalize_result stub path when a result is huge + huge = {"disposition": "halt_feature", "confirmed": False, "reason": "r", + "evidence": None, "ballots": [{"reason": "z" * 300000}], + "families_present": [], "families_agreeing": [], "challenge_id": "c", + "epic_id": "e", "feature": "f", "attempt": 1, "audit_path": "/x", "recorded_at": "t"} + fin = _finalize_result(huge) + check("R2#5: an over-cap result is trimmed/stubbed under the hard byte cap", + len(json.dumps(fin).encode("utf-8")) <= AUDIT_MAX_ONE_BYTES) + finally: + shutil.rmtree(r5, ignore_errors=True) + + # R2#6 — an existing-but-unreadable record must fail closed on --prepare (not resurrect). + r6 = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(r6, attempts=1) + run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + # corrupt the persisted challenge record in place + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + fd = os.open("featA-1.challenge.json", os.O_WRONLY | os.O_TRUNC, dir_fd=arbiter_fd) + os.write(fd, b"{ this is not valid json") + os.close(fd) + finally: + os.close(arbiter_fd) + _, e, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + check("R2#6: unreadable existing record -> prepare fails closed (no resurrection)", + e is not None and "unreadable" in e) + # probe helper distinguishes absent vs unreadable + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + st_absent, _ = _probe_record(arbiter_fd, "no-such.challenge.json", 4096) + st_bad, _ = _probe_record(arbiter_fd, "featA-1.challenge.json", 4096) + finally: + os.close(arbiter_fd) + check("R2#6: probe distinguishes absent vs unreadable", + st_absent == "absent" and st_bad == "unreadable") + finally: + shutil.rmtree(r6, ignore_errors=True) + + # R2#7 — an in_progress record with an existing audit re-emits WITHOUT a second egress. + r7 = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(r7, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + cid = out["challenge_id"] + ballot = os.path.join(r7, "b.json") + with open(ballot, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 1, + "challenge_id": cid, "disposition": "halt_epic", "reason": "systemic"}, fh) + first, _, _ = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid, claude_ballot=ballot)) + # manually rewind the record to in_progress (simulate a crash after audit, before consume) + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + rec = _read_json_at(arbiter_fd, "featA-1.challenge.json", CHALLENGE_RECORD_MAX_BYTES) + rec["status"] = "in_progress" + rec.pop("consumed_at", None) + _write_json_at(arbiter_fd, "featA-1.challenge.json", rec) + finally: + os.close(arbiter_fd) + # a DIFFERENT ballot on replay must be IGNORED — we re-emit the existing audit. + ballot2 = os.path.join(r7, "b2.json") + with open(ballot2, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 1, + "challenge_id": cid, "disposition": "retry_fix", "reason": "changed"}, fh) + again, _, _ = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid, claude_ballot=ballot2)) + check("R2#7: in_progress+existing audit re-emits the ORIGINAL verdict (no re-egress)", + again["disposition"] == first["disposition"] == "halt_feature") # halt_epic single-ballot -> halt_feature + finally: + shutil.rmtree(r7, ignore_errors=True) + + # R2#8 — input bounds: overlong id, numeric arg validation, marker-vs-cap. + check("R2#8: overlong id rejected by _id_ok", not _id_ok("a" * (MAX_ID_LEN + 1))) + check("R2#8: max-len id accepted", _id_ok("a" * MAX_ID_LEN)) + check("R2#8: _cap_bytes_with_marker never exceeds a tiny cap", + len(_cap_bytes_with_marker("Z" * 100, 3).encode("utf-8")) <= 3) + check("R2#8: overlong feature id -> controlled ArbiterError (no crash)", + run_cmd(cmd_prepare, _args(state=os.path.join(_tempfile.gettempdir(), "x"), + feature="f" * (MAX_ID_LEN + 1), attempt=1))[1] is not None) + check("R2#8: main() rejects a non-positive --max-output-bytes", + _run_main_expect_error(["--classify", "--max-output-bytes", "0", "--feature", "f", + "--challenge", "c", "--state", "/nope"])) + check("R2#8: main() rejects a non-positive --codex-timeout", + _run_main_expect_error(["--classify", "--codex-timeout", "0", "--feature", "f", + "--challenge", "c", "--state", "/nope"])) + + # =================================================================================== # + # ROUND 3 — Codex third security review + # =================================================================================== # + + # R3#1 — an ESCAPED closing quote does NOT terminate a labelled secret -> fail closed. + check("R3#1: token=\"secret\\\" (escaped quote, still open) -> fail closed", + redact('token="secret\\"', 10000) is None) + check("R3#1: single-quote escaped, still open -> fail closed", + redact("password='secret\\'", 10000) is None) + check("R3#1: token=\"secret\\\"\" (escaped THEN real close) -> handled (not None), redacted", + redact('token="secret\\""', 10000) is not None + and "secret" not in redact('token="secret\\""', 10000)) + check("R3#1: a plain closed value still redacts", + redact('token="plainval"', 10000) is not None and "plainval" not in redact('token="plainval"', 10000)) + check("R3#1: _has_unescaped_quote respects backslash escaping", + _has_unescaped_quote('abc"', 0, '"') is True + and _has_unescaped_quote('abc\\"', 0, '"') is False + and _has_unescaped_quote('abc\\""', 0, '"') is True) + + # R3#2 — oversized --challenge is rejected at the boundary; the stub is ALWAYS under cap. + r31 = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(r31, attempts=1) + _, e, _ = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge="c" * (MAX_ID_LEN + 1))) + check("R3#2: oversized --challenge rejected (controlled error)", e is not None and "too long" in e) + # _finalize_result stub stays under cap even if challenge_id/epic_id are pathologically huge. + huge = {"disposition": "halt_feature", "confirmed": False, "reason": "r", "evidence": None, + "ballots": [{"reason": "z" * 300000}], "families_present": [], "families_agreeing": [], + "challenge_id": "C" * 300000, "epic_id": "E" * 300000, "feature": "F" * 300000, + "attempt": 1, "audit_path": "/p" * 100000, "recorded_at": "t"} + fin = _finalize_result(huge) + check("R3#2: finalized stub is under the hard byte cap despite huge id fields", + len(json.dumps(fin).encode("utf-8")) <= AUDIT_MAX_ONE_BYTES) + finally: + shutil.rmtree(r31, ignore_errors=True) + + # R3#3 — printed bytes == persisted file bytes EXACTLY (not just parsed-equal). + r33 = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(r33, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + cid = out["challenge_id"] + ballot = os.path.join(r33, "b.json") + with open(ballot, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 1, + "challenge_id": cid, "disposition": "retry_fix", "reason": "flaky"}, fh) + raw_stdout, _, _ = run_cmd_raw(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid, claude_ballot=ballot)) + audit_path = json.loads(raw_stdout)["audit_path"] + with open(audit_path, "r") as fh: + persisted_bytes = fh.read() + check("R3#3: printed stdout bytes == persisted audit file bytes (exact)", + raw_stdout == persisted_bytes) + # crash-recovery re-emit is also byte-exact + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + rec = _read_json_at(arbiter_fd, "featA-1.challenge.json", CHALLENGE_RECORD_MAX_BYTES) + rec["status"] = "in_progress" + _write_json_at(arbiter_fd, "featA-1.challenge.json", rec) + finally: + os.close(arbiter_fd) + raw2, _, _ = run_cmd_raw(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid, claude_ballot=ballot)) + check("R3#3: crash-recovery re-emit bytes == persisted file bytes (exact)", + raw2 == persisted_bytes) + finally: + shutil.rmtree(r33, ignore_errors=True) + + # =================================================================================== # + # C1 — --resume-challenge: idempotent read-only crash-recovery lookup (v2.10 fix) + # =================================================================================== # + + # C1a — absent: no arbiter dir / no record at all -> {"state": "absent"}. + c1a = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1a, attempts=1) + res, e, rc = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=1)) + check("C1a: absent (no arbiter dir yet) -> {'state':'absent'}", + rc == 0 and res == {"state": "absent"}) + # Also absent when the arbiter dir exists (from an unrelated prior attempt) but this + # exact (feature, attempt) record was never issued. + run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + res_other, _, rc_other = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=99)) + check("C1a: absent for an unissued attempt even once the arbiter dir exists", + rc_other == 0 and res_other == {"state": "absent"}) + finally: + shutil.rmtree(c1a, ignore_errors=True) + + # C1b — issued -> reported "in_progress" with the SAME prompt --prepare issued. + c1b = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1b, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + res, e, rc = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=1)) + check("C1b: issued challenge -> state in_progress", rc == 0 and res["state"] == "in_progress") + check("C1b: in_progress carries the SAME challenge_id", res["challenge_id"] == out["challenge_id"]) + check("C1b: in_progress carries the SAME prompt --prepare issued", res["prompt"] == out["prompt"]) + finally: + shutil.rmtree(c1b, ignore_errors=True) + + # C1c — a durably in_progress record (classify claimed it, crashed before an audit) also + # reports "in_progress" (the driver re-dispatches and re-classifies; --classify already + # handles that fall-through safely). + c1c = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1c, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + rec = _read_json_at(arbiter_fd, "featA-1.challenge.json", CHALLENGE_RECORD_MAX_BYTES) + rec["status"] = "in_progress" + _write_json_at(arbiter_fd, "featA-1.challenge.json", rec) + finally: + os.close(arbiter_fd) + res, _, rc = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=1)) + check("C1c: in_progress-with-no-audit-yet -> state in_progress", + rc == 0 and res["state"] == "in_progress" and res["challenge_id"] == out["challenge_id"]) + finally: + shutil.rmtree(c1c, ignore_errors=True) + + # C1d — consumed -> re-emits the FROZEN persisted result verbatim, with ZERO model calls + # (asserted via monkeypatching poll_codex to fail loudly if ever invoked). + c1d = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1d, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + cid = out["challenge_id"] + ballot = os.path.join(c1d, "b.json") + with open(ballot, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 1, + "challenge_id": cid, "disposition": "retry_fix", "reason": "flaky"}, fh) + classified, _, _ = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid, claude_ballot=ballot)) + with open(classified["audit_path"]) as fh: + persisted = json.load(fh) + + module_ns = globals() + _orig_poll_codex = module_ns["poll_codex"] + _poll_called = {"v": False} + + def _boom_poll_codex(*a, **kw): + _poll_called["v"] = True + raise AssertionError("poll_codex must NEVER be called by --resume-challenge") + + module_ns["poll_codex"] = _boom_poll_codex + try: + res, e, rc = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=1)) + finally: + module_ns["poll_codex"] = _orig_poll_codex + + check("C1d: consumed -> state consumed, no crash", rc == 0 and res is not None + and res["state"] == "consumed") + check("C1d: consumed carries the SAME challenge_id", res is not None and res["challenge_id"] == cid) + check("C1d: ZERO model calls on a consumed resume", _poll_called["v"] is False) + check("C1d: re-emitted result is the IDENTICAL frozen persisted result", + res is not None + and json.dumps(res["result"], sort_keys=True) == json.dumps(persisted, sort_keys=True)) + finally: + shutil.rmtree(c1d, ignore_errors=True) + + # C1e — a tampered / oversized / symlinked challenge record -> controlled ArbiterError, + # never a crash (traceback) and never a fabricated resume state. + c1e = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1e, attempts=1) + run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + fd = os.open("featA-1.challenge.json", os.O_WRONLY | os.O_TRUNC, dir_fd=arbiter_fd) + os.write(fd, b"{ this is not valid json") + os.close(fd) + finally: + os.close(arbiter_fd) + _, e, rc = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=1)) + check("C1e: corrupted-JSON record -> controlled ArbiterError, not a crash", + e is not None and e.startswith("ERR:") and rc == 1) + finally: + shutil.rmtree(c1e, ignore_errors=True) + + c1f = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1f, attempts=1) + run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + fd = os.open("featA-1.challenge.json", os.O_WRONLY | os.O_TRUNC, dir_fd=arbiter_fd) + os.write(fd, b'{"pad":"' + b"A" * (CHALLENGE_RECORD_MAX_BYTES + 1000) + b'"}') + os.close(fd) + finally: + os.close(arbiter_fd) + _, e, rc = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=1)) + check("C1e: oversized record -> controlled ArbiterError, not a crash", + e is not None and e.startswith("ERR:") and rc == 1) + finally: + shutil.rmtree(c1f, ignore_errors=True) + + c1g = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1g, attempts=1) + run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + outside_g = _tempfile.mkdtemp() + try: + decoy = os.path.join(outside_g, "decoy.json") + with open(decoy, "w") as fh: + json.dump({"status": "issued"}, fh) + rec_path = os.path.join(epic_dir, "arbiter", "featA-1.challenge.json") + os.unlink(rec_path) + os.symlink(decoy, rec_path) + _, e, rc = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=1)) + check("C1e: symlinked record -> controlled ArbiterError, not a crash", + e is not None and e.startswith("ERR:") and rc == 1) + finally: + shutil.rmtree(outside_g, ignore_errors=True) + finally: + shutil.rmtree(c1g, ignore_errors=True) + + # C1h — a record whose tuple has been tampered with (fields don't match the filename's + # (feature, attempt) or the keyed challenge_id) is also refused, fail-closed. + c1h = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1h, attempts=1) + run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + rec = _read_json_at(arbiter_fd, "featA-1.challenge.json", CHALLENGE_RECORD_MAX_BYTES) + rec["challenge_id"] = "ch-attacker0000000" + _write_json_at(arbiter_fd, "featA-1.challenge.json", rec) + finally: + os.close(arbiter_fd) + _, e, rc = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=1)) + check("C1h: tuple-tampered record -> controlled ArbiterError, not a crash", + e is not None and e.startswith("ERR:") and rc == 1) + finally: + shutil.rmtree(c1h, ignore_errors=True) + + # C1i — main() end-to-end wiring for the new flag: argparse recognizes --resume-challenge + # and dispatches to the same behavior (belt-and-suspenders on the CLI plumbing itself). + c1i = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1i, attempts=1) + buf = io.StringIO() + with redirect_stdout(buf): + rc = main(["--resume-challenge", "--state", state_path, "--feature", "featA", + "--attempt", "1", "--challenge-key", FIXED_KEY, "--now", NOW]) + check("C1i: main() wires --resume-challenge through and exits 0", + rc == 0 and json.loads(buf.getvalue().strip()) == {"state": "absent"}) + finally: + shutil.rmtree(c1i, ignore_errors=True) + + # C1j — the crash window this round's fix closes: --classify persisted the frozen + # canonical audit but died BEFORE the final issued/in_progress -> consumed write, so the + # challenge record is still (durably) "in_progress" even though the verdict already + # exists on disk. --resume-challenge must probe the canonical audit FIRST and report + # "consumed" with that persisted result, making ZERO model calls -- never re-report + # "in_progress" (which would make the driver re-dispatch Claude and re-classify, + # duplicating model work for a decision that was already made). + c1j = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1j, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + cid = out["challenge_id"] + ballot = os.path.join(c1j, "b.json") + with open(ballot, "w") as fh: + json.dump({"epic_id": "epic1", "feature": "featA", "attempt": 1, + "challenge_id": cid, "disposition": "retry_fix", "reason": "flaky"}, fh) + classified, _, _ = run_cmd(cmd_classify, _args(state=state_path, feature="featA", + challenge=cid, claude_ballot=ballot)) + with open(classified["audit_path"]) as fh: + persisted = json.load(fh) + + # Simulate the crash: the audit is durably on disk (classify wrote it and rotated + # BEFORE the final consumption write — see cmd_classify's own comment on that + # ordering), but the challenge record never made it past "in_progress". + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + rec = _read_json_at(arbiter_fd, "featA-1.challenge.json", CHALLENGE_RECORD_MAX_BYTES) + check("C1j: precondition -- classify durably consumed the record before we rewind it", + rec.get("status") == "consumed") + rec["status"] = "in_progress" + rec["consumed_at"] = None + _write_json_at(arbiter_fd, "featA-1.challenge.json", rec) + finally: + os.close(arbiter_fd) + + module_ns = globals() + _orig_poll_codex = module_ns["poll_codex"] + _poll_called = {"v": False} + + def _boom_poll_codex(*a, **kw): + _poll_called["v"] = True + raise AssertionError("poll_codex must NEVER be called by --resume-challenge") + + module_ns["poll_codex"] = _boom_poll_codex + try: + res, e, rc = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=1)) + finally: + module_ns["poll_codex"] = _orig_poll_codex + + check("C1j: in_progress record + existing valid audit -> reported as consumed " + "(crash-after-persist-before-mark-consumed window)", + rc == 0 and res is not None and res["state"] == "consumed") + check("C1j: consumed carries the SAME challenge_id", res is not None and res["challenge_id"] == cid) + check("C1j: ZERO model calls -- the verdict was already decided", _poll_called["v"] is False) + check("C1j: re-emitted result is the IDENTICAL frozen persisted audit", + res is not None + and json.dumps(res["result"], sort_keys=True) == json.dumps(persisted, sort_keys=True)) + finally: + shutil.rmtree(c1j, ignore_errors=True) + + # C1k — the ordinary sub-case stays exactly as before: an in_progress record with NO + # audit written yet (classify claimed the record but crashed before persisting anything) + # is still reported "in_progress" so the driver re-dispatches. (C1c already exercises + # this path; this re-asserts it explicitly alongside the new C1j crash-window fix so a + # future change to the in_progress branch can't silently flip both outcomes to + # "consumed".) + c1k = _tempfile.mkdtemp() + try: + epic_dir, state_path, state = _make_epic(c1k, attempts=1) + out, _, _ = run_cmd(cmd_prepare, _args(state=state_path, feature="featA", attempt=1)) + arbiter_fd = open_arbiter_dir_fd(epic_dir, create=False) + try: + rec = _read_json_at(arbiter_fd, "featA-1.challenge.json", CHALLENGE_RECORD_MAX_BYTES) + rec["status"] = "in_progress" + _write_json_at(arbiter_fd, "featA-1.challenge.json", rec) + finally: + os.close(arbiter_fd) + res, _, rc = run_cmd(cmd_resume_challenge, + _args(state=state_path, feature="featA", attempt=1)) + check("C1k: in_progress with NO audit on disk -> still state in_progress", + rc == 0 and res["state"] == "in_progress" and res["challenge_id"] == out["challenge_id"] + and res["prompt"] == out["prompt"]) + finally: + shutil.rmtree(c1k, ignore_errors=True) + + print("SELFTEST: %d ok, %d fail" % (ok, fail)) + return 0 if fail == 0 else 1 + + +# --------------------------------------------------------------------------------------- # +# main +# --------------------------------------------------------------------------------------- # + +def main(argv): + if hasattr(sys.stdout, "reconfigure"): + sys.stdout.reconfigure(encoding="utf-8", errors="replace") + if hasattr(sys.stderr, "reconfigure"): + sys.stderr.reconfigure(encoding="utf-8", errors="replace") + + p = argparse.ArgumentParser(description="Compound V epic arbiter — Codex+Claude panel.") + p.add_argument("--prepare", action="store_true") + p.add_argument("--classify", action="store_true") + p.add_argument("--resume-challenge", action="store_true") + p.add_argument("--selftest", action="store_true") + p.add_argument("--state") + p.add_argument("--feature") + p.add_argument("--attempt", type=int) + p.add_argument("--challenge") + p.add_argument("--evidence-file", help="RELATIVE to /arbiter/-/") + p.add_argument("--claude-ballot") + p.add_argument("--now") + p.add_argument("--challenge-key", help="fixed per-epic challenge key (tests / pinned driver)") + p.add_argument("--capabilities-path") + p.add_argument("--config") + p.add_argument("--explicit-codex-model") + p.add_argument("--codex-timeout", type=int, default=DEFAULT_CODEX_TIMEOUT_SEC) + p.add_argument("--max-output-bytes", type=int, default=DEFAULT_MAX_OUTPUT_BYTES) + p.add_argument("--codex-bin") + p.add_argument("--supervisor-path") + args = p.parse_args(argv) + + if args.selftest: + return _selftest() + + # Finding 8/#11: bound the numeric inputs at the argument boundary — positive and sane. + if args.max_output_bytes <= 0 or args.max_output_bytes > MAX_OUTPUT_BYTES_CEILING: + p.error("--max-output-bytes must be in 1..%d" % MAX_OUTPUT_BYTES_CEILING) + if args.codex_timeout <= 0 or args.codex_timeout > MAX_TIMEOUT_SEC: + p.error("--codex-timeout must be in 1..%d seconds" % MAX_TIMEOUT_SEC) + + try: + if args.prepare: + return cmd_prepare(args, p) + if args.classify: + return cmd_classify(args, p) + if args.resume_challenge: + return cmd_resume_challenge(args, p) + except ArbiterError as e: + _log(str(e)) + return 1 + except OSError as e: + # Finding 8/#11: convert a filesystem error at a security boundary (ENAMETOOLONG, + # EACCES, …) into a controlled one-line error, never a raw traceback. + _log("filesystem error: %s" % e) + return 1 + p.error("one of --prepare / --classify / --resume-challenge / --selftest is required") + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/compound-v-epic-state.py b/scripts/compound-v-epic-state.py index 072929d..16c2116 100755 --- a/scripts/compound-v-epic-state.py +++ b/scripts/compound-v-epic-state.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 """ -Compound V epic-state manager — multi-feature autonomous build (PRD §8 / v1.1). +Compound V epic-state manager — multi-feature autonomous build (PRD §8 / v1.1 + v2.10 marathon). A v1.0 run executes ONE plan (one feature). "Epic mode" chains several: an ordered set of features, each run through the full v1.0 pipeline (spec -> pre-flights -> plan -> @@ -8,17 +8,87 @@ is the deterministic state spine for that meta-loop — one level up from state.json, the same shape of discipline (resumable, topological, no daemon). -epic-state.json: +epic-state.json (checkpoint stance — default, UNCHANGED since v1.1): {"epic_id", "title", "status": "running|done|blocked", "features": [{"id","title","depends_on":[...],"status":"pending|running|done|failed", "run_id": }]} -The orchestrator drives the loop; this script owns the bookkeeping: - --init build epic-state.json from a features list (validates refs + cycles) - --next print the next RUNNABLE feature (pending, all deps done) or a stop reason - --update set a feature's status/run_id +epic-state.json (marathon stance — v2.10, OPT-IN, additive on top of the above): + top-level adds: "autonomy":{"stance":"marathon","max_attempts_per_feature", + "max_no_progress_cycles","max_total_attempts","max_wall_clock_hours","started_at"}, + "final_review":{"status":"pending|passed|failed"}, "blocker_ledger":[...], + "no_progress_cycles", "total_attempts". + per-feature adds: "attempts","last_error","disposition". + Absent "autonomy" => every legacy/checkpoint code path is untouched; all new fields are + read via `.get(..., default)`. + +The orchestrator drives the loop; this script owns the bookkeeping. + +## CLI contract (v2.10 — B/C read this) + +Checkpoint (default, unchanged): + --init build epic-state.json from a features list (validates refs + cycles + epic_id) + --next -> {"feature": f|null, "reason": str} (read-only) + --update --feature F --status pending|running|done|failed|blocked [--run-id R] + -> {"feature","status","epic_status"} (atomic write) --summary render the feature table - --selftest + --stats -> {"epic_id","status","total","done","pending","running","failed","blocked", + "remaining"} (read-only) + --check-specs / --lint / --selftest + +Marathon (opt-in, additive). Every command below REJECTS a non-marathon state (controlled +nonzero, no write); a negative/non-numeric cap is rejected at --init: + --init --stance marathon [--max-attempts-per-feature N] + [--max-no-progress-cycles N] [--max-total-attempts N] [--max-wall-clock-hours H] + [--now ISO] [--start-sha SHA] (a cap may be an explicit null = unbounded on that axis; + a MISSING cap uses its documented default, never unbounded; --start-sha is OPTIONAL and + stored as autonomy.start_sha for the halt-page accumulated-diff command — a checkpoint + --init rejects --start-sha) + --next --autonomous -> {"feature": f|null, "reason": str, "blocked_by": [ids]} (read-only) + `reason` embeds the literal terminal-state token when terminal: "done: ...", + "blocked_needing_human: ...", "running_with_failures: ...", or the reconcile/runnable + text carried over from the default routing style. + --can-retry --feature F -> {"can_retry","attempts","cap"} (read-only) + --record-disposition --feature F --disposition retry_fix|halt_feature|halt_epic| + blocked_external [--reason R] [--families-agreeing a,b] [--confirmed true|false] + -> {"feature","disposition"} (atomic write; --confirmed true is HARD-REJECTED in v2.10; + the stored disposition is stamped with the feature's CURRENT attempts count and is only + ever honored by `--next --autonomous` while that attempt is still current — a stale + disposition left over from a prior attempt is treated as no disposition at all) + --update --status blocked --feature F [--blocker-reason R] [--blocker-confirmed true|false] + [--families-agreeing a,b] [--evidence E] + -> ledger append/REACTIVATE, idempotent by (feature, attempt), exactly one active entry + per blocked feature; --blocker-confirmed true is HARD-REJECTED in v2.10 (a blocker is + always confirmed:false — SUSPECTED, never caller-confirmed). `blocked` is marathon-only. + --update --status failed --feature F [--last-error "..."] -> persists last_error (cleared + on a subsequent ->running retry or ->done) + --update --status pending --feature F -> resolves that feature's active ledger entry + Any marathon --update INVALIDATES a passed final_review back to pending. + --record-final-review --status pending|passed|failed -> {"final_review","epic_status"} + (atomic write; --status passed is REJECTED unless ALL features are done; "done" requires + all-features-done AND final_review.status=="passed") + --breaker-check [--now ISO] -> {"tripped","which":[...],"detail":{...}} (read-only) + --trip-breaker [--now ISO] -> {"tripped","which","detail"} (atomic write IFF tripped; + sets epic status to "blocked_needing_human") + --record-progress-cycle --cycle-id C [--now ISO] -> {"cycle_id","no_progress_cycles", + "replayed"} (atomic write unless replayed; idempotent by cycle_id) + --clear-breaker [--now ISO] [--reset-wall-clock] [--set-max-total-attempts N] -> a JSON + summary of what was cleared/re-armed (atomic write). The human's re-arm after a + breaker trip / halt: clears the `blocked_needing_human` latch (removes any + `breaker_trip` record, resets `no_progress_cycles` to 0, recomputes top status). + --reset-wall-clock restarts `autonomy.started_at`; --set-max-total-attempts re-arms + that cap (N or an explicit null for unbounded). Still clears the latch even if the + (possibly re-armed) state would immediately re-trip — prints a loud stderr warning + naming the axis in that case. + --clear-disposition --feature F -> {"feature","disposition":null} (atomic write). Clears + a feature's stored disposition (the override for a sticky halt_epic/halt_feature verdict) + so `next_feature_autonomous` no longer short-circuits on it. + --record-audit-failed --feature F [--last-error "..."] -> {"feature","status", + "sample_audit_due","final_review","epic_status"} (ONE atomic write; marathon-only) — + reverts a done feature's sample-audit-ISSUES verdict in a single state write: status -> + failed, sample_audit_due -> false, optional last_error, and a passed final_review -> + pending. Replaces the driver's old two-write clear-due-then-mark-failed sequence, whose + crash window between writes could leave a feature 'done' with no audit obligation. Usage: compound-v-epic-state.py --init --features features.json --epic-id E --title T --out S @@ -27,16 +97,38 @@ compound-v-epic-state.py --summary --state S `features.json` is a JSON array: [{"id","title","depends_on":[...]}, ...]. -Python 3.9-safe, stdlib only. +Python 3.9-safe, stdlib only. No fabricated cost/token metrics anywhere in this file — +breakers bound counts and wall-clock hours only. """ import argparse import json +import math import os import sys +import tempfile +from datetime import datetime, timezone ID_RE_OK = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._-" -STATUSES = ("pending", "running", "done", "failed") +# Checkpoint (legacy/default) status set — UNCHANGED since v1.1. `blocked` is a MARATHON-only +# status: a checkpoint `--update --status blocked` is rejected so the legacy fail-fast router +# (next_feature) never sees a status it doesn't understand (Codex review #2). +CHECKPOINT_STATUSES = ("pending", "running", "done", "failed") +# Marathon superset. Module-level name STATUSES is kept (used in --status help text); it now +# names the marathon set. Per-stance acceptance is enforced in `apply_update`. +STATUSES = ("pending", "running", "done", "failed", "blocked") + +# Documented cap defaults (fail-SAFE: a MISSING cap key falls back to these, NEVER to +# unbounded — Codex review #4). max_total_attempts has a feature-count-derived default, +# computed per-state in `_default_total_attempts`. +_CAP_DEFAULTS = { + "max_attempts_per_feature": 2, + "max_no_progress_cycles": 3, + "max_wall_clock_hours": 10, +} +# Bound on the remembered processed-cycle-id set (Codex review #5) — global idempotency +# without unbounded growth over an all-night run. +_PROCESSED_CYCLE_CAP = 512 def _id_ok(s): @@ -92,6 +184,14 @@ def validate_features(features): sp = f.get("spec_path") if sp is not None and not isinstance(sp, str): errs.append("feature %r spec_path must be a string or absent" % fid) + # Codex round-3 #7: a non-list / non-string-element depends_on later crashes + # _detect_cycle (list(int)) and the dangling-ref loop (for d in int) with a raw + # TypeError — reject it here so --init fails with a controlled error, not a traceback. + deps = f.get("depends_on") + if deps is not None and (not isinstance(deps, list) + or not all(isinstance(d, str) for d in deps)): + errs.append("feature %r depends_on must be a list of id strings (got %r)" + % (fid, deps)) dup = sorted({x for x in ids if ids.count(x) > 1}) if dup: errs.append("duplicate feature ids: %s" % ", ".join(dup)) @@ -99,27 +199,147 @@ def validate_features(features): for f in features: if not isinstance(f, dict): continue - for d in (f.get("depends_on") or []): - if d not in idset: + deps = f.get("depends_on") + if not isinstance(deps, list): # a malformed depends_on is already flagged above + continue + for d in deps: + if isinstance(d, str) and d not in idset: errs.append("feature %r depends_on unknown id %r" % (f.get("id"), d)) - cyc = _detect_cycle([f for f in features if isinstance(f, dict) and _id_ok(str(f.get("id")))]) + cyc = _detect_cycle([f for f in features if isinstance(f, dict) and _id_ok(str(f.get("id"))) + and isinstance(f.get("depends_on", []), list)]) if cyc: errs.append("dependency cycle: %s" % " -> ".join(cyc)) return errs -def build_state(features, epic_id, title): +def _cap_or_default(caps, key, default): + """Distinguish "not provided" (use `default`) from an EXPLICIT null (unbounded on that + axis) — the key must be present in `caps` for an explicit None to take effect.""" + if key in caps: + return caps[key] + return default + + +def _is_number(v): + return isinstance(v, (int, float)) and not isinstance(v, bool) + + +def _is_nonneg_int(v): + return isinstance(v, int) and not isinstance(v, bool) and v >= 0 + + +# Sentinel distinguishing "argparse arg not supplied" from an explicit value (incl. an +# explicit null) — Codex round-3 #4/#6. +_UNSET = object() + + +def _cap_arg(kind): + """argparse `type` for a --max-* cap: the literal 'null'/'none'/'unbounded' resolves to + None (an EXPLICIT unbounded axis, honored with a loud warning at check time); anything + else is parsed as `kind` (int/float). Lets --init express the same explicit-null a + hand-edited/programmatic state can (Codex round-3 #6).""" + def parse(s): + if isinstance(s, str) and s.strip().lower() in ("null", "none", "unbounded"): + return None + try: + return kind(s) + except (TypeError, ValueError): + raise argparse.ArgumentTypeError( + "must be a %s, or 'null'/'none' for an unbounded axis" % kind.__name__) + return parse + + +def _validate_cap_value(name, value): + """A single numeric cap must be either None (EXPLICIT unbounded, honored only via the + explicit-null mechanism) or a non-negative number. A negative/non-numeric cap is a hard + error, rejected at --init rather than silently persisted-then-treated-as-unbounded + (Codex review #4). Returns an error string or None.""" + if value is None: + return None + if not _is_number(value): + return "%s must be a non-negative number or null (got %r)" % (name, value) + # Codex round-2 #2: NaN/inf pass the numeric+non-negative test but silently disable a + # breaker (NaN >= x is always False; inf caps never trip) — reject non-finite caps. + if not math.isfinite(value): + return "%s must be finite (got %r) — NaN/inf would silently disable that cap; use " \ + "null for an explicit unbounded axis" % (name, value) + if value < 0: + return "%s must be >= 0 (got %r) — a negative cap is rejected, not treated as " \ + "unbounded" % (name, value) + return None + + +def _default_total_attempts(state): + feats = [f for f in state.get("features", []) if isinstance(f, dict)] + return max(6, 3 * len(feats)) + + +def build_state(features, epic_id, title, stance=None, caps=None): + """Build epic-state.json content. `stance`/`caps` are OPT-IN: every marathon-only field + (top-level `autonomy`/`final_review`/`blocker_ledger`/`no_progress_cycles`/ + `total_attempts`, per-feature `attempts`/`last_error`/`disposition`) is written ONLY + when stance=="marathon" — a plain `build_state(features, epic_id, title)` call (no + stance) is byte-for-byte what v1.1 always produced. Validates `epic_id` via `_id_ok` + (Sol-R4#10) unconditionally — this only rejects a new class of previously-undefined- + behavior input (a traversal-style epic_id), it never changes output for a valid id.""" + if not _id_ok(str(epic_id)): + raise ValueError("invalid epic_id %r (allowed: A-Za-z0-9._-)" % (epic_id,)) feats = [] for f in features: - feats.append({ + feat = { "id": f["id"], "title": f.get("title", f["id"]), "depends_on": list(f.get("depends_on", []) or []), "spec_path": f.get("spec_path"), "status": "pending", "run_id": None, - }) - return {"epic_id": epic_id, "title": title, "status": "running", "features": feats} + } + if stance == "marathon": + feat["attempts"] = 0 + feat["last_error"] = None + feat["disposition"] = None + feats.append(feat) + state = {"epic_id": epic_id, "title": title, "status": "running", "features": feats} + if stance == "marathon": + caps = caps or {} + # Fail-SAFE cap validation at build/--init time (Codex review #4): reject a + # negative/non-numeric cap up front, so it can never be persisted and then silently + # mis-treated as unbounded at check time. + for _cap_name in ("max_attempts_per_feature", "max_no_progress_cycles", + "max_total_attempts", "max_wall_clock_hours"): + if _cap_name in caps: + _err = _validate_cap_value(_cap_name, caps[_cap_name]) + if _err: + raise ValueError(_err) + started_at_raw = _cap_or_default(caps, "started_at", None) + started_at = _now_iso(_parse_iso(started_at_raw)) if started_at_raw else _now_iso() + state["autonomy"] = { + "stance": "marathon", + # NOTE: no "max_features" knob — v2.10 never enforces a feature-count cap + # anywhere (the spec listed it, but a stored-and-unenforced field is a dead + # knob per cross-model review; drop it rather than fake an enforcement path). + "max_attempts_per_feature": _cap_or_default(caps, "max_attempts_per_feature", 2), + "max_no_progress_cycles": _cap_or_default(caps, "max_no_progress_cycles", 3), + "max_total_attempts": _cap_or_default(caps, "max_total_attempts", + max(6, 3 * len(feats))), + "max_wall_clock_hours": _cap_or_default(caps, "max_wall_clock_hours", 10), + "started_at": started_at, + } + # start_sha (v2.10 resume support) is OPTIONAL and OMITTED entirely when not + # supplied — a plain marathon build_state(...) call (no caps["start_sha"]) keeps the + # existing 6-key autonomy schema byte-for-byte (Codex/cross-model review golden + # tests assert `set(autonomy.keys())` exactly). The driver captures + # `git rev-parse HEAD` and passes it for the halt-page accumulated-diff command. + if "start_sha" in caps: + start_sha = caps["start_sha"] + if not isinstance(start_sha, str): + raise ValueError("start_sha must be a string (got %r)" % (start_sha,)) + state["autonomy"]["start_sha"] = start_sha + state["final_review"] = {"status": "pending"} + state["blocker_ledger"] = [] + state["no_progress_cycles"] = 0 + state["total_attempts"] = 0 + return state def check_specs(features, base_dir=""): @@ -206,15 +426,20 @@ def lint_decomposition(features): def stats(state): feats = state.get("features", []) - by = {"pending": 0, "running": 0, "done": 0, "failed": 0} + by = {"pending": 0, "running": 0, "done": 0, "failed": 0, "blocked": 0} for f in feats: s = f.get("status") if s in by: by[s] += 1 - return {"epic_id": state.get("epic_id"), "status": state.get("status"), - "total": len(feats), "done": by["done"], "pending": by["pending"], - "running": by["running"], "failed": by["failed"], - "remaining": by["pending"] + by["running"]} + out = {"epic_id": state.get("epic_id"), "status": state.get("status"), + "total": len(feats), "done": by["done"], "pending": by["pending"], + "running": by["running"], "failed": by["failed"], + "remaining": by["pending"] + by["running"]} + # Codex round-3 #5: `blocked` is a marathon-only status — only break it out for marathon + # states so checkpoint --stats output stays byte-compatible (no new key). + if _is_marathon(state): + out["blocked"] = by["blocked"] + return out def next_feature(state): @@ -223,6 +448,10 @@ def next_feature(state): The order of the guards encodes the documented stop/resume model (commands/v-epic.md): a failure or a crashed run HALTS the epic until a human reconciles it — the loop never autonomously routes around a failed/stale feature. + + UNTOUCHED by the v2.10 marathon work — `next_feature_autonomous` below is a SEPARATE + function; this one, its guard order, and its 2-key JSON shape are byte-for-byte what + v1.1 always produced. """ feats = [f for f in state.get("features", []) if isinstance(f, dict)] # .get("status"): a hand-made state entry missing the key must not KeyError (A7) — it @@ -260,6 +489,999 @@ def next_feature(state): % ", ".join(f["id"] for f in pending)) +def _reverse_deps_graph(feats): + """Adjacency map: feature id -> ids that directly depend_on it (reverse of depends_on). + Same graph-building idiom as `_detect_cycle`, reused for reachability instead of cycle + detection.""" + feats = [f for f in feats if isinstance(f, dict)] + forward = {f.get("id"): list(f.get("depends_on", []) or []) for f in feats} + reverse = {fid: [] for fid in forward} + for fid, deps in forward.items(): + for d in deps: + if d in reverse: + reverse[d].append(fid) + return reverse + + +def _transitive_closure(reverse, seeds): + out = set() + stack = list(seeds) + while stack: + cur = stack.pop() + if cur in out: + continue + out.add(cur) + stack.extend(reverse.get(cur, [])) + return out + + +def _transitive_dependents(feats, feature_id): + """All feature ids reachable by following depends_on backwards from `feature_id` + (i.e. everything that depends on it, directly or indirectly) — NOT including + `feature_id` itself.""" + reverse = _reverse_deps_graph(feats) + return sorted(_transitive_closure(reverse, reverse.get(feature_id, []))) + + +def next_feature_autonomous(state): + """Read-only autonomous routing (marathon `--next --autonomous`): DAG-transitive- + dependent cascading instead of the default's whole-epic fail-fast. A failed/blocked + feature removes only its transitive DEPENDENTS from the runnable set — independent + pending features stay runnable. Returns (feature|None, reason, blocked_by:[ids]). + + `blocked_by` is derived fresh every call (never persisted) — it is the transitive + dependents of any currently failed/blocked feature. Reopening a source feature + (`--update --status pending`) simply makes it, and everything it was blocking, + disappear from `blocked_by` on the next call — there is no cascade to reverse. + + Terminal-state resolution embeds the literal v2.10 token as a prefix of `reason` so + callers can match on it exactly like the default function's callers match on + "reconcile"/"complete"/"blocked": "done: ...", "blocked_needing_human: ...", + "running_with_failures: ...". v2.10 NEVER emits "done_with_blockers" (that terminal + state needs a 2nd safe external family — v2.11). + + FIX 1 (crash/breaker-window resume safety, v2.10 follow-up): a `failed` feature is no + longer blanket-treated as abandoned — it is routed BY its stored `disposition`: + - `retry_fix` AND still under the retry cap -> the feature ITSELF is runnable again + (the arbiter approved a retry and it hasn't re-run yet); its transitive dependents + are explicitly NOT blocked, because it is about to be retried, not abandoned. + - `retry_fix` but the retry cap is exhausted, or `halt_feature` -> abandoned, exactly + like the pre-fix blanket behavior (dependents blocked, independents continue). + - `halt_epic` -> handled by the pre-existing unconditional whole-epic halt check above. + - no/None disposition (the crash-in-arbitration window: the feature failed but the + arbiter never recorded a verdict before a breaker trip/crash) -> this must NOT be + silently treated as settled. It halts ALL routing (not just its own dependents) with + a dedicated "needs_arbitration" reason, so the driver re-runs the arbiter on resume + instead of quietly handing out unrelated independent work over an untriaged failure. + + FIX A (attempt-binding, v2.10 correctness follow-up): a stored disposition is honored + ONLY when `disposition.attempt == feature.attempts` — i.e. it was recorded for the + feature's CURRENT (still-failed) attempt, not a prior one. A disposition recorded at a + STALE attempt (the retry re-ran and failed again, or attempts otherwise advanced, without + a fresh arbiter verdict) is treated exactly like NO disposition at all — the same + needs_arbitration halt-all-routing path above. This applies uniformly to retry_fix, + halt_feature, halt_epic, and blocked_external: a stale verdict of ANY kind is not honored + — re-arbitrate, which is always safe. A legacy disposition with no stored `attempt` key + (pre-FIX-A data) is treated as attempt 0. + + FIX 2 (v2.10, cross-unit integration review follow-up): a `failed` feature whose + ATTEMPT-MATCHED disposition is `blocked_external` is NOT the "arbiter already settled it" + case FIX 1 originally assumed — it is a distinct crash window: the driver recorded the + verdict (`--record-disposition blocked_external`) but crashed before completing the + `--update --status blocked` transition (which flips status and appends the + `blocker_ledger` entry). Treating it as abandoned would silently lose the blocker + transition. Instead this surfaces a dedicated "needs_blocker_recording" reason — analogous + priority to needs_arbitration: it takes precedence over handing out other work, so the + driver completes the transition first. The `--update --status blocked` ledger append is + already idempotent by (feature, attempt), so re-running it is always safe. A feature that + has ALREADY transitioned to `status == "blocked"` never reaches this classification loop + (which only inspects `status == "failed"`) and stays on the normal blocked/ledger path, + unchanged. An attempt-MISMATCHED blocked_external disposition still falls into the + needs_arbitration bucket above (FIX A), unchanged. + """ + feats = [f for f in state.get("features", []) if isinstance(f, dict)] + ids = {f.get("id") for f in feats} + status_by = {f.get("id"): f.get("status") for f in feats} + done_ids = {f["id"] for f in feats if f.get("status") == "done"} + running_ids = sorted(f["id"] for f in feats if f.get("status") == "running") + pending = [f for f in feats if f.get("status") == "pending"] + + # FIX 1: classify every `failed` feature by its stored disposition instead of treating + # "failed" as a single undifferentiated blocking state. + retry_runnable = [] # retry_fix + still under cap -> the feature itself is runnable + abandoned_ids = [] # retry_fix (cap exhausted) or halt_feature -> abandoned, as before + needs_arb_ids = [] # no/None disposition -> crash-in-arbitration window, halt routing + needs_blocker_recording_ids = [] # FIX 2: blocked_external verdict recorded, but the + # blocked/ledger transition crashed mid-flight + for f in feats: + if f.get("status") != "failed": + continue + fid = f.get("id") + disp = f.get("disposition") + disp_kind = disp.get("disposition") if isinstance(disp, dict) else None + # FIX A: a disposition only counts if it was recorded for the feature's CURRENT + # attempt (legacy dispositions with no stored "attempt" key are treated as attempt 0). + # A kind/attempt mismatch (no disposition at all, or a stale one from a prior attempt) + # is treated identically — the crash-in-arbitration needs_arbitration path below. + disp_attempt_matches = (isinstance(disp, dict) + and disp.get("attempt", 0) == f.get("attempts", 0)) + if disp_kind is None or not disp_attempt_matches: + needs_arb_ids.append(fid) + elif disp_kind == "retry_fix": + info = can_retry_info(state, fid) + if info and info.get("can_retry"): + retry_runnable.append(f) + else: + abandoned_ids.append(fid) + elif disp_kind == "halt_feature": + abandoned_ids.append(fid) + elif disp_kind == "blocked_external": + # FIX 2: blocked_external landing on a still-`failed` feature (attempt-matched) + # means the arbiter's verdict was recorded but the blocked/ledger transition did + # NOT complete before a crash — this is unresolved, not a settled abandon. + needs_blocker_recording_ids.append(fid) + elif disp_kind == "halt_epic": + pass # handled by the unconditional (also attempt-gated) halt_epic check below + retry_runnable.sort(key=lambda rf: rf.get("id") or "") + abandoned_ids = sorted(abandoned_ids) + needs_arb_ids = sorted(needs_arb_ids) + needs_blocker_recording_ids = sorted(needs_blocker_recording_ids) + + # `blocking_ids` retains its pre-fix meaning (used for messaging + dependents-blocking) + # EXCEPT a retry-runnable failed feature is explicitly excluded — its dependents are not + # blocked, per FIX 1. A needs_blocker_recording feature is included here too: it is + # unresolved (not abandoned, not runnable), so its dependents must stay blocked exactly + # like an abandoned or needs_arbitration feature's would. + blocking_ids = sorted( + [fid for fid in status_by if status_by.get(fid) == "blocked"] + abandoned_ids + + needs_arb_ids + needs_blocker_recording_ids + ) + + reverse = _reverse_deps_graph(feats) + dependents_blocked = set() + for bid in blocking_ids: + dependents_blocked |= _transitive_closure(reverse, reverse.get(bid, [])) + # `blocked_by` = the transitive dependents that are actually WAITING (pending/running) on + # a blocked/failed upstream — a done dependent is not "blocked" and must not appear + # (Codex review #3). The blocking features themselves are reported via `blocking_ids`. + blocked_by = sorted(fid for fid in dependents_blocked + if status_by.get(fid) in ("pending", "running")) + + # Whole-epic halt: an explicit breaker trip already recorded, or a halt_epic verdict. + if state.get("status") == "blocked_needing_human": + return None, "blocked_needing_human: epic halted (breaker tripped or halt_epic)", blocked_by + # FIX A: also attempt-gated — a halt_epic disposition recorded at a since-superseded + # attempt is stale and must not keep halting the whole epic (the cd1/clear-disposition + # selftest below covers the common case: a feature that never re-ran keeps attempts at + # its record-time value, so its halt_epic disposition still matches and still halts). + halt_epic_ids = sorted(f["id"] for f in feats + if isinstance(f.get("disposition"), dict) + and f["disposition"].get("disposition") == "halt_epic" + and f["disposition"].get("attempt", 0) == f.get("attempts", 0)) + if halt_epic_ids: + return None, ("blocked_needing_human: halt_epic disposition on %s" + % ", ".join(halt_epic_ids)), blocked_by + + # Crash recovery for a 'running' feature is still the existing running->reconcile path + # (Component "survives a fall": "the existing running->reconcile path on re-entry") — + # autonomous routing does not invent a different crash-recovery mechanism. + if running_ids: + return None, ("epic needs reconcile: feature(s) still 'running' (%s) — a prior run " + "crashed; mark each --status failed (abandon) or pending (retry), then " + "re-run" % ", ".join(running_ids)), blocked_by + + # FIX 1: a failed feature with no recorded disposition is the crash-in-arbitration + # window — do NOT silently abandon it, and do NOT hand out other work as if the epic + # were settled. Halt ALL routing until the arbiter re-runs. This takes priority over a + # retry-runnable feature elsewhere: an unresolved arbitration gap isn't closed by other + # work being available. + if needs_arb_ids: + return None, ("needs_arbitration: feature %s failed without a recorded disposition — " + "re-run the arbiter" % ", ".join(needs_arb_ids)), blocked_by + + # FIX 2: an attempt-matched blocked_external disposition on a still-`failed` feature is + # the "verdict recorded, transition incomplete" crash window — surface it with the same + # priority as needs_arbitration above (it also halts ALL routing, not just this feature's + # dependents), so the driver completes the blocked/ledger transition before any other work + # is handed out. + if needs_blocker_recording_ids: + return None, ("needs_blocker_recording: feature(s) %s — arbiter said blocked_external " + "but the blocked/ledger transition is incomplete; re-run --update " + "--status blocked" % ", ".join(needs_blocker_recording_ids)), blocked_by + + # FIX 1: a retry_fix-approved failed feature that is still under the retry cap is itself + # directly runnable again — the driver re-runs it. This is checked before the normal + # pending-DAG routing below because it is not "new" work, it is the resumption of + # in-flight (arbiter-approved) work. + if retry_runnable: + return retry_runnable[0], "runnable", blocked_by + + if not pending: + if ids and done_ids == ids: + # FIX 2: a due-but-not-yet-passed sample audit gates terminal completion, even + # once every feature is 'done' and final_review has passed — a breaker/crash + # between marking a sampled SUCCESS 'done' and running its PASS sample-audit must + # not silently lose that obligation on resume. + due_ids = sorted(f["id"] for f in feats if f.get("sample_audit_due") is True) + if due_ids: + return None, ("sample_audit_due: feature(s) %s awaiting sample-audit" + % ", ".join(due_ids)), blocked_by + fr = state.get("final_review") + fr = fr if isinstance(fr, dict) else {} + if fr.get("status") == "passed": + return None, "done: all features done and final_review passed", blocked_by + return None, ("running_with_failures: all features done, awaiting final_review " + "(status=%s) before 'done'" % fr.get("status", "pending")), blocked_by + if blocking_ids: + return None, ("blocked_needing_human: no runnable feature — blocked/failed " + "feature(s) %s exhaust reachable work" % ", ".join(blocking_ids)), \ + blocked_by + return None, "epic complete: all features done", blocked_by # defensive fallback + + # A pending feature that is a transitive dependent of ANY blocked/failed feature is NOT + # runnable, even if its immediate deps happen to be done (Codex review #3: {A failed, + # B done, C pending, A->B->C} must not hand out C). + runnable = [f for f in pending + if f.get("id") not in dependents_blocked + and all(d in done_ids for d in (f.get("depends_on") or []))] + if runnable: + if blocking_ids: + return runnable[0], ("running_with_failures: runnable (feature(s) %s blocked/failed " + "independently)" % ", ".join(blocking_ids)), blocked_by + return runnable[0], "runnable", blocked_by + + if blocking_ids: + return None, ("blocked_needing_human: no runnable feature — remaining pending %s all " + "blocked by %s" % (", ".join(sorted(f["id"] for f in pending)), + ", ".join(blocking_ids))), blocked_by + return None, ("epic blocked: no runnable feature — unsatisfiable dependencies among %s" + % ", ".join(f["id"] for f in pending)), blocked_by + + +def _now_iso(dt=None): + """Emit an ISO-8601 UTC timestamp with a `+00:00` offset (never a bare `Z`, which + Python 3.9 cannot re-parse). An AWARE datetime in any zone is normalized to UTC + (Codex review #11 — an injected `+05:00` is emitted as `+00:00`); a naive datetime is + assumed UTC.""" + if dt is None: + dt = datetime.now(timezone.utc) + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + else: + dt = dt.astimezone(timezone.utc) + return dt.isoformat() + + +def _parse_iso(s): + """Parse an ISO-8601 string on Python 3.9, which cannot parse a trailing 'Z' — + normalize Z -> +00:00 first (never emit bare Z either; `_now_iso` always yields + +00:00 via an aware UTC datetime). Naive datetimes are assumed UTC.""" + if isinstance(s, str) and s.endswith("Z"): + s = s[:-1] + "+00:00" + dt = datetime.fromisoformat(s) + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return dt + + +def _resolve_now(now_arg): + """CLI --now resolution: an injected ISO string (deterministic tests), else the real + clock.""" + if now_arg: + return _parse_iso(now_arg) + return datetime.now(timezone.utc) + + +def _parse_csv_list(s): + return [x.strip() for x in (s or "").split(",") if x.strip()] + + +def _is_marathon(state): + autonomy = state.get("autonomy") + return isinstance(autonomy, dict) and autonomy.get("stance") == "marathon" + + +def _find_feature(state, feature_id): + feats = state.get("features") + if not isinstance(feats, list): + return None + for f in feats: + if isinstance(f, dict) and f.get("id") == feature_id: + return f + return None + + +def _total_attempts(state): + """The invariant: total_attempts == sum(feature.attempts) — ALWAYS recomputed, never + tracked as an independently-incremented counter (a caller-controlled `--attempt` bump + would drift from the real per-feature counts; this can't drift because it IS the sum).""" + return sum(f.get("attempts", 0) for f in state.get("features", []) if isinstance(f, dict)) + + +def _recompute_top_status(state): + """Marathon top-level status recompute: 'done' iff every feature is done AND + final_review.status=="passed" AND no feature has a due sample-audit (A6/Sol-R4#1, FIX 2); + otherwise 'running' — UNLESS a breaker/halt_epic already parked it at + 'blocked_needing_human' (only --trip-breaker sets that; this function never clears or + overwrites it). This deliberately REPLACES the checkpoint rollup's "any failed -> blocked" + fail-fast rule for marathon states, because that rule is exactly what autonomous DAG + routing (`next_feature_autonomous`) exists to route around — a single failed/blocked + feature must not blanket-halt a marathon epic while independent features are still + runnable. + + FIX 2: a due sample-audit gates 'done' HERE too (not just in `next_feature_autonomous` + and `record_final_review`) — so `state["status"]` itself is never misleadingly 'done' + while a sampled SUCCESS's audit obligation is still outstanding, whichever call site + triggers the recompute.""" + if state.get("status") == "blocked_needing_human": + return + feats = [f for f in state.get("features", []) if isinstance(f, dict)] + sts = [f.get("status") for f in feats] + fr = state.get("final_review") + fr = fr if isinstance(fr, dict) else {} + audit_due = any(f.get("sample_audit_due") is True for f in feats) + if sts and all(s == "done" for s in sts) and fr.get("status") == "passed" and not audit_due: + state["status"] = "done" + else: + state["status"] = "running" + + +def apply_update(state, feature_id, status, run_id=None, now_dt=None, + blocker_reason=None, blocker_confirmed=False, + families_agreeing=None, evidence=None, last_error=None): + """Mutate `state` in place for `--update`. Returns (ok, error|None). + + - checkpoint (non-marathon): rollup UNCHANGED — any failed -> blocked; all done -> done; + else running. The ->running transition is unrestricted, exactly as before. `blocked` + is NOT an accepted status (marathon-only — Codex review #2). + - marathon: attempts increment + a documented transition table gate ->running (A3: + legal sources pending/failed only); a ->blocked transition appends/REACTIVATES an + idempotent blocker-ledger entry keyed by (feature, attempt) — guaranteeing exactly one + ACTIVE entry per currently-blocked feature (Codex review #6) — and HARD-REJECTS + blocker_confirmed=True (A4/A5, Sol-R4#2 — v2.10 blockers are always confirmed:false, + SUSPECTED not caller-asserted); a ->pending transition resolves that feature's active + ledger entry; a ->failed persists `last_error` and ->running/->done clears it + (Codex review #7); a ->done resets no_progress_cycles to 0 (Codex review #8); + ANY marathon feature-state change INVALIDATES a passed final_review back to pending + (Codex review #1); the top-level status is recomputed via `_recompute_top_status` (A6) + instead of the checkpoint fail-fast rollup; total_attempts is recomputed (Codex #9). + """ + marathon = _is_marathon(state) + allowed = STATUSES if marathon else CHECKPOINT_STATUSES + if status not in allowed: + return False, ("--status %r is not valid for the %s stance (allowed: %s)" + % (status, "marathon" if marathon else "checkpoint", + ", ".join(allowed))) + # Codex round-2 #5: a non-marathon --update must REJECT marathon-only args rather than + # silently discard them (e.g. --last-error on a checkpoint state was accepted then + # dropped). Gate the whole marathon-field surface on stance. + if not marathon: + offenders = [] + if last_error is not None: + offenders.append("--last-error") + if blocker_reason is not None: + offenders.append("--blocker-reason") + if evidence is not None: + offenders.append("--evidence") + if families_agreeing: + offenders.append("--families-agreeing") + if blocker_confirmed: + offenders.append("--blocker-confirmed") + if offenders: + return False, ("%s %s marathon-only — not valid on a checkpoint-stance epic" + % (", ".join(offenders), "is" if len(offenders) == 1 else "are")) + hit = _find_feature(state, feature_id) + if hit is None: + return False, "no feature %r" % feature_id + prev_status = hit.get("status") # Codex round-2 #4: needed to detect a real status change + now_dt = now_dt or datetime.now(timezone.utc) + now_s = _now_iso(now_dt) + + if status == "running" and marathon: + prev = hit.get("status") + if prev not in ("pending", "failed"): + return False, ("illegal transition to 'running' from %r (legal sources: pending, " + "failed)" % prev) + hit["attempts"] = hit.get("attempts", 0) + 1 + + if status == "blocked" and marathon: + if blocker_confirmed: + return False, ("--blocker-confirmed true is rejected in v2.10 — confirmation is " + "derived from >=2 stored known-family arbiter ballots, never " + "caller-asserted; a v2.10 blocker is always confirmed:false " + "(SUSPECTED)") + attempt = hit.get("attempts", 0) + ledger = state.setdefault("blocker_ledger", []) + # Exactly-one-active invariant (Codex review #6): deactivate every OTHER ledger entry + # for this feature, then append-or-REACTIVATE the entry for the current attempt. + entry = None + for e in ledger: + if e.get("feature") != feature_id: + continue + if e.get("attempt") == attempt: + entry = e + elif e.get("active"): + e["active"] = False + if not e.get("resolved_at"): + e["resolved_at"] = now_s + if entry is None: + ledger.append({ + "feature": feature_id, + "attempt": attempt, + "confirmed": False, + "reason": blocker_reason or "", + "evidence": evidence, + "families_agreeing": list(families_agreeing or []), + "first_seen_at": now_s, + "blocks": _transitive_dependents(state.get("features", []), feature_id), + "active": True, + "resolved_at": None, + }) + elif not entry.get("active"): + # Reblock of a previously-resolved (feature, attempt): reactivate it so a + # currently-blocked feature is never invisible to the human report. + entry["active"] = True + entry["resolved_at"] = None + if blocker_reason: + entry["reason"] = blocker_reason + if evidence is not None: + entry["evidence"] = evidence + if families_agreeing: + entry["families_agreeing"] = list(families_agreeing) + # else: already active for this attempt — idempotent replay, no change. + + if status == "pending" and marathon: + for e in (state.get("blocker_ledger") or []): + if e.get("feature") == feature_id and e.get("active"): + e["active"] = False + e["resolved_at"] = now_s + + if marathon: + # last_error lifecycle (Codex review #7): set on ->failed (when supplied), cleared + # on a retry (->running) or success (->done). + if status == "failed": + if last_error is not None: + hit["last_error"] = last_error + elif status in ("running", "done"): + hit["last_error"] = None + # no_progress reset on a feature REACHING done (Codex review #8 / Component 4). + # Codex round-3 #1: gate on a real transition — a done->done replay is not progress + # and must not reset the stall counter (which would postpone the breaker). + if status == "done" and prev_status != "done": + state["no_progress_cycles"] = 0 + + hit["status"] = status + if run_id is not None: + hit["run_id"] = run_id + + if marathon: + # A REAL feature-state change invalidates a passed final_review (Codex review #1) — a + # fresh --record-final-review passed is required before the epic can be 'done' again. + # Codex round-2 #4: an idempotent replay (prev_status == status, e.g. done->done) + # is NOT a change and must NOT flip a passed review back to pending. + if prev_status != status: + fr = state.get("final_review") + if isinstance(fr, dict) and fr.get("status") == "passed": + state["final_review"] = {"status": "pending"} + state["total_attempts"] = _total_attempts(state) # recompute, never trust a stale field + _recompute_top_status(state) + else: + sts = [f["status"] for f in state["features"]] + if all(s == "done" for s in sts): + state["status"] = "done" + elif any(s == "failed" for s in sts): + state["status"] = "blocked" + else: + state["status"] = "running" + return True, None + + +def record_disposition(state, feature_id, disposition, reason=None, + families_agreeing=None, confirmed=False, now_dt=None): + """Store the arbiter's verdict on a feature (A4). `confirmed=True` is HARD-REJECTED in + v2.10 (Sol-R4#2) — confirmation of a blocked_external verdict is derived from >=2 stored + known-family ballots elsewhere (Unit B), never asserted by the caller of this CLI. + + FIX A (attempt-binding, v2.10 correctness follow-up): the disposition is stamped with + `attempt` = the feature's CURRENT `attempts` count at record time. `next_feature_autonomous` + only honors a stored disposition when it was recorded for the feature's still-current + attempt — this is what stops a stale `retry_fix` (approved for a prior, already-retried + attempt) from being silently re-honored as an unapproved second retry after the retry + re-runs, fails again, and crashes before a fresh arbitration.""" + if disposition not in ("retry_fix", "halt_feature", "halt_epic", "blocked_external"): + return False, "invalid --disposition %r" % (disposition,) + if confirmed: + return False, ("--confirmed true is rejected in v2.10 — confirmation is derived from " + ">=2 stored known-family arbiter ballots, never caller-asserted") + hit = _find_feature(state, feature_id) + if hit is None: + return False, "no feature %r" % feature_id + now_dt = now_dt or datetime.now(timezone.utc) + hit["disposition"] = { + "disposition": disposition, + "attempt": hit.get("attempts", 0), + "confirmed": False, + "reason": reason or "", + "families_agreeing": list(families_agreeing or []), + "recorded_at": _now_iso(now_dt), + } + state["total_attempts"] = _total_attempts(state) # recompute on every marathon mutation (#9) + return True, None + + +def record_final_review(state, status): + """A6: persist the final cross-feature re-verification gate. `next_feature_autonomous` + (A2) only reports 'done' once all features are done AND this is 'passed'. + + `passed` is REJECTED unless EVERY feature is currently done (Codex review #1) — a review + can't pass over incomplete work, so only record_final_review(...,'passed') on a fully- + done epic can ever flip the top-level status to 'done'. + + FIX 2 (v2.10 follow-up): `passed` is ALSO rejected while any feature has + `sample_audit_due == True` — a due sample-audit is a durable obligation that a passed + final_review must never gate past, breaker trip or not.""" + if status not in ("pending", "passed", "failed"): + return False, "--status must be one of: pending, passed, failed" + if status == "passed": + feats = [f for f in state.get("features", []) if isinstance(f, dict)] + not_done = [f.get("id") for f in feats if f.get("status") != "done"] + if not feats or not_done: + return False, ("cannot record final_review=passed while features are not done " + "(not done: %s)" % (", ".join(map(str, not_done)) or "none")) + due_ids = sorted(f.get("id") for f in feats if f.get("sample_audit_due") is True) + if due_ids: + return False, ("cannot record final_review=passed while sample_audit_due is true " + "for feature(s) %s" % ", ".join(due_ids)) + state["final_review"] = {"status": status} + state["total_attempts"] = _total_attempts(state) # recompute on every marathon mutation (#9) + _recompute_top_status(state) + return True, None + + +def can_retry_info(state, feature_id): + """A3: {"can_retry","attempts","cap"} — read-only. Non-marathon states have no stored + attempts/cap; this degrades to attempts=0 against the marathon default cap (2) rather + than crashing, consistent with the `.get(...,default)` reading discipline.""" + hit = _find_feature(state, feature_id) + if hit is None: + return None + attempts = hit.get("attempts", 0) + autonomy = state.get("autonomy") if isinstance(state.get("autonomy"), dict) else {} + # Fail-safe cap resolution (Codex round-2 #3): a MISSING key -> the documented default; + # an EXPLICIT null -> unbounded (can always retry), NEVER compared against with `<` + # (attempts < None raises on Python 3). + cap = _effective_cap(autonomy, "max_attempts_per_feature", + _CAP_DEFAULTS["max_attempts_per_feature"]) + can_retry = True if cap is None else (attempts < cap) + return {"can_retry": can_retry, "attempts": attempts, "cap": cap} + + +def _effective_cap(autonomy, name, default): + """Fail-SAFE cap resolution (Codex review #4): + - MISSING key -> the documented `default` (NEVER unbounded); + - EXPLICIT null -> unbounded on that axis, with a LOUD stderr warning each check; + - a number -> that number (validated non-negative at --init / post-load). + Returns the effective cap (a number, or None for the explicit-unbounded path).""" + if name not in autonomy: + return default + val = autonomy[name] + if val is None: + print("epic warning: %s is explicitly null — that cap axis is UNBOUNDED " + "(no limit will ever apply on it)" % name, file=sys.stderr) + return None + return val + + +def breaker_check(state, now_dt): + """A7: READ-ONLY. {"tripped","which":[...],"detail":{...}}. Trips on + total_attempts>=max_total_attempts, no_progress_cycles>=max_no_progress_cycles, or + wall-clock(now - autonomy.started_at)>=max_wall_clock_hours. Counts and hours only — + never a fabricated cost. A non-marathon state (no `autonomy`) never trips. Caps are + resolved fail-SAFE: a missing cap key uses the documented default, not unbounded + (Codex review #4).""" + autonomy = state.get("autonomy") if isinstance(state.get("autonomy"), dict) else {} + which = [] + detail = {} + if not autonomy: + return {"tripped": False, "which": which, "detail": detail} + + # Never trust a possibly-stale stored counter for the trip decision itself — derive it + # live from the features (the invariant total_attempts == sum(feature.attempts) holds + # by construction this way, not by hoping every writer kept the stored field in sync). + total_attempts = _total_attempts(state) + max_total = _effective_cap(autonomy, "max_total_attempts", _default_total_attempts(state)) + if max_total is not None and total_attempts >= max_total: + which.append("max_total_attempts") + detail["max_total_attempts"] = {"value": total_attempts, "cap": max_total} + + no_progress = state.get("no_progress_cycles", 0) + max_no_progress = _effective_cap(autonomy, "max_no_progress_cycles", + _CAP_DEFAULTS["max_no_progress_cycles"]) + if max_no_progress is not None and no_progress >= max_no_progress: + which.append("max_no_progress_cycles") + detail["max_no_progress_cycles"] = {"value": no_progress, "cap": max_no_progress} + + max_hours = _effective_cap(autonomy, "max_wall_clock_hours", + _CAP_DEFAULTS["max_wall_clock_hours"]) + started_at = autonomy.get("started_at") + if max_hours is not None: + # Codex round-3 #3: when a wall-clock cap is set, a missing/unparseable started_at + # must FAIL SAFE — TRIP the breaker (never silently skip a safety limit). A valid + # start time trips normally on elapsed >= cap. + elapsed_hours = None + if not isinstance(started_at, str) or not started_at: + print("epic-breaker warning: max_wall_clock_hours is set but autonomy.started_at " + "is missing %r — TRIPPING the wall-clock breaker fail-safe (never silently " + "skipped)" % (started_at,), file=sys.stderr) + which.append("max_wall_clock_hours") + detail["max_wall_clock_hours"] = {"value": None, "cap": max_hours, + "failsafe": "started_at missing"} + else: + try: + started_dt = _parse_iso(started_at) + elapsed_hours = (now_dt - started_dt).total_seconds() / 3600.0 + except (ValueError, TypeError): + print("epic-breaker warning: autonomy.started_at %r is unparseable — TRIPPING " + "the wall-clock breaker fail-safe (never silently skipped)" + % (started_at,), file=sys.stderr) + which.append("max_wall_clock_hours") + detail["max_wall_clock_hours"] = {"value": None, "cap": max_hours, + "failsafe": "started_at unparseable"} + if elapsed_hours is not None and elapsed_hours >= max_hours: + which.append("max_wall_clock_hours") + detail["max_wall_clock_hours"] = {"value": round(elapsed_hours, 4), + "cap": max_hours} + + return {"tripped": bool(which), "which": which, "detail": detail} + + +def trip_breaker(state, now_dt): + """A7: re-runs `breaker_check`; if tripped, atomically parks the epic at + 'blocked_needing_human' and records which breaker + detail. A no-op (besides the read) + when nothing is tripped — callers check `result["mutated"]` to decide whether to persist.""" + result = breaker_check(state, now_dt) + if result["tripped"] and state.get("status") != "blocked_needing_human": + state["status"] = "blocked_needing_human" + state["breaker_trip"] = {"which": result["which"], "detail": result["detail"], + "tripped_at": _now_iso(now_dt)} + state["total_attempts"] = _total_attempts(state) # recompute on every mutation (#9) + result["mutated"] = True + else: + result["mutated"] = False + return result + + +def record_progress_cycle(state, cycle_id): + """A7: one atomic, idempotent progress-boundary marker keyed by `cycle_id`. Compares + the current `done` count against the last-recorded count: more done -> reset + no_progress_cycles to 0; same or fewer -> increment it. + + Idempotency is GLOBAL, not just against the immediately-preceding id (Codex review #5): + ANY previously-accepted cycle-id (within the bounded remembered set) replays as a no-op, + so cycle-1 -> cycle-2 -> replay(cycle-1) does not double-count.""" + processed = state.get("processed_cycle_ids") + processed = list(processed) if isinstance(processed, list) else [] + if cycle_id in processed: + return {"cycle_id": cycle_id, "no_progress_cycles": state.get("no_progress_cycles", 0), + "replayed": True} + done_count = sum(1 for f in state.get("features", []) + if isinstance(f, dict) and f.get("status") == "done") + prior = state.get("last_done_count") + if prior is None: + # First cycle ever: nothing to compare against — establish the baseline without + # counting it as a no-progress cycle. + state["no_progress_cycles"] = state.get("no_progress_cycles", 0) + elif done_count > prior: + state["no_progress_cycles"] = 0 + else: + state["no_progress_cycles"] = state.get("no_progress_cycles", 0) + 1 + state["last_done_count"] = done_count + processed.append(cycle_id) + if len(processed) > _PROCESSED_CYCLE_CAP: + processed = processed[-_PROCESSED_CYCLE_CAP:] + state["processed_cycle_ids"] = processed + state["total_attempts"] = _total_attempts(state) # recompute on every marathon mutation (#9) + return {"cycle_id": cycle_id, "no_progress_cycles": state["no_progress_cycles"], + "replayed": False} + + +# Human-facing hint per breaker axis for the --clear-breaker fail-safe re-trip warning: what +# the human must actually go raise/restart for that specific axis to stop re-tripping +# immediately. no_progress_cycles has no hint because clear_breaker unconditionally resets it +# to 0, so it can never be part of an immediate re-trip. +_BREAKER_AXIS_HINTS = { + "max_total_attempts": "raise the cap with --set-max-total-attempts", + "max_wall_clock_hours": "restart the clock with --reset-wall-clock (or raise " + "autonomy.max_wall_clock_hours by hand)", +} + + +def clear_breaker(state, now_dt, reset_wall_clock=False, set_max_total_attempts=_UNSET): + """The human's re-arm after a breaker trip / halt (v2.10 resume support). Marathon-only. + + Clears the `blocked_needing_human` latch: removes any `breaker_trip` record, resets + `no_progress_cycles` to 0, then recomputes the top-level status via + `_recompute_top_status` (-> 'running' or 'done'). `_recompute_top_status` itself treats + 'blocked_needing_human' as sticky and refuses to overwrite it, so the status is force-set + to 'running' first so the recompute can actually re-derive it. + + `reset_wall_clock=True` restarts the wall-clock axis (`autonomy.started_at = now`). + `set_max_total_attempts` (an int, None for explicit-unbounded, or the `_UNSET` sentinel + for "leave it alone") re-arms that axis, validated with the same non-negative/finite + rules as --init. + + Fail-safe: after clearing, `breaker_check` is re-run against the (possibly re-armed) + state; if it would IMMEDIATELY re-trip (e.g. total_attempts is still >= max_total_attempts + and no --set-max-total-attempts was given), a loud stderr warning names exactly which + axis and what to raise — but the latch is cleared regardless; the human is in control. + + Returns (ok, error|None, summary|None).""" + if not _is_marathon(state): + return False, "--clear-breaker requires a marathon-stance epic", None + if set_max_total_attempts is not _UNSET: + err = _validate_cap_value("max_total_attempts", set_max_total_attempts) + if err: + return False, err, None + + had_trip = state.pop("breaker_trip", None) is not None + prior_no_progress = state.get("no_progress_cycles", 0) + state["no_progress_cycles"] = 0 + + autonomy = state.setdefault("autonomy", {}) + if reset_wall_clock: + autonomy["started_at"] = _now_iso(now_dt) + if set_max_total_attempts is not _UNSET: + autonomy["max_total_attempts"] = set_max_total_attempts + + # Force off the sticky latch so _recompute_top_status can re-derive the real status + # instead of its no-op early-return for 'blocked_needing_human'. + state["status"] = "running" + state["total_attempts"] = _total_attempts(state) # recompute on every mutation (#9) + _recompute_top_status(state) + + summary = { + "cleared_breaker_trip": had_trip, + "no_progress_cycles_reset_from": prior_no_progress, + "wall_clock_reset": bool(reset_wall_clock), + "max_total_attempts_set": (set_max_total_attempts + if set_max_total_attempts is not _UNSET else None), + "epic_status": state["status"], + } + + recheck = breaker_check(state, now_dt) + summary["would_immediately_retrip"] = recheck["tripped"] + summary["would_retrip_which"] = recheck["which"] + if recheck["tripped"]: + hints = "; ".join("%s (%s)" % (axis, _BREAKER_AXIS_HINTS.get(axis, "raise that cap")) + for axis in recheck["which"]) + print("epic-clear-breaker warning: the latch is cleared, but this state would " + "IMMEDIATELY re-trip on the next --breaker-check: %s" % hints, file=sys.stderr) + + return True, None, summary + + +def clear_disposition(state, feature_id): + """Clears a feature's stored `disposition` (v2.10 resume support). Marathon-only. This + is the override for a sticky `halt_epic`/`halt_feature` verdict — once cleared, + `next_feature_autonomous` no longer short-circuits on it (a fresh disposition can still + be recorded later via `record_disposition`). Returns (ok, error|None).""" + if not _is_marathon(state): + return False, "--clear-disposition requires a marathon-stance epic" + hit = _find_feature(state, feature_id) + if hit is None: + return False, "no feature %r" % feature_id + hit["disposition"] = None + state["total_attempts"] = _total_attempts(state) # recompute on every mutation (#9) + return True, None + + +def mark_sample_audit_due(state, feature_id): + """FIX 2 (v2.10 crash-safety follow-up): sets `feature.sample_audit_due = True` — a + durable obligation recorded BEFORE the PASS sample-audit runs, so a breaker trip/crash + in that window (after a sampled SUCCESS is marked 'done' but before its sample-audit + runs) is never silently lost on resume. `next_feature_autonomous` and + `record_final_review` both gate terminal completion on this flag. Marathon-only. + Returns (ok, error|None).""" + if not _is_marathon(state): + return False, "--mark-sample-audit-due requires a marathon-stance epic" + hit = _find_feature(state, feature_id) + if hit is None: + return False, "no feature %r" % feature_id + hit["sample_audit_due"] = True + state["total_attempts"] = _total_attempts(state) # recompute on every mutation (#9) + # A stale 'done' (set by an earlier passed final_review, before this obligation was + # recorded) must not survive marking a sample-audit due — recompute now, not just at the + # next unrelated mutation. + _recompute_top_status(state) + return True, None + + +def clear_sample_audit_due(state, feature_id): + """The counterpart to `mark_sample_audit_due` — the sample-audit passed, so the + obligation is cleared and terminal completion (and a passed final_review) can proceed + again. Marathon-only. Returns (ok, error|None).""" + if not _is_marathon(state): + return False, "--clear-sample-audit-due requires a marathon-stance epic" + hit = _find_feature(state, feature_id) + if hit is None: + return False, "no feature %r" % feature_id + hit["sample_audit_due"] = False + state["total_attempts"] = _total_attempts(state) # recompute on every mutation (#9) + _recompute_top_status(state) + return True, None + + +def record_audit_failed(state, feature_id, last_error=None): + """FIX B (atomic audit-failed, v2.10 correctness follow-up): the driver's sample-audit- + ISSUES path used to be TWO separate CLI writes — clear_sample_audit_due(...) and a + separate --update --status failed — so a crash between them could leave a feature 'done' + with its sample_audit_due obligation already cleared, and `next_feature_autonomous` / + `record_final_review` / `_recompute_top_status` gate terminal completion ONLY on that + flag, not on the audit's actual verdict. A crash in that window silently loses the audit + finding: final_review could pass over a feature that never actually got fixed. + + This performs every mutation of that sequence against ONE in-memory state object, so the + caller's single `_atomic_write_json` persists all of it in one write or none of it: + - feature.status -> "failed" + - feature.last_error (set iff `last_error` is supplied — same lifecycle as `apply_update`) + - feature.sample_audit_due -> False (the obligation this audit was discharging) + - a passed final_review is invalidated back to pending (same invariant `apply_update` + maintains: ANY real feature-state change invalidates a passed review) + - total_attempts / top-level status recomputed, exactly like every other mutator here + + Deliberately does NOT touch `attempts` — reverting an audit-failed 'done' back to 'failed' + is not a new retry attempt (no ->running transition occurred), so incrementing attempts + here would over-count against max_attempts_per_feature for work that hasn't actually + re-run yet. + + Marathon-only; an unknown feature is a controlled error. Returns (ok, error|None).""" + if not _is_marathon(state): + return False, "--record-audit-failed requires a marathon-stance epic" + hit = _find_feature(state, feature_id) + if hit is None: + return False, "no feature %r" % feature_id + prev_status = hit.get("status") + hit["status"] = "failed" + if last_error is not None: + hit["last_error"] = last_error + hit["sample_audit_due"] = False + if prev_status != "failed": + fr = state.get("final_review") + if isinstance(fr, dict) and fr.get("status") == "passed": + state["final_review"] = {"status": "pending"} + state["total_attempts"] = _total_attempts(state) # recompute on every mutation (#9) + _recompute_top_status(state) + return True, None + + +def validate_marathon_state(state): + """Defensive integrity check for a LOADED marathon state (Codex review #10) — a legacy + or hand-edited epic-state.json can carry corruption the builder would never produce. + Returns a list of error strings (empty = clean). main() aborts with a controlled nonzero + on any error rather than letting a downstream KeyError/TypeError crash the process. + Only applied when `_is_marathon(state)` — a checkpoint state is untouched.""" + errs = [] + feats = state.get("features") + if not isinstance(feats, list): + return ["marathon state has no valid 'features' list"] + for i, f in enumerate(feats): + if not isinstance(f, dict): + errs.append("feature at index %d is not an object" % i) + continue + fid = f.get("id") + if not isinstance(fid, str) or not _id_ok(fid): + errs.append("feature at index %d has a missing/invalid id %r" % (i, fid)) + att = f.get("attempts", 0) + if not _is_nonneg_int(att): + errs.append("feature %r has a malformed 'attempts' %r (want a non-negative int)" + % (f.get("id"), att)) + # Codex round-2 #1: a non-list / non-string-element depends_on later crashes + # _transitive_closure / next_feature_autonomous with a TypeError — reject it here. + deps = f.get("depends_on", []) + if not isinstance(deps, list) or not all(isinstance(d, str) for d in deps): + errs.append("feature %r has a malformed 'depends_on' %r (want a list of id strings)" + % (f.get("id"), deps)) + # FIX 2: sample_audit_due is bool-or-absent — accept a missing key, reject anything + # present that isn't an actual bool (including None). + if "sample_audit_due" in f and not isinstance(f.get("sample_audit_due"), bool): + errs.append("feature %r has a malformed 'sample_audit_due' %r (want a bool or " + "absent)" % (f.get("id"), f.get("sample_audit_due"))) + # FIX A: disposition is an object-or-null; when present its 'attempt' key is + # OPTIONAL (a legacy pre-FIX-A disposition with no 'attempt' key is accepted — treated + # as attempt 0 everywhere it's read, never a crash) but if present must be a + # non-negative int, exactly like a blocker_ledger entry's 'attempt' below. + disp = f.get("disposition") + if disp is not None: + if not isinstance(disp, dict): + errs.append("feature %r has a malformed 'disposition' %r (want an object or " + "null)" % (f.get("id"), disp)) + elif "attempt" in disp and not _is_nonneg_int(disp.get("attempt")): + errs.append("feature %r disposition has a malformed 'attempt' %r (want a " + "non-negative int)" % (f.get("id"), disp.get("attempt"))) + npc = state.get("no_progress_cycles", 0) + if not _is_nonneg_int(npc): + errs.append("'no_progress_cycles' is malformed %r (want a non-negative int)" % (npc,)) + ta = state.get("total_attempts", 0) + if not _is_nonneg_int(ta): + errs.append("'total_attempts' is malformed %r (want a non-negative int)" % (ta,)) + # Codex round-2 #1: a string last_done_count crashes the `done_count > prior` compare in + # --record-progress-cycle; a non-string processed_cycle_ids element breaks replay lookup. + ldc = state.get("last_done_count") + if ldc is not None and not _is_nonneg_int(ldc): + errs.append("'last_done_count' is malformed %r (want a non-negative int or absent)" + % (ldc,)) + pci = state.get("processed_cycle_ids") + if pci is not None and (not isinstance(pci, list) + or not all(isinstance(c, str) for c in pci)): + errs.append("'processed_cycle_ids' is malformed %r (want a list of strings)" % (pci,)) + # Codex round-3 #2: a non-list blocker_ledger crashes at ledger.append; a non-dict entry + # crashes on e.get(...). Require a list of dicts carrying the core keys. + ledger = state.get("blocker_ledger") + if ledger is not None: + if not isinstance(ledger, list): + errs.append("'blocker_ledger' is malformed %r (want a list)" % (ledger,)) + else: + for j, e in enumerate(ledger): + if not isinstance(e, dict): + errs.append("blocker_ledger entry at index %d is not an object" % j) + continue + if not isinstance(e.get("feature"), str): + errs.append("blocker_ledger entry at index %d has a non-string 'feature'" + % j) + if not _is_nonneg_int(e.get("attempt", 0)): + errs.append("blocker_ledger entry at index %d has a malformed 'attempt'" % j) + if not isinstance(e.get("active", False), bool): + errs.append("blocker_ledger entry at index %d has a non-bool 'active'" % j) + autonomy = state.get("autonomy") + if isinstance(autonomy, dict): + for cap_name in ("max_attempts_per_feature", "max_no_progress_cycles", + "max_total_attempts", "max_wall_clock_hours"): + if cap_name in autonomy: + e = _validate_cap_value("autonomy.%s" % cap_name, autonomy[cap_name]) + if e: + errs.append(e) + # Codex round-3 #3: a marathon state must carry a VALID started_at — the wall-clock + # breaker is a safety limit and must never be silently disabled by a missing/bad + # timestamp. Require it here; breaker_check additionally FAILS SAFE (trips) at runtime. + started_at = autonomy.get("started_at") + if not isinstance(started_at, str): + errs.append("autonomy.started_at is missing/non-string %r — the wall-clock breaker " + "needs a valid start time" % (started_at,)) + else: + try: + _parse_iso(started_at) + except (ValueError, TypeError): + errs.append("autonomy.started_at %r is not a valid ISO-8601 timestamp" + % (started_at,)) + # start_sha (v2.10) is string-or-absent — absent is normal (older marathon states, + # or a driver that didn't pass --start-sha at --init time) and not an error. + if "start_sha" in autonomy and not isinstance(autonomy["start_sha"], str): + errs.append("autonomy.start_sha is malformed %r (want a string or absent)" + % (autonomy["start_sha"],)) + return errs + + +def _atomic_write_json(path, obj): + """Atomic state write: tmp file in the SAME directory + os.replace (mirrors the idiom + at compound-v-fastpath-run.py:704) — a reader can never observe a truncated write. + No cross-process lock in v2.10 (the marathon is single-process; see the design spec's + Global Constraints).""" + d = os.path.dirname(os.path.abspath(path)) or "." + if not os.path.isdir(d): + os.makedirs(d) + fd, tmp = tempfile.mkstemp(prefix=".epic-state-", suffix=".json", dir=d) + try: + with os.fdopen(fd, "w") as fh: + fh.write(json.dumps(obj, indent=2) + "\n") + os.replace(tmp, path) + except Exception: + try: + os.unlink(tmp) + except OSError: + pass + raise + + def _read_json(path): with open(path, "r", errors="replace") as fh: return json.load(fh) @@ -342,18 +1564,20 @@ def check(name, cond): # check_specs (#1): missing path + nonexistent-but-contained file (existence + the # containment rule are exercised more fully in the containment block below) - import tempfile + import tempfile as _tempfile check("no spec_path -> error", any("no spec_path" in e for e in check_specs([{"id": "a", "depends_on": []}]))) check("nonexistent contained spec -> error", any("does not exist" in e for e in check_specs( - [{"id": "a", "spec_path": "nope.md"}], base_dir=tempfile.gettempdir()))) + [{"id": "a", "spec_path": "nope.md"}], base_dir=_tempfile.gettempdir()))) # stats (#4) s = stats(build_state([{"id": "a", "depends_on": []}, {"id": "b", "depends_on": ["a"]}], "e", "E")) check("stats total/remaining", s["total"] == 2 and s["remaining"] == 2 and s["done"] == 0) + # Codex round-3 #5: checkpoint --stats must NOT carry a `blocked` key (byte-compat). + check("stats: checkpoint output has NO 'blocked' key", "blocked" not in s) # containment (#2) + check_state_specs (#3) import shutil - d = tempfile.mkdtemp() + d = _tempfile.mkdtemp() try: os.makedirs(os.path.join(d, "specs")) with open(os.path.join(d, "specs", "a.md"), "w") as fh: @@ -378,8 +1602,9 @@ def check(name, cond): nf, _ = next_feature({"features": ["junk", {"id": "a", "status": "pending", "depends_on": []}]}) check("next_feature survives malformed entry", nf is not None and nf["id"] == "a") - # A7: a hand-made feature entry MISSING "status" must not KeyError — it matches no - # bucket (not done/failed/running/pending), and the rest of the epic still advances. + # A7 (legacy #): a hand-made feature entry MISSING "status" must not KeyError — it + # matches no bucket (not done/failed/running/pending), and the rest of the epic still + # advances. nf2, why2 = next_feature({"features": [{"id": "nostatus"}, {"id": "b", "status": "pending", "depends_on": []}]}) check("next_feature survives missing status", nf2 is not None and nf2["id"] == "b") @@ -399,11 +1624,1100 @@ def check(name, cond): small = [{"id": "a", "depends_on": []}, {"id": "b", "depends_on": []}, {"id": "c", "depends_on": ["a", "b"]}] check("small-graph 2 deps not over-coupled", not any("LAYER" in w for w in lint_decomposition(small))) + # ================================================================ + # v2.10 marathon — Unit A (Tasks A1-A7) + # ================================================================ + + # --- A1: marathon schema in build_state + --init --stance --------------------------- + plain = build_state(feats, "e", "Epic") + check("A1 checkpoint build_state: no autonomy key", "autonomy" not in plain) + check("A1 checkpoint build_state: exact top-level keys", + set(plain.keys()) == {"epic_id", "title", "status", "features"}) + check("A1 checkpoint build_state: exact feature keys", + set(plain["features"][0].keys()) == + {"id", "title", "depends_on", "spec_path", "status", "run_id"}) + + mstate = build_state(feats, "e", "Epic", stance="marathon", + caps={"max_wall_clock_hours": 8, + "started_at": "2026-01-01T00:00:00+00:00"}) + check("A1 marathon build_state: autonomy present", "autonomy" in mstate) + auto = mstate["autonomy"] + check("A1 marathon autonomy: every field exactly once", set(auto.keys()) == { + "stance", "max_attempts_per_feature", "max_no_progress_cycles", + "max_total_attempts", "max_wall_clock_hours", "started_at"}) + check("A1: no dead max_features knob in the schema", "max_features" not in auto) + check("A1 marathon autonomy: stance marathon", auto["stance"] == "marathon") + check("A1 marathon autonomy: cap override applied", auto["max_wall_clock_hours"] == 8) + check("A1 marathon autonomy: started_at from caps", + auto["started_at"] == "2026-01-01T00:00:00+00:00") + check("A1 marathon autonomy: default max_total_attempts = max(6,3xfeatures)", + auto["max_total_attempts"] == max(6, 3 * len(feats))) + check("A1 marathon build_state: final_review pending", mstate["final_review"] == {"status": "pending"}) + check("A1 marathon build_state: blocker_ledger empty", mstate["blocker_ledger"] == []) + check("A1 marathon build_state: counters zeroed", + mstate["no_progress_cycles"] == 0 and mstate["total_attempts"] == 0) + check("A1 marathon feature: attempts/last_error/disposition present", + mstate["features"][0]["attempts"] == 0 + and mstate["features"][0]["last_error"] is None + and mstate["features"][0]["disposition"] is None) + + try: + build_state(feats, "../evil", "Epic") + check("A1: traversal epic_id rejected", False) + except ValueError: + check("A1: traversal epic_id rejected", True) + + gd = _tempfile.mkdtemp() + try: + gp = os.path.join(gd, "epic-state.json") + _atomic_write_json(gp, build_state(feats, "e", "Epic")) + check("A1 golden: exactly one artifact (no stray tmp file)", os.listdir(gd) == ["epic-state.json"]) + with open(gp) as fh: + ondisk = json.load(fh) + check("A1 golden: no autonomy key on disk", "autonomy" not in ondisk) + finally: + shutil.rmtree(gd, ignore_errors=True) + + # --- A2: --next --autonomous (DAG-transitive routing, terminal states) -------------- + indep2 = [{"id": "x", "depends_on": []}, {"id": "y", "depends_on": []}, + {"id": "z", "depends_on": ["x"]}] + sta2 = build_state(indep2, "e", "E", stance="marathon", caps={}) + sta2["features"][0]["status"] = "failed" # x failed + # FIX 1: an arbitrated abandon (halt_feature) — the pre-fix "bare failed" behavior for a + # feature that HAS been triaged. The no-disposition crash-window case is covered + # separately below (FIX 1 tests). + sta2["features"][0]["disposition"] = {"disposition": "halt_feature"} + fa, whya, blocked_by = next_feature_autonomous(sta2) + check("A2: independent y still runnable despite x failed", fa is not None and fa["id"] == "y") + check("A2: z (transitive dependent of x) is in blocked_by", blocked_by == ["z"]) + check("A2: running_with_failures token in reason", "running_with_failures" in whya) + + sta2["features"][0]["status"] = "pending" # reopen the source + fa2, whya2, blocked_by2 = next_feature_autonomous(sta2) + check("A2: blocked_by re-derives to empty after reopen", blocked_by2 == []) + + allgraph = [{"id": "a", "depends_on": []}] + st_done = build_state(allgraph, "e", "E", stance="marathon", caps={}) + st_done["features"][0]["status"] = "done" + fa3, whya3, _ = next_feature_autonomous(st_done) + check("A2: all-done but final_review pending -> NOT done", + fa3 is None and "running_with_failures" in whya3 and "final_review" in whya3) + st_done["final_review"]["status"] = "passed" + fa4, whya4, _ = next_feature_autonomous(st_done) + check("A2: all-done + final_review passed -> done", fa4 is None and whya4.startswith("done")) + + st_halt = build_state(indep2, "e", "E", stance="marathon", caps={}) + st_halt["features"][0]["status"] = "failed" + st_halt["features"][0]["disposition"] = {"disposition": "halt_epic"} + fa5, whya5, _ = next_feature_autonomous(st_halt) + check("A2: halt_epic disposition -> blocked_needing_human", + fa5 is None and whya5.startswith("blocked_needing_human")) + + st_tripped = build_state(indep2, "e", "E", stance="marathon", caps={}) + st_tripped["status"] = "blocked_needing_human" + fa6, whya6, _ = next_feature_autonomous(st_tripped) + check("A2: pre-tripped state -> blocked_needing_human", + fa6 is None and whya6.startswith("blocked_needing_human")) + + only_dep = [{"id": "x", "depends_on": []}, {"id": "z", "depends_on": ["x"]}] + st_exh = build_state(only_dep, "e", "E", stance="marathon", caps={}) + st_exh["features"][0]["status"] = "failed" + st_exh["features"][0]["disposition"] = {"disposition": "halt_feature"} # arbitrated abandon + fa7, whya7, blocked_by7 = next_feature_autonomous(st_exh) + check("A2: exhausted reachable work -> blocked_needing_human", + fa7 is None and whya7.startswith("blocked_needing_human")) + check("A2: blocked_by includes the exhausted dependent", blocked_by7 == ["z"]) + + check("A2: default --next 2-tuple shape unchanged", + next_feature(sta2) == next_feature(sta2)) # sanity: still unpacks as 2-tuple + + # A2 precedence (cross-model review correction #1, CRITICAL): a runnable INDEPENDENT + # feature must be returned BEFORE any terminal blocked_needing_human — a suspected + # blocker must never halt the epic while independent work is still runnable. + prec = [{"id": "A", "depends_on": []}, {"id": "B", "depends_on": ["A"]}, + {"id": "C", "depends_on": []}] + st_prec = build_state(prec, "e", "E", stance="marathon", caps={}) + st_prec["features"][0]["status"] = "blocked" # A blocked; B depends on A; C independent + fp, whyp, blocked_by_p = next_feature_autonomous(st_prec) + check("A2 precedence: returns runnable independent C, not blocked_needing_human", + fp is not None and fp["id"] == "C") + check("A2 precedence: reason is running_with_failures, NOT blocked_needing_human", + "running_with_failures" in whyp and not whyp.startswith("blocked_needing_human")) + check("A2 precedence: B (dependent on A) is in blocked_by, C is not", + blocked_by_p == ["B"]) + + # --- FIX 1 (crash/breaker-window resume safety): a failed feature routes BY its stored + # disposition instead of being blanket-treated as abandoned ------------------------- + fix1_feats = [{"id": "x", "depends_on": []}, {"id": "y", "depends_on": []}, + {"id": "z", "depends_on": ["x"]}] + + st_f1 = build_state(fix1_feats, "e", "E", stance="marathon", caps={}) + st_f1["features"][0]["status"] = "failed" + st_f1["features"][0]["attempts"] = 1 # < default cap (2) -> can_retry True + # FIX A: disposition.attempt must match the feature's current attempts to be honored. + st_f1["features"][0]["disposition"] = {"disposition": "retry_fix", "attempt": 1} + f_f1, why_f1, blocked_by_f1 = next_feature_autonomous(st_f1) + check("FIX1: retry_fix + can_retry -> the failed feature itself is runnable", + f_f1 is not None and f_f1["id"] == "x" and why_f1 == "runnable") + check("FIX1: retry_fix + can_retry does NOT block its dependents (about to be retried)", + blocked_by_f1 == []) + + st_f2 = build_state(fix1_feats, "e", "E", stance="marathon", caps={}) + st_f2["features"][0]["status"] = "failed" + st_f2["features"][0]["attempts"] = 2 # == default cap (2) -> can_retry False + # FIX A: disposition.attempt must match the feature's current attempts to be honored. + st_f2["features"][0]["disposition"] = {"disposition": "retry_fix", "attempt": 2} + f_f2, why_f2, blocked_by_f2 = next_feature_autonomous(st_f2) + check("FIX1: retry_fix + cap exhausted -> abandoned, independent y still runnable", + f_f2 is not None and f_f2["id"] == "y") + check("FIX1: retry_fix cap-exhausted blocks the transitive dependent z", + blocked_by_f2 == ["z"]) + check("FIX1: retry_fix cap-exhausted reason carries running_with_failures", + "running_with_failures" in why_f2) + + st_f3 = build_state(fix1_feats, "e", "E", stance="marathon", caps={}) + st_f3["features"][0]["status"] = "failed" + st_f3["features"][0]["disposition"] = {"disposition": "halt_feature"} + f_f3, why_f3, blocked_by_f3 = next_feature_autonomous(st_f3) + check("FIX1: halt_feature -> abandoned, independent y still runnable", + f_f3 is not None and f_f3["id"] == "y") + check("FIX1: halt_feature blocks the transitive dependent z", blocked_by_f3 == ["z"]) + + st_f4 = build_state(fix1_feats, "e", "E", stance="marathon", caps={}) + st_f4["features"][0]["status"] = "failed" # disposition left None (build_state default) + f_f4, why_f4, blocked_by_f4 = next_feature_autonomous(st_f4) + check("FIX1: failed with NO recorded disposition -> needs_arbitration, no work handed out", + f_f4 is None and why_f4.startswith("needs_arbitration")) + check("FIX1: needs_arbitration names the untriaged feature", + "feature x failed without a recorded disposition" in why_f4) + + # --- FIX A (attempt-binding, v2.10 correctness follow-up): a stored disposition is + # honored ONLY for the attempt it was recorded against; a stale disposition left over + # from a prior (already-retried) attempt is treated as no disposition at all -------- + _T_FA = _parse_iso("2026-01-01T00:00:00+00:00") + + fa_feats = [{"id": "a", "depends_on": []}] + st_fa1 = build_state(fa_feats, "e", "E", stance="marathon", caps={"max_attempts_per_feature": 5}) + apply_update(st_fa1, "a", "running", now_dt=_T_FA) # attempts -> 1 + apply_update(st_fa1, "a", "failed", now_dt=_T_FA) + ok_fa1, _ = record_disposition(st_fa1, "a", "retry_fix", now_dt=_T_FA) # attempt stamped = 1 + check("FIX A: record_disposition stamps attempt = feature's current attempts", + ok_fa1 and st_fa1["features"][0]["disposition"]["attempt"] == 1) + f_fa1a, why_fa1a, _ = next_feature_autonomous(st_fa1) + check("FIX A: retry_fix recorded at the current attempt -> runnable", + f_fa1a is not None and f_fa1a["id"] == "a" and why_fa1a == "runnable") + + # The driver re-runs the retry (attempts -> 2) and it fails AGAIN, crashing before a + # fresh arbitration — the disposition on disk is still stamped attempt=1, now stale. + apply_update(st_fa1, "a", "running", now_dt=_T_FA) # attempts -> 2 + apply_update(st_fa1, "a", "failed", now_dt=_T_FA) + f_fa1b, why_fa1b, _ = next_feature_autonomous(st_fa1) + check("FIX A: a stale retry_fix (attempt mismatch) after re-run -> needs_arbitration, " + "NOT a silently-honored second retry", + f_fa1b is None and why_fa1b.startswith("needs_arbitration")) + check("FIX A: needs_arbitration (stale disposition) names the untriaged feature", + "feature a failed without a recorded disposition" in why_fa1b) + + fb_feats = [{"id": "a", "depends_on": []}, {"id": "b", "depends_on": []}] + st_fb = build_state(fb_feats, "e", "E", stance="marathon", caps={}) + apply_update(st_fb, "a", "running", now_dt=_T_FA) # attempts -> 1 + apply_update(st_fb, "a", "failed", now_dt=_T_FA) + record_disposition(st_fb, "a", "halt_feature", now_dt=_T_FA) # attempt stamped = 1 + f_fb1, why_fb1, blocked_by_fb1 = next_feature_autonomous(st_fb) + check("FIX A: a matching halt_feature (attempt matches) -> abandoned, independent b runnable", + f_fb1 is not None and f_fb1["id"] == "b" and "running_with_failures" in why_fb1) + + # Re-run + fail again WITHOUT a fresh disposition -> the recorded halt_feature is now stale. + apply_update(st_fb, "a", "running", now_dt=_T_FA) # attempts -> 2 + apply_update(st_fb, "a", "failed", now_dt=_T_FA) + f_fb2, why_fb2, _ = next_feature_autonomous(st_fb) + check("FIX A: an attempt-mismatch halt_feature -> needs_arbitration (stale verdict, " + "re-arbitrate), NOT a silent abandon", + f_fb2 is None and why_fb2.startswith("needs_arbitration")) + + # --- FIX A: validate_marathon_state accepts a legacy disposition with no 'attempt' key + # (treated as attempt 0) but flags a malformed one -------------------------------------- + st_fa_legacy = build_state(fa_feats, "e", "E", stance="marathon", caps={}) + st_fa_legacy["features"][0]["disposition"] = {"disposition": "halt_feature"} # no attempt key + check("FIX A: validate_marathon_state accepts a legacy disposition with no 'attempt' key", + validate_marathon_state(st_fa_legacy) == []) + st_fa_bad = build_state(fa_feats, "e", "E", stance="marathon", caps={}) + st_fa_bad["features"][0]["disposition"] = {"disposition": "halt_feature", "attempt": -1} + check("FIX A: validate_marathon_state flags a malformed (negative) disposition.attempt", + validate_marathon_state(st_fa_bad) != []) + st_fa_bad2 = build_state(fa_feats, "e", "E", stance="marathon", caps={}) + st_fa_bad2["features"][0]["disposition"] = "not-an-object" + check("FIX A: validate_marathon_state flags a non-object disposition", + validate_marathon_state(st_fa_bad2) != []) + + # --- FIX 2 (v2.10, cross-unit integration review follow-up): a `failed` feature whose + # attempt-matched disposition is blocked_external must surface needs_blocker_recording + # (the blocked/ledger transition crashed mid-flight), NOT be treated as abandoned -------- + fix2_feats = [{"id": "a", "depends_on": []}, {"id": "b", "depends_on": []}, + {"id": "c", "depends_on": ["a"]}] + st_fx1 = build_state(fix2_feats, "e", "E", stance="marathon", caps={}) + apply_update(st_fx1, "a", "running", now_dt=_T_FA) # attempts -> 1 + apply_update(st_fx1, "a", "failed", now_dt=_T_FA) + record_disposition(st_fx1, "a", "blocked_external", now_dt=_T_FA) # attempt stamped = 1 + f_fx1, why_fx1, blocked_by_fx1 = next_feature_autonomous(st_fx1) + check("FIX 2: failed + attempt-matched blocked_external -> needs_blocker_recording, " + "not abandoned, no work handed out", + f_fx1 is None and why_fx1.startswith("needs_blocker_recording")) + check("FIX 2: needs_blocker_recording names the untriaged feature and points at the fix", + "feature(s) a" in why_fx1 and "--update --status blocked" in why_fx1) + check("FIX 2: needs_blocker_recording halts ALL routing (independent b not handed out " + "as if settled), mirroring needs_arbitration's priority", + f_fx1 is None) + check("FIX 2: transitive dependent c is reported as blocked_by", + blocked_by_fx1 == ["c"]) + + # Completing the transition (driver re-runs --update --status blocked) moves the feature + # OFF the needs_blocker_recording path and onto the normal blocked/ledger path -> b + # becomes runnable, independently of the still-blocked a/c. + apply_update(st_fx1, "a", "blocked", blocker_reason="vendor outage", now_dt=_T_FA) + check("FIX 2: completing the transition sets status=blocked and appends one ledger entry", + st_fx1["features"][0]["status"] == "blocked" and len(st_fx1["blocker_ledger"]) == 1) + f_fx1b, why_fx1b, blocked_by_fx1b = next_feature_autonomous(st_fx1) + check("FIX 2: after the transition completes, a blocked-status feature stays on the " + "normal blocked/ledger path (unchanged) -- independent b is runnable", + f_fx1b is not None and f_fx1b["id"] == "b" and "running_with_failures" in why_fx1b) + check("FIX 2: after the transition completes, c is still blocked_by a", + blocked_by_fx1b == ["c"]) + + # Re-running --update --status blocked a second time (the idempotent-recovery case the + # fix explicitly relies on) must not create a second ledger entry or otherwise change + # anything observable. + apply_update(st_fx1, "a", "blocked", blocker_reason="vendor outage (replay)", now_dt=_T_FA) + check("FIX 2: re-running --update --status blocked is idempotent by (feature, attempt) " + "-- still exactly one ledger entry", + len(st_fx1["blocker_ledger"]) == 1) + + # An attempt-MISMATCHED blocked_external disposition is unaffected by FIX 2 -- it still + # falls into needs_arbitration (FIX A, unchanged). + st_fx2 = build_state(fa_feats, "e", "E", stance="marathon", caps={"max_attempts_per_feature": 5}) + apply_update(st_fx2, "a", "running", now_dt=_T_FA) # attempts -> 1 + apply_update(st_fx2, "a", "failed", now_dt=_T_FA) + record_disposition(st_fx2, "a", "blocked_external", now_dt=_T_FA) # attempt stamped = 1 + apply_update(st_fx2, "a", "running", now_dt=_T_FA) # attempts -> 2, disposition now stale + apply_update(st_fx2, "a", "failed", now_dt=_T_FA) + f_fx2, why_fx2, _ = next_feature_autonomous(st_fx2) + check("FIX 2: attempt-mismatched blocked_external -> needs_arbitration (unchanged from " + "FIX A), NOT needs_blocker_recording", + f_fx2 is None and why_fx2.startswith("needs_arbitration")) + + # A feature that is ALREADY status=='blocked' with its ledger entry present never enters + # the failed-feature classification loop at all -- normal path, unaffected by FIX 2. + st_fx3 = build_state(fa_feats, "e", "E", stance="marathon", caps={}) + apply_update(st_fx3, "a", "running", now_dt=_T_FA) + apply_update(st_fx3, "a", "failed", now_dt=_T_FA) + record_disposition(st_fx3, "a", "blocked_external", now_dt=_T_FA) + apply_update(st_fx3, "a", "blocked", blocker_reason="vendor outage", now_dt=_T_FA) + f_fx3, why_fx3, _ = next_feature_autonomous(st_fx3) + check("FIX 2: an already-blocked-status feature (transition complete) stays on the " + "normal blocked path -- not needs_blocker_recording", + not (why_fx3 or "").startswith("needs_blocker_recording")) + check("FIX 2: an already-blocked-status single-feature epic -> blocked_needing_human " + "(no other work), exactly as before this fix", + f_fx3 is None and why_fx3.startswith("blocked_needing_human")) + + # --- A3: attempts + --can-retry + transition table ----------------------------------- + feats3 = [{"id": "a", "depends_on": []}] + st3m = build_state(feats3, "e", "E", stance="marathon", caps={"max_attempts_per_feature": 2}) + ok3, err3 = apply_update(st3m, "a", "running", now_dt=_parse_iso("2026-01-01T00:00:00+00:00")) + check("A3: pending->running ok + attempts=1", ok3 and st3m["features"][0]["attempts"] == 1) + check("A3: total_attempts tracks the feature increment", st3m["total_attempts"] == 1) + ok3b, _ = apply_update(st3m, "a", "done") + check("A3: running->done ok", ok3b) + ok3c, err3c = apply_update(st3m, "a", "running") + check("A3: done->running rejected", not ok3c and "illegal transition" in err3c) + info3 = can_retry_info(st3m, "a") + check("A3: can_retry attempts=1 cap=2 -> true", + info3["can_retry"] is True and info3["attempts"] == 1 and info3["cap"] == 2) + apply_update(st3m, "a", "failed") + apply_update(st3m, "a", "running") # attempts -> 2 + info3b = can_retry_info(st3m, "a") + check("A3: can_retry flips at the cap", info3b["can_retry"] is False and info3b["attempts"] == 2) + + st3c = build_state(feats3, "e", "E") # checkpoint (non-marathon) + apply_update(st3c, "a", "running") + check("A3 checkpoint: no attempts field ever added", "attempts" not in st3c["features"][0]) + apply_update(st3c, "a", "done") + ok3d, _ = apply_update(st3c, "a", "running") + check("A3 checkpoint: transition table is marathon-only (done->running not rejected)", ok3d) + + # total_attempts invariant (cross-model review correction #4, HIGH): it is ALWAYS the + # live sum of feature.attempts, never an independently-incremented counter that could + # drift — exercise it across several features and several transitions. + feats3d = [{"id": "a", "depends_on": []}, {"id": "b", "depends_on": []}] + st3d = build_state(feats3d, "e", "E", stance="marathon", caps={}) + apply_update(st3d, "a", "running") + apply_update(st3d, "a", "failed") + apply_update(st3d, "a", "running") + apply_update(st3d, "b", "running") + check("A3: total_attempts == sum(feature.attempts) after several transitions", + st3d["total_attempts"] == _total_attempts(st3d) == 3) + + # --- A4: --record-disposition --------------------------------------------------------- + feats4 = [{"id": "a", "depends_on": []}] + st4 = build_state(feats4, "e", "E", stance="marathon", caps={}) + ok4, err4 = record_disposition(st4, "a", "retry_fix", reason="flaky test", + families_agreeing=["gpt"]) + check("A4: record_disposition round-trips", + ok4 and st4["features"][0]["disposition"]["disposition"] == "retry_fix") + check("A4: confirmed is always false", st4["features"][0]["disposition"]["confirmed"] is False) + ok5, err5 = record_disposition(st4, "a", "blocked_external", confirmed=True) + check("A4: --confirmed true hard-rejected", not ok5 and "confirmed" in err5.lower()) + check("A4: prior disposition untouched by the rejected call", + st4["features"][0]["disposition"]["disposition"] == "retry_fix") + ok6, _ = record_disposition(st4, "a", "halt_epic") + check("A4: halt_epic round-trips", ok6) + fah, whyah, _ = next_feature_autonomous(st4) + check("A4: halt_epic disposition drives autonomous routing", + fah is None and whyah.startswith("blocked_needing_human")) + + st4b = build_state(feats4, "e", "E", stance="marathon", caps={}) + okb, errb = apply_update(st4b, "a", "blocked", blocker_confirmed=True) + check("A4/A5: --blocker-confirmed true hard-rejected on --update too", + not okb and "confirmed" in errb.lower()) + + # --- A5: blocker-ledger lifecycle ------------------------------------------------------ + feats5 = [{"id": "a", "depends_on": []}, {"id": "b", "depends_on": ["a"]}] + st5 = build_state(feats5, "e", "E", stance="marathon", caps={}) + ok5a, _ = apply_update(st5, "a", "blocked", blocker_reason="waiting on vendor", + now_dt=_parse_iso("2026-01-01T00:00:00+00:00")) + check("A5: blocked appends one ledger entry", ok5a and len(st5["blocker_ledger"]) == 1) + entry = st5["blocker_ledger"][0] + check("A5: entry active + confirmed:false + blocks derived", + entry["active"] is True and entry["confirmed"] is False and entry["blocks"] == ["b"]) + apply_update(st5, "a", "blocked", blocker_reason="dup") # replay, same attempt (0) + check("A5: replay of the same (feature,attempt) is idempotent", len(st5["blocker_ledger"]) == 1) + apply_update(st5, "a", "pending", now_dt=_parse_iso("2026-01-02T00:00:00+00:00")) + check("A5: --status pending resolves the active entry", + st5["blocker_ledger"][0]["active"] is False + and st5["blocker_ledger"][0]["resolved_at"] is not None) + apply_update(st5, "a", "running") + apply_update(st5, "a", "done") + check("A5: --stats no longer counts a since-succeeded blocker", stats(st5)["blocked"] == 0) + st5["features"][0]["status"] = "failed" # simulate a later failure -> attempts now 1 + apply_update(st5, "a", "blocked", blocker_reason="again") + check("A5: a block at a NEW attempt count is a new ledger entry", len(st5["blocker_ledger"]) == 2) + check("A5: --stats breaks out the currently-blocked feature", stats(st5)["blocked"] == 1) + + # --- A6: final-review gate -------------------------------------------------------------- + feats6 = [{"id": "a", "depends_on": []}] + st6 = build_state(feats6, "e", "E", stance="marathon", caps={}) + apply_update(st6, "a", "running") + apply_update(st6, "a", "done") + check("A6: all-done but final_review pending -> top status stays 'running'", + st6["status"] == "running") + fa6b, why6b, _ = next_feature_autonomous(st6) + check("A6: autonomous NOT 'done' while final_review pending", + fa6b is None and "running_with_failures" in why6b) + okr, _ = record_final_review(st6, "passed") + check("A6: record_final_review flips top status to done", okr and st6["status"] == "done") + fa6c, why6c, _ = next_feature_autonomous(st6) + check("A6: autonomous 'done' once final_review passed", + fa6c is None and why6c.startswith("done")) + okr2, _ = record_final_review(st6, "bogus") + check("A6: invalid final-review status rejected", not okr2) + + # --- FIX 2 (crash/breaker-window resume safety): a durable sample-audit obligation + # gates terminal completion -------------------------------------------------------------- + feats_sa = [{"id": "a", "depends_on": []}] + st_sa = build_state(feats_sa, "e", "E", stance="marathon", caps={}) + apply_update(st_sa, "a", "running") + apply_update(st_sa, "a", "done") + record_final_review(st_sa, "passed") + fa_sa0, why_sa0, _ = next_feature_autonomous(st_sa) + check("FIX2 setup: autonomous routing reports 'done' before any sample-audit is due", + fa_sa0 is None and why_sa0.startswith("done")) + + ok_mark, err_mark = mark_sample_audit_due(st_sa, "a") + check("FIX2: mark_sample_audit_due ok", ok_mark and err_mark is None) + check("FIX2: feature.sample_audit_due is True", st_sa["features"][0]["sample_audit_due"] is True) + fa_sa1, why_sa1, _ = next_feature_autonomous(st_sa) + check("FIX2: mark due -> next_feature_autonomous no longer reports 'done'", + fa_sa1 is None and why_sa1.startswith("sample_audit_due")) + check("FIX2: sample_audit_due reason names the awaiting feature", + "feature(s) a awaiting sample-audit" in why_sa1) + + ok_fr_reject, err_fr_reject = record_final_review(st_sa, "passed") + check("FIX2: record_final_review passed REJECTED while sample_audit_due is true", + ok_fr_reject is False and "sample_audit_due" in err_fr_reject) + check("FIX2: a rejected final_review=passed does not flip status to done", + st_sa["status"] != "done") + + ok_clear, err_clear = clear_sample_audit_due(st_sa, "a") + check("FIX2: clear_sample_audit_due ok", ok_clear and err_clear is None) + check("FIX2: feature.sample_audit_due is False after clear", + st_sa["features"][0]["sample_audit_due"] is False) + fa_sa2, why_sa2, _ = next_feature_autonomous(st_sa) + check("FIX2: clear due -> normal completion resumes ('done')", + fa_sa2 is None and why_sa2.startswith("done")) + ok_fr2, err_fr2 = record_final_review(st_sa, "passed") + check("FIX2: record_final_review passed accepted again once due is cleared", + ok_fr2 and err_fr2 is None) + + # Crash simulation: a 'done' feature with sample_audit_due left set (as if the process + # died between marking the sampled SUCCESS 'done' and clearing the audit) must be + # surfaced on the very next --next --autonomous — never silently skipped to 'done', and + # never allowed to acquire a passed final_review in the meantime. + st_sa2 = build_state(feats_sa, "e", "E", stance="marathon", caps={}) + apply_update(st_sa2, "a", "running") + apply_update(st_sa2, "a", "done") + mark_sample_audit_due(st_sa2, "a") + fa_sa3, why_sa3, _ = next_feature_autonomous(st_sa2) + check("FIX2 crash-sim: sample_audit_due surfaces immediately on resume " + "(final_review still pending)", + fa_sa3 is None and why_sa3.startswith("sample_audit_due")) + ok_fr3, _ = record_final_review(st_sa2, "passed") + check("FIX2 crash-sim: final_review=passed still rejected on resume", ok_fr3 is False) + + # --- --mark/clear-sample-audit-due: error handling --------------------------------------- + ok_unk, err_unk = mark_sample_audit_due(st_sa, "does-not-exist") + check("FIX2: mark_sample_audit_due on unknown feature -> controlled error", + ok_unk is False and "no feature" in err_unk) + ok_unk2, err_unk2 = clear_sample_audit_due(st_sa, "does-not-exist") + check("FIX2: clear_sample_audit_due on unknown feature -> controlled error", + ok_unk2 is False and "no feature" in err_unk2) + + st_sa_chk = build_state(feats_sa, "e", "E") # checkpoint (non-marathon) + ok_chk, err_chk = mark_sample_audit_due(st_sa_chk, "a") + check("FIX2: mark_sample_audit_due on a checkpoint state -> controlled error", + ok_chk is False and "marathon" in err_chk) + ok_chk2, err_chk2 = clear_sample_audit_due(st_sa_chk, "a") + check("FIX2: clear_sample_audit_due on a checkpoint state -> controlled error", + ok_chk2 is False and "marathon" in err_chk2) + + # --- validate_marathon_state: sample_audit_due is bool-or-absent ------------------------- + good_sa = build_state(feats_sa, "e", "E", stance="marathon", caps={}) + check("FIX2: absent sample_audit_due validates clean", + validate_marathon_state(good_sa) == []) + good_sa["features"][0]["sample_audit_due"] = True + check("FIX2: a bool True sample_audit_due validates clean", + validate_marathon_state(good_sa) == []) + good_sa["features"][0]["sample_audit_due"] = False + check("FIX2: a bool False sample_audit_due validates clean", + validate_marathon_state(good_sa) == []) + bad_sa = build_state(feats_sa, "e", "E", stance="marathon", caps={}) + bad_sa["features"][0]["sample_audit_due"] = "yes" + check("FIX2: a non-bool sample_audit_due -> validation error", + validate_marathon_state(bad_sa) != []) + + # --- FIX B (atomic audit-failed, v2.10 correctness follow-up): --record-audit-failed + # reverts a done+due feature to failed, clears sample_audit_due, and invalidates a + # passed final_review, all in ONE state mutation — the driver's old two-write + # clear-due-then-mark-failed sequence had a crash window between the two writes that + # could leave a feature 'done' with no audit obligation at all. ----------------------- + feats_raf = [{"id": "a", "depends_on": []}] + st_raf = build_state(feats_raf, "e", "E", stance="marathon", caps={}) + apply_update(st_raf, "a", "running") + apply_update(st_raf, "a", "done") + mark_sample_audit_due(st_raf, "a") + record_final_review(st_raf, "passed") # rejected (sample_audit_due) — stays pending + ok_raf, err_raf = record_audit_failed(st_raf, "a", last_error="sample audit found a regression") + check("FIX B: record_audit_failed ok", ok_raf and err_raf is None) + check("FIX B: single write yields status=failed", st_raf["features"][0]["status"] == "failed") + check("FIX B: single write yields sample_audit_due=False (never done-without-due)", + st_raf["features"][0]["sample_audit_due"] is False) + check("FIX B: last_error persisted", st_raf["features"][0]["last_error"] + == "sample audit found a regression") + check("FIX B: final_review is not 'passed' after the atomic revert", + st_raf["final_review"]["status"] != "passed") + check("FIX B: top-level status is not 'done'", st_raf["status"] != "done") + # No intermediate 'done-without-due' state was ever observable: this is the SAME state + # object that would be serialized in one _atomic_write_json call by the CLI, so there is + # only ever one post-write state to inspect (simulating "no crash window"). + check("FIX B: never observes done-without-due (single atomic write, no split state)", + not (st_raf["features"][0]["status"] == "done" + and not st_raf["features"][0]["sample_audit_due"])) + + # A feature that passed its audit cleanly (never marked done here, so its final_review is + # untouched pending) still gets reverted correctly when audit-failed is recorded directly + # from 'running' too (not just from 'done') — attempts are NOT bumped by this call. + st_raf2 = build_state(feats_raf, "e", "E", stance="marathon", caps={}) + apply_update(st_raf2, "a", "running") + prior_attempts = st_raf2["features"][0]["attempts"] + ok_raf2, _ = record_audit_failed(st_raf2, "a") + check("FIX B: record_audit_failed does not touch attempts (not a new retry)", + ok_raf2 and st_raf2["features"][0]["attempts"] == prior_attempts) + check("FIX B: last_error left untouched when --last-error is not supplied", + st_raf2["features"][0]["last_error"] is None) + + ok_raf_unk, err_raf_unk = record_audit_failed(st_raf, "does-not-exist") + check("FIX B: record_audit_failed on unknown feature -> controlled error", + ok_raf_unk is False and "no feature" in err_raf_unk) + + st_raf_chk = build_state(feats_raf, "e", "E") # checkpoint (non-marathon) + ok_raf_chk, err_raf_chk = record_audit_failed(st_raf_chk, "a") + check("FIX B: record_audit_failed on a checkpoint state -> controlled error", + ok_raf_chk is False and "marathon" in err_raf_chk) + + # --- A7: global breakers ------------------------------------------------------------------ + feats7 = [{"id": "a", "depends_on": []}, {"id": "b", "depends_on": []}] + st7 = build_state(feats7, "e", "E", stance="marathon", + caps={"max_total_attempts": 3, "max_no_progress_cycles": 2, + "max_wall_clock_hours": 1, + "started_at": "2026-01-01T00:00:00+00:00"}) + now0 = _parse_iso("2026-01-01T00:00:00+00:00") + check("A7: breaker-check not tripped at baseline", breaker_check(st7, now0)["tripped"] is False) + # total_attempts is DERIVED live from feature.attempts (the invariant), never a + # trust-the-stored-counter check — set feature attempts directly, not the top-level field. + st7["features"][0]["attempts"] = 2 + st7["features"][1]["attempts"] = 0 + check("A7: total_attempts (sum of feature.attempts) just under cap -> not tripped", + breaker_check(st7, now0)["tripped"] is False) + st7["features"][0]["attempts"] = 3 + r_ta = breaker_check(st7, now0) + check("A7: total_attempts >= cap trips exactly at the boundary", + r_ta["tripped"] and "max_total_attempts" in r_ta["which"]) + check("A7: the derived total_attempts equals sum(feature.attempts)", + r_ta["detail"]["max_total_attempts"]["value"] == _total_attempts(st7) == 3) + check("A7: --breaker-check never mutates state", st7.get("status") != "blocked_needing_human") + st7["features"][0]["attempts"] = 0 + st7["features"][1]["attempts"] = 0 + st7["no_progress_cycles"] = 1 + check("A7: no_progress_cycles just under cap -> not tripped", + breaker_check(st7, now0)["tripped"] is False) + st7["no_progress_cycles"] = 2 + r_np = breaker_check(st7, now0) + check("A7: no_progress_cycles >= cap trips", + r_np["tripped"] and "max_no_progress_cycles" in r_np["which"]) + st7["no_progress_cycles"] = 0 + r_wc_ok = breaker_check(st7, _parse_iso("2026-01-01T00:59:00+00:00")) + check("A7: wall-clock just under cap -> not tripped", r_wc_ok["tripped"] is False) + r_wc_trip = breaker_check(st7, _parse_iso("2026-01-01T01:00:00+00:00")) + check("A7: wall-clock >= cap trips", + r_wc_trip["tripped"] and "max_wall_clock_hours" in r_wc_trip["which"]) + + st7b = build_state(feats7, "e", "E", stance="marathon", caps={"max_total_attempts": 1}) + res_notrip = trip_breaker(st7b, now0) + check("A7: trip_breaker is a no-op when not tripped", + res_notrip["tripped"] is False and st7b["status"] != "blocked_needing_human") + st7b["features"][0]["attempts"] = 1 + res_trip = trip_breaker(st7b, now0) + check("A7: trip_breaker sets blocked_needing_human when tripped", + res_trip["tripped"] and st7b["status"] == "blocked_needing_human") + + feats7c = [{"id": "a", "depends_on": []}, {"id": "b", "depends_on": []}] + st7c = build_state(feats7c, "e", "E", stance="marathon", caps={}) + r1 = record_progress_cycle(st7c, "cycle-1") + check("A7: first progress cycle establishes a baseline (no false trip)", + r1["no_progress_cycles"] == 0 and r1["replayed"] is False) + r2 = record_progress_cycle(st7c, "cycle-2") + check("A7: no new done -> no_progress_cycles increments", r2["no_progress_cycles"] == 1) + st7c["features"][0]["status"] = "done" + r3 = record_progress_cycle(st7c, "cycle-3") + check("A7: a new done resets no_progress_cycles", r3["no_progress_cycles"] == 0) + r3_replay = record_progress_cycle(st7c, "cycle-3") + check("A7: a replayed cycle-id is an idempotent no-op", + r3_replay["replayed"] is True and r3_replay["no_progress_cycles"] == 0) + + # Codex review #4: a negative cap is now REJECTED at --init/build, not silently treated + # as unbounded. + try: + build_state(feats7, "e", "E", stance="marathon", caps={"max_total_attempts": -1}) + check("#4: negative cap rejected at build/--init", False) + except ValueError: + check("#4: negative cap rejected at build/--init", True) + try: + build_state(feats7, "e", "E", stance="marathon", caps={"max_wall_clock_hours": "lots"}) + check("#4: non-numeric cap rejected at build/--init", False) + except ValueError: + check("#4: non-numeric cap rejected at build/--init", True) + st7e = build_state(feats7, "e", "E", stance="marathon", caps={"max_wall_clock_hours": None}) + check("A7: an explicit null wall-clock cap is unbounded", + breaker_check(st7e, _parse_iso("2027-01-01T00:00:00+00:00"))["tripped"] is False) + # #4: a MISSING cap key falls back to the documented default (fail-SAFE), never unbounded. + st7f = build_state(feats7, "e", "E", stance="marathon", + caps={"started_at": "2026-01-01T00:00:00+00:00"}) + del st7f["autonomy"]["max_wall_clock_hours"] # simulate a hand-edited state missing the key + r_missing = breaker_check(st7f, _parse_iso("2026-01-01T11:00:00+00:00")) + check("#4: a MISSING cap key uses the documented default (10h), not unbounded", + r_missing["tripped"] is True and "max_wall_clock_hours" in r_missing["which"]) + + # ================================================================ + # Codex cross-model review — the 11 findings, one+ selftest each + # ================================================================ + T0 = _parse_iso("2026-01-01T00:00:00+00:00") + + # --- #1 CRITICAL: final-review bypass ------------------------------------------------ + r1feats = [{"id": "a", "depends_on": []}, {"id": "b", "depends_on": []}] + r1 = build_state(r1feats, "e", "E", stance="marathon", caps={}) + apply_update(r1, "a", "running", now_dt=T0) + apply_update(r1, "a", "done", now_dt=T0) # b still pending + okp, errp = record_final_review(r1, "passed") + check("#1: final_review=passed REJECTED unless all features done", + okp is False and "not done" in errp) + check("#1: rejected review did not flip status to done", r1["status"] != "done") + apply_update(r1, "b", "running", now_dt=T0) + apply_update(r1, "b", "done", now_dt=T0) + okp2, _ = record_final_review(r1, "passed") + check("#1: passes once all done -> status done", okp2 and r1["status"] == "done") + # a feature --update after a pass INVALIDATES the review and drops status off 'done' + apply_update(r1, "a", "pending", now_dt=T0) + check("#1: --update after pass resets final_review to pending", + r1["final_review"]["status"] == "pending") + check("#1: ...and top status leaves 'done'", r1["status"] != "done") + + # --- #2 HIGH: marathon commands bound to persisted stance ---------------------------- + r2chk = build_state([{"id": "a", "depends_on": []}], "e", "E") # checkpoint + okb2, errb2 = apply_update(r2chk, "a", "blocked") + check("#2: checkpoint --update --status blocked is rejected", + okb2 is False and "checkpoint" in errb2) + check("#2: checkpoint status set unchanged (still 4)", + CHECKPOINT_STATUSES == ("pending", "running", "done", "failed")) + # can_retry/breaker helpers still importable, but the CLI-level rejection is what #2 asks; + # exercise the persisted-stance gate via a round-trip in the CLI smoke below is heavy — + # here assert the discriminator the guard uses: + check("#2: _is_marathon discriminates checkpoint vs marathon", + _is_marathon(r2chk) is False and _is_marathon(r1) is True) + + # --- #3 HIGH: runnable EXCLUDES transitive dependents -------------------------------- + r3feats = [{"id": "A", "depends_on": []}, {"id": "B", "depends_on": ["A"]}, + {"id": "C", "depends_on": ["B"]}] + r3 = build_state(r3feats, "e", "E", stance="marathon", caps={}) + r3["features"][0]["status"] = "failed" # A failed + r3["features"][1]["status"] = "done" # B done (immediate dep of C satisfied) + fr3, whyr3, bbr3 = next_feature_autonomous(r3) + check("#3: C (transitive dependent of failed A) is NOT handed out as runnable", + fr3 is None or fr3.get("id") != "C") + check("#3: C is reported in blocked_by (waiting), B (done) is not", + bbr3 == ["C"]) + + # --- #4 HIGH: covered above (reject-at-init, missing-key default) -------------------- + check("#4: (see negative/non-numeric/missing-key checks above)", True) + + # --- #5 HIGH: progress-cycle idempotency is GLOBAL ---------------------------------- + r5 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + record_progress_cycle(r5, "c1") # baseline + record_progress_cycle(r5, "c2") # no progress -> 1 + npc_before = r5["no_progress_cycles"] + rep = record_progress_cycle(r5, "c1") # replay of a NON-adjacent earlier id + check("#5: replay of ANY earlier cycle-id is a no-op (global, not just last)", + rep["replayed"] is True and r5["no_progress_cycles"] == npc_before) + + # --- #6 HIGH: ledger reblock reactivates --------------------------------------------- + r6 = build_state([{"id": "a", "depends_on": []}, {"id": "b", "depends_on": ["a"]}], + "e", "E", stance="marathon", caps={}) + apply_update(r6, "a", "blocked", blocker_reason="vendor", now_dt=T0) # attempt 0, active + apply_update(r6, "a", "pending", now_dt=T0) # resolves -> inactive + check("#6: after reopen the entry is inactive", r6["blocker_ledger"][0]["active"] is False) + apply_update(r6, "a", "blocked", blocker_reason="vendor again", now_dt=T0) # reblock, same attempt 0 + active_entries = [e for e in r6["blocker_ledger"] + if e.get("feature") == "a" and e.get("active")] + check("#6: reblock at the same attempt REACTIVATES (no new entry)", + len(r6["blocker_ledger"]) == 1 and len(active_entries) == 1 + and active_entries[0]["resolved_at"] is None) + # invariant: every currently-blocked feature has exactly one active entry + blocked_feats = [f["id"] for f in r6["features"] if f.get("status") == "blocked"] + for bf in blocked_feats: + acts = [e for e in r6["blocker_ledger"] if e.get("feature") == bf and e.get("active")] + check("#6: blocked feature %r has exactly one active ledger entry" % bf, len(acts) == 1) + + # --- #7 HIGH: --last-error persist + clear ------------------------------------------ + r7 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + apply_update(r7, "a", "running", now_dt=T0) + apply_update(r7, "a", "failed", last_error="boom: NPE at L42", now_dt=T0) + check("#7: --status failed --last-error persists last_error", + r7["features"][0]["last_error"] == "boom: NPE at L42") + apply_update(r7, "a", "running", now_dt=T0) # retry + check("#7: last_error cleared on the retry (->running)", r7["features"][0]["last_error"] is None) + apply_update(r7, "a", "failed", last_error="again", now_dt=T0) + apply_update(r7, "a", "running", now_dt=T0) + apply_update(r7, "a", "done", now_dt=T0) + check("#7: last_error cleared on ->done", r7["features"][0]["last_error"] is None) + + # --- #8 MED: no_progress_cycles reset inside apply_update on ->done ------------------- + r8 = build_state([{"id": "a", "depends_on": []}, {"id": "b", "depends_on": []}], + "e", "E", stance="marathon", caps={}) + r8["no_progress_cycles"] = 2 # simulate accumulated stall + apply_update(r8, "a", "running", now_dt=T0) + apply_update(r8, "a", "done", now_dt=T0) + check("#8: a feature reaching done resets no_progress_cycles inside --update", + r8["no_progress_cycles"] == 0) + + # --- #9 MED: total_attempts recomputed on EVERY marathon mutation -------------------- + r9 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + apply_update(r9, "a", "running", now_dt=T0) # attempts 1 + r9["total_attempts"] = 999 # corrupt the stored counter + record_disposition(r9, "a", "retry_fix", now_dt=T0) + check("#9: record_disposition recomputes total_attempts from features", + r9["total_attempts"] == 1) + r9["total_attempts"] = 999 + record_progress_cycle(r9, "z1") + check("#9: record_progress_cycle recomputes total_attempts", r9["total_attempts"] == 1) + r9["total_attempts"] = 999 + trip_breaker(r9, T0) # not tripped here, but should still be a no-op on status; recompute only on trip + r9["total_attempts"] = 999 + record_final_review(r9, "pending") + check("#9: record_final_review recomputes total_attempts", r9["total_attempts"] == 1) + + # --- #10 MED: defensive normalization of a loaded marathon state --------------------- + good = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + check("#10: a well-formed marathon state validates clean", validate_marathon_state(good) == []) + bad_att = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + bad_att["features"][0]["attempts"] = -3 + check("#10: negative attempts -> error", validate_marathon_state(bad_att) != []) + bad_att2 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + bad_att2["features"][0]["attempts"] = None + check("#10: null attempts -> error", validate_marathon_state(bad_att2) != []) + bad_id = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + del bad_id["features"][0]["id"] + check("#10: missing feature id -> error", validate_marathon_state(bad_id) != []) + bad_npc = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + bad_npc["no_progress_cycles"] = "lots" + check("#10: non-int no_progress_cycles -> error", validate_marathon_state(bad_npc) != []) + bad_cap = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + bad_cap["autonomy"]["max_total_attempts"] = -5 # bypasses --init, hand-edited + check("#10: persisted negative cap -> error (not silent unbounded)", + validate_marathon_state(bad_cap) != []) + chk = build_state([{"id": "a", "depends_on": []}], "e", "E") # checkpoint + check("#10: checkpoint state is NOT subjected to marathon validation (no-op path)", + _is_marathon(chk) is False) + + # --- #11 LOW: _now_iso normalizes aware datetimes to UTC ----------------------------- + plus5 = _now_iso(_parse_iso("2026-01-01T05:00:00+05:00")) + check("#11: an injected +05:00 is emitted as +00:00 (normalized to UTC)", + plus5 == "2026-01-01T00:00:00+00:00") + check("#11: naive datetime assumed UTC", _now_iso(datetime(2026, 1, 1)) == "2026-01-01T00:00:00+00:00") + r11 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", + caps={"started_at": "2026-01-01T05:00:00+05:00"}) + check("#11: build_state started_at is normalized to UTC too", + r11["autonomy"]["started_at"] == "2026-01-01T00:00:00+00:00") + + # ================================================================ + # Codex round-2 review — 5 remaining edge cases + # ================================================================ + + # --- R2#1 HIGH: validate_marathon_state type gaps ------------------------------------ + r2_deps = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + r2_deps["features"][0]["depends_on"] = 5 # numeric, not a list + check("R2#1: numeric depends_on -> validation error", + any("depends_on" in e for e in validate_marathon_state(r2_deps))) + r2_deps2 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + r2_deps2["features"][0]["depends_on"] = [1, 2] # list of non-strings + check("R2#1: non-string depends_on element -> error", + any("depends_on" in e for e in validate_marathon_state(r2_deps2))) + r2_ldc = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + r2_ldc["last_done_count"] = "three" + check("R2#1: string last_done_count -> error", + any("last_done_count" in e for e in validate_marathon_state(r2_ldc))) + r2_pci = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + r2_pci["processed_cycle_ids"] = [1, 2] + check("R2#1: non-string processed_cycle_ids element -> error", + any("processed_cycle_ids" in e for e in validate_marathon_state(r2_pci))) + # ...and a well-formed state with real depends_on/last_done_count still validates clean + r2_ok = build_state([{"id": "a", "depends_on": []}, {"id": "b", "depends_on": ["a"]}], + "e", "E", stance="marathon", caps={}) + r2_ok["last_done_count"] = 1 + r2_ok["processed_cycle_ids"] = ["c1", "c2"] + check("R2#1: well-formed depends_on/last_done_count/processed_cycle_ids stays clean", + validate_marathon_state(r2_ok) == []) + + # --- R2#2 HIGH: non-finite caps rejected -------------------------------------------- + for bad in (float("nan"), float("inf")): + try: + build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", + caps={"max_wall_clock_hours": bad}) + check("R2#2: non-finite cap %r rejected at build" % bad, False) + except ValueError: + check("R2#2: non-finite cap %r rejected at build" % bad, True) + # a persisted NaN cap (hand-edited, bypassing --init) is caught by the post-load validator + r2_nan = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + r2_nan["autonomy"]["max_total_attempts"] = float("nan") + check("R2#2: persisted NaN cap -> validation error", + any("finite" in e for e in validate_marathon_state(r2_nan))) + + # --- R2#3 HIGH: explicit-null max_attempts_per_feature does not crash --can-retry ---- + r2_null = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", + caps={"max_attempts_per_feature": None}) + apply_update(r2_null, "a", "running", now_dt=T0) # attempts -> 1 + info_null = can_retry_info(r2_null, "a") + check("R2#3: null max_attempts_per_feature -> can_retry True, cap None (no crash)", + info_null["can_retry"] is True and info_null["cap"] is None and info_null["attempts"] == 1) + + # --- R2#4 MED: idempotent done-replay must NOT invalidate a passed final_review ------- + r2_rep = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + apply_update(r2_rep, "a", "running", now_dt=T0) + apply_update(r2_rep, "a", "done", now_dt=T0) + record_final_review(r2_rep, "passed") + check("R2#4: precondition — epic is done after pass", r2_rep["status"] == "done") + apply_update(r2_rep, "a", "done", now_dt=T0) # idempotent replay (prev==status==done) + check("R2#4: replay of done on an already-done feature keeps final_review passed", + r2_rep["final_review"]["status"] == "passed") + check("R2#4: ...and top status stays 'done'", r2_rep["status"] == "done") + # a REAL change still invalidates (regression guard for #1) + apply_update(r2_rep, "a", "pending", now_dt=T0) + check("R2#4: a real status change still invalidates the review", + r2_rep["final_review"]["status"] == "pending" and r2_rep["status"] != "done") + + # --- R2#5 MED: --last-error (and other marathon args) rejected on a checkpoint state -- + r2_chk = build_state([{"id": "a", "depends_on": []}], "e", "E") # checkpoint + okle, errle = apply_update(r2_chk, "a", "failed", last_error="boom") + check("R2#5: checkpoint --update --last-error rejected (nonzero, no silent discard)", + okle is False and "--last-error" in errle and "marathon-only" in errle) + check("R2#5: ...and the checkpoint feature status was NOT changed", + r2_chk["features"][0]["status"] == "pending") + # a plain checkpoint --update with no marathon args still works + okp3, _ = apply_update(r2_chk, "a", "failed") + check("R2#5: plain checkpoint --update (no marathon args) still succeeds", okp3) + + # ================================================================ + # Codex round-3 review — 8 malformed-state / argparse robustness fixes + # ================================================================ + + # --- R3#1 HIGH: no_progress reset must NOT fire on a done->done replay ---------------- + r3_np = build_state([{"id": "a", "depends_on": []}, {"id": "b", "depends_on": []}], + "e", "E", stance="marathon", caps={}) + apply_update(r3_np, "a", "running", now_dt=T0) + apply_update(r3_np, "a", "done", now_dt=T0) # real ->done resets (was 0 anyway) + r3_np["no_progress_cycles"] = 2 # simulate accumulated stall on OTHER work + apply_update(r3_np, "a", "done", now_dt=T0) # idempotent replay: prev==status==done + check("R3#1: done->done replay does NOT reset no_progress_cycles", + r3_np["no_progress_cycles"] == 2) + # a REAL transition to done still resets (regression guard) + apply_update(r3_np, "b", "running", now_dt=T0) + apply_update(r3_np, "b", "done", now_dt=T0) + check("R3#1: a real ->done still resets no_progress_cycles", r3_np["no_progress_cycles"] == 0) + + # --- R3#2 HIGH: validate_marathon_state validates blocker_ledger --------------------- + r3_led = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + r3_led["blocker_ledger"] = "not a list" + check("R3#2: non-list blocker_ledger -> error", + any("blocker_ledger" in e for e in validate_marathon_state(r3_led))) + r3_led2 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + r3_led2["blocker_ledger"] = ["junk"] # non-dict entry + check("R3#2: non-dict ledger entry -> error", + any("blocker_ledger" in e for e in validate_marathon_state(r3_led2))) + r3_led3 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + r3_led3["blocker_ledger"] = [{"feature": 5, "attempt": 0, "active": True}] # bad feature type + check("R3#2: ledger entry with non-string feature -> error", + any("blocker_ledger" in e for e in validate_marathon_state(r3_led3))) + # a well-formed ledger validates clean + r3_ledok = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + apply_update(r3_ledok, "a", "blocked", blocker_reason="x", now_dt=T0) + check("R3#2: a builder-produced ledger validates clean", + validate_marathon_state(r3_ledok) == []) + + # --- R3#3 HIGH: started_at safety — validate + fail-safe TRIP ------------------------- + r3_sa = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + del r3_sa["autonomy"]["started_at"] + check("R3#3: missing started_at -> validation error", + any("started_at" in e for e in validate_marathon_state(r3_sa))) + r3_sa2 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", caps={}) + r3_sa2["autonomy"]["started_at"] = "not-a-timestamp" + check("R3#3: unparseable started_at -> validation error", + any("started_at" in e for e in validate_marathon_state(r3_sa2))) + # breaker_check FAILS SAFE: max_wall_clock_hours set but started_at missing -> TRIPPED + r3_sa3 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", + caps={"max_wall_clock_hours": 10, + "started_at": "2026-01-01T00:00:00+00:00"}) + r3_sa3["autonomy"]["started_at"] = None # corrupt after build + r_fs = breaker_check(r3_sa3, T0) + check("R3#3: missing started_at + wall-clock cap -> breaker TRIPS (never silently skipped)", + r_fs["tripped"] is True and "max_wall_clock_hours" in r_fs["which"]) + r3_sa4 = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", + caps={"max_wall_clock_hours": 10, + "started_at": "2026-01-01T00:00:00+00:00"}) + r3_sa4["autonomy"]["started_at"] = "garbage" + r_fs2 = breaker_check(r3_sa4, T0) + check("R3#3: unparseable started_at + wall-clock cap -> breaker TRIPS fail-safe", + r_fs2["tripped"] is True and "max_wall_clock_hours" in r_fs2["which"]) + + # --- R3#4 MED: presence-based rejection of marathon args (not truthiness) ------------- + # apply_update alone can't see an explicit-false bool; the CLI presence check does. Here + # assert the discriminator: --blocker-confirmed's arg default is None (a sentinel) so + # `false` is distinguishable from absent. (Full CLI reject proven in the live smoke.) + check("R3#4: (blocker_confirmed sentinel default enables presence detection — see CLI smoke)", + True) + + # --- R3#5 MED: checkpoint --stats has no 'blocked' key; marathon does ----------------- + chk_stats = stats(build_state([{"id": "a", "depends_on": []}], "e", "E")) + check("R3#5: checkpoint stats omits 'blocked'", "blocked" not in chk_stats) + mar_stats = stats(build_state([{"id": "a", "depends_on": []}], "e", "E", + stance="marathon", caps={})) + check("R3#5: marathon stats includes 'blocked'", "blocked" in mar_stats) + + # --- R3#6 MED: explicit-null cap via the CLI arg parser ------------------------------- + parse_int_cap = _cap_arg(int) + check("R3#6: cap arg 'null' -> None (explicit unbounded)", parse_int_cap("null") is None) + check("R3#6: cap arg 'none' -> None", parse_int_cap("none") is None) + check("R3#6: cap arg '5' -> 5", parse_int_cap("5") == 5) + try: + parse_int_cap("abc") + check("R3#6: cap arg garbage rejected", False) + except argparse.ArgumentTypeError: + check("R3#6: cap arg garbage rejected", True) + # an explicit-null cap in build_state is honored as unbounded + r3_null = build_state([{"id": "a", "depends_on": []}], "e", "E", stance="marathon", + caps={"max_total_attempts": None}) + check("R3#6: explicit-null cap persists as null (unbounded)", + r3_null["autonomy"]["max_total_attempts"] is None) + + # --- R3#7 MED: --init depends_on TypeError -> controlled error ------------------------ + check("R3#7: non-list depends_on -> validation error (no crash)", + any("depends_on" in e for e in validate_features([{"id": "a", "depends_on": 5}]))) + check("R3#7: non-string depends_on element -> validation error", + any("depends_on" in e for e in validate_features([{"id": "a", "depends_on": [1]}]))) + # validate_features must NOT crash on the bad input (it returns errors, doesn't raise) + try: + validate_features([{"id": "a", "depends_on": 5}]) + check("R3#7: validate_features survives non-iterable depends_on (no TypeError)", True) + except TypeError: + check("R3#7: validate_features survives non-iterable depends_on (no TypeError)", False) + + # --- R3#8 MED: malformed --now resolution -> controlled error (proven at CLI) --------- + try: + _parse_iso("not-a-timestamp") + check("R3#8: _parse_iso raises on bad input (caught by the CLI guard)", False) + except (ValueError, TypeError): + check("R3#8: _parse_iso raises on bad input (caught by the CLI guard)", True) + + # ================================================================ + # v2.10 marathon — human resume after a breaker/halt latch + # (--clear-breaker, --clear-disposition, --init --start-sha) + # ================================================================ + + # --- --clear-breaker: clears a tripped blocked_needing_human -> status recomputes ------ + cb_feats = [{"id": "a", "depends_on": []}] + cbA = build_state(cb_feats, "e", "E", stance="marathon", caps={"max_no_progress_cycles": 2}) + cbA["status"] = "blocked_needing_human" + cbA["breaker_trip"] = {"which": ["max_no_progress_cycles"], "detail": {}, + "tripped_at": "2026-01-01T00:00:00+00:00"} + cbA["no_progress_cycles"] = 5 + okA, errA, sumA = clear_breaker(cbA, T0) + check("clear-breaker: ok + no error", okA and errA is None) + check("clear-breaker: status recomputes off blocked_needing_human ('running')", + cbA["status"] == "running") + check("clear-breaker: breaker_trip record removed", "breaker_trip" not in cbA) + check("clear-breaker: no_progress_cycles reset to 0", cbA["no_progress_cycles"] == 0) + check("clear-breaker: summary reports cleared_breaker_trip", sumA["cleared_breaker_trip"] is True) + check("clear-breaker: summary reports the prior no_progress_cycles", + sumA["no_progress_cycles_reset_from"] == 5) + check("clear-breaker: no immediate re-trip when caps are fine", + sumA["would_immediately_retrip"] is False and sumA["would_retrip_which"] == []) + + # status recompute can also land back on 'done' (all-done + final_review passed) + cbD = build_state(cb_feats, "e", "E", stance="marathon", caps={}) + apply_update(cbD, "a", "running", now_dt=T0) + apply_update(cbD, "a", "done", now_dt=T0) + record_final_review(cbD, "passed") + check("clear-breaker setup: status is 'done' pre-trip", cbD["status"] == "done") + cbD["status"] = "blocked_needing_human" # simulate a later-tripped latch + cbD["breaker_trip"] = {"which": ["max_wall_clock_hours"], "detail": {}, "tripped_at": "x"} + okD, _, sumD = clear_breaker(cbD, T0) + check("clear-breaker: recompute can land back on 'done'", okD and cbD["status"] == "done") + + # --- --clear-breaker --reset-wall-clock: moves started_at ------------------------------ + cb2 = build_state(cb_feats, "e", "E", stance="marathon", + caps={"started_at": "2020-01-01T00:00:00+00:00"}) + ok2, _, sum2 = clear_breaker(cb2, T0, reset_wall_clock=True) + check("clear-breaker --reset-wall-clock: started_at moved to 'now'", + ok2 and cb2["autonomy"]["started_at"] == _now_iso(T0)) + check("clear-breaker --reset-wall-clock: summary reflects the reset", + sum2["wall_clock_reset"] is True) + cb2b = build_state(cb_feats, "e", "E", stance="marathon", + caps={"started_at": "2020-01-01T00:00:00+00:00"}) + ok2b, _, sum2b = clear_breaker(cb2b, T0) # no --reset-wall-clock + check("clear-breaker: started_at untouched without --reset-wall-clock", + ok2b and cb2b["autonomy"]["started_at"] == "2020-01-01T00:00:00+00:00" + and sum2b["wall_clock_reset"] is False) + + # --- --clear-breaker --set-max-total-attempts: re-arms that axis ----------------------- + cb3 = build_state(cb_feats, "e", "E", stance="marathon", caps={"max_total_attempts": 1}) + ok3, err3, sum3 = clear_breaker(cb3, T0, set_max_total_attempts=50) + check("clear-breaker --set-max-total-attempts: cap updated", + ok3 and cb3["autonomy"]["max_total_attempts"] == 50) + check("clear-breaker --set-max-total-attempts: summary reflects the new cap", + sum3["max_total_attempts_set"] == 50) + ok3b, _, _ = clear_breaker(cb3, T0, set_max_total_attempts=None) + check("clear-breaker --set-max-total-attempts null: explicit-unbounded honored", + ok3b and cb3["autonomy"]["max_total_attempts"] is None) + ok3c, err3c, sum3c = clear_breaker(cb3, T0, set_max_total_attempts=-5) + check("clear-breaker --set-max-total-attempts: a negative cap is rejected (same rules as --init)", + ok3c is False and sum3c is None and err3c) + + # --- --clear-breaker: clearing WITHOUT re-arming a still-over total_attempts axis warns - + cb4 = build_state(cb_feats, "e", "E", stance="marathon", caps={"max_total_attempts": 1}) + apply_update(cb4, "a", "running", now_dt=T0) # attempts=1 == cap + trip_res4 = trip_breaker(cb4, T0) + check("clear-breaker setup: over-cap trip actually latched", + trip_res4["tripped"] and cb4["status"] == "blocked_needing_human") + ok4, _, sum4 = clear_breaker(cb4, T0) # no re-arm supplied + check("clear-breaker: the latch is still cleared even though it would re-trip", + ok4 and cb4["status"] != "blocked_needing_human") + check("clear-breaker: summary flags the immediate re-trip on max_total_attempts", + sum4["would_immediately_retrip"] is True + and "max_total_attempts" in sum4["would_retrip_which"]) + + # --- --clear-breaker: non-marathon state -> controlled error --------------------------- + cb5 = build_state(cb_feats, "e", "E") # checkpoint + ok5, err5, sum5 = clear_breaker(cb5, T0) + check("clear-breaker: non-marathon state -> controlled error, no write", + ok5 is False and sum5 is None and "marathon" in err5) + + # --- --clear-disposition: undoes a sticky halt_epic so autonomous routing resumes ------ + cd_feats = [{"id": "a", "depends_on": []}, {"id": "b", "depends_on": ["a"]}] + cd1 = build_state(cd_feats, "e", "E", stance="marathon", caps={}) + record_disposition(cd1, "a", "halt_epic", now_dt=T0) + fah_before, whyah_before, _ = next_feature_autonomous(cd1) + check("clear-disposition setup: a halt_epic disposition halts the epic", + fah_before is None and whyah_before.startswith("blocked_needing_human")) + okcd, errcd = clear_disposition(cd1, "a") + check("clear-disposition: ok + no error", okcd and errcd is None) + check("clear-disposition: the feature's disposition is cleared to null", + cd1["features"][0]["disposition"] is None) + fah_after, whyah_after, _ = next_feature_autonomous(cd1) + check("clear-disposition: next_feature_autonomous no longer short-circuits on it", + fah_after is not None and fah_after["id"] == "a") + + okcd2, errcd2 = clear_disposition(cd1, "does-not-exist") + check("clear-disposition: unknown feature -> controlled error", + okcd2 is False and "no feature" in errcd2) + + cd_chk = build_state(cd_feats, "e", "E") # checkpoint + okcd3, errcd3 = clear_disposition(cd_chk, "a") + check("clear-disposition: non-marathon state -> controlled error, no write", + okcd3 is False and "marathon" in errcd3) + + # --- --init --stance marathon --start-sha: stored as autonomy.start_sha ---------------- + sha_feats = [{"id": "a", "depends_on": []}] + sha_state = build_state(sha_feats, "e", "E", stance="marathon", caps={"start_sha": "abc123"}) + check("start-sha: marathon build_state stores autonomy.start_sha", + sha_state["autonomy"]["start_sha"] == "abc123") + check("start-sha: a stored string start_sha validates clean", + validate_marathon_state(sha_state) == []) + sha_absent = build_state(sha_feats, "e", "E", stance="marathon", caps={}) + check("start-sha: absent start_sha is fine (no key, no validation error)", + "start_sha" not in sha_absent["autonomy"] + and validate_marathon_state(sha_absent) == []) + sha_bad = build_state(sha_feats, "e", "E", stance="marathon", caps={}) + sha_bad["autonomy"]["start_sha"] = 12345 # simulate a hand-edited/corrupt state + check("start-sha: a non-string start_sha on a loaded state -> validation error", + any("start_sha" in e for e in validate_marathon_state(sha_bad))) + try: + build_state(sha_feats, "e", "E", stance="marathon", caps={"start_sha": 12345}) + check("start-sha: a non-string start_sha is rejected at build time", False) + except ValueError: + check("start-sha: a non-string start_sha is rejected at build time", True) + # checkpoint --init stays unaffected: build_state without a marathon stance never looks + # at caps["start_sha"] at all (structurally impossible to leak into the checkpoint shape). + plain_sha = build_state(sha_feats, "e", "E") + check("start-sha: checkpoint build_state carries no autonomy/start_sha (unaffected)", + "autonomy" not in plain_sha) + print("SELFTEST: %d ok, %d fail" % (ok, fail)) return 0 if fail == 0 else 1 def main(argv): + # LANG=C-clean: any non-ASCII in stdout/stderr must not crash the process. + if hasattr(sys.stdout, "reconfigure"): + sys.stdout.reconfigure(encoding="utf-8", errors="replace") + if hasattr(sys.stderr, "reconfigure"): + sys.stderr.reconfigure(encoding="utf-8", errors="replace") + p = argparse.ArgumentParser(description="Compound V epic-state manager.") p.add_argument("--init", action="store_true") p.add_argument("--next", dest="want_next", action="store_true") @@ -415,8 +2729,10 @@ def main(argv): p.add_argument("--title", default="") p.add_argument("--state", help="path to epic-state.json") p.add_argument("--out", help="output path for --init (default = --state)") - p.add_argument("--feature", help="feature id (for --update)") - p.add_argument("--status", choices=STATUSES) + p.add_argument("--feature", help="feature id (for --update / --can-retry / " + "--record-disposition)") + p.add_argument("--status", help="(--update: %s) (--record-final-review: pending|passed|" + "failed)" % "|".join(STATUSES)) p.add_argument("--run-id") p.add_argument("--lint", action="store_true", help="(with --features) print structural decomposition warnings + validation errors") @@ -426,6 +2742,63 @@ def main(argv): help="(with --init) every feature must carry an existing spec_path") p.add_argument("--check-specs", action="store_true", help="(with --state) verify every non-done feature still has an existing, contained spec_path (resume guard)") + # -- v2.10 marathon flags -------------------------------------------------------------- + p.add_argument("--stance", choices=("checkpoint", "marathon"), + help="(with --init) opt into the marathon autonomy schema") + # Caps accept an integer/float OR the literal 'null'/'none' for an explicit unbounded + # axis (Codex round-3 #6). Default is a sentinel so "absent" (use the documented default) + # is distinct from "explicit null" (unbounded). + p.add_argument("--max-attempts-per-feature", type=_cap_arg(int), default=_UNSET) + p.add_argument("--max-no-progress-cycles", type=_cap_arg(int), default=_UNSET) + p.add_argument("--max-total-attempts", type=_cap_arg(int), default=_UNSET) + p.add_argument("--max-wall-clock-hours", type=_cap_arg(float), default=_UNSET) + p.add_argument("--now", help="inject an ISO-8601 timestamp instead of the real clock " + "(deterministic tests); default = real clock") + p.add_argument("--autonomous", action="store_true", + help="(with --next) marathon DAG-transitive routing + terminal-state resolution") + p.add_argument("--can-retry", action="store_true") + p.add_argument("--record-disposition", action="store_true") + p.add_argument("--disposition", choices=("retry_fix", "halt_feature", "halt_epic", + "blocked_external")) + p.add_argument("--reason") + p.add_argument("--families-agreeing", help="comma-separated model family names") + p.add_argument("--confirmed", choices=("true", "false"), default="false", + help="(with --record-disposition) MUST be false in v2.10 — hard-rejected if true") + p.add_argument("--record-final-review", action="store_true") + p.add_argument("--record-progress-cycle", action="store_true") + p.add_argument("--cycle-id") + p.add_argument("--breaker-check", action="store_true") + p.add_argument("--trip-breaker", action="store_true") + p.add_argument("--blocker-reason") + p.add_argument("--blocker-confirmed", choices=("true", "false"), default=None, + help="(with --update --status blocked) MUST be false in v2.10 — hard-rejected if true") + p.add_argument("--evidence", help="(with --update --status blocked) the missing external fact, if known") + p.add_argument("--last-error", help="(with --update --status failed, marathon-only) persist the " + "feature's last error; cleared on a retry/done") + p.add_argument("--start-sha", help="(with --init --stance marathon) git rev-parse HEAD, " + "stored as autonomy.start_sha; marathon-only") + p.add_argument("--clear-breaker", action="store_true", + help="(marathon) human re-arm after a breaker trip/halt — clears the " + "blocked_needing_human latch") + p.add_argument("--reset-wall-clock", action="store_true", + help="(with --clear-breaker) reset autonomy.started_at to now") + p.add_argument("--set-max-total-attempts", type=_cap_arg(int), default=_UNSET, + help="(with --clear-breaker) re-arm autonomy.max_total_attempts; N or " + "'null'/'none' for an explicit unbounded axis") + p.add_argument("--clear-disposition", action="store_true", + help="(marathon) clear a feature's stored disposition, e.g. to undo a " + "sticky halt_epic/halt_feature verdict") + p.add_argument("--mark-sample-audit-due", action="store_true", + help="(marathon) record a durable sample-audit obligation on --feature F " + "before a PASS sample-audit runs; gates terminal completion") + p.add_argument("--clear-sample-audit-due", action="store_true", + help="(marathon) clear --feature F's sample-audit obligation once the " + "audit has passed") + p.add_argument("--record-audit-failed", action="store_true", + help="(marathon) ONE atomic write: revert --feature F from done to failed, " + "clear its sample_audit_due obligation, optionally set --last-error, " + "and invalidate a passed final_review back to pending — the fix for a " + "crash between the driver's separate clear-due/mark-failed writes") args = p.parse_args(argv) if args.selftest: @@ -457,15 +2830,43 @@ def main(argv): for e in spec_errs: print("epic-init spec error: %s" % e, file=sys.stderr) return 1 - state = build_state(feats, args.epic_id, args.title or args.epic_id) + stance = "marathon" if args.stance == "marathon" else None + caps = None + if stance == "marathon": + caps = {} + # `is not _UNSET` distinguishes an omitted cap (use the default) from an EXPLICIT + # value — including an explicit null (unbounded), which must land in `caps` so + # build_state records it (Codex round-3 #6). + if args.max_attempts_per_feature is not _UNSET: + caps["max_attempts_per_feature"] = args.max_attempts_per_feature + if args.max_no_progress_cycles is not _UNSET: + caps["max_no_progress_cycles"] = args.max_no_progress_cycles + if args.max_total_attempts is not _UNSET: + caps["max_total_attempts"] = args.max_total_attempts + if args.max_wall_clock_hours is not _UNSET: + caps["max_wall_clock_hours"] = args.max_wall_clock_hours + if args.now: + caps["started_at"] = args.now + if args.start_sha: + caps["start_sha"] = args.start_sha + elif args.start_sha: + # Non-marathon --init must REJECT --start-sha (not silently discard it) — same + # presence-based-rejection discipline as every other marathon-only arg in this + # file, and it keeps a plain checkpoint --init byte-identical (this path never + # reaches build_state with a caps dict at all). + print("epic-init error: --start-sha is marathon-only — not valid without " + "--stance marathon", file=sys.stderr) + return 1 + try: + state = build_state(feats, args.epic_id, args.title or args.epic_id, + stance=stance, caps=caps) + except ValueError as e: + print("epic-init error: %s" % e, file=sys.stderr) + return 1 out = args.out or args.state if not out: p.error("--init needs --out (or --state)") - od = os.path.dirname(out) - if od and not os.path.isdir(od): - os.makedirs(od) - with open(out, "w") as fh: - fh.write(json.dumps(state, indent=2) + "\n") + _atomic_write_json(out, state) print("wrote %s (%d features)" % (out, len(state["features"]))) return 0 @@ -473,6 +2874,37 @@ def main(argv): p.error("--state is required and must exist") state = _read_json(args.state) + # Codex review #10: a loaded marathon state must be structurally sound before any + # marathon command reads/mutates it — a controlled nonzero, never a raw KeyError/ + # TypeError crash. Checkpoint states are untouched (validator is a no-op there). + if _is_marathon(state): + merrs = validate_marathon_state(state) + if merrs: + for e in merrs: + print("epic-state error: %s" % e, file=sys.stderr) + return 1 + + # Codex round-3 #8: a malformed --now must fail with a controlled error, never an + # uncaught ValueError from datetime.fromisoformat on the breaker/disposition/update + # paths. Validate once up front (init handled its own --now via build_state above). + if args.now: + try: + _parse_iso(args.now) + except (ValueError, TypeError) as e: + print("epic error: --now %r is not a valid ISO-8601 timestamp (%s)" + % (args.now, e), file=sys.stderr) + return 1 + + def _needs_marathon(cmd): + """Codex review #2: a marathon-only command must REJECT (nonzero, no write) when the + persisted state is not marathon stance.""" + if not _is_marathon(state): + print("epic error: %s requires a marathon-stance epic (this state has no " + "'autonomy' block — re-init with --stance marathon)" % cmd, + file=sys.stderr) + return False + return True + if args.check_specs: errs = check_state_specs(state, base_dir=os.path.dirname(os.path.abspath(args.state))) if errs: @@ -486,40 +2918,199 @@ def main(argv): print(json.dumps(stats(state))) return 0 + if args.can_retry: + if not _needs_marathon("--can-retry"): + return 1 + if not args.feature: + p.error("--can-retry needs --feature") + info = can_retry_info(state, args.feature) + if info is None: + print("epic error: no feature %r" % args.feature, file=sys.stderr) + return 1 + print(json.dumps(info)) + return 0 + + if args.breaker_check: + if not _needs_marathon("--breaker-check"): + return 1 + now_dt = _resolve_now(args.now) + print(json.dumps(breaker_check(state, now_dt))) + return 0 + + if args.trip_breaker: + if not _needs_marathon("--trip-breaker"): + return 1 + now_dt = _resolve_now(args.now) + result = trip_breaker(state, now_dt) + mutated = result.pop("mutated", False) + if mutated: + _atomic_write_json(args.state, state) + print(json.dumps(result)) + return 0 + + if args.clear_breaker: + if not _needs_marathon("--clear-breaker"): + return 1 + now_dt = _resolve_now(args.now) + ok, err, summary = clear_breaker(state, now_dt, + reset_wall_clock=args.reset_wall_clock, + set_max_total_attempts=args.set_max_total_attempts) + if not ok: + print("epic-clear-breaker error: %s" % err, file=sys.stderr) + return 1 + _atomic_write_json(args.state, state) + print(json.dumps(summary)) + return 0 + + if args.clear_disposition: + if not _needs_marathon("--clear-disposition"): + return 1 + if not args.feature: + p.error("--clear-disposition needs --feature") + ok, err = clear_disposition(state, args.feature) + if not ok: + print("epic-clear-disposition error: %s" % err, file=sys.stderr) + return 1 + _atomic_write_json(args.state, state) + print(json.dumps({"feature": args.feature, "disposition": None})) + return 0 + + if args.mark_sample_audit_due: + if not _needs_marathon("--mark-sample-audit-due"): + return 1 + if not args.feature: + p.error("--mark-sample-audit-due needs --feature") + ok, err = mark_sample_audit_due(state, args.feature) + if not ok: + print("epic-sample-audit error: %s" % err, file=sys.stderr) + return 1 + _atomic_write_json(args.state, state) + print(json.dumps({"feature": args.feature, "sample_audit_due": True})) + return 0 + + if args.clear_sample_audit_due: + if not _needs_marathon("--clear-sample-audit-due"): + return 1 + if not args.feature: + p.error("--clear-sample-audit-due needs --feature") + ok, err = clear_sample_audit_due(state, args.feature) + if not ok: + print("epic-sample-audit error: %s" % err, file=sys.stderr) + return 1 + _atomic_write_json(args.state, state) + print(json.dumps({"feature": args.feature, "sample_audit_due": False})) + return 0 + + if args.record_audit_failed: + if not _needs_marathon("--record-audit-failed"): + return 1 + if not args.feature: + p.error("--record-audit-failed needs --feature") + ok, err = record_audit_failed(state, args.feature, last_error=args.last_error) + if not ok: + print("epic-audit-failed error: %s" % err, file=sys.stderr) + return 1 + _atomic_write_json(args.state, state) + hit = _find_feature(state, args.feature) + print(json.dumps({"feature": args.feature, "status": hit["status"], + "sample_audit_due": hit.get("sample_audit_due"), + "final_review": state.get("final_review"), + "epic_status": state["status"]})) + return 0 + + if args.record_progress_cycle: + if not _needs_marathon("--record-progress-cycle"): + return 1 + if not args.cycle_id: + p.error("--record-progress-cycle needs --cycle-id") + result = record_progress_cycle(state, args.cycle_id) + if not result.get("replayed"): + _atomic_write_json(args.state, state) + print(json.dumps(result)) + return 0 + + if args.record_disposition: + if not _needs_marathon("--record-disposition"): + return 1 + if not args.feature or not args.disposition: + p.error("--record-disposition needs --feature and --disposition") + now_dt = _resolve_now(args.now) + ok, err = record_disposition(state, args.feature, args.disposition, reason=args.reason, + families_agreeing=_parse_csv_list(args.families_agreeing), + confirmed=(args.confirmed == "true"), now_dt=now_dt) + if not ok: + print("epic-disposition error: %s" % err, file=sys.stderr) + return 1 + _atomic_write_json(args.state, state) + print(json.dumps({"feature": args.feature, + "disposition": _find_feature(state, args.feature)["disposition"]})) + return 0 + + if args.record_final_review: + if not _needs_marathon("--record-final-review"): + return 1 + if not args.status: + p.error("--record-final-review needs --status pending|passed|failed") + ok, err = record_final_review(state, args.status) + if not ok: + print("epic-final-review error: %s" % err, file=sys.stderr) + return 1 + _atomic_write_json(args.state, state) + print(json.dumps({"final_review": state["final_review"], "epic_status": state["status"]})) + return 0 + if args.want_next: - f, why = next_feature(state) - print(json.dumps({"feature": f, "reason": why})) - # INTENTIONALLY always 0 (A7): the JSON on stdout is the contract — a null feature - # with a stop reason is information, not failure (commands/v-epic.md: "--next is - # read-only and never an error"). Exit codes here are NOT a signal channel; the - # driver branches on "reason", and a nonzero would read as a script fault. + if args.autonomous: + if not _needs_marathon("--next --autonomous"): + return 1 + f, why, blocked_by = next_feature_autonomous(state) + print(json.dumps({"feature": f, "reason": why, "blocked_by": blocked_by})) + else: + f, why = next_feature(state) + print(json.dumps({"feature": f, "reason": why})) + # INTENTIONALLY always 0 (A7 legacy): the JSON on stdout is the contract — a null + # feature with a stop reason is information, not failure (commands/v-epic.md: + # "--next is read-only and never an error"). Exit codes here are NOT a signal + # channel; the driver branches on "reason", and a nonzero would read as a script + # fault. --next --autonomous keeps the same convention. return 0 if args.update: if not args.feature or not args.status: p.error("--update needs --feature and --status") - hit = None - for f in state["features"]: - if f["id"] == args.feature: - hit = f - break - if hit is None: - print("epic-update error: no feature %r" % args.feature, file=sys.stderr) + # Codex round-3 #4: reject ANY marathon-only arg SUPPLIED against a checkpoint state — + # detected by argument PRESENCE (sentinel/None defaults), so `--blocker-confirmed + # false` and an explicit empty `--families-agreeing` are caught too, not just truthy + # values. Presence, not truthiness. + if not _is_marathon(state): + supplied = [] + if args.last_error is not None: + supplied.append("--last-error") + if args.blocker_reason is not None: + supplied.append("--blocker-reason") + if args.evidence is not None: + supplied.append("--evidence") + if args.families_agreeing is not None: + supplied.append("--families-agreeing") + if args.blocker_confirmed is not None: + supplied.append("--blocker-confirmed") + if supplied: + print("epic-update error: %s %s marathon-only — not valid on a checkpoint-" + "stance epic" % (", ".join(supplied), + "is" if len(supplied) == 1 else "are"), file=sys.stderr) + return 1 + now_dt = _resolve_now(args.now) + ok, err = apply_update(state, args.feature, args.status, run_id=args.run_id, now_dt=now_dt, + blocker_reason=args.blocker_reason, + blocker_confirmed=(args.blocker_confirmed == "true"), + families_agreeing=_parse_csv_list(args.families_agreeing), + evidence=args.evidence, last_error=args.last_error) + if not ok: + print("epic-update error: %s" % err, file=sys.stderr) return 1 - hit["status"] = args.status - if args.run_id is not None: - hit["run_id"] = args.run_id - # roll up epic status (fail-fast: any failure blocks the epic immediately) - sts = [f["status"] for f in state["features"]] - if all(s == "done" for s in sts): - state["status"] = "done" - elif any(s == "failed" for s in sts): - state["status"] = "blocked" - else: - state["status"] = "running" - with open(args.state, "w") as fh: - fh.write(json.dumps(state, indent=2) + "\n") - print(json.dumps({"feature": args.feature, "status": args.status, "epic_status": state["status"]})) + _atomic_write_json(args.state, state) + print(json.dumps({"feature": args.feature, "status": args.status, + "epic_status": state["status"]})) return 0 if args.summary: @@ -529,7 +3120,11 @@ def main(argv): print(" [%-7s] %-20s deps=%s run=%s" % (f["status"], f["id"], deps, f["run_id"] or "-")) return 0 - p.error("one of --init / --next / --update / --summary / --selftest is required") + p.error("one of --init / --next / --update / --summary / --stats / --check-specs / " + "--lint / --can-retry / --breaker-check / --trip-breaker / --clear-breaker / " + "--record-progress-cycle / --record-disposition / --clear-disposition / " + "--mark-sample-audit-due / --clear-sample-audit-due / --record-audit-failed / " + "--record-final-review / --selftest is required") if __name__ == "__main__": diff --git a/skills/compound-v/epic-mode.md b/skills/compound-v/epic-mode.md index de2b6ab..46cc204 100644 --- a/skills/compound-v/epic-mode.md +++ b/skills/compound-v/epic-mode.md @@ -113,21 +113,78 @@ When `--next` returns `epic complete` (all features `done`), run a **final integ --- +## Marathon stance (v2.10, opt-in) + +Everything above this section is the **checkpoint** stance — the unchanged default. `marathon` is an opt-in alternative, chosen only at `--init` time (`--stance marathon`; no in-place upgrade of an existing checkpoint `epic-state.json`), that chews the **whole runnable feature DAG in one `/v:epic` invocation** instead of stopping at every `MAX_FEATURES` checkpoint. The full driver sequence lives in [`v-epic.md`](../../commands/v-epic.md) "Autonomous marathon loop" — this section is the authority for *what* marathon is and *why* it's shaped this way; read the command doc for the exact command-by-command steps. + +**Schema (marathon-only, additive on top of the checkpoint shape above).** Absent `autonomy` ⇒ every checkpoint code path is untouched — new fields are read via `.get(..., default)` everywhere: + +```json +{ + "autonomy": {"stance": "marathon", "max_attempts_per_feature": 2, "max_no_progress_cycles": 3, + "max_total_attempts": 12, "max_wall_clock_hours": 10, "started_at": "2026-07-12T00:00:00+00:00", + "start_sha": ""}, + "final_review": {"status": "pending"}, + "blocker_ledger": [], + "no_progress_cycles": 0, + "total_attempts": 0 +} +``` + +Per feature, marathon adds `"attempts": 0, "last_error": null, "disposition": null`. + +**CLI additions** (every one below REJECTS a non-marathon state; see [`compound-v-epic-state.py`](../../scripts/compound-v-epic-state.py)'s docstring "## CLI contract" for the authoritative, full argument list): + +| Command | Effect | +|---|---| +| `--init --stance marathon [--max-attempts-per-feature N] [--max-no-progress-cycles N] [--max-total-attempts N] [--max-wall-clock-hours H] [--start-sha ]` | writes the marathon block once; checkpoint `--init` (no `--stance`) stays byte-identical. `--start-sha` (driver passes `git rev-parse HEAD`) is stored as `autonomy.start_sha`, the accumulated-diff baseline | +| `--next --autonomous --state S` | `{"feature","reason","blocked_by":[ids]}` — DAG-transitive routing: an abandoned/blocked feature removes only its transitive **dependents**, never its independents; a runnable independent is always returned before any terminal escalation. Routes a `failed` feature **by its stored `disposition`** — which is **attempt-bound** (honored only when its `attempt` == the feature's current `attempts`; a stale disposition from an earlier attempt is ignored): `retry_fix`+can-retry → returned **runnable** (re-run); `retry_fix`+cap-exhausted or `halt_feature` → abandoned; **no valid (current-attempt) disposition → reason `"needs_arbitration: ..."`** (a crash mid-arbitration, or a stale-attempt verdict — the driver runs the resume ladder, see below); **a `blocked_external` disposition whose `--update --status blocked` ledger transition never completed → reason `"needs_blocker_recording: ..."`** (a crash between `--record-disposition` and the ledger write — the driver finishes the interrupted transition idempotently). While any feature has `sample_audit_due` it reports `"sample_audit_due: ..."` **before** `final_review` | +| `--mark-sample-audit-due --feature F --state S` / `--clear-sample-audit-due --feature F --state S` | set/clear a durable PASS-integrity audit obligation. While any is set, `--next --autonomous` surfaces `"sample_audit_due: ..."` and `--record-final-review passed` is **rejected** — the obligation outlives a crash because it is persisted, not held in driver memory | +| `--record-audit-failed --feature F [--last-error S] --state S` | **ONE atomic write** for a failed sample-audit: status→`failed` + records `last_error` + clears `sample_audit_due` + invalidates a passed `final_review` — no crash window where the feature is `done`-without-obligation (replaces the unsafe clear-then-revert two-step) | +| `--can-retry --feature F --state S` | `{"can_retry","attempts","cap"}` (read-only) | +| `--record-disposition --feature F --disposition retry_fix\|halt_feature\|halt_epic\|blocked_external [--reason R] [--families-agreeing a,b]` | stores the arbiter's verdict; `--confirmed true` is **hard-rejected** in v2.10. **Omit `--families-agreeing` when the arbiter returns an empty `families_agreeing`** (argparse needs a value — drop the flag, or pass `""`) | +| `--update --status blocked --feature F [--blocker-reason R] [--families-agreeing a,b] [--evidence E]` | appends/reactivates an idempotent blocker-ledger entry; `--blocker-confirmed true` is **hard-rejected** — v2.10 blockers are always `confirmed:false` (SUSPECTED, never caller-asserted) | +| `--update --status failed --feature F [--last-error "..."]` | persists the failure reason (cleared on the next successful retry/done) | +| `--record-final-review --status pending\|passed\|failed --state S` | `passed` requires every feature `done` **and no `sample_audit_due` obligation outstanding**; the epic reaches top-level `done` **only** via all-done AND `final_review.status=="passed"` — feature completion alone is never enough | +| `--breaker-check [--now ISO] --state S` | read-only → `{"tripped","which":[...],"detail":{...}}` | +| `--trip-breaker [--now ISO] --state S` | atomic write **iff** tripped — parks the epic at `blocked_needing_human` | +| `--record-progress-cycle --cycle-id C [--now ISO] --state S` | idempotent by `cycle_id`; compares the pass's `done` count to the prior one, resets/increments `no_progress_cycles` | +| `--clear-breaker --state S [--reset-wall-clock] [--set-max-total-attempts N]` | **human recovery:** clears the `blocked_needing_human` latch + re-arms the tripped caps so the next `/v:epic` **resumes the marathon**; `--reset-wall-clock` re-stamps `started_at`, `--set-max-total-attempts` raises the attempt cap | +| `--clear-disposition --feature F --state S` | **human recovery:** clears a sticky `halt_epic` disposition on a feature so `--next --autonomous` routes normally again | + +**The arbiter panel** ([`compound-v-epic-arbiter.py`](../../scripts/compound-v-epic-arbiter.py), NEW in v2.10) classifies a feature FAILURE via a two-phase exchange — `--prepare --state S --feature F --attempt N` issues a bounded Claude ballot-task prompt + challenge id (bound to `{epic_id,feature,attempt,challenge_id}`; a mismatched/replayed/stale challenge is dropped before any model call), then `--classify --state S --feature F --challenge [--evidence-file REL] [--claude-ballot FILE]` polls Codex (through the timeout supervisor, read-only sandbox, redacted+capped evidence) if the capabilities file says it's usable, validates the Claude ballot the **driver** supplies (the arbiter cannot itself launch an in-harness Claude Task — that's the driver's job), and aggregates with a complete deterministic truth table: a parse-fail/errored ballot is **dropped, never a fabricated halt vote**; empty/tied → conservative `halt_feature`; `retry_fix` past the per-feature `--can-retry` cap is masked to `halt_feature`; a `blocked_external` verdict on this Codex+Claude panel is always **SUSPECTED** — only two *distinct known external* families (`GPT`/`Gemini`/`Grok`) can CONFIRM one, and Claude-self can never count as an independent confirming family (same family as the implementer). Every ballot + the resolved family + the aggregation reason is written to a frozen audit JSON under `docs/superpowers/execution/epics//arbiter/-.json`. The driver **records the verdict (`--record-disposition`) and commits it before spending any retry** — so the retry intent is a persisted `disposition`, not an in-memory decision: a breaker trip or a hard crash between the verdict and the re-run loses nothing, and on resume `--next --autonomous` re-routes the feature by its stored disposition. A `failed` feature found with *no* valid (current-attempt) disposition — a crash *during* arbitration, or a stale-attempt verdict — is surfaced as `"needs_arbitration: ..."`. Its symmetric one-step-later sibling is `"needs_blocker_recording: ..."` — a `blocked_external` verdict WAS recorded but the crash landed before the `--update --status blocked` ledger write; the driver finishes that interrupted transition idempotently (the ledger append is keyed by `(feature, attempt)`), never re-arbitrating. Both are "finish the interrupted transition on resume" cases, driven purely by persisted state. + +**Idempotent arbitration resume (`--resume-challenge --state S --feature F --attempt N` → `{"state":"absent"|"in_progress"|"consumed", "challenge_id"?, "prompt"?, "result"?}`).** On a `"needs_arbitration"` re-entry the driver must NOT blindly restart from `--prepare` — a crash can leave a challenge already `in_progress` or `consumed`, and `--prepare`/`--classify` reject those, which would deadlock. `--resume-challenge` recovers idempotently: `consumed` returns the already-computed `result` so the driver records the verdict with **no new model call / no re-egress**; `in_progress` returns the `prompt` to re-dispatch Claude and re-`--classify` on the bound `challenge_id`; `absent` means run the exchange fresh. The full driver ladder is in [`v-epic.md`](../../commands/v-epic.md) §2 (`needs_arbitration`). + +**The blocker ledger** — "do everything you can" credo: finish everything reachable, isolate only the genuinely impossible, escalate with proof, never halt the rest. A `blocked_external` disposition marks the feature `--status blocked` (ledger entry, always `confirmed:false` in v2.10); `--next --autonomous` treats `blocked` as a benign skip — only its transitive dependents drop out — and never trips a whole-epic halt by itself. The epic only resolves to `blocked_needing_human` once no other reachable work remains, or a `halt_epic` verdict or a tripped breaker fires (those two *do* halt the whole epic immediately, on purpose — a panel-level "stop everything" vote, or a hard resource limit, is not something the DAG should route around). + +**Global circuit breakers** — the honest bound on "how much can `/v:epic` do unattended": `total_attempts >= max_total_attempts` (default `max(6, 3×features)`), `no_progress_cycles >= max_no_progress_cycles` (default 3 — a full pass that advances `done` by zero counts as one), or wall-clock elapsed since `autonomy.started_at >= max_wall_clock_hours` (default 10). Counts and hours only — **never a fabricated cost**. Breakers are re-checked before every feature *and* after every attempt *and* before every model call (the arbiter, the final review) — not only once per pass; a single in-flight pipeline phase may still overrun its check window before the next boundary catches it (an honest, not a hard real-time, guarantee). + +**PASS integrity (spec Component 5).** A marathon SUCCESS is not blindly trusted. Two guards: (1) the driver **sample-audits a deterministic fraction of PASSes** — a fresh adversarial `compound-v:spec-reviewer` re-review (QUALITY + the 2.5 reward-hack check in [`agents/spec-reviewer.md`](../../agents/spec-reviewer.md)) on a sampled successful feature; the concrete rule (every 3rd `done`, plus always the first success of the invocation) lives in [`v-epic.md`](../../commands/v-epic.md) §4, and a failed sample-audit reverts the feature via the single atomic `--record-audit-failed` (status→`failed` + clears the obligation + invalidates a passed review in one write) then routes it through the arbiter path; (2) the **final cross-feature re-verification** (`--record-final-review`) gates terminal `done`. Both are model calls, so both are preceded by a breaker re-check. **The sample-audit obligation is durable, not a driver memo:** a sampled feature's `--mark-sample-audit-due` is persisted+committed together with (or before) its `done`, so `done` is never on disk without the pending obligation; the state script then blocks `--record-final-review passed` and makes `--next --autonomous` surface `"sample_audit_due: ..."` until the audit runs — so a crash between "mark done" and "run the audit" cannot let an unaudited success slip through to `done`. **A crash-recovered run is subject to the same sampling** — the marathon `epic needs reconcile` path routes a recovered success into §4's success handler (sample-decide, then mark-due-before-done), never the checkpoint reconcile's direct `done` write, so recovery can't smuggle an unsampled first-success past the gate. + +**Terminal states:** `done` (all features done **and** `final_review.status=="passed"` — feature completion alone is never `done`); `blocked_needing_human` (a `halt_epic` verdict, a tripped breaker, or exhausted reachable work — the v2.10 blocker terminal); `running_with_failures` (non-terminal, work still runnable, or all done and awaiting final review). v2.10 never emits `done_with_blockers` — that terminal needs a second safe external confirming family, deferred to v2.11. + +**Human recovery from a halt (resume the marathon — not a fallback to checkpoint).** `blocked_needing_human` is a latch, but it is not a dead end: a human resolves the root cause and **re-runs `/v:epic ` to resume the marathon**. A tripped breaker is cleared and re-armed with `--clear-breaker` (`--reset-wall-clock` for the wall-clock cap, `--set-max-total-attempts N` for the attempt cap — without re-arming, the same cap re-trips on the first pass); a sticky `halt_epic` verdict is cleared with `--clear-disposition --feature F`; a mid-pipeline crash is recovered in place with `/v:resume ` (the feature's recorded `run_id`), distinct from a full `--update --status pending` restart from the spec. All human-gated — **nothing un-trips or re-runs automatically in v2.10** (that is the deferred v2.11 auto-watcher). The exact runbook (every field + copy-paste commands) is [`v-epic.md`](../../commands/v-epic.md) §7. + +--- + ## Honesty boundary State this to the user — epic mode is bounded, not magic: - **Autonomous *chaining*, not "guess a product from one sentence."** Each feature still needs a **real spec** — brainstormed and human-approved up front (carried as `spec_path`); the per-feature pre-flights and partition do the work, the epic layer only orders and chains. -- **Bounded, not unbounded.** An epic is *N full v1.0 runs*; it runs under a `MAX_FEATURES` budget (default 1) and **STOPS at a human checkpoint** after the budget is spent — not a fire-and-forget overnight build. The checkpoint is a **driver-enforced cadence** (default: stop after every feature), the human-in-the-loop point — not a script-enforced token meter. +- **Bounded, not unbounded (checkpoint stance).** An epic is *N full v1.0 runs*; it runs under a `MAX_FEATURES` budget (default 1) and **STOPS at a human checkpoint** after the budget is spent — not a fire-and-forget overnight build. The checkpoint is a **driver-enforced cadence** (default: stop after every feature), the human-in-the-loop point — not a script-enforced token meter. - **Large epics run sequentially, feature-by-feature.** Parallelism is *within* a feature (the v1.0 batch dispatch); features advance one runnable-front at a time in topological order. Independent features at the same depth still run one after another — there is **no cross-feature parallel dispatch** in v1.1. - **Quality is bounded by per-feature spec + partition quality.** A weak decomposition (overlapping features, missed deps) produces a weak epic. The state spine guarantees **order and resumability**, not that your decomposition was right. +- **Marathon (v2.10, opt-in) — "survives a fall" is honest, not magic.** *In-session:* the loop continues past a soft per-feature error to the next runnable feature automatically, within the one live `/v:epic` invocation; a crashed feature is caught by the existing `running` → reconcile path on the next pass. *Hard death* (quota, closed terminal, crashed machine): a **human re-invokes `/v:epic `**, re-entrant, resuming from `epic-state.json`. **There is no automatic resurrection in v2.10** — nothing revives this epic while you're away; that is the deferred v2.11 auto-watcher (Execution Lease + Two-Tier Watcher + generation-fenced execution — its own spec, its own review pass). No fabricated cost/token metrics anywhere in either stance. --- ## Cross-references - Epic state spine (CLI + validation): [`scripts/compound-v-epic-state.py`](../../scripts/compound-v-epic-state.py) -- Driver command: [`commands/v-epic.md`](../../commands/v-epic.md) (`/v:epic`) +- Marathon arbiter panel (v2.10): [`scripts/compound-v-epic-arbiter.py`](../../scripts/compound-v-epic-arbiter.py) +- Driver command: [`commands/v-epic.md`](../../commands/v-epic.md) (`/v:epic`) — the marathon loop's exact command sequence - Per-run state machine + crash-resume (one level down): [`state-machine.md`](state-machine.md) - The per-feature manifest contract: [`execution-manifest.md`](execution-manifest.md) - The main skill: [`SKILL.md`](SKILL.md)