Update documentation with 11 changed files (#4558)

docs/howto/diagnose-a-red-canary-unit-test-gate.md

@@ -0,0 +1,131 @@
+---
+title: How to diagnose a red-canary unit-test gate
+description: Operator runbook for the case where every self-deploy is refused on failing_gate="unit-test" while `cargo test --lib` passes clean standalone — recognise the environment-induced false red, read the named failing test from the enriched failing_detail, confirm the gate now runs hermetically in an isolated state root, and verify the self-deploy loop advances past the stuck SHA.
+last_updated: 2026-07-24
+review_schedule: as-needed
+owner: simard
+doc_type: howto
+status: active
+related:
+  - ../reference/hermetic-unit-test-gate.md
+  - ../reference/canary-gate-convergence.md
+  - ../reference/overseer-deploy-canary-diagnostics.md
+  - ../reference/state-root-resolution.md
+  - ./converge-a-stuck-red-canary-self-deploy.md
+  - ../safe-self-update.md
+---
+
+# How to diagnose a red-canary unit-test gate
+
+> **Status: active.** This describes shipped behaviour: the hermetic
+> `unit-test` gate and the named `failing_detail`. For the full design and API,
+> see [Hermetic unit-test canary gate](../reference/hermetic-unit-test-gate.md).
+
+Use this runbook when the Overseer refuses **every** self-deploy on the
+`unit-test` gate — `DeployDrift` climbs, `running_commit` is pinned — yet the
+test suite passes when you run it by hand.
+
+## 1. Confirm the symptom
+
+The #4558 signature is an identical `unit-test` refusal on every tick, with a
+fast exit-101 abort:
+
+```bash
+journalctl --user -u simard -o cat \
+  | grep -E 'overseer::deploy' | tail -n 40
+```
+
+You are looking for a repeating refusal against the **same** `target_commit`:
+
+```
+WARN overseer::deploy: self-deploy refused by deploy gate
+    target_commit=7d0964f running_commit=7d0964f
+    failing_gate="unit-test"
+    failing_detail="tests failed (exit exit status: 101): <failing test block>"
+```
+
+## 2. Rule out a genuine regression
+
+Run the same suite the gate runs, **standalone**, from the repo root:
+
+```bash
+cargo test --lib
+```
+
+- **If it fails** — this is a real regression. Read the named test from
+  `failing_detail` (see step 3) and fix the source. Stop here.
+- **If it passes clean** (e.g. `9279 passed; 0 failed`) but the gate reddens —
+  this is the **environment-induced false red** the hermetic gate fixes. The
+  in-process lib-test was aborting because it bound the live daemon's socket or
+  locked its shared WAL / cognitive-store under the daemon's state root. With the
+  hermetic gate shipped, this no longer happens; if you still see it, continue.
+
+## 3. Read the named failing test
+
+The gate now captures **both** stdout and stderr and extracts the failing test
+name into `failing_detail` (clamped to 4096B at the gate, 512B downstream). You
+should see a real marker block, **not** a truncated `Drop t…` spinner fragment:
+
+```
+failing_detail="tests failed (exit status: 101): failures:
+    self_relaunch::gates::tests::extract_failure_detail_names_test
+panicked at src/self_relaunch/gates.rs:412: assertion failed …"
+```
+
+If `failing_detail` still shows only `tests failed (exit …)` with no test name,
+the running binary predates #4558 — deploy a build that includes the hermetic
+gate.
+
+## 4. Confirm the gate runs hermetically
+
+The `unit-test` gate spawns `cargo test` with the four isolation keys overridden
+to a fresh per-run temp dir and `current_dir` set to the manifest dir:
+
+`SIMARD_STATE_ROOT`, `SIMARD_HOME`, `HOME`, `TMPDIR` → a private
+`tempfile::TempDir`.
+
+`CARGO_HOME` / `RUSTUP_HOME` are **pinned** to absolute paths resolved from the
+real (pre-override) `HOME` *before* `HOME` is redirected, so cargo/rustup still
+find the toolchain — without this, the `HOME` override would itself cause a fresh
+exit-101 abort of the same class.
+
+That temp state root is empty, so the in-process suite resolves its own WAL /
+cognitive-store / socket path
+([state-root resolution](../reference/state-root-resolution.md)) and cannot
+collide with the live daemon. To verify locally that a running daemon no longer
+red-canaries a green tree, run the gate's fixture test:
+
+```bash
+# Green fixture passes even with a simulated live daemon holding the shared root;
+# red fixture's failing_detail names the failing test.
+cargo test -p simard self_relaunch::gates
+cargo test --test unit_test_gate_fixture   # integration fixture
+```
+
+If the temp-dir/env setup ever fails, the gate **fails closed** — it returns a
+`unit-test gate could not create an isolated state root: …` failure rather than
+silently falling back to the live daemon's state root. That is expected
+fail-closed behaviour, not the #4558 bug; fix the temp/disk condition and retry.
+
+## 5. Verify convergence
+
+Once the gate renders a true green verdict, the guarded deploy gate stops
+returning `RedCanary`, the swap proceeds, and drift returns to 0:
+
+```bash
+journalctl --user -u simard -o cat | grep -E 'overseer::(tick|deploy)' | tail -n 20
+```
+
+You should see the deploy **succeed** and the next drift observation report
+`DeployDrift == 0` — the loop advances past the previously stuck target SHA
+instead of re-queuing the identical `unit-test` refusal.
+
+## See also
+
+- [Hermetic unit-test canary gate](../reference/hermetic-unit-test-gate.md) —
+  the design, the four isolation keys, `extract_failure_detail`, and the
+  fail-closed / truncation contracts.
+- [How to converge a stuck red-canary self-deploy](./converge-a-stuck-red-canary-self-deploy.md) —
+  the sibling runbook for a red canary on any gate.
+- [Overseer deploy red-canary diagnostics](../reference/overseer-deploy-canary-diagnostics.md) —
+  the `failing_gate` / `failing_detail` surface this runbook reads.

docs/index.md

@@ -160,6 +160,8 @@ Bare `simard` prints the unified help text instead of attempting a hidden enviro
 - [Overseer tick self-healing reference](./reference/overseer-tick-self-healing.md) - The transient-failure self-healing rung on the `overseer` meta-thread's per-tick health: a `cycle_failed` tick whose `run_cycle()` error is a transient upstream blip (5xx, timeout, connection reset, rate-limit) routes to a self-clearing `"backoff"` for one cadence instead of `"erroring"`. Covers the additive `transient_cycle_failure` field, the fail-closed `is_transient` classifier, the bounded consecutive-transient escalation ceiling, and the safety invariants (#893).
 - [Overseer deploy red-canary diagnostics reference](./reference/overseer-deploy-canary-diagnostics.md) - The additive diagnostics that make a refused self-deploy diagnosable: the reddening canary gate name and detail surfaced into the `overseer::deploy` WARN and OTel attributes (`failing_gate` / `failing_detail`), the `CanaryResult.failing_gate` / `failing_detail` fields, `CanaryResult::refusal_reason`, the enriched `deploy_refused` notification and `Capability` detail, and the fail-closed `is_transient` `deploy_gate` / `target_canary` guard that stops a red canary being retried as a transient blip (#4420).
 - [Canary gate isolation and self-deploy convergence reference](./reference/canary-gate-convergence.md) - The #4440 root-cause repair that acts on those diagnostics so a healthy candidate self-deploys and `DeployDrift` returns to 0: per-gate `self_relaunch::gate` tracing/OTel spans in `verify_canary`, the additive `RelaunchConfig.canary_env` narrow allow-list plus `scrub_gate_env` that supplies a gate's legitimately-missing signal without weakening fail-closed semantics, and the self-deploy loop advancing past the stuck target SHA. See the [convergence runbook](./howto/converge-a-stuck-red-canary-self-deploy.md).
+- [Hermetic unit-test canary gate reference](./reference/hermetic-unit-test-gate.md) - The #4558 repair that stops the live daemon from red-canarying a green tree: `run_unit_test_gate` runs `cargo test` in a fail-closed, per-run isolated state root (`SIMARD_STATE_ROOT`/`SIMARD_HOME`/`HOME`/`TMPDIR` → a fresh `tempfile::TempDir` + manifest `current_dir`) so an in-process lib-test cannot bind the daemon's socket or lock its shared WAL/cognitive-store, plus `extract_failure_detail` capturing BOTH stdout and stderr and lifting the failing test name into `failing_detail` (4096B gate clamp, 512B downstream) instead of a truncated `Drop t…` fragment. See the [diagnosis runbook](./howto/diagnose-a-red-canary-unit-test-gate.md).
+- [How to diagnose a red-canary unit-test gate](./howto/diagnose-a-red-canary-unit-test-gate.md) - Operator runbook for a self-deploy refused on `failing_gate="unit-test"` while `cargo test --lib` passes standalone: recognise the environment-induced false red, read the named failing test from the enriched `failing_detail`, confirm the gate now runs hermetically in an isolated state root, and verify the self-deploy loop advances past the stuck SHA (#4558).
 - [How to review the Overseer's workstream gaps](./howto/review-overseer-workstream-gaps.md) - Read, act on, and tune the Overseer's recurring "what workstreams are we missing?" gap-scan — the uncovered high-priority goals, high-signal issues, and unaddressed anomalies it flags each tick, where the deduped notification appears, and the `SIMARD_OVERSEER_GAP_SCAN` knobs (#2630).
 - [Overseer workstream gap-scan reference](./reference/overseer-workstream-gap-scan.md) - The additive Observe→Orient→Act gap-scan: the `Signal::WorkstreamGap`/`GapItem`/`ProblemKind::WorkstreamCoverage` model, the coverage-set detection contract, the deduped NotifyOperator act path, the `SIMARD_OVERSEER_GAP_SCAN` configuration, and the additive `OverseerTickReport.workstream_gaps_detected` counter (#2630).
 - [Overseer recipe-launch idempotency reference](./reference/overseer-recipe-launch-idempotency.md) - The launcher-level rail that makes `AmplihackRecipeRunner::spawn` idempotent per task signature: the pure `recipe_signature` normalization (`target_repo` + `task_description`, trim/lowercase/whitespace-collapse, `\u{1F}` separator), the reap-then-dedup order, the fail-visible `overseer::recipe` suppressed-launch warning, the shared-handle `probe` semantics, and the injectable `ChildSpawner`/`SpawnedChild` test seam — so a still-blocked signature no longer spawns a byte-identical `smart-orchestrator` every tick (#4125).