Update Cargo.toml with 40 changed files (#4809)

docs/concepts/agentic-disk-reclamation.md

@@ -1,7 +1,7 @@
 ---
 title: Agentic disk reclamation
 description: Design rationale for Simard's fully agentic disk-reclamation capability — why the reclaim agent proposes candidates while a deterministic Rust executor disposes of them, the non-bypassable protected-path rails, and how the capability self-heals disk pressure without per-cycle hand-crafted cleanup heuristics.
-last_updated: 2026-07-07
+last_updated: 2026-07-27
 review_schedule: as-needed
 owner: simard
 doc_type: concept
@@ -122,7 +122,47 @@ Only three candidate classes can *ever* be reclaimed, and only after passing
 
 1. **Tracked worktrees** whose PR is MERGED or CLOSED and which are idle,
 2. **Orphaned, de-registered** (untracked) worktree directories,
-3. **Stale build caches** (`target/` and shared cargo target dirs).
+3. **Stale build caches** — per-managed-repo `<repo>/target/` (its `debug/`,
+   `release/`, `llvm-cov-target/`, and incremental caches) plus the shared cargo
+   target dirs under the state root.
+
+### Routine reclaim frees `target/` between emergency passes
+
+The reclamation **allow-root set** includes `<repo>/target` for every managed
+repo (rooted at the `target/` *parent* so the guard's strict-inside containment
+still confines removal to descendants of `target/`, never `<repo>/src` or
+`<repo>/.git`). This is what lets **routine** (non-emergency) reclaim actually
+free rebuildable build artifacts.
+
+Without that root, routine reclaim had nothing in scope: every `target/debug`
+candidate the agent proposed was rejected as *outside allow-root* and pushed to
+the human-review list, producing the "freed 0 bytes, 0 paths removed, N skipped
+for review" no-op. Disk then climbed unchecked until the deterministic
+`emergency_cleanup` (Tier 1) fired at ~95% and freed space in a burst — a
+~30-minute saw-tooth that rode the partition at 94–99% and risked `ENOSPC`
+between passes. With `<repo>/target` in scope, routine reclaim removes the same
+rebuildable artifacts *proactively and incrementally*, so the partition no longer
+depends on the emergency backstop to avoid filling. `emergency_cleanup` remains
+the deterministic hard stop, but it should now fire rarely rather than every
+cycle.
+
+Widening to `target/` stays **additive and non-breaking**: the artifacts are the
+same rebuildable class emergency cleanup already treats as safe, removal is still
+gated by every hard rail below (the live-PID rail refuses any candidate that a
+running process is sitting *inside* via `/proc/<pid>/cwd`; the protected deny-set
+still overrides the allow-root), and no new candidate *category* is introduced — only the containment
+scope of the existing `StaleBuildCache` kind is corrected.
+
+### Observability: per-candidate skip reasons
+
+Every rejected candidate is logged with a structured `tracing` event carrying its
+`path`, the closed `RejectReason` enum, and its `CandidateKind` — turning the
+opaque "N skipped for review" summary into a per-path audit trail. An operator
+can now tell a `target/debug` skipped for **Live process** (a build is running;
+expected) apart from one skipped for **Outside allow-root** (a scope bug worth
+investigating). The agent's free-text rationale is never logged as a field (anti
+log-forging); only the enum reason is. See
+[disk-reclaim telemetry](../reference/disk-reclaim-telemetry.md).
 
 ### Fail-closed, everywhere
 

docs/concepts/goal-reblock-backoff-dedup.md

@@ -0,0 +1,161 @@
+---
+title: Goal-reblock backoff & stewardship dedup — the Overseer stops relaunching blocked goals every cycle
+description: >
+  Why the Overseer no longer re-observes and re-enqueues an already-blocked goal
+  on every ~15-minute cycle, and no longer files a fresh stewardship issue for
+  the same block (issues #4817, #4828). Explains the observed 8h+ churn
+  (identical "GoalHygiene ... blocked (0 no-action cycle(s))" from
+  2026-07-26T18:54Z through 2026-07-27T02:19Z, repeatedly hitting "held:
+  per-cycle launch cap reached"), why GoalHygiene briefs escaped the existing
+  gap-scan backoff, how a dedicated `goal_reblock_backoff` BackoffGate keyed on
+  goal:blocked:{goal_id} suppresses the relaunch, and how stabilising the
+  failure signature folds the duplicate stewardship issues into one.
+last_updated: 2026-07-27
+review_schedule: as-needed
+owner: simard
+doc_type: concept
+status: implemented
+related:
+  - ./gap-scan-backoff-dedup.md
+  - ./blocked-goal-escalation-backoff.md
+  - ./ooda-reinvestigate-blocked-goals.md
+  - ./no-progress-breaker-storm-suppression.md
+  - ../reference/goal-reblock-backoff-api.md
+  - ../reference/overseer-backoff-gate-api.md
+  - ../reference/stewardship-api.md
+  - ../howto/diagnose-recurring-goal-reblock-churn.md
+---
+
+# Goal-reblock backoff & stewardship dedup
+
+> **Status: implemented (issues
+> [#4817](https://github.com/rysweet/Simard/issues/4817),
+> [#4828](https://github.com/rysweet/Simard/issues/4828)).** The Overseer's
+> goal-hygiene path now runs through a dedicated `goal_reblock_backoff`
+> `BackoffGate` and files exactly one stewardship issue per still-blocked goal.
+> Primary sources:
+> [`src/overseer/mod.rs`](https://github.com/rysweet/Simard/blob/main/src/overseer/mod.rs)
+> (`goal_reblock_backoff` field + `gate()`/`act()` wiring, and the `GoalBlocked`
+> signature text),
+> [`src/overseer/guardrails.rs`](https://github.com/rysweet/Simard/blob/main/src/overseer/guardrails.rs)
+> (the shared `BackoffGate`), and
+> [`src/stewardship/dedup.rs`](https://github.com/rysweet/Simard/blob/main/src/stewardship/dedup.rs)
+> (`normalize_for_signature` counter redaction). API surface:
+> [goal-reblock backoff reference](../reference/goal-reblock-backoff-api.md).
+
+## The defect this fixes
+
+The Overseer's goal-hygiene step observes blocked goals and, for each, decides
+whether to relaunch a covering recipe workstream and/or file a stewardship
+issue. Its `dedup_key` and human-facing text are built here:
+
+```rust
+// ProblemKind::GoalHygiene
+format!("goal:blocked:{goal_id}"),                                    // dedup_key
+format!("goal {goal_id} blocked ({consecutive_no_action} no-action cycle(s))"), // signature text
+```
+
+Two things went wrong at once:
+
+1. **No relaunch suppression for GoalHygiene.** The
+   [gap-scan backoff](./gap-scan-backoff-dedup.md) and the in-flight guard were
+   scoped to `WORKSTREAM_COVERAGE_GROUP`; `GoalHygiene` briefs have
+   `sequence_group = None` and slipped straight past them. So every ~15-minute
+   cycle the overseer **re-observed the same two already-in-flight/blocked
+   goals and relaunched their Simard recipe workstreams**, repeatedly hitting
+   `held: per-cycle launch cap reached` without ever clearing the block. This ran
+   for **8h+** — identical `GoalHygiene … blocked (0 no-action cycle(s))` lines
+   from `2026-07-26T18:54Z` through `2026-07-27T02:19Z`.
+
+2. **The signature fluctuated, so dedup never matched.** The
+   `({consecutive_no_action} no-action cycle(s))` counter is part of the
+   **signature-bearing error text**, and `failure_signature()` hashes that text
+   (via `normalize_for_signature`). Each time the counter ticked (`0`, then `1`,
+   …) the signature changed, so `find_existing()` never matched the prior issue
+   and the overseer **self-filed duplicate stewardship issues** — the
+   `recurring_goal_reblock` clusters reported as #4817 and #4828.
+
+The net effect: a goal that was *already blocked and already being worked* got
+re-observed, re-enqueued, and re-issued forever, hammering the launch cap and
+spamming duplicate stewardship issues.
+
+## The fix, part 1: relaunch backoff keyed per goal
+
+A dedicated gate suppresses per-cycle relaunch of a still-blocked goal:
+
+```rust
+goal_reblock_backoff: BackoffGate,   // in the Overseer struct
+```
+
+keyed on a stable per-goal key:
+
+```rust
+let key = format!("overseer-obs:goal:blocked:{goal_id}");
+```
+
+It reuses the same bounded-exponential-backoff semantics as the
+[gap-scan `BackoffGate`](../reference/overseer-backoff-gate-api.md):
+
+- The **first** observation of a blocked goal admits (arms the base window).
+- A re-observation **inside** the window is **suppressed** — the goal is not
+  relaunched — honoring the in-flight/held state and the
+  `consecutive_no_action` counter.
+- The window grows `× multiplier` per re-hit, capped, so a persistently blocked
+  goal is retried on a *slowing* cadence rather than every cycle.
+- The gate **re-admits immediately when the block clears** (a long silence, ≥ 2×
+  the window, resets to the base window), so a goal that becomes actionable again
+  resurfaces promptly.
+
+`gate()` **peeks** (decide without recording) and `act()` **commits** only after
+a successful relaunch, so a launch that is itself held (cap reached) does not
+consume the dedup slot.
+
+> **Suppression is relaunch-only.** A blocked goal that genuinely needs human
+> attention still surfaces **exactly one** stewardship issue for visibility — the
+> gate only silences the wasteful *relaunch churn*, never the human signal.
+
+## The fix, part 2: a stable signature folds duplicate issues into one
+
+The fluctuating counter is moved **out of the signature-bearing error text** and
+kept only in the issue **body/title annotation** (where the human still sees the
+full context). The hashed signature input therefore no longer changes as the
+counter ticks, so `find_existing()` matches the prior issue and dedup fires
+across cycles → **one** stewardship issue per blocked goal.
+
+As defense-in-depth, `normalize_for_signature` (`src/stewardship/dedup.rs`)
+additionally **redacts** residual counter patterns — `(<N> no-action cycle(s))`
+and `no progress for <N> cycles` — so any counter that leaks into a signature
+input still folds to a single signature. This mirrors the existing UUID-redaction
+contract that already scrubs volatile session/run ids.
+
+```mermaid
+flowchart LR
+    obs["Observe blocked goal_id"] --> key["key = overseer-obs:goal:blocked:{goal_id}"]
+    key --> peek{"goal_reblock_backoff.peek()"}
+    peek -->|Suppress| hold["skip relaunch this cycle (no launch-cap churn)"]
+    peek -->|Admit| relaunch["relaunch workstream → commit() on success"]
+    obs --> sig["failure_signature(GoalBlocked, text w/o counter)"]
+    sig --> dedup{"find_existing()?"}
+    dedup -->|match| one["reuse the single open stewardship issue"]
+    dedup -->|none| file["file exactly one issue"]
+```
+
+## Safety & scope
+
+- **Additive / non-breaking.** New gate field + signature-text change +
+  redaction rule; no API or PRD change, no `print!`/`println!` — structured
+  `tracing` + OTel only.
+- **Untrusted `goal_id`.** The goal id is normalized (bounded length,
+  `[A-Za-z0-9._:-]`) before it is embedded in a dedup key or an issue
+  title/body, preventing signature collisions and issue-body injection.
+- **Bounded state.** Backoff state is saturating and evicts on block-clear/TTL,
+  so per-goal keys cannot grow unbounded on a long-running daemon.
+- **Exactly-once escalation.** Relaunch suppression never suppresses the single
+  stewardship issue — information availability is preserved for the human.
+
+## See also
+
+- [Gap-scan dedup & backoff](./gap-scan-backoff-dedup.md) — the sibling rail for coverage gaps.
+- [Blocked-goal escalation backoff](./blocked-goal-escalation-backoff.md) — the WhisperGate backoff for escalations.
+- [Goal-reblock backoff reference](../reference/goal-reblock-backoff-api.md) — the typed API.
+- [Diagnose recurring goal-reblock churn](../howto/diagnose-recurring-goal-reblock-churn.md) — the operator runbook.

docs/concepts/reclaim-effectiveness-backoff.md

@@ -0,0 +1,157 @@
+---
+title: Reclaim effectiveness backoff — disk-reclaim stops churning when it frees nothing
+description: >
+  Why the OODA daemon no longer re-runs agentic disk-reclamation every cycle
+  when reclamation keeps freeing 0 bytes (issues #4809, #4825, #4810). Explains
+  the observed churn on the ~94%-full /tmp/state host, why the plain
+  %-used trigger re-fired a proven-ineffective run every ~15 minutes, how the
+  new `ReclaimEffectivenessGate` applies bounded exponential cooldown after a
+  streak of no-op reclaims, how a genuinely filling disk still bypasses the
+  cooldown at a hard %-used ceiling, and why the gate is suppress-only (it can
+  never turn a dry-run into an apply).
+last_updated: 2026-07-27
+review_schedule: as-needed
+owner: simard
+doc_type: concept
+status: implemented
+related:
+  - ./agentic-disk-reclamation.md
+  - ./automated-disk-health.md
+  - ../reference/reclaim-effectiveness-gate-api.md
+  - ../reference/disk-reclaim-telemetry.md
+  - ../reference/disk-reclaim-api.md
+  - ../operations/reclaim-effectiveness-kill-switch.md
+  - ../howto/configure-reclaim-effectiveness.md
+  - ../howto/configure-disk-reclamation.md
+---
+
+# Reclaim effectiveness backoff
+
+> **Status: implemented (issues
+> [#4809](https://github.com/rysweet/Simard/issues/4809),
+> [#4825](https://github.com/rysweet/Simard/issues/4825),
+> [#4810](https://github.com/rysweet/Simard/issues/4810)).** The OODA daemon's
+> disk-reclaim trigger now consults a `ReclaimEffectivenessGate` before firing,
+> so a reclamation run that keeps freeing nothing is not re-attempted on every
+> cycle. Primary sources:
+> [`src/disk_reclaim/effectiveness.rs`](https://github.com/rysweet/Simard/blob/main/src/disk_reclaim/effectiveness.rs)
+> (the gate),
+> [`src/operator_commands_ooda/daemon/mod.rs`](https://github.com/rysweet/Simard/blob/main/src/operator_commands_ooda/daemon/mod.rs)
+> (the daemon trigger wiring), and
+> [`src/disk_reclaim/mod.rs`](https://github.com/rysweet/Simard/blob/main/src/disk_reclaim/mod.rs)
+> (`emit_reclaim_telemetry`). API surface:
+> [reclaim effectiveness gate reference](../reference/reclaim-effectiveness-gate-api.md).
+
+## The defect this fixes
+
+The daemon's Tier-3 self-heal step (issue #2704, see
+[agentic disk reclamation](./agentic-disk-reclamation.md)) fires whenever a
+cheap `df` probe reports `%-used ≥ SIMARD_DISK_RECLAIM_PCT`. On the production
+OODA host the working partition sat at **~94–99% used with ~12 GiB free of
+196 GiB**, permanently above the trigger threshold, while every reclamation run
+freed **0 bytes** — the candidates the analysis agent proposed were all
+undeletable (protected paths, live processes, uncommitted/unpushed worktrees)
+and every rail correctly refused them.
+
+The trigger had **no memory that the previous run accomplished nothing**. So
+each ~15-minute daemon cycle:
+
+1. observed `used ≥ threshold`,
+2. re-invoked the full agentic reclaim capability (a brain call + a
+   recipe-runner scratch dir + per-cycle run artifacts),
+3. freed 0 bytes,
+4. and — because the run itself consumed scratch space — sometimes left the
+   partition *fuller* than before.
+
+The result is the churn reported in #4809 (*"routine disk-reclaim is
+ineffective"*), #4825 (*"systemic disk-reclaim churn in the OODA daemon"*), and
+#4810 (*"OODA daemon rides disk at 94–99%"*): reclamation re-scanned the same
+undeletable paths forever without ever reclaiming space, and the churn itself
+added disk pressure.
+
+## The fix: effectiveness-aware exponential cooldown
+
+Reclamation is now gated on **whether it recently worked**, not only on
+**how full the disk is**. The daemon records the outcome of each run and, after
+a streak of ineffective runs, backs off exponentially before trying again.
+
+```mermaid
+flowchart TD
+    tick["Daemon cycle: used_pct ≥ SIMARD_DISK_RECLAIM_PCT?"] -->|no| noop["log 'under threshold', done"]
+    tick -->|yes| ceiling{"used_pct ≥ hard ceiling?"}
+    ceiling -->|yes| run["RUN reclaim (bypass cooldown)"]
+    ceiling -->|no| gate{"ReclaimEffectivenessGate.peek()"}
+    gate -->|Suppress| skip["SKIP this cycle; WARN + suppressed_cycles metric"]
+    gate -->|Admit| run
+    run --> eff{"effective? (bytes_freed > 0 or used_pct dropped)"}
+    eff -->|yes| reset["record success → reset noop_streak & cooldown"]
+    eff -->|no| grow["record no-op → grow noop_streak & cooldown window"]
+```
+
+The gate reuses the same bounded-exponential-backoff semantics as the
+Overseer's [`BackoffGate`](../reference/overseer-backoff-gate-api.md):
+
+- **First no-op** arms a base cooldown window (default 15 min).
+- **Each further no-op** grows the window `× multiplier` (default ×2), hard-capped
+  (default 4 h). While inside the window, the daemon **skips** the reclaim run
+  entirely — no brain call, no scratch dir, no artifacts.
+- **A run that frees space** (positive `bytes_freed`, or a measured drop in
+  `used_pct`) is *effective*: it resets the streak and the cooldown so genuine
+  reclamation stays responsive.
+- **A long silence** since the last attempt (≥ 2× the current window) also resets
+  to the base window, so a disk that fills again after a quiet period resurfaces
+  promptly.
+
+### Genuine fill-ups are never masked
+
+Suppression is bypassed whenever locally-observed `%-used` crosses a **hard
+ceiling** (`SIMARD_DISK_RECLAIM_HARD_CEILING_PCT`, default `97`). The bypass
+authority is derived from a *fresh local `df` sample* — never from re-ingested
+telemetry — so a real, accelerating fill-up always triggers reclamation
+regardless of the cooldown. The cooldown only silences the pathological case:
+*already above the trigger, but reclamation demonstrably cannot help.*
+
+Every suppressed cycle is visible: it emits a `WARN` daemon log line and
+increments a `simard.disk.reclaim.suppressed_cycles` counter, and the
+per-run telemetry gains `noop_streak` / `suppressed_cycles` / `effective`
+attributes (see [disk-reclaim telemetry](../reference/disk-reclaim-telemetry.md)).
+Operators can therefore see *"reclaim is being deliberately held back because it
+keeps freeing nothing"* rather than silence.
+
+## Cross-cycle skip memory (don't re-propose the same undeletable path)
+
+Independently of the cooldown, the executor now remembers the **canonicalized**
+paths a rail rejected and refuses to re-propose them on the next cycle, so a
+single undeletable worktree is not re-vetted every run. Paths are
+canonicalized *before* both the guard check and the skip-memory lookup, which
+prevents a symlink or `..` alias from smuggling a protected path past the guard.
+See the [reclaim effectiveness gate reference](../reference/reclaim-effectiveness-gate-api.md#cross-cycle-skip-memory).
+
+## Safety posture
+
+- **Suppress-only.** The gate can only *skip* a run. It never changes the
+  destructive posture: the daemon stays **dry-run by default** and
+  `SIMARD_DISK_RECLAIM_DAEMON_APPLY=1` remains the sole apply opt-in. A unit
+  test asserts the gate can never transition dry-run → apply.
+- **Fail safe, not open.** Any parse/validation/canonicalization error treats a
+  candidate as *not authorized to delete* and does **not** suppress a cycle
+  where suppression could hide real pressure.
+- **Bounded arithmetic.** The no-op streak, cooldown exponent, and window are
+  saturating so a long-running daemon cannot overflow into an absurd cooldown.
+- **Additive.** No existing metric, env var, or behavior changes; the PRD is
+  preserved and there is no `print!`/`println!` — the gate emits structured
+  `tracing` + OTel only.
+
+## Turning it off
+
+The whole effectiveness gate is a single kill switch,
+`SIMARD_DISK_RECLAIM_EFFECTIVENESS_GATE=off`, which reverts to the previous
+"fire on every over-threshold cycle" behavior. See the
+[reclaim-effectiveness kill switch](../operations/reclaim-effectiveness-kill-switch.md).
+
+## See also
+
+- [Configure reclaim effectiveness](../howto/configure-reclaim-effectiveness.md) — the operator knobs.
+- [Agentic disk reclamation](./agentic-disk-reclamation.md) — the propose/dispose design this gates.
+- [Automated disk health](./automated-disk-health.md) — the daemon step the gate lives in.
+- [Reclaim effectiveness gate reference](../reference/reclaim-effectiveness-gate-api.md) — the typed API.