fix(disk): relocate cargo target off 28G / volume + anti-thrash cleanup gate (#4803)

docs/concepts/root-disk-saturation-relocation.md

@@ -0,0 +1,118 @@
+---
+title: Root-disk saturation relocation and thrash guard
+description: Why Simard relocates cargo build artifacts off the small root volume, adds hysteresis to emergency disk cleanup, and refuses build-heavy dispatch under disk pressure — the fix for the #4803 crash-loop.
+last_updated: 2026-07-27
+review_schedule: as-needed
+owner: simard
+doc_type: explanation
+related:
+  - ./automated-disk-health.md
+  - ./agentic-disk-reclamation.md
+  - ../howto/relocate-build-artifacts-off-root-volume.md
+  - ../reference/build-artifact-relocation-and-disk-thrash-guard.md
+  - ../reference/resource-admission-api.md
+---
+
+# Root-disk saturation relocation and thrash guard
+
+Fixes [#4803](https://github.com/rysweet/Simard/issues/4803).
+
+## The failure this prevents
+
+On hosts where `/home` and `~/.simard` live on a small root volume (a 28 GiB
+`/` in the reference incident), cargo build artifacts written under
+`~/.simard` — chiefly `target/debug` and `target/llvm-cov-target` — filled the
+root filesystem to **0 bytes free**. That single condition cascaded across the
+whole daemon:
+
+- `cognitive-open-lock` refusals and typed `database is locked` errors,
+- `memory-ipc` write failures in goal-session engineers,
+- stalled goal advancement across every workstream.
+
+The daemon's Tier-1 **emergency disk cleanup** re-fired roughly every 25
+minutes (observed: 21:58 → 99%, 23:00 → 94%, 23:35 → 94%, 00:10 → 96%, 00:42
+→ 97%). Each run deleted `target/debug` + `target/llvm-cov-target` (0.7–2.6 GB)
+and pruned backups, but `/` refilled within one cycle because cargo
+immediately rebuilt into the same location. Cleanup was an **ineffective
+band-aid that thrashed** — it never removed the *cause*, only the symptom, and
+each pass burned I/O and starved progress.
+
+Meanwhile a 196 GiB `/tmp` volume sat with ~26 GiB free, unused for build
+artifacts.
+
+## Root cause
+
+Two independent defaults pointed the fastest-refilling artifacts at the small
+root volume:
+
+1. **`default_cargo_target_for_worktree`** (in `src/agent_supervisor/tmux.rs`)
+   defaulted `CARGO_TARGET_DIR` to `$HOME/.cargo-targets/<worktree>` — i.e.
+   under `~` on the 28 GiB `/`.
+2. A divergent hardcoded fallback in
+   `src/agent_supervisor/lifecycle/spawn.rs` set
+   `CARGO_TARGET_DIR=/tmp/simard-engineer-target` for the single-process spawn
+   path, so the two spawn paths disagreed about where artifacts lived.
+
+Emergency cleanup could not win against a build target that lived on the volume
+it was trying to protect.
+
+## The fix, in three moves
+
+The fix is **additive and non-breaking** — it changes defaults and adds guard
+rails; it does not change any public CLI surface.
+
+### 1. Relocate build artifacts onto the large volume (primary)
+
+The default cargo target root moves off `/`. The existing
+`SIMARD_CARGO_TARGETS_ROOT` override still wins; when it is unset the default
+now resolves to the large-volume fallback (`/tmp/simard-cargo-targets`) instead
+of `$HOME/.cargo-targets`. Both spawn paths delegate to **one** resolver, so
+they can no longer diverge. This alone stops `/` from re-saturating.
+
+See [`default_cargo_target_for_worktree`](../reference/build-artifact-relocation-and-disk-thrash-guard.md#c1-cargo-target-root-relocation).
+
+### 2. Give emergency cleanup hysteresis (stops the thrash)
+
+Emergency cleanup gains a **high/low watermark** (trigger at ≥ 95% used, do
+not re-fire until usage falls back below a distinct low watermark) plus a
+**persistent backoff marker** under `<state_root>/disk-health/`. Cleanup can no
+longer re-fire on every timer tick inside a single build window. After the
+relocation in move 1, `/` is no longer the fill target, so cleanup becomes
+rare rather than perpetual.
+
+See [emergency-cleanup hysteresis](../reference/build-artifact-relocation-and-disk-thrash-guard.md#c3-emergency-cleanup-hysteresis--backoff).
+
+### 3. Refuse build-heavy dispatch under disk pressure (belt)
+
+Before dispatching a build-heavy goal session, a **preflight** probes `/`
+through the existing [`disk_pressure`](../reference/resource-admission-api.md)
+gate. That gate classifies against a single min-free threshold `T`
+(`SIMARD_DISK_PRESSURE_MIN_FREE_GB`, default **20 GiB**) in two bands:
+`Warn` when free space is below `T`, and `Refuse` when it drops below `T/2`.
+So at defaults the preflight **warns below 20 GiB free and refuses below
+10 GiB free**. On a `Refuse`, dispatch is **loudly skipped for this cycle**
+(retried next cycle) rather than writing the daemon into ENOSPC again. This
+reuses the existing min-free threshold and 90% admission ceiling; it introduces
+no parallel constants.
+
+See [dispatch preflight](../reference/build-artifact-relocation-and-disk-thrash-guard.md#c4-build-heavy-dispatch-preflight).
+
+## Design principles honoured
+
+- **No silent fallbacks.** The preflight refuses loudly (`warn!`) and the
+  worktree allocator returns a hard `Err`; a degraded spawn is never preferred
+  over a visible refusal.
+- **Structured tracing + OTel only.** No `print!`/`println!`/`eprintln!` is
+  added; every signal flows through `tracing`.
+- **Containment before deletion.** Every `remove_dir_all` site is guarded by a
+  `symlink_metadata` check and a `starts_with(repo_root | state_root)`
+  containment assertion, so a hostile `target -> /` symlink cannot make cleanup
+  escape the tree.
+- **Fail-open on the marker, never fail-shut.** A corrupted or unreadable
+  backoff marker allows cleanup to proceed (logged), so the guard can never
+  suppress cleanup forever.
+
+## Where to go next
+
+- Operators: [Relocate build artifacts off the root volume](../howto/relocate-build-artifacts-off-root-volume.md).
+- Engineers: [Build-artifact relocation and disk-thrash guard — reference](../reference/build-artifact-relocation-and-disk-thrash-guard.md).

docs/howto/relocate-build-artifacts-off-root-volume.md

@@ -0,0 +1,153 @@
+---
+title: "How to relocate build artifacts off the root volume"
+description: Move cargo build artifacts off a small root volume, tune the emergency-cleanup backoff, and verify the daemon no longer crash-loops on a full disk.
+last_updated: 2026-07-27
+review_schedule: as-needed
+owner: simard
+doc_type: howto
+related:
+  - ../concepts/root-disk-saturation-relocation.md
+  - ../reference/build-artifact-relocation-and-disk-thrash-guard.md
+  - ./configure-disk-health-check.md
+  - ./configure-disk-reclamation.md
+  - ./reclaim-disk-space-and-run-low-space-rust-builds.md
+---
+
+# How to relocate build artifacts off the root volume
+
+By default Simard writes cargo build artifacts to
+`/tmp/simard-cargo-targets/<worktree>` so they land on a large volume instead
+of the small root filesystem that hosts `~/.simard`. This guide shows how to
+confirm the relocation, point it at a different volume, tune the emergency
+cleanup, and verify the [#4803](https://github.com/rysweet/Simard/issues/4803)
+crash-loop is gone.
+
+## When to use this
+
+Use this guide when:
+
+- `/home` and `~/.simard` share a small root volume that keeps filling up.
+- `~/.simard/ooda.log` shows emergency disk cleanup re-firing every ~25 minutes.
+- You see `database is locked`, `cognitive-open-lock` refusals, or
+  `memory-ipc` write failures that correlate with `/` at 0 bytes free.
+- You want build artifacts on a specific data volume rather than `/tmp`.
+
+## Step 1: Confirm which volume is filling up
+
+```bash
+df -h / /tmp
+```
+
+If `/` is at or near 100% while `/tmp` (or another data volume) has room, the
+default relocation applies. On the reference host this looked like a 28 GiB `/`
+at 100% next to a 196 GiB `/tmp` with ~26 GiB free.
+
+## Step 2: Confirm where cargo artifacts are being written
+
+With no override set, the default target root is `/tmp/simard-cargo-targets`.
+Verify the daemon is using it:
+
+```bash
+ls -d /tmp/simard-cargo-targets/*/ 2>/dev/null
+```
+
+You should see one directory per active engineer worktree, each containing a
+`debug/` (and, during coverage runs, `llvm-cov-target/`) subtree. If instead
+you find large artifacts under `~/.cargo-targets` or `~/.simard/**/target`, the
+daemon is running an old build — deploy the #4803 fix and restart.
+
+## Step 3: (Optional) point relocation at a specific volume
+
+To use a dedicated data volume instead of `/tmp`, set
+`SIMARD_CARGO_TARGETS_ROOT` before launching the daemon:
+
+```bash
+export SIMARD_CARGO_TARGETS_ROOT=/data/simard-cargo-targets
+```
+
+Rules:
+
+- The override **wins** over the default.
+- An **empty** value is ignored (treated as unset) so it can never resolve to
+  `/<worktree>` and re-saturate `/`.
+- A per-worktree basename is appended automatically; point this at the volume
+  root, not at a single worktree.
+
+To pin an exact directory for a one-off local build instead, set
+`CARGO_TARGET_DIR` directly — it is honoured verbatim by both spawn paths.
+
+## Step 4: Tune the emergency-cleanup backoff (optional)
+
+Emergency cleanup now uses hysteresis (trigger at ≥ 95% used, do not re-fire
+until usage drops below 85%) plus a minimum re-fire interval. Adjust the
+interval with:
+
+```bash
+# Minimum seconds between two emergency cleanups (default 900, clamped 0–86400).
+export SIMARD_DISK_EMERGENCY_MIN_REFIRE_SECS=1800
+```
+
+Leave this at the default unless cleanup still fires too often after
+relocation — post-relocation, `/` is no longer the fill target, so cleanup
+should become rare on its own.
+
+The backoff marker lives under `<state_root>/disk-health/`. If it is deleted or
+corrupted, cleanup **fails open** (runs anyway) and logs a warning — it can
+never be stuck suppressed.
+
+## Step 5: Confirm the dispatch preflight threshold
+
+Build-heavy goal dispatch is gated by the `disk_pressure` classifier, which
+compares `/` free space against a single min-free threshold `T` set by
+`SIMARD_DISK_PRESSURE_MIN_FREE_GB` (default **20 GiB**):
+
+- **`Warn`** — free space below `T` (below **20 GiB** at defaults): dispatch
+  still proceeds, but logs a `warn!`.
+- **`Refuse`** — free space below `T/2` (below **10 GiB** at defaults):
+  dispatch is skipped this cycle.
+
+Note the `Refuse` line is **half** the `min-free` knob, not equal to it. Tune
+the knob if you want a different floor:
+
+```bash
+# T = 20 GiB → Warn below 20 GiB, Refuse below 10 GiB.
+export SIMARD_DISK_PRESSURE_MIN_FREE_GB=20
+```
+
+When the preflight refuses, the daemon logs a `warn!` and **retries the goal
+next cycle** — it does not block or fail the goal.
+
+## Step 6: Verify the crash-loop is gone
+
+After deploying and restarting the daemon:
+
+```bash
+# Emergency cleanup should NOT re-fire every ~25 minutes anymore.
+grep -i "emergency" ~/.simard/ooda.log | tail -10
+
+# Root volume should hold steady well below 95%.
+watch -n 60 'df -h /'
+
+# Preflight refusals (if any) are visible and benign:
+grep -i "disk pressure" ~/.simard/ooda.log | tail -10
+```
+
+Success looks like: `/` stays below the high watermark, emergency cleanup
+entries become sparse instead of appearing every cycle, and the
+`database is locked` / `cognitive-open-lock` / `memory-ipc` errors clear.
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---------|-------------|-----|
+| Artifacts still under `~/.cargo-targets` | Old binary before the #4803 fix | Redeploy and restart the daemon. |
+| `SIMARD_CARGO_TARGETS_ROOT` set but ignored | Value is empty | Set a non-empty absolute path (empty is intentionally ignored). |
+| Cleanup still thrashes | `/tmp` itself is small / is `/` | Point `SIMARD_CARGO_TARGETS_ROOT` at a genuinely large volume. |
+| Coverage jobs can't find artifacts | Tooling hardcodes `./target` | Point llvm-cov at the relocated `CARGO_TARGET_DIR` (env-consistent). |
+| Goals never dispatch | `/` chronically below the refuse line | Reclaim space (see [reclaim disk](./reclaim-disk-space-and-run-low-space-rust-builds.md)); the preflight is protecting you from ENOSPC. |
+
+## See also
+
+- Concept: [Root-disk saturation relocation and thrash guard](../concepts/root-disk-saturation-relocation.md)
+- Reference: [Build-artifact relocation and disk-thrash guard](../reference/build-artifact-relocation-and-disk-thrash-guard.md)
+- [Configure and monitor the disk health check](./configure-disk-health-check.md)