fix(overseer): watchdog reclaims hung tick guard to prevent missed ticks (Problem #3)

[rysweet]

Observed Problem #3 — Overseer missing scheduled ticks

Cadence is every 15 min, but simard status showed the last acting-overseer
tick at 2026-07-28T19:23:39Z (next expected 19:38:39Z) with the overseer
marked (stale), and the tick history showed large gaps:
17:57:55Z -> 19:23:39Z (~86 min) and 19:23:39Z -> ~20:46Z with no
intervening tick.

Root cause

The acting-overseer tick uses a bare AtomicBool overlap guard
(overseer_tick_running) that is cleared by a spawned thread's ClearOnDrop.
If a tick hangs on a long gh/network call, ClearOnDrop never runs, so
the guard stays true forever and every subsequent scheduled tick is
dropped → staleness + multi-tick gaps. There was no watchdog to reclaim the
stuck guard.

Fix (additive / non-breaking)

• Add a pure, clock-injected TickWatchdog + guard_generation_matches
  helper next to OverseerCadence in src/overseer/wiring.rs. The watchdog
  bounds max in-flight time (three cadence intervals, floored at 10 min) and
  is fully unit-tested.
• Wire it into the daemon: record arm time + generation when a tick arms;
  before the cadence check, if a tick has been in flight past the bound,
  reclaim the guard, bump the generation, and emit tracing::warn! + a
  daemon_log staleness/liveness signal so a catch-up tick can fire.
• Generation-token guard on ClearOnDrop: a late-finishing hung tick only
  clears the guard if its captured generation still matches current, avoiding
  the stale-clear race against a catch-up tick.

Tests

6 new unit tests in overseer::wiring::tests:

• watchdog_reclaims_a_tick_that_exceeds_its_budget
• watchdog_does_not_reclaim_a_tick_within_its_budget
• watchdog_budget_is_floored_to_avoid_same_second_respawn
• watchdog_inflight_is_monotonic_safe
• generation_guard_prevents_the_stale_clear_race
• a_hung_tick_does_not_block_subsequent_ticks (end-to-end virtual-clock model)

All 50 overseer::wiring::tests pass. Structured tracing + OTel only (no
print!/println!). Preserves PRD and OODA cadence semantics.

Step 13: Local Testing Results

Detected toolchains:

• Rust 2024/Cargo (Cargo.toml, Cargo.lock): all seven changed files are Rust daemon/Overseer sources, and the affected runtime entry point is simard ooda run.
• Node/npm (package.json, package-lock.json, bin.js): provides the npm launcher and Playwright dashboard scripts, but neither the launcher nor dashboard changed, so this toolchain is not part of the focused validation surface.

Chosen validation strategy:

• Run locked native Rust tests against the actual watchdog and owned worker lifecycle. This is the smallest hermetic strategy that safely proves lease expiry and the externally observable daemon guarantee: a hung tick cannot publish stale state, overlap its replacement, or suppress a later tick. NODE_OPTIONS=--max-old-space-size=32768 was set on both commands as required.

Scenario 1 — Watchdog expires the observed hung-tick duration

Command: NODE_OPTIONS=--max-old-space-size=32768 cargo test --locked --lib 'overseer::wiring::tests::watchdog_expires_a_tick_that_exceeds_its_budget' -- --exact --nocapture
Result: PASS
Output: test overseer::wiring::tests::watchdog_expires_a_tick_that_exceeds_its_budget ... ok; test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 9946 filtered out

Scenario 2 — Hung worker is fenced, reaped, and followed by a clean replacement

Command: NODE_OPTIONS=--max-old-space-size=32768 cargo test --locked --lib 'overseer::wiring::tests::expired_worker_exits_before_replacement_and_cannot_publish' -- --exact --nocapture
Result: PASS
Output: test overseer::wiring::tests::expired_worker_exits_before_replacement_and_cannot_publish ... ok; test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 9946 filtered out

[github-actions]

📊 Coverage Summary

Generated by cargo llvm-cov --workspace --summary-only (nightly, excluding test files)

Module	Lines	Covered	Coverage
Total	213480	180350	84.5%

_{Coverage data from CI run. Test files matching tests?/ are excluded from line counts.}