Skip to content

Purge stale actor_sessions at OODA daemon startup (self-heal cross-host/observe_only leases) - #5009

Merged
rysweet merged 1 commit into
mainfrom
feat/issue-5005-harden-the-typed-ooda-daemon-so-stale-actorsession
Jul 30, 2026
Merged

Purge stale actor_sessions at OODA daemon startup (self-heal cross-host/observe_only leases)#5009
rysweet merged 1 commit into
mainfrom
feat/issue-5005-harden-the-typed-ooda-daemon-so-stale-actorsession

Conversation

@rysweet

@rysweet rysweet commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Problem

On the deva3 host migration, every OODA goal cycle failed with
actor session is already bound to a different identity or authorization scope
(AuthorizationScopeViolation, src/typed_ooda/ledger.rs).

Root cause: the migrated ledger (typed-ooda/outcomes.sqlite3) carried 3,976 stale
actor_sessions rows
from the previous host, all created in observe_only=0 mode and
future-dated (so the existing DELETE ... WHERE expires_at < now sweep never cleared them).
The new host runs observe_only=1, and scope_key includes observe_only — so every reused
per-goal session_id collided with a stale row and was rejected forever. This recurs on any
restart / migration / observe_only change; it is a latent bug, not a one-off.

Fix

Purge the transient actor_sessions table once at authoritative daemon startup, before any
goal cycle runs. At boot no actor session can still be in flight, so the leases are safe to
clear; runtime scope enforcement is completely unchanged.

  • Ledger::purge_actor_sessions()DELETE FROM actor_sessions (ledger.rs)
  • purge_actor_sessions_on_startup() wired into the daemon init path, immediately after
    create_dir_all(&state_root) and before the draining-flag clear / freshness gate (daemon/mod.rs)
  • Test startup_purge_removes_stale_actor_session_but_preserves_live_scope_guard proves a
    stale future-dated lease is removed and that a genuine in-process cross-scope collision
    is still rejected (AuthorizationScopeViolation) afterwards.
  • Reference doc docs/reference/actor-session-startup-purge.md.

Closes #5005


Merge-ready evidence

Crusty-old-engineer review: PASS. Fix rests on the invariant "no actor_session is
meaningful across a restart"; verified orphaned engineer subprocesses are tracked separately
in engineer_claims (fail-closed liveness gate), so purging authorization leases cannot cause
a double-actor. Fresh-install safe: CapabilityHandler::open() runs schema::initialize()
eagerly and schema.rs:73 creates actor_sessions with CREATE TABLE IF NOT EXISTS, so the
DELETE can't fail on first boot. Fail-closed ? posture is correct for a daemon that must not
run on a broken ledger.

QA-team: PASS (Rust-native harness). This is a Rust CLI repo (Cargo.toml at root), so
cargo test is the QA harness per the qa-team skill's repo-type detection. Ran on the PR branch:

  • startup_purge_removes_stale_actor_session_but_preserves_live_scope_guard — pass
  • full actor_session group (7 tests) and typed_ooda::ledger group (19 tests) — all pass,
    including changed_observe_only_on_reused_session_is_rejected (proves the scope guard that was
    the root of the migration bug still rejects live scope changes).

Quality-audit: PASS (3 cycles, final cycle clean).

  • Cycle 1 (stubs / error-swallowing / shell+async anti-patterns on the diff): 0 findings —
    no todo!/unimplemented!/bail!/panic!/TODO markers; production paths use ? + map_err(persistence) (fail-visible); the 8 expect() are all test-only.
  • Cycle 2 (docs / structure / error visibility): 0 findings — new pub(crate) fn is documented,
    both new functions are small (~5 and ~13 lines), errors surfaced not swallowed.
  • Cycle 3 (boot-ordering correctness + mandatory final zero-stubs scan across all 7 changed files): 0 findings — purge ordered before any goal cycle; zero stubs across all changed files.
  • No new high/critical/medium findings emerged → stopped at the 3-cycle minimum on a clean cycle.

Checks/build: all required CI green for head; mergeable=MERGEABLE, mergeStateStatus=CLEAN.
Docs: updated — new actor-session-startup-purge.md plus related reference/how-to pages.
Reviews/policies: main is not branch-protected; no required approvals outstanding; no unresolved threads.
Linked issue: Closes #5005.
Scope: diff limited to src/typed_ooda/ledger.rs, src/operator_commands_ooda/daemon/mod.rs, and docs — no unrelated changes.

Verdict: MERGE_READY.

Clear transient actor-session leases before daemon goal-cycle work while preserving durable typed-OODA ledger data and live-session scope enforcement. Keep the purge crate-internal and document its lifecycle boundary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

📊 Coverage Summary

Generated by cargo llvm-cov --workspace --summary-only (nightly, excluding test files)

Module Lines Covered Coverage
Total 214406 181169 84.5%

Coverage data from CI run. Test files matching tests?/ are excluded from line counts.

@rysweet

rysweet commented Jul 30, 2026

Copy link
Copy Markdown
Owner Author

Crusty review + merge-ready evidence

Crusty review: PASS.

  • Root-cause fix (self-healing transient leases): clears the actor_sessions table once at authoritative daemon startup, when no actor session can be in flight; runtime scope enforcement is unchanged.
  • Fresh-install safe — verified CapabilityHandler::open() calls schema::initialize() eagerly, and schema.rs:73 creates actor_sessions with CREATE TABLE IF NOT EXISTS. So DELETE FROM actor_sessions runs against an existing (empty) table on a brand-new install; it cannot fail with "no such table" and cannot brick first boot.
  • Fail-closed posture is correct: the startup purge uses ?, so a purge failure aborts boot rather than starting on a broken ledger. For a daemon that must not run on corrupt lease state, failing loudly is the right call.
  • Directly tested: startup_purge_removes_stale_actor_session_but_preserves_live_scope_guard proves a stale future-dated lease is removed and that a genuine cross-identity/scope collision is still rejected (AuthorizationScopeViolation) afterwards.
  • Naming/visibility/comments clean; pub(crate) scoping appropriate.

Merge-ready criteria:

  • Checks/build: all CI green for head revision; mergeable=MERGEABLE, mergeStateStatus=CLEAN.
  • Docs: updated — new docs/reference/actor-session-startup-purge.md plus related reference/how-to pages.
  • QA/behavior coverage: change has no external/CLI/API/UI surface (internal daemon-startup step); behavior is covered by the dedicated Rust unit test above. Produced by a complete default-workflow run (design → TDD → refactor-review → precommit-test, all green).
  • Metadata/scope: PR describes problem/fix/validation; Closes #5005; diff is tightly scoped (ledger.rs, daemon/mod.rs, docs) with no unrelated changes.
  • No merge conflicts; no outstanding review threads; no required-review block.

Verdict: merge-ready.

@rysweet
rysweet merged commit 60411c0 into main Jul 30, 2026
18 checks passed
@rysweet
rysweet deleted the feat/issue-5005-harden-the-typed-ooda-daemon-so-stale-actorsession branch July 30, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant