Skip to content

fix(scheduler): persist exact-fire acceptance receipts - #1004

Merged
olegbrok merged 1 commit into
mainfrom
agent/scheduler-receipt-ledger
Aug 5, 2026
Merged

fix(scheduler): persist exact-fire acceptance receipts#1004
olegbrok merged 1 commit into
mainfrom
agent/scheduler-receipt-ledger

Conversation

@bradbrok

@bradbrok bradbrok commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Summary

  • atomically claim each non-direct scheduler fire and create its durable exact-fire ledger/outbox row in the same SQLite transaction
  • carry (schedule_id, fired_at) through the API into tmux and Codex, committing accepted_at at the real transport acceptance edge before resolving the process-local Future
  • retain accepted and quarantined rows as forensic outcomes while active replay queries exclude them
  • let durable acceptance win teardown races, preventing both false FIRED BUT UNDELIVERED verdicts and replay of work that already entered the agent
  • expose GET /scheduler/wake-ledger with agent/state/time filters for direct fleet-health queries
  • classify attempt-cap and zombie outcomes as queryable quarantines; soften the cold-boot readiness timeout message so it remains a timing observation rather than a false hook-failure verdict

Audit finding

The positive receipt was previously only an in-memory Future. tmux/Codex resolved it at transport acceptance, but AgentScheduler wrote durable state later. Teardown or process death in that interval could resolve/log negative state or leave an outbox row that replayed work which had already run. Conversely, successful rows were deleted, so last_delivered and log strings could not prove the outcome of an exact fire.

Evidence: 08-04 replay incident, TOD false-loss incident.

Misfire policy decision

Fire age is not treated as confirmation. Retiring anything older than its cron period would silently lose imperative one-shots and misclassify late success. This repair is confirmation-state-first. Age can become a guardrail only when a schedule has an explicit V3-style SKIP, FIRE_ONCE, or QUARANTINE policy (V3 policy map).

Crash-seam drill

The regression commits accepted_at while the Future is deliberately still unresolved, kills the delivery group in that exact post-accept/pre-scheduler-confirm window, reopens the database, and starts a new scheduler. The accepted fire is never replayed or logged lost. A second unconfirmed fire is replayed to its cap and ends quarantined. The final ledger contains exactly one receipted-ran-once and one quarantined terminal record, with no active outbox rows.

Validation

  • uv run ruff check .
  • 129 passed — scheduler/registry, including schema migration and crash-seam drill
  • 382 passed — tmux transport
  • 89 passed — Codex transport
  • 15 passed — focused API/ledger/owner-alert slice
  • git diff --check

A broader local run reached 773 passing tests before the pre-existing manual-dream test escaped its SDK mock into a real tmux dream because the shared MCP port was already occupied; that environmental failure is unrelated to this patch.

Fixes #991

Comment thread src/pinky_daemon/api.py
scheduler_kwargs["on_accept"] = (
schedule_receipt.accept
)
except (TypeError, ValueError):

@olegbrok olegbrok left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — four-leg review against the audit's acceptance criteria:

  1. The crash gap is genuinely closed. claim_schedule_fire does CAS + exact-fire row in one transaction under the RMW lock (UNIQUE(schedule_id, fired_at) backs the idempotent insert), and the drill freezes at the true seam — durable receipted-ran-once asserted while the process-local Future is still unresolved, then kill, reopen, zero replay. That is the #991 criterion verbatim, not an approximation.
  2. The hard edge case is decided correctly. Persist-failure after transport acceptance resolves the Future True with a loud named log (SCHEDULER_RECEIPT_PERSIST_FAILURE) — at-most-once for accepted work is right, replay there would duplicate into the pane; and teardown negative accounting defers to durable receipts ("durable receipt won teardown race").
  3. Scope choices are explicit and sound: direct_send keeps the plain CAS lane (its delivery evidence is the platform send, not a wake receipt); fire-age is rejected as truth per the audit; gate-timeout wording is timing-only. /scheduler/wake-ledger sits behind the global auth middleware like its siblings.
  4. Fleet impact: the ledger query lets the 2am health check replace both log-string greps with state inspection — I'll rewrite that section of the check after this ships in a release.

The broad-suite local caveat (dream mock/MCP port collision) is env-shaped and unrelated; CI arbitrates.

🤖 Reviewed by Barsik

@olegbrok
olegbrok marked this pull request as ready for review August 5, 2026 13:09
@olegbrok
olegbrok merged commit 4a68f23 into main Aug 5, 2026
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants