fix(scheduler): persist exact-fire acceptance receipts - #1004
Merged
Conversation
| scheduler_kwargs["on_accept"] = ( | ||
| schedule_receipt.accept | ||
| ) | ||
| except (TypeError, ValueError): |
olegbrok
approved these changes
Aug 5, 2026
olegbrok
left a comment
Collaborator
There was a problem hiding this comment.
APPROVE — four-leg review against the audit's acceptance criteria:
- The crash gap is genuinely closed.
claim_schedule_firedoes CAS + exact-fire row in one transaction under the RMW lock (UNIQUE(schedule_id, fired_at) backs the idempotent insert), and the drill freezes at the true seam — durablereceipted-ran-onceasserted while the process-local Future is still unresolved, then kill, reopen, zero replay. That is the #991 criterion verbatim, not an approximation. - The hard edge case is decided correctly. Persist-failure after transport acceptance resolves the Future True with a loud named log (
SCHEDULER_RECEIPT_PERSIST_FAILURE) — at-most-once for accepted work is right, replay there would duplicate into the pane; and teardown negative accounting defers to durable receipts ("durable receipt won teardown race"). - Scope choices are explicit and sound: direct_send keeps the plain CAS lane (its delivery evidence is the platform send, not a wake receipt); fire-age is rejected as truth per the audit; gate-timeout wording is timing-only.
/scheduler/wake-ledgersits behind the global auth middleware like its siblings. - Fleet impact: the ledger query lets the 2am health check replace both log-string greps with state inspection — I'll rewrite that section of the check after this ships in a release.
The broad-suite local caveat (dream mock/MCP port collision) is env-shaped and unrelated; CI arbitrates.
🤖 Reviewed by Barsik
olegbrok
marked this pull request as ready for review
August 5, 2026 13:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
(schedule_id, fired_at)through the API into tmux and Codex, committingaccepted_atat the real transport acceptance edge before resolving the process-local FutureFIRED BUT UNDELIVEREDverdicts and replay of work that already entered the agentGET /scheduler/wake-ledgerwith agent/state/time filters for direct fleet-health queriesAudit finding
The positive receipt was previously only an in-memory Future. tmux/Codex resolved it at transport acceptance, but
AgentSchedulerwrote durable state later. Teardown or process death in that interval could resolve/log negative state or leave an outbox row that replayed work which had already run. Conversely, successful rows were deleted, solast_deliveredand log strings could not prove the outcome of an exact fire.Evidence: 08-04 replay incident, TOD false-loss incident.
Misfire policy decision
Fire age is not treated as confirmation. Retiring anything older than its cron period would silently lose imperative one-shots and misclassify late success. This repair is confirmation-state-first. Age can become a guardrail only when a schedule has an explicit V3-style
SKIP,FIRE_ONCE, orQUARANTINEpolicy (V3 policy map).Crash-seam drill
The regression commits
accepted_atwhile the Future is deliberately still unresolved, kills the delivery group in that exact post-accept/pre-scheduler-confirm window, reopens the database, and starts a new scheduler. The accepted fire is never replayed or logged lost. A second unconfirmed fire is replayed to its cap and ends quarantined. The final ledger contains exactly onereceipted-ran-onceand onequarantinedterminal record, with no active outbox rows.Validation
uv run ruff check .129 passed— scheduler/registry, including schema migration and crash-seam drill382 passed— tmux transport89 passed— Codex transport15 passed— focused API/ledger/owner-alert slicegit diff --checkA broader local run reached 773 passing tests before the pre-existing manual-dream test escaped its SDK mock into a real tmux dream because the shared MCP port was already occupied; that environmental failure is unrelated to this patch.
Fixes #991