Skip to content

fix(cron): grace window must defer a fire, not drop it - #20

Merged
nyem69 merged 1 commit into
mainfrom
fix/cron-catchup-grace-window-silent-drop
Jul 14, 2026
Merged

fix(cron): grace window must defer a fire, not drop it#20
nyem69 merged 1 commit into
mainfrom
fix/cron-catchup-grace-window-silent-drop

Conversation

@nyem69

@nyem69 nyem69 commented Jul 14, 2026

Copy link
Copy Markdown
Owner

The bug

The catch-up sweep skipped any fire inside the 90 s grace window on the reasoning that "node-cron has not yet had its chance." On a post-sleep wake that reasoning is backwards: node-cron's minute-tick was suspended through that minute and never replays it — which is the entire reason this module exists.

The sweep then wrote writeCheckpoint(now) unconditionally in its finally, pushing lastCheck past the skipped fire. The next sweep rejected it via prevFire <= lastCheck. The fire was gone.

Net: a 90-second-wide blind spot immediately before every wake sweep. Recurring jobs hide it (they just fire again next interval); a date-pinned one-shot loses its fire until the schedule next comes round.

Observed in production

manamurah-ayam-hub-verdict-ONESHOT (30 9 14 7 *, Asia/Kuala_Lumpur) was due 2026-07-14T01:30:00Z. The host woke and swept at 01:30:59Z — 59 s < 90 s grace — so it was skipped, then buried. Jobs on either side of it replayed normally:

01:00  manamurah-kv-snapshot      replayed
01:22  collect-tiktok-selangor    replayed
01:30  ayam-hub-verdict           <-- silently dropped
01:40  plan-url-queue-processor   replayed

Its next natural fire would have been July 2027. Found because the one-shot was a deliberate re-check that never happened.

The fix

computeMissedFires now returns deferred: DeferredFire[] alongside replay/tooOld, and checkpointFloor(now, deferred) clamps the persisted checkpoint to min(now, earliestDeferred - 1).

The fire is reconsidered on the next sweep, by which point the grace has passed and the run-log dedup makes the real decision:

  • node-cron fired it on time → dedup vetoes → no replay. This is the guard that fixed the 2026-06-07 sitrep double-fire, and it is unchanged.
  • the host slept through it → no run-log entry → replayed once.

The grace window keeps its anti-race role but can no longer destroy a fire. Every skip path in this module is now replayed, deferred, or warned — none silent.

Tests

catchup.test.ts drives two consecutive sweeps and pins both failure modes:

  • holds the checkpoint behind the deferred fire instead of advancing to now
  • replays the fire on the next sweep when the host slept through it ← the drop
  • does NOT replay when node-cron fired it on time (no double-fire) ← the June regression
  • checkpointFloor — earliest-wins, never advances past now

Verified against the real production values (actual schedule + the actual 01:30:59Z wake from gateway.log): the old checkpoint buries the fire on both sweeps; the new one replays it. 798/798 tests pass, tsc --noEmit clean. Built and running on the live gateway (boot 54cfa5d3).

🤖 Generated with Claude Code

https://claude.ai/code/session_01B849gvnP1gpazo2U4R9hPN

The catch-up sweep skipped any fire inside the 90s grace window on the
reasoning that "node-cron has not yet had its chance". On a post-sleep wake
that is backwards: node-cron's minute-tick was suspended through that minute
and never replays it, which is the whole reason catch-up exists. The sweep
then wrote writeCheckpoint(now) unconditionally, pushing lastCheck past the
skipped fire so the next sweep rejected it via `prevFire <= lastCheck`.

Net: a 90s-wide blind spot immediately before every wake sweep. Recurring
jobs hide it (they fire again next interval); a date-pinned one-shot loses
its fire until the schedule next comes round.

Observed: manamurah-ayam-hub-verdict-ONESHOT (`30 9 14 7 *`, MYT) was due
2026-07-14T01:30:00Z. The host swept at 01:30:59Z -- 59s < 90s grace -- so it
was skipped, then buried. Jobs at 01:22 and 01:40 replayed normally. Its next
natural fire would have been July 2027.

computeMissedFires now reports `deferred` alongside replay/tooOld, and
checkpointFloor() clamps the persisted checkpoint to just before the earliest
deferred fire. The fire is reconsidered next sweep, where the run-log dedup
makes the real call: fired on time -> vetoed (this is what guards the
2026-06-07 sitrep double-fire, and is unchanged); slept through -> replayed
once. Every skip path is now replayed, deferred, or warned -- none silent.

Tests drive two consecutive sweeps and pin both failure modes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B849gvnP1gpazo2U4R9hPN
@nyem69
nyem69 merged commit 642f677 into main Jul 14, 2026
2 checks passed
@nyem69
nyem69 deleted the fix/cron-catchup-grace-window-silent-drop branch July 14, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant