fix(events): key legacy-Timeline seeding dedup instead of a newest-200 window (handoff to @Jason) - #161
Open
stonexer wants to merge 1 commit into
Open
fix(events): key legacy-Timeline seeding dedup instead of a newest-200 window (handoff to @Jason)#161stonexer wants to merge 1 commit into
stonexer wants to merge 1 commit into
Conversation
Seeded Timeline rows carry the entry's HISTORICAL `at`, so they are the
OLDEST rows in a loop's event stream. Deduping against
`listEvents(loopId, {limit: 200})` therefore stopped seeing them once a
loop accumulated 200 newer events, after which every task-file sync
re-inserted the entire Timeline as fresh `note` rows - unbounded growth
(reproduced live: 4 events -> 230 after one re-sync past the window).
`scripts/migrate-v2-split.ts --execute` carried the identical defect,
which falsified its "re-run is a no-op" idempotency claim.
- new `gateway/timelineSeed.ts`: the ONE seeder both the live ingest and
the migration script import, so they cannot dedup differently.
- dedup is keyed and unbounded: `store.seededTimelineKeys` selects
DISTINCT (day, text) over exactly the seeder's own rows (type=note
carrying data.source='timeline'). DISTINCT bounds the result to the
number of distinct entries, not the row count.
- the insert additionally uses a deterministic primary key derived from
the same identity + ON CONFLICT DO NOTHING (`store.addEventIfAbsent`),
so concurrent syncs and half-finished retries converge on one row.
- MESSAGE_CAP moves to the leaf `gateway/http.ts` (re-exported from
index.ts) so the seeder shares one clip budget without a cycle.
- migrate-v2-split: `executeSeed`/`main` exported and the auto-run
guarded by a direct-invocation check, so the pass is testable.
- migrate-v2-split rehearsal: the loss detector's separator class omitted
`|`, the separator the daemon's own appendTimeline writes, so it
reported a false LOSS (exit 1) on every daemon-authored row. Aligned
with docSplit's TIMELINE_LINE.
Regression tests seed a loop past 200 events and assert zero new rows on
re-sync, on both the gateway ingest and the migration path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
ingestTaskFileContent(gateway/index.ts) seeded legacy## Timelineentries into the event stream, deduping againststore.listEvents(loopId, {limit: 200})— the newest 200 events.Seeded rows carry the Timeline entry's historical
at(the line's date), so they are the oldest rows in the stream. As soon as a loop accumulates 200 newer events, the seeded rows are no longer in that window — and from then on every task-file sync re-inserts the entire Timeline as freshnoterows. Unbounded growth; the web/CLI timeline fills with duplicates.A daily loop emits
run-started+run-returnedper run, so it crosses 200 events in roughly 100 runs (~3 months). Real loops carry 10–30 Timeline lines.Reproduced live against a dev server on this branch (loop
old-spike-closed, a 2-line Timeline):scripts/migrate-v2-split.ts(~line 80) had the identical defect, which falsified its ownre-run is a no-opdocstring and the matching AGENTS.md claim. The plan's rehearsal number ("416 events / 80 rows, second run 0") was only clean because those loops had <200 events at the time.The fix
New
packages/server/src/gateway/timelineSeed.ts— the one seeder both write surfaces import (the live ingest andmigrate-v2-split --execute), same anti-drift discipline asgateway/validate.ts. The two can no longer dedup differently.Dedup is keyed and unbounded, enforced twice:
store.seededTimelineKeys(loopId)—SELECT DISTINCT substr(at,1,10), text FROM events WHERE loop_id = ? AND type = 'note' AND data->>'source' = 'timeline'. It asks for exactly the rows this seeder wrote, so nothing can age out.DISTINCTbounds the result to the number of distinct entries (itself bounded by the doc'sWIRE_TEXT_CAP), not to the row count — so even an already-duplicated prod stream collapses to one key per entry rather than loading every duplicate.ev-tl-<sha256(loopId + day + clipped text)>) inserted viastore.addEventIfAbsent(ON CONFLICT DO NOTHING).Why both. (1) alone is correct but is a read-then-write, so two concurrent syncs — or the migration racing a live watcher flush — can both see an empty key set and both insert. (2) alone is race-free and survives a pass that dies half-way and is retried, but it does not dedup rows seeded before this fix, which carry random ids. Together they cover both: the query handles the existing prod rows, the deterministic id makes every future write idempotent regardless of what the read saw.
Why not the high-water-mark alternative. A per-loop "last seeded Timeline date" is simpler to write but strictly weaker here: Timeline lines are not append-ordered by date (a legacy README is authored newest-first, and agents backfill), it needs a new column, and it does not survive partial seeding — a pass that writes 8 of 12 entries and dies leaves the mark either too far forward (4 entries lost permanently) or unmoved (nothing recovered on the retry). Content-keyed identity has no such failure mode.
Also in this PR:
MESSAGE_CAPmoved fromgateway/index.tsinto the leafgateway/http.ts(re-exported fromindex.ts, socli.ts's existing import is untouched) — the seeder needs the same clip budget and must not import the run-lifecycle core it is imported by.layout.test.tsstays green.migrate-v2-split.ts:executeSeed/mainexported, and the auto-run wrapped in a direct-invocation guard so the pass is importable by a test without connecting to a DB or callingprocess.exit. Verified the script still runs normally astsx scripts/migrate-v2-split.ts [--execute].lostLinesseparator class omitted|— the separator the daemon's ownappendTimelinewrites (- 2026-07-01 | text). Every daemon-authored Timeline line therefore kept a|glued to its tail, never matched the splitter's event text, and was reported as a LOSS, which exits 1. That made the rehearsal read as "unsafe to migrate" on essentially every legacy loop. Aligned withdocSplit'sTIMELINE_LINE, which already includes|with a comment explaining exactly this. Happy to split it out if you would rather keep it separate.Not touched: the seeding call site stays on the sync path (a task-file snapshot landed) and never runs on the idle poll hot path — the documented budget is unaffected. Nothing about the doc plane changed; seeding remains additive-only.
Test evidence
New
src/gateway/timelineSeed.test.ts(7 cases) encodes the actual failure rather than a small-stream approximation:source: timeline) still dedup — pins the keyed query specifically, not just the deterministic id;ingestTaskFileContentcalls converge on one row per entry — pins theON CONFLICThalf;executeSeed()past 200 events, re-run twice → zero new rows;docSplitaccepts.I confirmed these fail against the old logic — temporarily restoring the newest-200 window + plain insert turns 5 of the 7 red (the two that stay green are the pure-helper and clipping cases, which do not depend on the window).
End-to-end against a scratch pglite DB with a padded loop:
Suites:
788 = your 781 plus the 7 new cases; no existing test changed.
One thing to do before you trust the migration
Re-run the rehearsal against current prod before executing. The original "second run: 0" figure was measured on a mirror whose loops were all under 200 events, so it did not exercise the defect. If any prod loop already crossed the window on a deployed build of this branch, its stream will hold duplicate seeded rows today; this fix stops the growth but does not retro-clean them. Say the word and I will write the one-time dedupe (the deterministic id makes the target set trivially identifiable) as a follow-up.
packages/server/AGENTS.mdis updated with the invariant, so the next agent does not reintroduce a windowed dedup.