Skip to content

fix(events): key legacy-Timeline seeding dedup instead of a newest-200 window (handoff to @Jason) - #161

Open
stonexer wants to merge 1 commit into
feat/task-tree-v2from
fm/event-dedup-fix-b2
Open

fix(events): key legacy-Timeline seeding dedup instead of a newest-200 window (handoff to @Jason)#161
stonexer wants to merge 1 commit into
feat/task-tree-v2from
fm/event-dedup-fix-b2

Conversation

@stonexer

@stonexer stonexer commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

The bug

ingestTaskFileContent (gateway/index.ts) seeded legacy ## Timeline entries into the event stream, deduping against store.listEvents(loopId, {limit: 200}) — the newest 200 events.

Seeded rows carry the Timeline entry's historical at (the line's date), so they are the oldest rows in the stream. As soon as a loop accumulates 200 newer events, the seeded rows are no longer in that window — and from then on every task-file sync re-inserts the entire Timeline as fresh note rows. Unbounded growth; the web/CLI timeline fills with duplicates.

A daily loop emits run-started + run-returned per run, so it crosses 200 events in roughly 100 runs (~3 months). Real loops carry 10–30 Timeline lines.

Reproduced live against a dev server on this branch (loop old-spike-closed, a 2-line Timeline):

after first sync           :   4 events   (2 seeded)
pad to 228 events, re-sync : 230 events   ← the 2 legacy entries seeded AGAIN

scripts/migrate-v2-split.ts (~line 80) had the identical defect, which falsified its own re-run is a no-op docstring and the matching AGENTS.md claim. The plan's rehearsal number ("416 events / 80 rows, second run 0") was only clean because those loops had <200 events at the time.

The fix

New packages/server/src/gateway/timelineSeed.ts — the one seeder both write surfaces import (the live ingest and migrate-v2-split --execute), same anti-drift discipline as gateway/validate.ts. The two can no longer dedup differently.

Dedup is keyed and unbounded, enforced twice:

  1. store.seededTimelineKeys(loopId)SELECT DISTINCT substr(at,1,10), text FROM events WHERE loop_id = ? AND type = 'note' AND data->>'source' = 'timeline'. It asks for exactly the rows this seeder wrote, so nothing can age out. DISTINCT bounds the result to the number of distinct entries (itself bounded by the doc's WIRE_TEXT_CAP), not to the row count — so even an already-duplicated prod stream collapses to one key per entry rather than loading every duplicate.
  2. A deterministic primary key (ev-tl-<sha256(loopId + day + clipped text)>) inserted via store.addEventIfAbsent (ON CONFLICT DO NOTHING).

Why both. (1) alone is correct but is a read-then-write, so two concurrent syncs — or the migration racing a live watcher flush — can both see an empty key set and both insert. (2) alone is race-free and survives a pass that dies half-way and is retried, but it does not dedup rows seeded before this fix, which carry random ids. Together they cover both: the query handles the existing prod rows, the deterministic id makes every future write idempotent regardless of what the read saw.

Why not the high-water-mark alternative. A per-loop "last seeded Timeline date" is simpler to write but strictly weaker here: Timeline lines are not append-ordered by date (a legacy README is authored newest-first, and agents backfill), it needs a new column, and it does not survive partial seeding — a pass that writes 8 of 12 entries and dies leaves the mark either too far forward (4 entries lost permanently) or unmoved (nothing recovered on the retry). Content-keyed identity has no such failure mode.

Also in this PR:

  • MESSAGE_CAP moved from gateway/index.ts into the leaf gateway/http.ts (re-exported from index.ts, so cli.ts's existing import is untouched) — the seeder needs the same clip budget and must not import the run-lifecycle core it is imported by. layout.test.ts stays green.
  • migrate-v2-split.ts: executeSeed/main exported, and the auto-run wrapped in a direct-invocation guard so the pass is importable by a test without connecting to a DB or calling process.exit. Verified the script still runs normally as tsx scripts/migrate-v2-split.ts [--execute].
  • Adjacent fix, flagging it explicitly since it was not in the original scope: the rehearsal's lostLines separator class omitted | — the separator the daemon's own appendTimeline writes (- 2026-07-01 | text). Every daemon-authored Timeline line therefore kept a | glued to its tail, never matched the splitter's event text, and was reported as a LOSS, which exits 1. That made the rehearsal read as "unsafe to migrate" on essentially every legacy loop. Aligned with docSplit's TIMELINE_LINE, which already includes | with a comment explaining exactly this. Happy to split it out if you would rather keep it separate.

Not touched: the seeding call site stays on the sync path (a task-file snapshot landed) and never runs on the idle poll hot path — the documented budget is unaffected. Nothing about the doc plane changed; seeding remains additive-only.

Test evidence

New src/gateway/timelineSeed.test.ts (7 cases) encodes the actual failure rather than a small-stream approximation:

  • the repro — seed a 2-line Timeline, bulk-insert 226 unrelated newer events (228 total), re-sync the same doc → still 228 (was 230), and flat across further syncs;
  • a new Timeline entry still seeds after the loop is past the window (proves the mechanism still works, not just that it stopped writing);
  • rows seeded before this fix (random ids, source: timeline) still dedup — pins the keyed query specifically, not just the deterministic id;
  • two concurrent ingestTaskFileContent calls converge on one row per entry — pins the ON CONFLICT half;
  • over-cap text keys on the clipped body (the pre-existing rule, kept);
  • the migration path: executeSeed() past 200 events, re-run twice → zero new rows;
  • the rehearsal loss check accepts every separator docSplit accepts.

I confirmed these fail against the old logic — temporarily restoring the newest-200 window + plain insert turns 5 of the 7 red (the two that stay green are the pure-helper and clipping cases, which do not depend on the window).

End-to-end against a scratch pglite DB with a padded loop:

--- rehearsal ---
old-spike  119B → doc 10B · 2 events
rows: 1 · losses: 0 · drain-gate flagged: 0        (was: losses: 1, exit 1)
exit=0

--- execute x3 ---
seeded 2 events across 1 rows (re-run is a no-op)
seeded 0 events across 1 rows (re-run is a no-op)
seeded 0 events across 1 rows (re-run is a no-op)

Suites:

pnpm --filter @loopany/server test  → 76 passed | 1 skipped (77 files) · 788 passed | 3 skipped
pnpm -r typecheck                   → Done (both packages)

788 = your 781 plus the 7 new cases; no existing test changed.

One thing to do before you trust the migration

Re-run the rehearsal against current prod before executing. The original "second run: 0" figure was measured on a mirror whose loops were all under 200 events, so it did not exercise the defect. If any prod loop already crossed the window on a deployed build of this branch, its stream will hold duplicate seeded rows today; this fix stops the growth but does not retro-clean them. Say the word and I will write the one-time dedupe (the deterministic id makes the target set trivially identifiable) as a follow-up.

packages/server/AGENTS.md is updated with the invariant, so the next agent does not reintroduce a windowed dedup.

Seeded Timeline rows carry the entry's HISTORICAL `at`, so they are the
OLDEST rows in a loop's event stream. Deduping against
`listEvents(loopId, {limit: 200})` therefore stopped seeing them once a
loop accumulated 200 newer events, after which every task-file sync
re-inserted the entire Timeline as fresh `note` rows - unbounded growth
(reproduced live: 4 events -> 230 after one re-sync past the window).

`scripts/migrate-v2-split.ts --execute` carried the identical defect,
which falsified its "re-run is a no-op" idempotency claim.

- new `gateway/timelineSeed.ts`: the ONE seeder both the live ingest and
  the migration script import, so they cannot dedup differently.
- dedup is keyed and unbounded: `store.seededTimelineKeys` selects
  DISTINCT (day, text) over exactly the seeder's own rows (type=note
  carrying data.source='timeline'). DISTINCT bounds the result to the
  number of distinct entries, not the row count.
- the insert additionally uses a deterministic primary key derived from
  the same identity + ON CONFLICT DO NOTHING (`store.addEventIfAbsent`),
  so concurrent syncs and half-finished retries converge on one row.
- MESSAGE_CAP moves to the leaf `gateway/http.ts` (re-exported from
  index.ts) so the seeder shares one clip budget without a cycle.
- migrate-v2-split: `executeSeed`/`main` exported and the auto-run
  guarded by a direct-invocation check, so the pass is testable.
- migrate-v2-split rehearsal: the loss detector's separator class omitted
  `|`, the separator the daemon's own appendTimeline writes, so it
  reported a false LOSS (exit 1) on every daemon-authored row. Aligned
  with docSplit's TIMELINE_LINE.

Regression tests seed a loop past 200 events and assert zero new rows on
re-sync, on both the gateway ingest and the migration path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant