fix: stop replaying historical events on JSONL switch + dedup by uuid - #17
Open
pangin wants to merge 2 commits into
Open
fix: stop replaying historical events on JSONL switch + dedup by uuid#17pangin wants to merge 2 commits into
pangin wants to merge 2 commits into
Conversation
When cq detects a different JSONL file has become the newest session,
the watcher used to:
1. Compare the candidate's mtime against a baseline (lastModTime) that
was frozen at startup, so any other file whose mtime exceeded that
frozen value would "win" — including stale historical sessions that
Claude Code occasionally touches for bookkeeping.
2. Reset lastPos to 0 and tail the switched-to file from byte zero,
dumping that file's entire historical content into the event channel.
In practice this meant: while a user was actively working in session A,
Claude Code touched an old session B (33MB, 9,522 events) → watcher
switched to B → all 9,522 historical events streamed through → each
EventReading/Writing/Bash/Thinking granted XP → "same context replaying
sequentially while XP rockets up".
This change tightens the switch decision and removes the historical
replay:
- tailFile now refreshes w.lastModTime after every successful read of
the current file, so the watcher's idea of the active file's mtime
stays current instead of frozen at startup.
- checkForNewerFile compares the candidate's mtime against a FRESH
stat of the current file, not the (potentially stale) cached
lastModTime. An actively-growing session is its own freshest
reference and cannot be lost to a one-off external touch.
- A new looksLikeActiveSession guard rejects candidates whose filename
isn't UUID-shaped or whose mtime is older than 30s, filtering out
touched-but-dormant historical files.
- On a real switch, lastPos is set to the new file's current EOF so
only events appended AFTER the switch are emitted. The pre-switch
history of that file (legitimately not "live" anymore) stays out of
the event stream.
Includes watcher_test.go covering: active file kept, stale baseline
ignored, touched historical file ignored, non-UUID filename ignored,
and a real switch lands at EOF rather than offset 0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Belt-and-braces protection against re-emission. The prior commit removed
the most common replay path (offset-zero re-read after a JSONL switch),
but the event pipeline still has no notion of identity — if any future
path causes the same line to be parsed twice, XP is granted twice. Add
a small bounded set of seen UUIDs inside the watcher and drop any line
whose identity has already been emitted.
- Watcher gains a `seenUUIDs *uuidSet` field, initialized to a 50k-
entry ring-buffer-backed set (~5 MB hard memory ceiling; oldest
entries auto-evicted, so it never grows unbounded across long
sessions).
- parseLine probes for `uuid` (regular messages) or `messageId`
(file-history-snapshot rows) before doing the full Unmarshal, and
returns nil on a hit — cheap fast path for duplicates.
- On a compact_boundary event, the set is reset and the current
line's UUID is re-added. compact is a natural conversation boundary
after which pre-compact UUIDs are no longer referenced (post-compact
rows are all new UUIDs with parentUuid chaining back to the compact
marker — verified against real session JSONLs), so dropping them
frees memory early at a semantically clean point.
Tests cover: basic dedup, ring-buffer eviction at capacity, full reset,
parseLine UUID/messageId dedup, and the compact-reset preserving the
compact line's own UUID while making pre-compact UUIDs fresh again.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Reported by a user running
cq --koreanagainst~/.claude/projects/<encoded-cwd>/*.jsonl: aftercqprintedthe same conversation context started replaying through the watcher repeatedly and XP began climbing without any real user action behind it.
Root cause
Two compounding watcher bugs in watcher.go:
1. Stale baseline for the "switch to newer file" decision
checkForNewerFile()decided whether to switch by comparing the candidate's mtime againstw.lastModTime, which was set once inFindProjectConversationand only refreshed on a prior switch — never while actively tailing. The active file could grow for hours whilelastModTimestayed at its startup snapshot, so any other session file whose mtime exceeded that frozen value (including stale historical sessions that Claude Code occasionally touches for bookkeeping) trivially won the comparison.2. Offset-zero re-read of the switched-to file
On switch,
w.lastPos = 0was unconditionally set, thentailFile()'s next tickSeek(0, SeekStart)and parsed the entire history of the new file. In the reported scenario the switched-to file was 33MB / ~9,500 events — every one of which streamed back through the event channel and was treated as a fresh live action by main.go'sHandleEvent, granting XP throughProfile.RecordRead/Write/Bash/Thinking/AgentComplete/....There is no event deduplication anywhere downstream, so once the replay started there was nothing to stop the XP from compounding until the switch eventually settled.
Fix
Split into two reviewable commits.
fix: stop replaying historical events on JSONL session switchtailFile()now refreshesw.lastModTime = info.ModTime()after every successful read of the current file. The watcher's idea of "what mtime is the active file at" stays current instead of frozen.checkForNewerFile()compares the candidate's mtime against a freshos.Statof the current file, not against the cached baseline. An actively-growing session is its own freshest reference and cannot be lost to a one-off external touch.looksLikeActiveSessionguard rejects candidates whose filename isn't UUID-shaped or whose mtime is older than 30s. Old historical sessions whose mtimes were merely touched (no real activity) no longer pass.w.lastPos = newInfo.Size()— seek to EOF rather than offset 0, so only events appended after the switch are emitted.feat: dedup events by uuid with compact-aware resetDefense-in-depth, so a future regression in the file-switching logic (or any other path that re-parses lines) cannot recreate the XP-replay symptom.
seenUUIDs *uuidSetfield: a ring-buffer-backed bounded set (50k entries, ~5 MB hard ceiling, oldest entries auto-evicted).parseLine()cheaply probes each line foruuidormessageIdand drops the line on a hit.compact_boundary, the set is reset (the compact line's own UUID is re-added so re-reading that line still no-ops). I confirmed against real session JSONLs that all post-compact rows have new UUIDs withparentUuidchaining back to the compact marker — so pre-compact UUIDs are genuinely orphan and the reset only frees memory at a semantically clean boundary.Tests
watcher_test.go(new file):TestCheckForNewerFile_ActiveFileNotSwitchedAway— active file is its own newest reference; no switch.TestCheckForNewerFile_StaleBaselineDoesNotTriggerSwitch— regression guard for the exact bug.TestCheckForNewerFile_TouchedHistoricalFileIgnored— historical session whose mtime got touched but is outside the active window is not eligible.TestCheckForNewerFile_NonUUIDFilenameIgnored— non-session files don't passlooksLikeActiveSession.TestCheckForNewerFile_RealSessionSwitchSeeksToEOF— a genuinely newer live session triggers a switch andlastPoslands at EOF, so the 10 pre-existing lines do not stream back as events.TestUUIDSet_AddIsDedup,TestUUIDSet_EvictsOldestAtCapacity,TestUUIDSet_ResetClearsAll— bounded-set semantics.TestParseLine_DedupesByUUID— same JSON line returns events first, nil second.TestParseLine_CompactResetsDedupSet— compact frees pre-compact UUIDs while keeping the compact line itself deduped.go test -cbuilds cleanly. (Fullgo testrequires raylib's DLL on PATH, which my CI doesn't have — I verified each behavior in addition with standalone harness programs that vendored the affected functions.)