feat: tracing pipeline (schema rename + recorder + viewer + docs)#23
Merged
Conversation
Naming rule: every record type follows <entity>.<verb> past tense, lowercase,
dot-separated. Payload key matches the first segment of type.
Renames:
transcript.started/forked/.compacted (JSON value) → session.*
RecordStarted/Forked/Compacted → SessionStarted/Forked/Compacted
RecordMessageAppended/RecordStatePatched → MessageAppended/StatePatched
SystemRecord (lifecycle payload) → SessionRecord
(the struct holds provider/model/parentId/boundaryId — name now matches
intent; it never carried system-prompt data)
Record.System field → Record.Session
Record.TranscriptID + json tag → SessionID / "sessionId"
*Command.TranscriptID → SessionID
ForkCommand.Source/NewTranscriptID → Source/NewSessionID
ListItem.TranscriptID, fileIndexEntry.TranscriptID → SessionID
ContentBlock gains a Source string field for provenance attribution
(populated in a later commit). The existing inline-image data field,
which used JSON tag "source", is renamed to ImageSource / "imageSource"
to free the namespace.
Projector's applyStatePatch now ignores unknown patch paths
(default: continue) instead of returning an error — keeps the projector
forward-compatible with patch paths added by later schema iterations.
The error-expecting test becomes IsIgnored, asserting subsequent ops
still apply.
Adds session.Recorder, a synchronous observer on core.Agent's event bus that translates lifecycle events into transcript records. Wired via core.Config.OnEvent (new BuildParams.OnEvent passthrough; constructed by Session.NewRecorder at agent start). New records per turn: inference.requested digests of system prompt / tool list / message chain inference.responded stop reason, latency, token usage System observability: Use(sec, caller), Drop(name, caller) on core.System require explicit caller strings (system:init, command:identity, subagent:init). System.SetObserver replays existing sections on attach so the event chain is complete from t0. Recorder writes system.section.added/removed. Tools observability: Same pattern on core.Tools: Add/Remove gain caller, SetObserver replays. MCP registrations pass caller="mcp:<toolname>". Both wrappers (permissionTools, progressTools) pass-through. Recorder writes tools.added (with schema) / tools.removed (with name). Content provenance: splitTextByProvenance splits user-message content on <system-reminder> boundaries and tags those blocks with Source="reminder". Round-trip safe: extractUserContent concatenates back, preserving core.Message.Content byte-for-byte. Non-blocking telemetry: agent.emitTelemetry uses select-default on the outbox so observer-fired events never deadlock the agent goroutine on a slow TUI consumer. Tests: 4 new Recorder tests + 2 provenance tests.
NewRecorder now calls FileStore.Start synchronously before returning, so
session.started lands on disk first. Previously the call order
1. app builds Recorder
2. core.NewAgent wires System/Tools.SetObserver
3. SetObserver replays existing members → AppendSystemSection/AppendTools
creates the transcript file
4. Store.Save → Start → file exists, no-op
left provider/model/parentID metadata absent from every transcript that
involved any observer replay (i.e. every main-agent session). Resume
and the projects index would show empty Provider/Model.
Fix lives in the recorder constructor because that's the latest point
before observers can fire. RecorderOptions gains Cwd and ProjectID so
Start can be called with the right metadata. Setup.NewRecorder threads
them through from Store.cwd/Store.projectID.
Test: TestRecorder_WritesSessionStartedBeforeTelemetry asserts that
after NewRecorder() returns and an OnSystemChange event is fired, the
first record on disk is session.started with provider+model populated.
A localhost-only web UI for inspecting session transcripts under
~/.gen/projects/<encoded-cwd>/transcripts/. Read-only, single binary
(assets embedded via go:embed), no build step on the frontend.
Backend (internal/trace):
- HTTP server with three endpoints:
GET / SPA shell
GET /api/sessions list transcripts in the project
GET /api/sessions/{id}/records paginated record fetch
GET /api/sessions/{id}/stream SSE live tail
- The wire format is the JSONL on disk verbatim — server doesn't
reshape records. One schema, not two.
- Polling-based tailer (500ms tick); no fsnotify dependency.
- Path-traversal guard rejects sessionIDs containing slashes.
Frontend (internal/trace/ui/assets):
- Vanilla JS, ~200 lines. EventSource for SSE.
- Sessions sidebar | colored timeline | JSON detail panel.
- Per-group filter checkboxes (state, tools, system, inference,
message).
- Auto-scrolls when near the bottom; pauses on user scroll-up.
CLI (cmd/gen/trace.go):
- `gen trace` binds 127.0.0.1 on a random port by default, opens the
browser, blocks until Ctrl-C.
- --addr to pin a port; refuses any non-loopback host.
- --no-open to skip the browser launch (for headless / CI).
Tests cover the records endpoint, empty-project case, and the
path-traversal guard. Smoke-tested locally against real session files.
3 tasks
Two issues in the viewer's frontend. P1 (defense-in-depth XSS): s.id from /api/sessions flows into the sidebar li's innerHTML unescaped. Today the ID is always a UUID by construction, but filesystem-derived strings should never be trusted as HTML — the title beside it is already escaped, the inconsistency suggests an oversight. Wrap s.id.slice(0, 12) in escapeHTML(). P2 (timeline duplicates on reconnect): browsers auto-reconnect EventSource on transient drops (laptop sleep, localhost blip). The server starts each stream from offset 0 and replays the entire file, so every reconnect re-emitted every record and onmessage pushed them all into state.records. After a sleep/wake the timeline doubled, tripled, etc. Fix: maintain a state.seenIDs Set keyed by each record's stable id field. Skip on duplicate. Cleared in openSession alongside records[]. The server-side replay strategy is left unchanged — a follow-up could honor Last-Event-ID to skip already-streamed bytes, but dedup-on-client is the correctness fix and lets the timeline survive reconnects today.
A first-principles description of the transcript event taxonomy, record envelope, payload shapes per group, replay algorithm, common troubleshooting recipes (jq one-liners), and the gen trace viewer's HTTP API. Companion to docs/transcriptstore.md, which covers storage layout and resume mechanics; this doc focuses on the event schema and the viewer.
yanmxa
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the rest of the tracing stack into main.
PRs #20, #21, #22 were merged into their stacked base branches (`trace/1-persistence`, `trace/2-schema-rename`, `trace/3-recorder`) rather than `main` — the stacked-PR base wasn't retargeted after #19 landed. This PR consolidates their cumulative content into main via a single merge.
What's in the diff (already reviewed)
Test
🤖 Generated with Claude Code