cc-explorer corpus-layer redesign: bounded cache + rg prefilter + mtime pruning (Phases 0-3)#54
Open
coryking wants to merge 3 commits into
Open
cc-explorer corpus-layer redesign: bounded cache + rg prefilter + mtime pruning (Phases 0-3)#54coryking wants to merge 3 commits into
coryking wants to merge 3 commits into
Conversation
The module-global parse cache was an unbounded dict retaining the fully parsed pydantic object graph of every transcript ever loaded. One cross-project search_projects call parses the whole ~3 GB corpus, which left a live MCP server instance at 5.5 GB RSS (raw bytes x ~1.8 heap factor, measured). Control instances that never ran a cross-project op sit at 80 MB. - TranscriptCache: cachetools.LRUCache keyed by resolved path, byte-capped via getsizeof; entries charged file_size x 1.8 (the measured heap factor). Cap: CC_EXPLORER_CACHE_MB env, default 200 MB. A single file larger than the cap is served uncached rather than erroring. - load_transcript keeps its exact signature; the cache instance is module-level and injectable for tests. - Tolerant parsing stops being silent: malformed/unknown lines are counted and reported once per file to stderr (never stdout — that's the stdio MCP channel). Known structural header lines (mode, permission-mode, custom-title, x-converter-provenance) skip silently — they're wire format, not data. Replaces the nonsense except-tuple (json.JSONDecodeError, ValueError, Exception). Phase 0 of the corpus-layer redesign (bounded memory + rg prefilter + mtime pruning); the prefilter and identity/derived type split land next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uning + orjson
Phases 1-3 of the corpus-layer redesign, on top of the bounded cache
(previous commit). Search cost now scales with the answer, not the corpus.
corpus.py (NEW) — the identity layer. SessionRef is a session known by
filename alone (replaces the _ArtifactSession shadow type); Corpus is the
refs in scope for one tool call, built from dir listings only (discover /
narrow_to_ids / narrow_to_artifact_ids / candidate_refs). Project
resolution + discovery (resolve_project(s), discover_projects,
ProjectInfo) move here from search.py — they were always identity, not
matching.
Prefilter: candidate_refs runs ripgrep over raw JSONL bytes (argv-batched;
rg has no --files-from) with a streaming Python fallback when rg is
missing. Raw bytes are a candidate SUPERSET of extracted text except where
JSON escaping rewrites characters — rg_safe gates quote/backslash/\s/
anchor/lookaround patterns to the scan-all path, so the prefilter can
over-select but never drop a true hit. The typed matcher (_entry_matches)
stays the matcher of record.
resolve.py (NEW) — id→artifact resolution out of mcp_server, operating on
SessionRefs (no parse). _ArtifactSession, _sessions_by_filename,
_projects_for_sessions, _narrow_projects_for_artifacts all deleted.
search.py — SessionInfo.load(ref) is the ONLY promotion path (parses
exactly one session); agents_present becomes a lazy cached property
(with_agents_present flag killed); load_sessions is a thin
discover+promote wrapper.
mcp_server.py rewired: search_projects and read_turn(no session) go
prefilter-first; every session-keyed tool resolves refs by filename and
promotes only its target; get_agent_detail narrows by agent-filename glob
instead of parsing every project; convert/rewind/delete resolve on refs.
activity.py — mtime-window pruning: append-only transcripts older than the
window start are skipped without parsing (compaction rewrites bump mtime,
so pruning errs safe); a session survives if any of its subagent bodies is
fresh even when the parent is cold.
parser.py — orjson for line parsing (2-6x on many-small-lines workloads).
Measured on the real corpus (1,472 sessions / ~3 GB):
cross-corpus search 0.5 s prefilter + 1.1 s triage, peak RSS 195 MB
(incident baseline: minutes, 5.5 GB RSS)
timeline 7d 1.0 s
read_turn (no sess) 0.4 s to locate the holding file
412 tests pass (43 new: rg_safe gate, scanner contracts incl. argv
batching, candidate superset incl. the stripped-XML/newline boundary case,
prefilter/full-scan triage equivalence, narrow_to_artifact_ids,
mtime-pruning parity + fresh-subagent survival).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
coryking
changed the title
…edesign Four parallel cleanup reviews (reuse / simplification / efficiency / altitude) over the branch diff, deduped and applied: - search.promote_refs(): the bulk form of SessionInfo.load — replaces the same walrus-filter idiom copy-pasted across load_sessions, _load_all_sessions, search_projects, read_turn, and get_agent_detail. - Corpus.narrow_to_artifact_ids: one recursive glob per encoded dir (was one glob per (dir, id) plus a second per-session glob), and narrow_to_ids computed once instead of twice. - corpus owns MIN_ID_LEN; resolve.py and get_agent_detail import it instead of re-declaring the literal 6. - SessionRef.transcript_files reuses subagents.resolve_subagents_dir instead of inlining the path formula. - browse_session: one Corpus.discover per call (the agent-id fallback reused it instead of re-walking ~/.claude/projects), and the whole flow is ref-only — browse_session_turns now takes a transcript Path, so the fabricated minimal SessionInfo (a bypass of the load()-only promotion invariant) is gone. - activity timeline: staleness prune is stat-only (raw agent-file walk) before collect_agent_files reads meta.json/provenance — only surviving sessions pay for provenance. 7d window on the real corpus: 0.98s → 0.44s. - parser._STRUCTURAL_LINE_TYPES covers the other known header types (ai-title, last-prompt, attachment, agent-name, relocated, worktree-state, pr-link) — they were already skipped, but were being reported as "unparseable" noise on stderr (394 lines in one session). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The incident
A live cc-explorer MCP server instance was found at 5.5 GB RSS: one
search_projectswith noprojectsscope parsed the entire~/.claude/projectscorpus into pydantic object graphs and retained them forever in an unbounded module-global cache. Control instances: 80 MB. This PR is the full approved redesign — "to find a needle, first you must drink the ocean" is dead.No database, no ingest pipeline, no hooks, no watcher. JSONL stays the only truth (Syncthing replicates it across boxes); instead the cache is bounded, searches are prefiltered with ripgrep so cost scales with answer size, and the timeline prunes by mtime.
Phase 0 — bound the cache
TranscriptCache:cachetools.LRUCachekeyed by resolved path, byte-capped viagetsizeof; entries chargedfile_size × 1.8(measured heap factor).CC_EXPLORER_CACHE_MB, default 200. Oversized single files served uncached.mode,permission-mode,custom-title,x-converter-provenance) skip silently. Theexcept (json.JSONDecodeError, ValueError, Exception)nonsense-tuple is gone.Phase 1 — corpus identity layer + rg prefilter
corpus.py(new):SessionRef= a session known by filename alone (deletes the_ArtifactSessionshadow type — the symptom ofSessionInfoconflating cheap identity with expensive derived state).Corpus= the refs in scope for one call:discover/narrow_to_ids/narrow_to_artifact_ids/candidate_refs. Project resolution + discovery moved here fromsearch.py.candidate_refsrunsrgover raw JSONL bytes (argv-batched — rg has no--files-from), Python streaming fallback if rg is missing. Raw bytes are a candidate superset of extracted text except where JSON string escaping rewrites characters —rg_safegates quote/backslash/\s/anchor/lookaround patterns to the scan-all path (memory-safe post Phase 0)._entry_matchesremains matcher of record; the prefilter can over-select, never drop a true hit.resolve.py(new): id→artifact resolution out of the tool layer, operating on refs — no parse ever.SessionInfo.load(ref)is the only promotion path;agents_presentis now a lazy cached property (with_agents_presentflag killed); every session-keyed tool resolves refs by filename and promotes only its target.get_agent_detailwithout a session narrows by agent-filename glob instead of parsing every project.Phase 2 — timeline mtime pruning
Append-only transcripts older than the window start can't contain in-window entries → skipped without parsing. Compaction rewrites bump mtime, so pruning errs safe. A session survives when any subagent body is fresh even if the parent is cold (background agents outlive their parent's last write).
Phase 3 — orjson + measurements
orjson for line parsing in the parser and the discovery head-scans.
Measured on the real corpus (1,472 sessions / ~3 GB), in-process:
search_projectsget_activity_timeline7dread_turn, bare turn uuidLive in-process smoke of every read tool against the real corpus:
list_projects, cross-corpussearch_projects,grep_session,read_turn(no session),browse_session,list_project_sessions,get_activity_timeline— all correct.Tests
412 passed (56 new across the PR): cache eviction/cap/oversize;
rg_safegate; scanner contracts incl. argv batching and rust-regex rejection →ScannerError→ scan-all; candidate-superset incl. the stripped-XML/newline boundary case; prefilter/full-scan triage equivalence;narrow_to_artifact_ids(incl. workflow-nested agents); mtime-pruning payload parity + stale-parent/fresh-subagent survival. The existingtriage-oracle equivalence test carries over and now proves the prefilter changes nothing.🤖 Generated with Claude Code