Allow a dataset to span multiple JSONL files#77
Merged
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
TraceStore.load_many unions independently-indexed files into one dataset — every query surface (view/search/filters/overview) treats the union as one dataset with no file boundaries visible. Trace ids must be unique across files (load fails fast otherwise). The public entrypoints accept trace_path as one Path or a sequence. The engine stays agnostic about what each file encodes — callers describe their files through dataset_context or their own prompt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RwnU9ZBJnrQC89tBumL6PY
francescov1
force-pushed
the
inf-4125-eval-results-dataset
branch
from
July 18, 2026 13:45
4ba8b87 to
9a1896a
Compare
francescov1
changed the title
After a dataset loads via TraceStore.load_many, the run_code sandbox still bootstrapped a single-file TraceStore from store.trace_path / store.index_path (the primary file only), so user code saw a trace_store missing every additional file while the host trace tools saw the union. Thread all (trace, index) sources through run_python → _run_protocol → bootstrap → halo_bootstrap: mount each file at an indexed virtual path and rebuild the store with load_many inside Pyodide. New integration test mounts two files and asserts the sandbox trace_store sees all 5 traces and can read one that lives only in the second file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RwnU9ZBJnrQC89tBumL6PY
…t path Two follow-ups to the multi-file sandbox change: - engine/main.py: a set trace_index.index_path was reused for every file in a multi-file dataset, so all files shared one sidecar index and load_many paired traces with the wrong byte offsets. Extract _resolve_trace_sources: derive a per-file sidecar (the default) and fail fast if a single index_path is pinned for a multi-file dataset. - test_sandbox_policy_denials: after mounts moved to /input/traces_0.jsonl, the host-immutability test still wrote to /input/traces.jsonl, so it no longer exercised a write against the mounted copy. Point it at the real virtual path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RwnU9ZBJnrQC89tBumL6PY
…model The multi-file bootstrap payload crossed the host↔Pyodide boundary as a loose JSON array of dicts, indexed by raw key on the Python side. Replace it with a shared TraceDatasetSource model (living in engine.traces.models so it's staged into Pyodide alongside the trace store): the host builds and model_dumps it, and halo_bootstrap validates the JSON back through a TypeAdapter instead of json.loads + dict access. The wire is still JSON (it has to cross into WASM as text) but now it's a typed, validated contract on both ends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RwnU9ZBJnrQC89tBumL6PY
… Path) tuples Replace the loose (trace_path, index_path) tuple that flowed through the whole multi-file path with the typed TraceDatasetSource model (now Path fields, serialized to strings only when crossing the JSON-RPC boundary): - TraceStore._sources / load_many / sources property - main._resolve_trace_sources - Sandbox.run_python / _run_protocol / bootstrap - halo_bootstrap validates straight into TraceStore.load_many, no tuple rebuild Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RwnU9ZBJnrQC89tBumL6PY
Drop the ``Path | Sequence[Path]`` union (and the isinstance branching it forced) from every public entrypoint. ``trace_paths: Sequence[Path]`` is now always a list — one element for the common single-file case, several for a multi-file dataset — so there's one code path and no single-vs-many special casing. Callers wrap a lone path as ``[path]``. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RwnU9ZBJnrQC89tBumL6PY
bf1d1ab made every entrypoint list-only, which TypeErrors when a caller passes a lone Path (the prior API) — a Path is not a Sequence[Path], so len()/iteration blow up at startup. Restore the single-path convenience on the public entrypoints via ``Path | Sequence[Path]`` and normalize once at the edge (``_as_trace_path_list``); _resolve_trace_sources and every internal path stay list-only. Callers that pass one file use the bare Path again; multi-file callers pass a list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RwnU9ZBJnrQC89tBumL6PY
…config
Index location is a per-file property, so move it off the dataset-wide
TraceIndexConfig (which caused the "one index_path can't serve N files"
wart) onto each dataset entry. Entrypoints now take
`Path | Sequence[TraceDataset]` where TraceDataset = {trace_path,
index_path: Path | None}: a lone Path is the single-file edge convenience
(index derived), and a multi-file dataset can pin each file's index
independently. TraceIndexConfig keeps only schema_version;
ensure_index_exists takes the per-file index_path as a param; the
multi-file guard is gone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RwnU9ZBJnrQC89tBumL6PY
…rride Revert the per-file TraceDataset index folding in favor of a simpler single index location that is a directory: TraceIndexConfig.index_dir. When set, each dataset file's sidecar index is a distinct file named after the trace inside that one dir, so a single index_dir serves an entire multi-file dataset (and doubles as a stable cache location, e.g. for Modal). When unset, each index derives next to its trace as before. Entrypoints revert to Path | Sequence[Path] (single Path at the edge, lists internally); TraceDataset is gone; ensure_index_exists derives the per-file name in index_dir; the multi-file guard is dropped since the folder handles N files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RwnU9ZBJnrQC89tBumL6PY
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 1f75c96. Configure here.
Two dataset files sharing a basename in different directories mapped to
the same {index_dir}/{name}.engine-index.jsonl, so one index clobbered
the other and multi-file runs mis-routed byte offsets or hit spurious
duplicate trace_id errors. Derive the in-dir index name from a hash of
the trace's full path (kept in a shared sidecar_index_path helper), so
distinct files get distinct indexes while the name stays deterministic
for stable caching.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RwnU9ZBJnrQC89tBumL6PY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Let a single analysis span multiple JSONL dataset files instead of exactly one.
TraceStore.load_many(sources)unions several independently-indexed files into one store; every public entrypoint inengine/main.pynow acceptstrace_path: Path | Sequence[Path]. Single-fileTraceStore.load(...)and the single-Pathcall sites are unchanged.This keeps the engine generic trace tooling — it has no notion of evals or any other domain. A caller that wants to analyze more than one export (e.g. conversations plus a second file of verdict spans) simply passes both paths; the agent reads them all through the ordinary trace tools and tells them apart by whatever attributes the caller documents in
dataset_context.How
engine/traces/trace_store.py:load_many(sources: list[tuple[Path, Path]])builds a per-file trace-id → file map (_path_by_trace_id,_trace_file_for), and every read (view_trace/view_spans/search_trace/search_span) opens the owning file._apply_filtersgroups matching rows by file and opens one handle per file. A trace id that appears in more than one file raisesValueError; an empty source list is rejected.engine/main.py: each entrypoint indexes every path (reusingTraceIndexBuilder) then callsload_many.tests/unit/traces/test_trace_store_multi_file.py(union counts, per-file reads, cross-file search/overview/filters, duplicate + empty rejection, single-file parity) and a generic second-file fixturetests/fixtures/tiny_traces_second_file.jsonl.Test plan
464 unit + 64 integrationgreen; ruff + pyright clean.🤖 Generated with Claude Code
Note
Medium Risk
Breaking public API (
trace_path→trace_paths,index_path→index_dir) and wide changes to trace I/O and sandbox bootstrap; behavior is guarded by new multi-file and integration tests but callers must update.Overview
Multi-file datasets are now first-class: all
engine/main.pyentrypoints taketrace_paths: Path | Sequence[Path](still a singlePathfor the common case), index each file, and load one logical store withTraceStore.load_many. Trace ids must be unique across files or load fails fast.TraceDatasetSourceis the shared(trace_path, index_path)model for the store,run_python, and sandbox bootstrap JSON-RPC.TraceIndexConfigdrops per-runindex_pathin favor of optionalindex_dir, withsidecar_index_pathplacing hashed, collision-safe index files when several traces share a cache directory.TraceStoreroutes reads to the owning JSONL file and treats filters/overview/search as one dataset.run_codemounts every file at indexed virtual paths (/input/traces_0.jsonl, …) and bootstraps the same union inside Pyodide as on the host.Reviewed by Cursor Bugbot for commit 7d93d8a. Bugbot is set up for automated code reviews on this repo. Configure here.