0.1.0: hardening round (schema, versioning, validation, CLI, reporting) - #3
Merged
Conversation
Adds an optional toolCallId to Step. ToolCallEvaluator now pairs a tool_result to its tool_call by id when present (authoritative: an unmatched id is an orphan, no name fallback), falling back to the previous name+FIFO only when the result has no id. Backward compatible: id-less fixtures keep the old behavior. The Python emitter generates a toolCallId per call (call-N) and copies it onto the result via the ToolCall handle, so every emitted trace pairs precisely even with repeated calls to the same tool. Schema doc updated. Adds id-pairing tests (same-name calls, out-of-order results, unknown id) on both sides. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validate now rejects non-negative violations (cost, inputTokens, outputTokens, durationMs), non-monotonic step timestamps, and steps falling outside the run's [startTime, endTime] interval, on top of the existing required-field checks. Strictness is pragmatic (validate-if-present): an absent version and an empty step list are still accepted, keeping older emitted traces valid. A missing step timestamp is flagged as missing but skipped for ordering so it cannot produce a spurious before-previous error. Schema-version compatibility stays out of scope here. Adds aggressive edge-case tests: negative quantities, out-of-order timestamps, steps before startTime / after endTime, the lenient-optional-fields policy, and the missing-timestamp-skips-ordering guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Evaluator.EvaluateRun now takes a context.Context as its first argument. Cancelling it aborts in-flight evaluation instead of running to completion: - The four compute evaluators (tool_calls, loops, cost_latency, node_transitions) check ctx.Err() on entry and return it promptly. - The LLM judge uses the caller's ctx as the parent of its per-call timeout, so a cancelled run also aborts the outbound Gemini HTTP request (previously it started from context.Background() and ignored the caller). - The runner propagates ctx to every file and every evaluator, short-circuiting a file to a context.Canceled FileError before touching disk. - The CLI wires signal.NotifyContext(os.Interrupt) so Ctrl+C cancels a run, which matters most for the network-bound judge. Adds cancellation tests: a runner-level test (cancelled ctx yields zero evaluations and a context.Canceled error per file) and an evaluator-level table test pinning that every evaluator honors a cancelled context. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Uncomment .idea/ so the JetBrains project directory is ignored, and remove the previously committed .idea files from the index. Also drop the duplicate *.so and .env entries the Python template added; both are still ignored by the Go template block above, so .env (which holds GEMINI_API_KEY) stays untracked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e tests Add trajectory/trace.schema.json (draft 2020-12) as the strict external contract for the trace format, without adding any dependency to the Go core (still stdlib-only). Keep it honest against the source of truth from both sides: a Go test fails the build if the schema's type enum or per-type required fields drift from the trajectory constants, and a Python test validates every fixture and the emitter output against it. Document the two intentional gaps vs Run.Validate: the schema forbids unknown fields (loader is lenient) and cannot express cross-field temporal invariants (those stay Go-only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the text and JSON renderers out of cmd/trazo/main.go into the report package alongside Markdown, so all three output formats live in one testable place and main.go just selects one. Behavior is unchanged. Add golden tests covering all three formats over a single fixture that exercises scored findings, run-level findings, Markdown escaping, empty findings, and file errors. Pin golden files to LF via .gitattributes so the byte-for-byte comparison holds on Windows (core.autocrlf=true) and CI alike. Also rename Evaluator.go to evaluator.go to match Go file-naming convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rework the trazo CLI beyond a single -dir scan: - Accept a positional PATH (a trace file or a directory); -dir stays as the fallback default. - Add -recursive to descend into subdirectories. - Add -validate for a structure-only pass (loads and validates, skips evaluators) with a concise "N valid, M invalid" summary. - Validate -format up front (fail fast) and add a proper -help/usage block. Refactor the runner to separate file discovery (CollectFiles) from execution (RunFiles); Run(dir) is now a thin wrapper. FileError carries the path as read so errors stay unambiguous across subdirectories. Tests cover recursive and non-recursive discovery and the validate summary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the flat per-finding text output with a human-facing report: findings grouped by run (agent shown in the header), severity up front in aligned columns, runs with no findings marked "clean", a one-line summary, and file errors last. The zero-value Score noise is gone. JSON and Markdown output are unchanged; JSON stays keyed on runId (the new Evaluation.Agent field is json:"-"). Update the golden text fixture and the schema doc's expected output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce real schema versioning. trajectory.SchemaVersion ("0.1.0") is the
canonical version, and Run.Validate now requires a semver version and rejects a
trace whose major differs from the build's supported major. Minor/patch
differences within the major stay compatible (adding toolCallId was such an
additive bump), so older 0.x traces still load. The JSON Schema marks version
required with a semver pattern, and a Go test keeps the schema, the
SchemaVersion constant, and the compatibility check in sync.
Normalize every fixture to 0.1.0 and flip the old "absent version is accepted"
test to assert the new gate, with explicit accept/reject version cases.
Python side: the emitter exposes SCHEMA_VERSION mirroring the Go constant and
stamps it by default; the agent no longer conflates its own version with the
schema version. Also fix a stale recorder docstring that predated toolCallId.
Update the schema doc with a Versioning section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
0.1.0 hardening round
Rounds out the trazo core from a working MVP to something closer to
production-grade, along the axes raised in a technical review: schema formality,
versioning, deeper validation, tool-call identity, cancellation, CLI ergonomics,
and reporting stability. The Go core stays standard-library only (no
go.sum); the only new dependency isjsonschema, and it is test-only on thePython side.
Every change ships with tests.
go build ./... && go test ./...and the Pythonsuite (
pytest, 35 tests) are green.Schema and versioning
trajectory/trace.schema.json, draft 2020-12) as thestrict external contract, derived from the Go types. A dependency-free Go test
fails the build if the schema's type enum or per-type required fields drift
from the
trajectoryconstants; a Python test validates every fixture and theemitter output against it.
versionis now required and semver-gated.trajectory.SchemaVersion(0.1.0) is canonical; a trace whose MAJOR differsfrom the supported major is rejected with an actionable message. MINOR/PATCH
stay compatible within a major, so older
0.xtraces still load. Documented inthe schema doc's Versioning section; the Python emitter mirrors the constant as
SCHEMA_VERSION.Validation and correctness
Run.Validate: non-negative cost/tokens/duration, monotonic steptimestamps, steps within the run interval, per-type required fields, plus the
version gate. All violations are collected and joined.
toolCallIdcorrelates atool_resultwith itstool_callauthoritatively, with name/FIFO as a documented fallback.context.Contextthreaded through theEvaluatorinterface, so Ctrl+C ora CI timeout aborts in-flight work (notably the network-bound LLM judge).
CLI and reporting
-recursive, a-validatestructure-only mode with a concise summary, up-front-formatvalidation, and a proper-help.CollectFiles) from execution(
RunFiles);FileErrornow carries the full path so errors are unambiguousacross subdirectories.
reportpackage and covered by golden tests (LF-pinned via.gitattributesfor Windows/CI). The text format is regrouped by run with asummary footer.
Polish
Evaluator.go->evaluator.go(Go file-naming convention).toolCallId; stop the agent fromconflating its own version with the schema version.
🤖 Generated with Claude Code