feat(results): slim index.jsonl target_execution/transcript_summary to sidecars - #1699
Merged
Conversation
…o sidecars index.jsonl rows previously inlined the full target_execution envelope (command argv, complete stdout/stderr logs, transcript) and the full transcript_summary (unbounded files/shell/web-fetch lists) on every row and every repeat-sample rollup entry, duplicating what already lives in target-execution.json and each sample's result.json sidecar. Drop both inline objects; keep target_execution_path/transcript_path (already present) as the sidecar pointers, and add a compact target_error_kind scalar (row-level and per-sample) so the existing target-error-kind table affordance doesn't need a sidecar read to know a run failed. av-cpl5.3
Deploying agentv with
|
| Latest commit: |
0c25bee
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://843f17b2.agentv.pages.dev |
| Branch Preview URL: | https://cpl5-index.agentv.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of av-cpl5 (results-artifact-layout tracker). Implements av-cpl5.3: slim
.internal/index.jsonlrows so they stay a Dashboard-ready manifest without inlining heavy sidecar payloads.IndexArtifactEntry.target_execution(fullTargetExecutionWireenvelope — command argv, complete stdout/stderr logs, transcript) is dropped from every row.target_execution_path(unchanged, already written totarget-execution.json) remains the sidecar pointer. A new compacttarget_error_kindscalar (row-level and per-sample, insamples[]) preserves the one bit of headline signal (error_kind) that was previously read out of the inlined envelope, without requiring a sidecar read.IndexArtifactEntry.transcript_summaryandTrialResultArtifact.transcript_summary(unboundedfiles_read/files_modified/shell_commands/web_fetches/errorslists) are dropped.transcript_path(unchanged) still points to the full transcript; each sample'sresult.jsonsidecar (referenced bysample_path) still carriestranscript_summaryin full, andapps/cli/src/commands/results/serve.tsalready has a fallback (objectField(trial, 'transcript_summary') ?? objectField(runResult, 'transcript_summary')) that reads it from there when absent inline — verified this fallback already covers both old (fat) and new (slim) row shapes with no server changes needed.projection_identityinline as-is (not moved to a sidecar):writeArtifactsFromResultsreadsprojection_identity.idback off previously-writtenindex.jsonlrows on disk (readExistingIndexRecords→existingRecordsByProjectionIdentity) to decide skip/update/error duplicate-policy across separateagentv evalinvocations appending to the same run. Moving it to a sidecar would require an N-file read on every append to reconstruct dedup state. It's already compact (fixed set of scalar fields + optional shortissueslist), so it doesn't need the same slimming astarget_execution/transcript_summary.samples[](per-trial rollup) keeps its existing compact shape (sample_path,score,status,scores,error,cost_usd,execution_status,failure_stage,failure_reason_code) plus the newtarget_error_kind.Handoff notes for av-cpl5.1 / av-cpl5.4
New/changed field names for the docs and dashboard-reader beads to build against:
target_error_kind?: string— new, row-level and per-samples[]-entry. Replaces readingtarget_execution.error_kind/target_execution.errorKinddirectly off the row (seeapps/dashboard/src/components/ResultTable.tsx'stargetErrorKind()helper, which still reads the old nested shape — that helper needs updating in av-cpl5.1 to readtarget_error_kindinstead; fulltarget_executionis no longer present on rows/samples).target_execution— removed fromIndexArtifactEntry/TrialResultArtifact. Full detail:target_execution_pathsidecar (unchanged path/shape).transcript_summary— removed fromIndexArtifactEntry/TrialResultArtifact. Full detail: each sample'sresult.json(path =<result_dir>/<sample_path>/result.json);serve.tsalready falls back there when missing inline.projection_identity— unchanged/still inline (see rationale above).Test plan
bun test apps/cli/test/commands/eval/artifact-writer.test.ts(74 pass) — updated fat-row assertions to match the new slim shape.bun test packages/core/test/evaluation/target-execution-artifacts.test.ts(1 pass) — updated to asserttarget_error_kindinline and full envelope only via thetarget_execution_pathsidecar.bun test apps/cli/test/eval.integration.test.ts -t "rejects removed --export"(1 pass) — updated to checktranscript_summarylives in the sample'sresult.json, not the index row.bun test apps/cli/test/commands/results/serve.test.ts(114 pass, unchanged) — confirms the dashboard-serving backward-compat fallback fortranscript_summary(old fat rows vs. new slim rows) already works with no server-side changes.bun test packages/core/test/evaluation/(2253 pass) andapps/cli/test/commands/eval/ apps/cli/test/commands/results/(437 pass + 1 pre-existing unrelated failure inbundle.test.ts, confirmed failing identically on an untouched sibling worktree at the same base commit — unrelated to this change).bun run typecheckclean for bothpackages/coreandapps/cli.--threshold 0.8):examples/features/rubric/evals/suite.yamltestsummary-taskagainstllmtarget → 100% pass; inspected.internal/index.jsonl: row has notarget_execution/transcript_summary, all existing sidecar path fields intact.examples/features/local-cli/evals/suite.yamltestcli-provider-echoagainst a real CLI-provider agent target (mock CLI subprocess) graded by the same live Azure grader → 100% pass; confirmedtarget_execution_pathsidecar (target-execution.json, 4.5KB with full command/stdout/stderr/transcript) exists and is loadable, while the index row itself stays slim (whole row ~3KB) with no inlined envelope; confirmedresult.jsonsidecar still carries the fulltranscript_summary.Closes av-cpl5.3 (leaving the bead open per process — coordinator closes after review/merge).