Skip to content

feat(results): slim index.jsonl target_execution/transcript_summary to sidecars - #1699

Merged
christso merged 1 commit into
mainfrom
cpl5-index
Jul 6, 2026
Merged

feat(results): slim index.jsonl target_execution/transcript_summary to sidecars#1699
christso merged 1 commit into
mainfrom
cpl5-index

Conversation

@christso

@christso christso commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Part of av-cpl5 (results-artifact-layout tracker). Implements av-cpl5.3: slim .internal/index.jsonl rows so they stay a Dashboard-ready manifest without inlining heavy sidecar payloads.

  • IndexArtifactEntry.target_execution (full TargetExecutionWire envelope — command argv, complete stdout/stderr logs, transcript) is dropped from every row. target_execution_path (unchanged, already written to target-execution.json) remains the sidecar pointer. A new compact target_error_kind scalar (row-level and per-sample, in samples[]) preserves the one bit of headline signal (error_kind) that was previously read out of the inlined envelope, without requiring a sidecar read.
  • IndexArtifactEntry.transcript_summary and TrialResultArtifact.transcript_summary (unbounded files_read/files_modified/shell_commands/web_fetches/errors lists) are dropped. transcript_path (unchanged) still points to the full transcript; each sample's result.json sidecar (referenced by sample_path) still carries transcript_summary in full, and apps/cli/src/commands/results/serve.ts already has a fallback (objectField(trial, 'transcript_summary') ?? objectField(runResult, 'transcript_summary')) that reads it from there when absent inline — verified this fallback already covers both old (fat) and new (slim) row shapes with no server changes needed.
  • Kept projection_identity inline as-is (not moved to a sidecar): writeArtifactsFromResults reads projection_identity.id back off previously-written index.jsonl rows on disk (readExistingIndexRecordsexistingRecordsByProjectionIdentity) to decide skip/update/error duplicate-policy across separate agentv eval invocations appending to the same run. Moving it to a sidecar would require an N-file read on every append to reconstruct dedup state. It's already compact (fixed set of scalar fields + optional short issues list), so it doesn't need the same slimming as target_execution/transcript_summary.
  • samples[] (per-trial rollup) keeps its existing compact shape (sample_path, score, status, scores, error, cost_usd, execution_status, failure_stage, failure_reason_code) plus the new target_error_kind.

Handoff notes for av-cpl5.1 / av-cpl5.4

New/changed field names for the docs and dashboard-reader beads to build against:

  • target_error_kind?: string — new, row-level and per-samples[]-entry. Replaces reading target_execution.error_kind / target_execution.errorKind directly off the row (see apps/dashboard/src/components/ResultTable.tsx's targetErrorKind() helper, which still reads the old nested shape — that helper needs updating in av-cpl5.1 to read target_error_kind instead; full target_execution is no longer present on rows/samples).
  • target_execution — removed from IndexArtifactEntry/TrialResultArtifact. Full detail: target_execution_path sidecar (unchanged path/shape).
  • transcript_summary — removed from IndexArtifactEntry/TrialResultArtifact. Full detail: each sample's result.json (path = <result_dir>/<sample_path>/result.json); serve.ts already falls back there when missing inline.
  • projection_identity — unchanged/still inline (see rationale above).

Test plan

  • bun test apps/cli/test/commands/eval/artifact-writer.test.ts (74 pass) — updated fat-row assertions to match the new slim shape.
  • bun test packages/core/test/evaluation/target-execution-artifacts.test.ts (1 pass) — updated to assert target_error_kind inline and full envelope only via the target_execution_path sidecar.
  • bun test apps/cli/test/eval.integration.test.ts -t "rejects removed --export" (1 pass) — updated to check transcript_summary lives in the sample's result.json, not the index row.
  • bun test apps/cli/test/commands/results/serve.test.ts (114 pass, unchanged) — confirms the dashboard-serving backward-compat fallback for transcript_summary (old fat rows vs. new slim rows) already works with no server-side changes.
  • bun test packages/core/test/evaluation/ (2253 pass) and apps/cli/test/commands/eval/ apps/cli/test/commands/results/ (437 pass + 1 pre-existing unrelated failure in bundle.test.ts, confirmed failing identically on an untouched sibling worktree at the same base commit — unrelated to this change).
  • bun run typecheck clean for both packages/core and apps/cli.
  • Live dogfood (real Azure OpenAI LLM target + real Azure LLM-rubric grader, --threshold 0.8):
    • examples/features/rubric/evals/suite.yaml test summary-task against llm target → 100% pass; inspected .internal/index.jsonl: row has no target_execution/transcript_summary, all existing sidecar path fields intact.
    • examples/features/local-cli/evals/suite.yaml test cli-provider-echo against a real CLI-provider agent target (mock CLI subprocess) graded by the same live Azure grader → 100% pass; confirmed target_execution_path sidecar (target-execution.json, 4.5KB with full command/stdout/stderr/transcript) exists and is loadable, while the index row itself stays slim (whole row ~3KB) with no inlined envelope; confirmed result.json sidecar still carries the full transcript_summary.

Closes av-cpl5.3 (leaving the bead open per process — coordinator closes after review/merge).

…o sidecars

index.jsonl rows previously inlined the full target_execution envelope
(command argv, complete stdout/stderr logs, transcript) and the full
transcript_summary (unbounded files/shell/web-fetch lists) on every row
and every repeat-sample rollup entry, duplicating what already lives in
target-execution.json and each sample's result.json sidecar. Drop both
inline objects; keep target_execution_path/transcript_path (already
present) as the sidecar pointers, and add a compact target_error_kind
scalar (row-level and per-sample) so the existing target-error-kind
table affordance doesn't need a sidecar read to know a run failed.

av-cpl5.3
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 0c25bee
Status: ✅  Deploy successful!
Preview URL: https://843f17b2.agentv.pages.dev
Branch Preview URL: https://cpl5-index.agentv.pages.dev

View logs

@christso
christso merged commit 16a21b3 into main Jul 6, 2026
8 checks passed
@christso
christso deleted the cpl5-index branch July 6, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant