results(summary): rename cases to tests, trim run metadata - #1697
Merged
Conversation
Run-root summary.json now uses tests/test terminology instead of cases/case: counts.total_cases/passed_cases/failed_cases become total_tests/passed_tests/failed_tests, total_instances/errored_instances become total_samples/errored_samples (sample-level counts alongside the new test-level ones), and the cases[] array is now tests[]. Also drops the metadata.run_id duplicate of the summary root's run_id field, with summaryRunId() reading the root field first and falling back to the old metadata location for historical bundles. av-cpl5.2
Deploying agentv with
|
| Latest commit: |
e1f2aa4
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://1fb62a04.agentv.pages.dev |
| Branch Preview URL: | https://cpl5-summary.agentv.pages.dev |
The programmatic API test asserted summary.metadata.run_id, which moved to the summary root in the cases->tests rename. Update the test's inline type and assertion to match. av-cpl5.2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
summary.json'scases[]totests[]and alignscountsto test/sample terminology:total_cases/passed_cases/failed_cases→total_tests/passed_tests/failed_tests,total_instances/errored_instances→total_samples/errored_samples.metadata.run_idduplicate of the summary root'srun_idfield.summaryRunId()(used when rebuilding cross-run indexes) reads the root field first and falls back to the oldmetadata.run_idlocation so historical bundles written before this change still resolve correctly.buildRunSummaryArtifact(casesByKey→testsByKey,instances→samples, etc.) to match the new terminology.Implements
av-cpl5.2(child of source-of-truth trackerav-cpl5). This is the writer side only —RunSummaryArtifact,buildRunSummaryArtifact,writeInitialRunSummaryArtifact, andaggregateRunDir's summary-writing path inpackages/core/src/evaluation/run-artifacts.ts. Dashboard/CLI reader compatibility for historical bundles (oldcases/total_casesnames) is scoped to the parallel beadav-cpl5.1; no reader code outsiderun-artifacts.tsreferenced these fields (verified via repo-wide search), so nothing else needed updating in this PR..internal/index.jsonlandIndexArtifactEntry(av-cpl5.3's territory) were left untouched.Known follow-up (out of scope here):
metadata.eval_fileis written as whatever absolute/relative path the CLI passes intobuildRunSummaryArtifact/writeArtifactsFromResults— in a live dogfood run it came through as an absolute host path, while the siblingmetadata.runtime_source.eval_filesis already repo-relative (normalized upstream inrun-eval.ts). Fixingeval_filewould need cwd/repoRoot plumbing through multiple caller files outside this bead's declared scope; flagging for a follow-up bead rather than expanding this PR's surface.Test plan
bun test apps/cli/test/commands/eval/artifact-writer.test.ts— 75 pass, including new/updated assertions fortests[],counts.total_tests/passed_tests/failed_tests/total_samples/errored_samples, and top-levelrun_id.bun test apps/cli/test/commands/eval/aggregate.test.ts packages/core/test/evaluation/target-execution-artifacts.test.ts packages/core/test/evaluation/orchestrator.test.ts— 115 pass.bun test apps/cli/test/commands/results/report.test.ts apps/cli/test/commands/results/export.test.ts— 39 pass.bun --filter @agentv/core typecheck— clean.bunx biome checkon touched files — clean.bun apps/cli/src/cli.ts eval run examples/features/rubric/evals/operators.eval.yaml --target llm --workers 1 --threshold 0.8against a live Azure LLM target with a livellm-rubricgrader (threshold 0.8, not a 0-threshold smoke check) — passed 100%, and the resulting.agentv/results/<run_id>/summary.jsonwas inspected end-to-end to confirm the newtests[]/counts.total_tests/etc. shape and the removedmetadata.run_idduplicate.🤖 Generated with Claude Code