Skip to content

docs(readme): modernize quick start eval and fix agentv compare example - #1572

Merged
christso merged 10 commits into
mainfrom
christso/readme-quickstart-cleanup
Jun 30, 2026
Merged

docs(readme): modernize quick start eval and fix agentv compare example#1572
christso merged 10 commits into
mainfrom
christso/readme-quickstart-cleanup

Conversation

@christso

@christso christso commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Replace the README opening install/YAML block and separate "What it does" section with a shorter outcome-focused intro for AgentV's broader target scope.
  • Rename "Why AgentV?" to "Why?" and tighten Core Concepts around eval suites, path-derived category, workspace context, target/model, policy, experiment as the named run condition, and concrete runs.
  • Promote the current quick start authoring shape to top-level name, top-level target, top-level model, and policy; experiment remains the result/run concept, not a top-level eval YAML or defineEval() field.
  • Keep public policy examples to runs, timeout_seconds/timeoutSeconds, threshold, and budget_usd/budgetUsd, matching PR [codex] Support eval policy config #1576.
  • Keep YAML and TypeScript examples on assertions, including string shorthand plus contains, code-grader, llm-grader, and threshold usage.
  • Clarify the Results layout around .agentv/results/<experiment>/<timestamp>/, resolved target identity subfolders such as copilot-sdk--claude-sonnet-4.6, index.jsonl, summary.json, and per-test artifacts/sidecars.
  • Rename the README tree's generated source snapshot from task/ to test/ for reader-facing terminology; tracked follow-up av-2ac7 covers the compatibility-safe code/artifact rename.
  • Add a compact Core Concepts Mermaid diagram covering eval suites/imports/tests, path-derived categories, workspace fixtures/graders, experiment conditions, target/model, policy, concrete runs, run artifacts, and Dashboard/compare/trend readers.
  • Expand the TypeScript SDK section with both paths: evaluate() for application-owned runs and defineEval() for TypeScript eval files that AgentV runs through the normal runner.
  • Update the eval-file validation schema and generated eval-schema.json so explicit type: rubrics accepts string shorthand criteria, matching runtime parsing and public docs.

Verification

  • Ran bun run generate:schema from packages/core.
  • Ran bun run lint.
  • Ran bun test packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-schema-sync.test.ts.
  • Ran bun test packages/core/test/evaluation/loaders/grader-parser.test.ts --test-name-pattern "string shorthand|criteria".
  • Validated the earlier README quick start YAML snippet against EvalFileSchema before the later target/policy README guidance update.
  • Ran git diff --check after the target/model/policy/workspace-isolation/runs README updates.
  • Ran git diff --check after the name + target/model/policy README correction.
  • Earlier in this PR: ran a one-case mock-provider eval and verified current generated artifacts include result.json, grading.json, metrics.json, timing.json, transcripts, and outputs.

Notes

  • This README PR should be read with implementation PR [codex] Support eval policy config #1576. PR [codex] Support eval policy config #1576 currently rejects top-level experiment authoring and accepts name + target + model + policy.
  • experiment remains the result namespace/run condition concept used by .agentv/results/<experiment>/<timestamp>/, compare, Dashboard, and application-owned evaluate() calls.
  • The code-level rename from task/ / task_dir to test/ / test_dir is intentionally tracked separately in Bead av-2ac7 because it touches the public artifact contract and readers.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 30, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: d0114d3
Status: ✅  Deploy successful!
Preview URL: https://4b328def.agentv.pages.dev
Branch Preview URL: https://christso-readme-quickstart-c.agentv.pages.dev

View logs

@christso
christso force-pushed the christso/readme-quickstart-cleanup branch 13 times, most recently from 268ad9d to d325172 Compare June 30, 2026 05:02
@christso
christso force-pushed the christso/readme-quickstart-cleanup branch 5 times, most recently from aee1399 to 44529b0 Compare June 30, 2026 06:25
christso and others added 9 commits June 30, 2026 09:21
- Drop test-level `criteria` shorthand in favor of an explicit `rubrics`
  assertion (rubric scoring under the hood), keeping a representative mix
  of contains, rubrics, code-grader, and llm-grader assertions.
- Add an `experiment:` runtime block to the quick start eval.
- Fix the `agentv compare` example: single-source compares targets within
  one run, which is misleading for the quick start. Show the canonical
  two-manifest pairwise form to compare two runs (before/after).
- Drop VS Code (no longer supported) and Azure OpenAI (not an agent) from
  the "Any agent" list.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Rename "Output formats" to "Results" and document the run bundle
  directory structure (.agentv/results/<experiment>/<timestamp>/...),
  modeled on agent-eval's presentation but using AgentV's own layout.
- Expand the TypeScript SDK example so it fully mirrors the quick start
  YAML eval: same test and the full assertions mix (contains, rubrics,
  code-grader, llm-grader) plus threshold.
- Use canonical array form for the code-grader command.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@christso
christso force-pushed the christso/readme-quickstart-cleanup branch from b40198e to 103cefe Compare June 30, 2026 07:23
@christso
christso merged commit ecaf328 into main Jun 30, 2026
8 checks passed
@christso
christso deleted the christso/readme-quickstart-cleanup branch June 30, 2026 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant