From 67377b5f5f0969791b9d0c16b2d4ffdc7ea94145 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Sat, 6 Jun 2026 14:27:27 +0200 Subject: [PATCH] docs(evals): document benchmark provenance patterns --- .../docs/docs/evaluation/eval-cases.mdx | 9 +- .../docs/docs/evaluation/eval-files.mdx | 2 + .../docs/docs/guides/benchmark-provenance.mdx | 264 ++++++++++++++++++ 3 files changed, 274 insertions(+), 1 deletion(-) create mode 100644 apps/web/src/content/docs/docs/guides/benchmark-provenance.mdx diff --git a/apps/web/src/content/docs/docs/evaluation/eval-cases.mdx b/apps/web/src/content/docs/docs/evaluation/eval-cases.mdx index a8601a214..e6787ae52 100644 --- a/apps/web/src/content/docs/docs/evaluation/eval-cases.mdx +++ b/apps/web/src/content/docs/docs/evaluation/eval-cases.mdx @@ -159,6 +159,8 @@ tests: The `metadata` field is included in the stdin JSON passed to lifecycle commands as `case_metadata`. Operational checkout state belongs under `workspace.repos[].checkout.base_commit`; `metadata.base_commit` is informational only. +For benchmark task packs with source pins, patches, generated rows, and +supporting files, see [Benchmark Provenance](/docs/guides/benchmark-provenance/). ## Per-Test Assertions @@ -389,7 +391,7 @@ tests: ## Metadata -Pass additional context to graders via the `metadata` field: +Pass additional context through the `metadata` field: ```yaml tests: @@ -400,3 +402,8 @@ tests: difficulty: medium input: Write a function to sort a list ``` + +`metadata` is passed to workspace lifecycle hooks as `case_metadata`, preserved +in result records, and available to in-process custom assertions. AgentV does +not interpret arbitrary metadata keys itself; use `workspace`, `execution`, +`input`, `expected_output`, and `assertions` for operational behavior. diff --git a/apps/web/src/content/docs/docs/evaluation/eval-files.mdx b/apps/web/src/content/docs/docs/evaluation/eval-files.mdx index c2b6e6817..d85a993c9 100644 --- a/apps/web/src/content/docs/docs/evaluation/eval-files.mdx +++ b/apps/web/src/content/docs/docs/evaluation/eval-files.mdx @@ -217,6 +217,8 @@ input: Fix the null check bug in parser.ts - **Suite-level config applies:** Suite-level `assertions`, `input`, `workspace`, and `execution` still apply to directory-discovered cases This pattern is useful for benchmarks with many cases, where each case benefits from its own directory for workspace templates, supporting files, or documentation. +For guidance on keeping provenance metadata, patches, oracle files, and generated +dataset rows out of oversized inline YAML, see [Benchmark Provenance](/docs/guides/benchmark-provenance/). ## Environment Variable Interpolation diff --git a/apps/web/src/content/docs/docs/guides/benchmark-provenance.mdx b/apps/web/src/content/docs/docs/guides/benchmark-provenance.mdx new file mode 100644 index 000000000..7c07d4b7b --- /dev/null +++ b/apps/web/src/content/docs/docs/guides/benchmark-provenance.mdx @@ -0,0 +1,264 @@ +--- +title: Benchmark Provenance +description: Patterns for source pins, task artifacts, hooks, and generated benchmark metadata. +sidebar: + order: 5 +--- + +Benchmark suites usually need more than a prompt and a score. They carry source +pins, task patches, generated dataset rows, oracle data, setup scripts, and +verification commands. AgentV represents that with existing primitives: + +- Put runtime behavior in `workspace`, `execution`, `input`, `expected_output`, + and `assertions`. +- Put provenance and classification in per-case `metadata`. +- Put bulky per-case artifacts in case directories and supporting files. + +These are documentation patterns, not special runtime schema keys. AgentV does +not interpret keys such as `source_commit`, `test_patch`, or `question_type` +unless your hook or custom assertion reads them. + +## Operational vs Informational Fields + +Use this split when deciding where a benchmark key belongs: + +| Field area | Operational? | What AgentV does | +|------------|--------------|------------------| +| `workspace.repos[]` | Yes | Clones or copies repositories and checks out the configured refs. | +| `workspace.template` | Yes | Copies a workspace template into the run workspace. | +| `workspace.hooks` | Yes | Runs lifecycle commands with workspace and case context on stdin. | +| `workspace.isolation`, `workspace.mode`, `workspace.path` | Yes | Controls workspace reuse and materialization. | +| `execution` | Yes | Selects targets, thresholds, dependencies, and default grader behavior. | +| `input`, `input_files`, `expected_output` | Yes | Builds the target prompt and passive reference answer. | +| `assertions` | Yes | Runs deterministic, LLM, composite, or code graders. | +| Top-level `name`, `version`, `tags`, `license`, `requires` | Informational | Identifies and categorizes the suite. | +| `tests[].metadata` | Informational to AgentV | Passes arbitrary case data through to results and hook stdin; in-process custom assertions can also read it. | + +`metadata` can still become operational inside your own hook scripts. For +example, a `before_each` hook can read `case_metadata.test_patch` and apply that +patch before the agent starts. The distinction is that AgentV itself only passes +the metadata along; the script owns the behavior. + +## Hook Payloads + +Lifecycle hooks receive JSON on stdin. Case-scoped hooks such as per-test +`before_all`, `before_each`, and `after_each` receive the current test's +metadata as `case_metadata`: + +```json +{ + "workspace_path": "/home/user/.agentv/workspaces/run-123/case-01", + "test_id": "case-01", + "eval_run_id": "run-123", + "case_input": "Fix the bug", + "case_metadata": { + "source_commit": "4f3e2d1", + "test_patch": "cases/case-01/test.patch" + } +} +``` + +Suite-level `before_all` hooks run once for the workspace, before any one test is +selected, so they should do suite setup only. Use `before_each` when setup depends +on per-case metadata such as a patch path, source row, or selected test list. + +## Task Artifact Anatomy + +Benchmark task packs map cleanly onto AgentV fields: + +| Task artifact | AgentV pattern | +|---------------|----------------| +| Prompt or instruction | `input`, usually with `type: file` blocks for long prompts | +| Source checkout | `workspace.repos[].source` and `workspace.repos[].checkout` | +| Per-case setup | `workspace.hooks.before_each` reading `case_metadata` | +| Gold answer | `expected_output` when the answer is passive reference data | +| Active verification | `assertions`, especially `code-grader` for commands or artifact checks | +| Provenance | `tests[].metadata` with source pins, generator rows, and curation labels | +| Bulky task files | `tests: ./cases/` with per-case directories and supporting files | + +This mirrors the common task shape used by filesystem-native benchmark harnesses: +Margin keeps each task's prompt, case metadata, tests, environment, and optional +oracle in a case directory; Terminal-Bench and Harbor keep task instructions, +container setup, run-test scripts, and result artifacts as separate files. In +AgentV, keep the same separation but bind it with eval YAML instead of adding a +large benchmark-specific schema. + +## SWE-Style Case + +A SWE-style benchmark usually needs a source repo, a commit pin, a patch that +adds or selects tests, and a list of failing tests that should pass after the +agent's fix. Keep the checkout operational under `workspace.repos`; keep the +benchmark provenance and per-case test selectors in `metadata`. + +```yaml +name: swe-style-regression +description: Regression tasks against pinned source commits. + +workspace: + isolation: per_test + repos: + - path: ./repo + source: + type: git + url: https://github.com/example/widget.git + checkout: + ref: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123 + clone: + depth: 1 + hooks: + before_each: + command: ["python", "./scripts/apply-test-patch.py"] + timeout_ms: 120000 + after_each: + reset: strict + +assertions: + - name: focused-tests + type: code-grader + command: ["python", "./graders/run-focused-tests.py"] + required: true + +tests: + - id: widget-1234 + criteria: Fix the widget parser regression without breaking existing behavior. + input: | + Work in repo/. Fix the parser regression described by the failing tests. + Do not change unrelated public APIs. + metadata: + repo_url: https://github.com/example/widget.git + source_commit: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123 + test_patch: cases/widget-1234/test.patch + fail_to_pass_tests: + - tests/parser.test.ts::handles-empty-widget + - tests/parser.test.ts::preserves-widget-id +``` + +In this example, `workspace.repos[].checkout.ref` is the actual checkout. The +matching `metadata.source_commit` is audit data that gets recorded with the case +and is available to scripts. `apply-test-patch.py` can read +`case_metadata.test_patch` and `case_metadata.fail_to_pass_tests`, then apply +the patch and write the selected test list into the workspace. The code grader +can read that workspace file through its `workspace_path` payload. + +## Finance-Style Generated Dataset + +Generated datasets often need stable row provenance more than workspace setup. +Keep the generated row identity in metadata, use `expected_output` for the gold +answer, and score with rubrics or an LLM/code grader. + +```yaml +name: finance-research-generated +description: Generated finance research cases with row-level provenance. + +assertions: + - name: answer-quality + type: llm-grader + prompt: ./graders/finance-answer.md + required: true + +tests: + - id: finance-agent-row-0042 + criteria: Answer the finance question with the correct conclusion and evidence. + input: | + Research the company filing and answer: + What drove the year-over-year change in gross margin? + expected_output: + - role: assistant + content: | + Gross margin improved because product mix shifted toward higher-margin + software revenue while fulfillment costs declined. + metadata: + source_repo: https://github.com/example/finance-research-dataset.git + source_commit: 05b8b2e9f071e8d0a6f1c2b3d4e5f60718293abc + source_file: data/generated/finance_agent.csv + source_row: 42 + question_type: margin_analysis +``` + +Here, `source_repo`, `source_commit`, `source_file`, `source_row`, and +`question_type` are informational metadata. They support audits, slices, and +regeneration checks. If a hook or grader needs the source file at runtime, clone +it through `workspace.repos` or make the generator output available as a normal +fixture file. + +## When to Split Into Case Directories + +Inline YAML is fine when a case has a short prompt, a short expected answer, and +a few metadata fields. Move away from inline YAML when the benchmark starts +accumulating task-local artifacts: + +- The case has patches, hidden tests, oracle JSON, screenshots, reports, or + fixture files. +- The prompt or expected output is long enough that YAML diffs become hard to + review. +- Each task needs a different workspace template or setup files. +- A generator emits many rows and reviewers need to inspect individual cases. +- Hook and grader scripts need stable file paths for per-case resources. + +Use an external YAML or JSONL file for many simple generated rows: + +```yaml +name: generated-finance +tests: ./cases.jsonl +``` + +Use case directories when each case needs supporting files: + +```text +swe-benchmark/ + EVAL.yaml + cases/ + widget-1234/ + case.yaml + prompt.md + test.patch + oracle.json + workspace/ + README.md +``` + +```yaml +# EVAL.yaml +name: swe-benchmark +workspace: + repos: + - path: ./repo + source: { type: git, url: https://github.com/example/widget.git } + checkout: { ref: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123 } +tests: ./cases/ +``` + +```yaml +# cases/widget-1234/case.yaml +criteria: Fix the widget parser regression. +input: + - role: user + content: + - type: file + value: cases/widget-1234/prompt.md +metadata: + repo_url: https://github.com/example/widget.git + source_commit: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123 + test_patch: cases/widget-1234/test.patch + oracle_file: cases/widget-1234/oracle.json +``` + +When `tests` points to a directory, AgentV discovers each immediate +subdirectory's `case.yaml`, uses the directory name as `id` if no `id` is set, +and automatically uses a `workspace/` subdirectory as that case's +`workspace.template`. File blocks still use the normal eval-file search roots, +so include the case directory in paths such as `cases/widget-1234/prompt.md`. +Metadata paths are not resolved by AgentV; resolve them in your hook or grader +script. + +## Authoring Rules + +- Do not add benchmark-specific fields when `metadata` plus hooks or custom + assertions can express the need. +- Do not duplicate operational checkout state only in metadata. Put the real + checkout under `workspace.repos`. +- Keep `metadata` snake_case because it crosses process and result boundaries. +- Prefer `expected_output` for passive gold answers and `code-grader` for active + commands, file checks, or generated artifact validation. +- Prefer case directories over long inline YAML once task artifacts become part + of the benchmark contract.