From cf8cbdf1e88a02abc25700a4c6270b1ac731fa7e Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Wed, 1 Jul 2026 04:28:56 +0200 Subject: [PATCH 1/5] docs(adr): stabilize eval authoring contract --- ...chmark-execution-behind-runner-boundary.md | 10 +- ...arate-experiments-from-eval-definitions.md | 11 +- ...benchmark-schema-on-existing-primitives.md | 10 +- .../0013-stabilize-eval-authoring-contract.md | 183 ++++++++++++++++++ 4 files changed, 201 insertions(+), 13 deletions(-) create mode 100644 docs/adr/0013-stabilize-eval-authoring-contract.md diff --git a/docs/adr/0002-keep-harbor-benchmark-execution-behind-runner-boundary.md b/docs/adr/0002-keep-harbor-benchmark-execution-behind-runner-boundary.md index 32b9ad02f..f44726b66 100644 --- a/docs/adr/0002-keep-harbor-benchmark-execution-behind-runner-boundary.md +++ b/docs/adr/0002-keep-harbor-benchmark-execution-behind-runner-boundary.md @@ -6,10 +6,12 @@ Date: 2026-06-17 Proposed -Superseded for eval authoring placement on 2026-06-30 by GitHub issue #1575 / -Bead `av-ogpn.1`: top-level `experiment:` is no longer an authored eval YAML -field. Use top-level `target` and `policy`; keep Harbor-specific runner options -behind adapter boundaries. +Superseded for the current eval authoring contract by +[ADR 0013](0013-stabilize-eval-authoring-contract.md): top-level +`experiment:` remains the optional string run/result grouping label, runtime +policy stays in explicit top-level fields such as `target`, `repeat`, +`timeout_seconds`, `budget_usd`, and `gate`, and Harbor-specific execution +still belongs behind adapter boundaries. ## Context diff --git a/docs/adr/0006-separate-experiments-from-eval-definitions.md b/docs/adr/0006-separate-experiments-from-eval-definitions.md index 0b7d42dd3..d37d8dc1d 100644 --- a/docs/adr/0006-separate-experiments-from-eval-definitions.md +++ b/docs/adr/0006-separate-experiments-from-eval-definitions.md @@ -17,11 +17,12 @@ Partially superseded by [ADR 0009](0009-eval-path-result-identity-and-default-experiment.md) for result experiment bucket precedence, result row identity, and run bundle path naming. -Partially superseded on 2026-06-30 by GitHub issue #1575 / Bead `av-ogpn.1`: -top-level `experiment:` is no longer an authored eval YAML field. The eval file -defines the experiment; top-level `name` is the result namespace, top-level -`target` identifies the system under test, and top-level `policy` owns -runtime/gating controls. +Superseded for the current eval authoring contract by +[ADR 0013](0013-stabilize-eval-authoring-contract.md): top-level +`experiment:` remains the optional string run/result grouping label, top-level +authoring `tags` are removed from the preferred contract, `cases` / `case_id` +replace `tests` / `test_id` as preferred vocabulary, and top-level `gate` +replaces scalar `threshold`. ## Context diff --git a/docs/adr/0009-keep-benchmark-schema-on-existing-primitives.md b/docs/adr/0009-keep-benchmark-schema-on-existing-primitives.md index 0b625b78c..242de16d5 100644 --- a/docs/adr/0009-keep-benchmark-schema-on-existing-primitives.md +++ b/docs/adr/0009-keep-benchmark-schema-on-existing-primitives.md @@ -6,10 +6,12 @@ Date: 2026-06-27 Proposed -Superseded for eval authoring placement on 2026-06-30 by GitHub issue #1575 / -Bead `av-ogpn.1`: top-level `experiment:` is no longer an authored eval YAML -field. Use top-level `target` and `policy`; keep the Dashboard/result -"Experiment" concept as a derived grouping label. +Superseded for the current eval authoring contract by +[ADR 0013](0013-stabilize-eval-authoring-contract.md): top-level +`experiment:` remains the optional string run/result grouping label, top-level +authoring `tags` are removed from the preferred contract, `cases` / `case_id` +replace `tests` / `test_id` as preferred vocabulary, and top-level `gate` +replaces scalar `threshold`. ## Context diff --git a/docs/adr/0013-stabilize-eval-authoring-contract.md b/docs/adr/0013-stabilize-eval-authoring-contract.md new file mode 100644 index 000000000..acc1fdfff --- /dev/null +++ b/docs/adr/0013-stabilize-eval-authoring-contract.md @@ -0,0 +1,183 @@ +# 13. Stabilize eval authoring around experiment, cases, and gate + +Date: 2026-07-01 + +## Status + +Accepted + +Supersedes the eval-authoring placement portions of +[ADR 0002](0002-keep-harbor-benchmark-execution-behind-runner-boundary.md), +[ADR 0006](0006-separate-experiments-from-eval-definitions.md), and +[ADR 0009](0009-keep-benchmark-schema-on-existing-primitives.md) that moved the +run label out of top-level `experiment`, introduced `policy` as the preferred +runtime wrapper, or treated authoring `tags` as a first-class eval-selection +surface. + +Complements [ADR 0012](0012-finalize-run-artifact-layout.md), which keeps +`experiment` as run metadata rather than a result path namespace. + +## Context + +AgentV's eval YAML needs to stay small enough for humans and agents to author +reliably while still supporting repo-native workspaces, real targets, repeat +runs, gating, and Dashboard comparison. + +Recent same-week proposals tried a few competing names: + +- removing top-level `experiment` in favor of `policy` or derived labels; +- adding or preserving top-level `tags` as another grouping primitive; +- keeping `tests` / `test_id` as the public case vocabulary; +- keeping scalar `threshold` as the CI gate. + +Those proposals made the contract less direct. Public peer systems also support +keeping `experiment` as the grouping concept: Vercel `agent-eval` names +experiment config files and result groups as experiments, while Convex Evals +stores and aggregates runs by experiment. In contrast, `tags` is usually a +many-valued classification or annotation mechanism, not the single condition +being compared. Agentskills' skill-eval examples call `with_skill` and +`without_skill` configurations, which is useful language for that project but +does not justify replacing AgentV's existing Dashboard and result vocabulary. + +AgentV already displays and queries run groups as experiments in Dashboard, +results APIs, compare flows, and CLI `--experiment`. Renaming that axis now +would create churn without simplifying the product. + +## Decision + +The preferred eval authoring contract is: + +```yaml +name: code-generation-quality +experiment: backend-with-skills +target: copilot-sdk +repeat: + count: 3 + strategy: pass_any + early_exit: false +timeout_seconds: 600 +budget_usd: 5 +gate: + min_case_pass_rate: 0.95 + max_execution_errors: 0 +cases: + - id: fizzbuzz + input: Write FizzBuzz in Python + assertions: + - type: contains + value: "fizz" +``` + +`name` is the optional top-level suite display name. When omitted, AgentV should +derive the display name from the eval file basename by removing the `.eval.yaml` +suffix, for example `code-generation-quality.eval.yaml` becomes +`code-generation-quality`. Suite `name` is metadata for display and reporting; +it must not drive run identity, experiment grouping, case selection, gating, +artifact routing, cache keys that should track executable behavior, or result +comparison semantics. Source identity belongs to `eval_path` and run metadata, +not to the display name. + +`experiment` remains the optional top-level string run/result grouping label. +It names the condition being measured, such as `baseline`, `candidate`, +`with-skills`, or `without-skills`. It is not a runtime-policy object, not a +separate artifact type, and not a storage path namespace. + +Top-level `description` is not part of the preferred eval authoring contract. +Existing files that contain it may be read as legacy display metadata, but it +must be ignored for run identity, experiment grouping, case selection, gating, +artifact routing, cache keys that should track executable behavior, and result +comparison semantics. + +`target` remains the system under test. Do not rename it to `agent`; AgentV +targets can be agents, model providers, gateways, replay targets, CLI wrappers, +transcript providers, or future service wrappers. + +`cases` is the preferred authored collection name, and each authored case uses +`id`. Inside `cases[]`, `case_id` is redundant because the object is already +case-scoped. + +`case_id` is the preferred flattened identity field where a record is not +already scoped to one case, including `index.jsonl`, Dashboard/API payloads, +gate command input, and other result rows. CLI filters use `--case-id` for the +same reason: the flag sits beside other dimensions such as eval path, target, +run, and project. + +`tests` and `test_id` are legacy compatibility names only. If an eval file uses +both `cases` and `tests`, validation should reject the file with an explicit +conflict instead of merging them. + +`gate` replaces scalar `threshold` in the preferred schema. Gate is a top-level +suite/run policy that evaluates the completed run. It is not an assertion and +not a per-case inline field. The v1 built-in surface is intentionally small: + +```yaml +gate: + min_case_pass_rate: 0.95 + max_execution_errors: 0 + command: ["bun", "./gates/case-policy.ts"] + timeout_ms: 60000 +``` + +The executable gate command receives structured completed-run JSON on stdin and +emits structured JSON with `passed`, optional `failures`, and optional +`warnings`. Non-zero exit, invalid JSON, or timeout is recorded as a gate +execution error distinct from assertion or grader failures. + +Top-level eval authoring `tags` are removed from the preferred contract. Do not +teach `tags` as a first-class eval YAML grouping or selection field. Use +repo paths, categories, eval file organization, `experiment`, `target`, and +explicit case metadata for durable identity and filtering needs. + +Mutable Dashboard or result annotations may still use tags as result metadata, +for example a local `tags.json` sidecar on a run bundle. Those tags are user +annotations over completed results, not authored eval inputs, and they must not +be confused with eval YAML schema. + +## Compatibility + +This ADR defines the preferred contract. Implementation work must still make an +explicit compatibility decision for existing shipped fields: + +- `tests` should remain readable as the legacy authored collection name during + the case-vocabulary migration, with `cases` preferred in new docs. +- `tests[].id` remains readable for legacy authored cases; new authored cases + should use `cases[].id`. +- `test_id` should remain readable for legacy flattened result rows, with + `case_id` preferred in new artifact/API/gate rows. +- `--test-id` should remain a deprecated alias for `--case-id` until the CLI + compatibility window is closed. +- `threshold` should be removed from examples and preferred schema docs, then + either hard-corrected or deprecated based on release evidence. +- Existing result tags and Dashboard tag mutation are out of scope for eval YAML + removal. They remain a result-annotation feature unless a separate ADR removes + them. + +## Consequences + +Positive: + +- AgentV keeps the same experiment vocabulary across CLI, Dashboard, run + metadata, compare, and peer-framework comparisons. +- The eval YAML contract is flatter and easier for coding agents to author. +- Tags stop competing with path/category and experiment grouping. +- Gating becomes a clear release-policy step over completed run artifacts rather + than a scalar hidden among runtime knobs. + +Negative: + +- Older ADRs and examples that mention `policy`, top-level `tags`, `tests`, or + scalar `threshold` need cleanup or explicit supersession notes. +- Implementations need compatibility readers until current users and run + artifacts have migrated. +- Teams that used eval authoring tags for ad hoc grouping need to move that + grouping to paths, metadata, or result annotations. + +## Non-Goals + +- Implementing the parser, artifact, CLI, or Dashboard migration in this ADR. +- Removing mutable Dashboard/result tags. +- Adding a `benchmark` top-level wrapper. +- Adding per-case `gate` fields. +- Replacing `target` with `agent`. +- Renaming `grader` to `scorer`. +- Introducing a separate `experiment.yaml` artifact. From 7d4113d47275b6efa2b70b392b6511d0bb6ee1bf Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Wed, 1 Jul 2026 06:33:39 +0200 Subject: [PATCH 2/5] docs: clarify eval experiment labels --- README.md | 55 +++++++++++-------- .../0013-stabilize-eval-authoring-contract.md | 54 +++++++++--------- 2 files changed, 59 insertions(+), 50 deletions(-) diff --git a/README.md b/README.md index 6090e3f0b..a36882033 100644 --- a/README.md +++ b/README.md @@ -18,8 +18,9 @@ Test AI targets on real repo tasks and measure what actually works. - **Category** is derived from where the eval lives, such as folder path and file name. Use paths to organize the corpus instead of repeating category labels in every eval. - **Workspace / fixtures / graders** are task-owned context: repos, setup scripts, files, fixtures, isolation, deterministic checks, and LLM grading prompts. - **Target** is the system under test: an agent, provider, gateway, replay target, CLI wrapper, transcript provider, or future app/service wrapper. Each eval selects one `target`, either by name from `targets.yaml` or with an eval-local target object. -- **Experiment** is the run/result grouping label being measured over that corpus, such as `backend-with-skills` or `backend-without-skills`. -- **Per-test defaults / run controls** configure inherited score cutoffs, repeats, timeouts, budgets, and completion hooks with fields such as `default_test.threshold`, `repeat`, `timeout_seconds`, `budget_usd`, and `on_run_complete`. +- **Experiment** is the run/result grouping label being measured over that corpus, such as `with-skills` or `without-skills`. Keep suite/category and target/model names out of this label. +- **Evaluate options** configure runner-level behavior such as repeat policy and optional timeouts under `evaluate_options`. +- **Default test** configures inherited per-test defaults such as score `threshold`. - **Run** is one concrete execution of an experiment against a resolved target that writes portable artifacts for readers such as Dashboard, compare, and trend. ```mermaid @@ -53,21 +54,29 @@ npm install -g agentv agentv init ``` -**2. Configure targets** in `.agentv/targets.yaml` — point to the system under test, such as an agent, provider, gateway, replay source, or CLI wrapper. +**2. Configure targets** in `.agentv/targets.yaml` — point to the system under test, such as an agent, provider, gateway, replay source, or CLI wrapper. Provider-specific budgets belong here: + +```yaml +targets: + - name: copilot-sdk + provider: anthropic + model: claude-sonnet-4.6 + max_budget_usd: 0.50 +``` **3. Create an eval** in `evals/`: ```yaml description: Code generation quality -experiment: backend-with-skills +experiment: with-skills target: copilot-sdk -repeat: - count: 3 - strategy: pass_any - early_exit: false -timeout_seconds: 600 +evaluate_options: + repeat: + count: 3 + strategy: pass_any + early_exit: false + default_test: threshold: 0.8 -budget_usd: 5 workspace: isolation: per_case @@ -88,16 +97,16 @@ tests: The target can be an eval-local object when this eval needs target settings of its own: ```yaml -description: Code generation quality with GPT-5 target settings -experiment: backend-with-skills-gpt5 +description: Code generation quality with Copilot target settings +experiment: with-skills target: - extends: codex-gpt5 - model: gpt-5.1 - reasoning_effort: high -repeat: - count: 2 - strategy: pass_any -timeout_seconds: 900 + extends: copilot-sdk + model: claude-sonnet-4.6 +evaluate_options: + repeat: + count: 2 + strategy: pass_any + default_test: threshold: 0.85 @@ -106,7 +115,7 @@ tests: input: Write FizzBuzz in Python ``` -`target: codex-gpt5` resolves the named target from `.agentv/targets.yaml` or `targets.yaml` and uses its default provider, model, hooks, and provider settings. The object form above starts from `codex-gpt5`, then applies the eval-local fields for this eval. If `extends` is omitted, the object defines the full target inline and must include enough provider configuration to run. AgentV records the resolved target information in run artifacts so results can be audited and replayed. +`target: copilot-sdk` resolves the named target from `.agentv/targets.yaml` or `targets.yaml` and uses its default provider, model, hooks, and provider settings. The object form above starts from `copilot-sdk`, then applies the eval-local fields for this eval. If `extends` is omitted, the object defines the full target inline and must include enough provider configuration to run. AgentV records the resolved target information in run artifacts so results can be audited and replayed. The experiment label stays `with-skills` because the condition is unchanged; the model/provider variation belongs to the resolved target metadata. Use `default_test.threshold` for the inherited per-test pass cutoff. Existing eval files with a top-level `threshold` still load during migration, and `--threshold` on the CLI still overrides YAML thresholds for a run. @@ -122,7 +131,7 @@ agentv compare .agentv/results//index.jsonl .agentv/results//`. In this example, `experiment: backend-with-skills` names the condition being measured and `target: copilot-sdk` selects the system under test from `targets.yaml`; both are recorded as metadata, not path segments. The root `index.jsonl` manifest is the portable row index used by scripts, CI, and `agentv compare`; per-case sidecars include the resolved eval and target configuration used for the run. +Each run writes a portable bundle directly under `.agentv/results//`. In this example, `experiment: with-skills` names the condition being measured and `target: copilot-sdk` selects the system under test from `targets.yaml`; both are recorded as metadata, not path segments. The root `index.jsonl` manifest is the portable row index used by scripts, CI, and `agentv compare`; per-case sidecars include the resolved eval and target configuration used for the run. ```bash agentv eval evals/my-eval.yaml @@ -162,7 +171,7 @@ Use `evaluate()` when your application owns the run: import { evaluate } from '@agentv/sdk'; const { results, summary } = await evaluate({ - experiment: 'backend-with-skills', + experiment: 'with-skills', task: async (input) => runMyAppTarget(input), threshold: 0.8, tests: [ @@ -189,7 +198,7 @@ import { defineEval } from '@agentv/sdk'; export default defineEval({ description: 'Code generation quality', - experiment: 'backend-with-skills', + experiment: 'with-skills', target: { extends: 'copilot-sdk', model: 'claude-sonnet-4.6', diff --git a/docs/adr/0013-stabilize-eval-authoring-contract.md b/docs/adr/0013-stabilize-eval-authoring-contract.md index acc1fdfff..b8aa62ba3 100644 --- a/docs/adr/0013-stabilize-eval-authoring-contract.md +++ b/docs/adr/0013-stabilize-eval-authoring-contract.md @@ -1,4 +1,4 @@ -# 13. Stabilize eval authoring around experiment, cases, and gate +# 13. Stabilize eval authoring around experiment, tests, and gate Date: 2026-07-01 @@ -27,13 +27,17 @@ Recent same-week proposals tried a few competing names: - removing top-level `experiment` in favor of `policy` or derived labels; - adding or preserving top-level `tags` as another grouping primitive; -- keeping `tests` / `test_id` as the public case vocabulary; +- renaming `tests` / `test_id` to `cases` / `case_id`; - keeping scalar `threshold` as the CI gate. Those proposals made the contract less direct. Public peer systems also support keeping `experiment` as the grouping concept: Vercel `agent-eval` names experiment config files and result groups as experiments, while Convex Evals -stores and aggregates runs by experiment. In contrast, `tags` is usually a +stores and aggregates runs by experiment. Promptfoo also keeps the authored +collection as `tests` while describing each item as a test case; AgentV's test +objects have richer workspace and harness semantics, but the suite-file -> +`tests[]` -> test-case hierarchy is still close enough that renaming the public +field would create more churn than clarity. In contrast, `tags` is usually a many-valued classification or annotation mechanism, not the single condition being compared. Agentskills' skill-eval examples call `with_skill` and `without_skill` configurations, which is useful language for that project but @@ -49,7 +53,7 @@ The preferred eval authoring contract is: ```yaml name: code-generation-quality -experiment: backend-with-skills +experiment: with-skills target: copilot-sdk repeat: count: 3 @@ -60,7 +64,7 @@ budget_usd: 5 gate: min_case_pass_rate: 0.95 max_execution_errors: 0 -cases: +tests: - id: fizzbuzz input: Write FizzBuzz in Python assertions: @@ -80,7 +84,9 @@ not to the display name. `experiment` remains the optional top-level string run/result grouping label. It names the condition being measured, such as `baseline`, `candidate`, `with-skills`, or `without-skills`. It is not a runtime-policy object, not a -separate artifact type, and not a storage path namespace. +separate artifact type, and not a storage path namespace. It should not repeat +the suite name, category, target, provider, or model; those are separate +dimensions in run metadata. Top-level `description` is not part of the preferred eval authoring contract. Existing files that contain it may be read as legacy display metadata, but it @@ -92,19 +98,16 @@ comparison semantics. targets can be agents, model providers, gateways, replay targets, CLI wrappers, transcript providers, or future service wrappers. -`cases` is the preferred authored collection name, and each authored case uses -`id`. Inside `cases[]`, `case_id` is redundant because the object is already -case-scoped. +`tests` remains the preferred authored collection name, and each authored test +case uses `id`. AgentV may use "case" or "test case" in prose when describing an +individual row, but the wire contract remains `tests[]` because that matches the +existing AgentV surface and the common suite-file -> tests -> test-case hierarchy +in peer declarative eval tools. -`case_id` is the preferred flattened identity field where a record is not -already scoped to one case, including `index.jsonl`, Dashboard/API payloads, -gate command input, and other result rows. CLI filters use `--case-id` for the -same reason: the flag sits beside other dimensions such as eval path, target, -run, and project. - -`tests` and `test_id` are legacy compatibility names only. If an eval file uses -both `cases` and `tests`, validation should reject the file with an explicit -conflict instead of merging them. +`test_id` remains the preferred flattened identity field where a record is not +already scoped to one test case, including `index.jsonl`, Dashboard/API +payloads, gate command input, and other result rows. CLI filters continue to use +`--test-id`. `gate` replaces scalar `threshold` in the preferred schema. Gate is a top-level suite/run policy that evaluates the completed run. It is not an assertion and @@ -138,14 +141,11 @@ be confused with eval YAML schema. This ADR defines the preferred contract. Implementation work must still make an explicit compatibility decision for existing shipped fields: -- `tests` should remain readable as the legacy authored collection name during - the case-vocabulary migration, with `cases` preferred in new docs. -- `tests[].id` remains readable for legacy authored cases; new authored cases - should use `cases[].id`. -- `test_id` should remain readable for legacy flattened result rows, with - `case_id` preferred in new artifact/API/gate rows. -- `--test-id` should remain a deprecated alias for `--case-id` until the CLI - compatibility window is closed. +- `tests` remains the preferred authored collection name. +- `tests[].id` remains the preferred authored test-case identifier. +- `test_id` remains the preferred flattened result/API/gate identity field. +- `--test-id` remains the preferred CLI filter for selecting authored test + cases. - `threshold` should be removed from examples and preferred schema docs, then either hard-corrected or deprecated based on release evidence. - Existing result tags and Dashboard tag mutation are out of scope for eval YAML @@ -165,7 +165,7 @@ Positive: Negative: -- Older ADRs and examples that mention `policy`, top-level `tags`, `tests`, or +- Older ADRs and examples that mention `policy`, top-level `tags`, `cases`, or scalar `threshold` need cleanup or explicit supersession notes. - Implementations need compatibility readers until current users and run artifacts have migrated. From 0955665e83b3c2bece710911b7890ba3819de7ab Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Wed, 1 Jul 2026 06:39:24 +0200 Subject: [PATCH 3/5] docs: remove arbitrary suite timeouts --- docs/adr/0013-stabilize-eval-authoring-contract.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/adr/0013-stabilize-eval-authoring-contract.md b/docs/adr/0013-stabilize-eval-authoring-contract.md index b8aa62ba3..4ff51d354 100644 --- a/docs/adr/0013-stabilize-eval-authoring-contract.md +++ b/docs/adr/0013-stabilize-eval-authoring-contract.md @@ -59,7 +59,6 @@ repeat: count: 3 strategy: pass_any early_exit: false -timeout_seconds: 600 budget_usd: 5 gate: min_case_pass_rate: 0.95 From 8908446dcbd06bc9f41d4fe1401fefbc1d2c0717 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Wed, 1 Jul 2026 06:47:22 +0200 Subject: [PATCH 4/5] docs: introduce default test threshold contract --- .../0013-stabilize-eval-authoring-contract.md | 39 ++++++++++++++----- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/docs/adr/0013-stabilize-eval-authoring-contract.md b/docs/adr/0013-stabilize-eval-authoring-contract.md index 4ff51d354..3ff7a9523 100644 --- a/docs/adr/0013-stabilize-eval-authoring-contract.md +++ b/docs/adr/0013-stabilize-eval-authoring-contract.md @@ -55,13 +55,16 @@ The preferred eval authoring contract is: name: code-generation-quality experiment: with-skills target: copilot-sdk -repeat: - count: 3 - strategy: pass_any - early_exit: false -budget_usd: 5 +evaluate_options: + repeat: + count: 3 + strategy: pass_any + early_exit: false + budget_usd: 5 +default_test: + threshold: 0.8 gate: - min_case_pass_rate: 0.95 + min_test_pass_rate: 0.95 max_execution_errors: 0 tests: - id: fizzbuzz @@ -97,6 +100,20 @@ comparison semantics. targets can be agents, model providers, gateways, replay targets, CLI wrappers, transcript providers, or future service wrappers. +`evaluate_options` is the preferred home for runner-level execution controls. +This follows promptfoo's separation between test definitions and evaluation +runner options while keeping AgentV's field names in snake_case. The v1 preferred +surface is intentionally small: `repeat`, `budget_usd`, and optional timeout +fields when a suite needs a finite cap. Omitted timeout fields mean unlimited by +default. Concurrency remains an operator/project/target setting, not authored +eval YAML. + +`default_test` is the preferred home for inherited per-test defaults. The first +default worth standardizing is `threshold`, because it is a per-test score cutoff +used to classify individual results, not a suite-level release gate. Additional +defaults such as inherited assertions may be added only when implementation work +defines clear merge and override rules. + `tests` remains the preferred authored collection name, and each authored test case uses `id`. AgentV may use "case" or "test case" in prose when describing an individual row, but the wire contract remains `tests[]` because that matches the @@ -110,11 +127,11 @@ payloads, gate command input, and other result rows. CLI filters continue to use `gate` replaces scalar `threshold` in the preferred schema. Gate is a top-level suite/run policy that evaluates the completed run. It is not an assertion and -not a per-case inline field. The v1 built-in surface is intentionally small: +not a per-test inline field. The v1 built-in surface is intentionally small: ```yaml gate: - min_case_pass_rate: 0.95 + min_test_pass_rate: 0.95 max_execution_errors: 0 command: ["bun", "./gates/case-policy.ts"] timeout_ms: 60000 @@ -145,8 +162,10 @@ explicit compatibility decision for existing shipped fields: - `test_id` remains the preferred flattened result/API/gate identity field. - `--test-id` remains the preferred CLI filter for selecting authored test cases. -- `threshold` should be removed from examples and preferred schema docs, then - either hard-corrected or deprecated based on release evidence. +- Top-level `repeat`, `budget_usd`, and timeout fields should remain readable + during migration, with `evaluate_options` preferred in new docs. +- Top-level `threshold` should remain readable during migration, with + `default_test.threshold` preferred in new docs. - Existing result tags and Dashboard tag mutation are out of scope for eval YAML removal. They remain a result-annotation feature unless a separate ADR removes them. From bf90b4e3ca95704de34c370f7c6d8cb87c22ca61 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Wed, 1 Jul 2026 06:50:11 +0200 Subject: [PATCH 5/5] docs: move budgets to target config --- README.md | 2 -- .../0013-stabilize-eval-authoring-contract.md | 19 ++++++++++++------- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index a36882033..cf44fb95f 100644 --- a/README.md +++ b/README.md @@ -208,9 +208,7 @@ export default defineEval({ strategy: 'pass_any', earlyExit: false, }, - timeoutSeconds: 600, threshold: 0.8, - budgetUsd: 5, workspace: { isolation: 'per_case', }, diff --git a/docs/adr/0013-stabilize-eval-authoring-contract.md b/docs/adr/0013-stabilize-eval-authoring-contract.md index 3ff7a9523..fc019796f 100644 --- a/docs/adr/0013-stabilize-eval-authoring-contract.md +++ b/docs/adr/0013-stabilize-eval-authoring-contract.md @@ -60,7 +60,6 @@ evaluate_options: count: 3 strategy: pass_any early_exit: false - budget_usd: 5 default_test: threshold: 0.8 gate: @@ -103,10 +102,14 @@ transcript providers, or future service wrappers. `evaluate_options` is the preferred home for runner-level execution controls. This follows promptfoo's separation between test definitions and evaluation runner options while keeping AgentV's field names in snake_case. The v1 preferred -surface is intentionally small: `repeat`, `budget_usd`, and optional timeout -fields when a suite needs a finite cap. Omitted timeout fields mean unlimited by -default. Concurrency remains an operator/project/target setting, not authored -eval YAML. +surface is intentionally small: `repeat` and optional timeout fields when a suite +needs a finite cap. Omitted timeout fields mean unlimited by default. Concurrency +remains an operator/project/target setting, not authored eval YAML. + +Budgets belong with the target/provider runtime configuration, such as +`max_budget_usd` in `targets.yaml`, because cost limits are provider-specific +execution policy rather than suite authoring metadata. Do not teach new eval YAML +examples with top-level `budget_usd` or `evaluate_options.budget_usd`. `default_test` is the preferred home for inherited per-test defaults. The first default worth standardizing is `threshold`, because it is a per-test score cutoff @@ -162,8 +165,10 @@ explicit compatibility decision for existing shipped fields: - `test_id` remains the preferred flattened result/API/gate identity field. - `--test-id` remains the preferred CLI filter for selecting authored test cases. -- Top-level `repeat`, `budget_usd`, and timeout fields should remain readable - during migration, with `evaluate_options` preferred in new docs. +- Top-level `repeat` and timeout fields should remain readable during migration, + with `evaluate_options` preferred in new docs. +- Top-level `budget_usd` should remain readable during migration, but new docs + should put provider/target budgets in `targets.yaml` as `max_budget_usd`. - Top-level `threshold` should remain readable during migration, with `default_test.threshold` preferred in new docs. - Existing result tags and Dashboard tag mutation are out of scope for eval YAML