diff --git a/CONCEPTS.md b/CONCEPTS.md index 972171008..0884cf49f 100644 --- a/CONCEPTS.md +++ b/CONCEPTS.md @@ -18,11 +18,13 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Matrix authoring** — The Promptfoo-compatible shape AgentV adopts where useful: `prompts x tests/vars x targets`, with repeat samples and retries applied as run policy after the authored matrix is resolved. AgentV uses this matrix model without copying Promptfoo wholesale. AgentV-native boundaries remain: `targets` identify systems under test, `provider` names the backend/adapter kind inside a target, `environment` recipes prepare coding-agent testbeds, `env` carries provider/eval variables, `extensions` are lifecycle hooks, reusable content uses field-local `file://` refs, and grouping uses tags plus run-bundle metadata. -**Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `environment`, assertions, expected references, and judge criteria. It can run directly or share reusable parts through field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, `defaults: file://...`, and `environment: file://...`. +**Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `environment`, assertions, expected references, and judge criteria. It runs directly or shares reusable parts through field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, `default_test: file://...`, and `environment: file://...`. -**Raw case file** — YAML, JSONL, or directory case data loaded with `tests: file://./cases.yaml`, string shorthand, or another supported field-local tests reference. Raw cases are reusable data inputs; they do not carry imported suite context such as shared `environment`, shared `input`, or shared `assertions`. +**Raw case file** — YAML, JSONL, or directory case data loaded with `tests: file://./cases.yaml`, string shorthand, or another supported field-local tests reference. Raw cases are reusable data inputs; they do not carry suite context such as shared `environment`, shared `prompts`, or shared `assertions`. -**Policy eval** — Eval YAML whose main job is to bind top-level runtime policy such as target selection, repeat count, timeout, budget, thresholds, and tags around explicit prompts/tests/targets refs. Policy evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. Use tags and run-bundle metadata for grouping rather than experiment path buckets, Vercel path layout, or model-as-experiment grouping. +**Policy eval** — Eval YAML whose main job is to bind top-level runtime policy such as target selection, repeat count, timeout, budget, thresholds, and tags around explicit prompts/tests/targets refs. Policy evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. Use tags and run-bundle metadata for grouping rather than experiment path buckets, Vercel path layout, model-as-experiment grouping, or wrapper-suite import semantics. + +**Multi-file run** — A CLI-selected set of eval YAML files. Use multiple eval files, tags, and CLI run selection to group suites under one run intent without adding wrapper-suite semantics to YAML. **Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Experiment is expressed as the reserved `tags.experiment` key (see **Tags**); there is no top-level `experiment` field. Runtime policy belongs in top-level eval fields or target objects; the experiment label is recorded in `summary.json` and `.internal/index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `extensions` or target hooks, not in a separate experiment artifact. diff --git a/apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx b/apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx index b1e1604c8..d43c58851 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx @@ -197,14 +197,18 @@ tests: ``` The `metadata` field is included in the stdin JSON passed to lifecycle commands as `case_metadata`. -Operational checkout state belongs under `workspace.repos[].commit`; matching metadata fields such as `source_commit` are informational only. -For historical repo-state evals, pin the checkout under `workspace.repos[]` -instead of only mentioning the SHA in prompt prose: +Operational checkout state belongs in `environment`; matching metadata fields +such as `source_commit` are informational only. For historical repo-state evals, +pin the checkout in an environment setup recipe instead of only mentioning the +SHA in prompt prose: ```yaml -workspace: - repos: - - path: ./agentv +environment: + type: host + workdir: ./agentv + setup: + command: ./scripts/materialize-repo.sh + args: repo: https://github.com/EntityProcess/agentv.git commit: 5e3c8f46d80fe66b1a75659e4fd94e38a7e09215 ``` @@ -503,7 +507,7 @@ tests: - Generates valid Python ``` -`metadata` is passed to workspace lifecycle hooks as `case_metadata`, preserved +`metadata` is passed to lifecycle hooks as `case_metadata`, preserved in result records, and available to in-process custom assertions. AgentV does -not interpret arbitrary metadata keys itself; use `workspace`, `execution`, -`input`, `expected_output`, and `assert` for operational behavior. +not interpret arbitrary metadata keys itself; use `environment`, `extensions`, +`prompts`, `expected_output`, and `assert` for operational behavior. diff --git a/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx b/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx index 665981e1f..0db5ad3f7 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx @@ -6,18 +6,17 @@ sidebar: slug: docs/evaluation/eval-files --- -Evaluation files define the test cases, graders, workspace lifecycle, and run +Evaluation files define the test cases, graders, environment, lifecycle hooks, and run controls for an evaluation run. The reserved `tags.experiment` key is the run/result grouping label, top-level `target` identifies the system under test, and fields such as `evaluate_options.repeat`, `threshold`, `timeout_seconds`, `evaluate_options.budget_usd`, and `evaluate_options.max_concurrency` control repeated -attempts and gates. Workspace lifetime belongs under `workspace.scope`; -repository provenance belongs under `workspace.repos`; Docker/container binding -belongs under `workspace.docker`. Non-provisioning setup commands belong in -top-level `extensions`; reset policy stays under -`workspace.hooks.after_each.reset`; runner-specific setup belongs in the -`target` object, in `targets`, or in project config. AgentV supports two eval -data formats: YAML and JSONL. +attempts and gates. Coding-agent testbeds, workdirs, Docker config, repository +materialization, setup, and reset policy belong in `environment`. Provider +environment-variable overrides belong in top-level `env`. Lifecycle hooks belong +in `extensions`; runner-specific setup belongs in the `target` object, in +`targets`, or in project config. AgentV supports two eval data formats: YAML and +JSONL. YAML is the canonical portable model. TypeScript helpers, generated fixtures, and Python scripts should lower to the same YAML/JSONL shapes rather than inventing a separate eval contract. Eval files describe the task, target binding, and run controls. Use @@ -35,41 +34,43 @@ surfaces. Eval YAML is AgentV's composable and runnable authoring primitive. It is a focused, shareable slice of the same config graph as `.agentv/config.yaml`. -Use ordinary `*.eval.yaml` files for direct task suites and for wrapper evals -that compose other suites. Raw case files are reusable data inputs, not a -second runnable experiment format. +Use ordinary `*.eval.yaml` files for direct task suites. Raw case files are +reusable data inputs, not a second runnable experiment format. -- A **task suite** is eval YAML that owns task context: `workspace`, shared - `input`, shared `assert`, fixtures, graders, and test cases. It can run - directly or be imported through `imports.suites`. +- A **task suite** is eval YAML that owns task context: `environment`, shared + `prompts`, shared `assert`, fixtures, graders, and test cases. It can run + directly. - A **raw case file** is a YAML, JSON, JSONL, CSV, script-backed dataset, - directory, or glob of cases. Import it with `imports.tests`, - `tests: ./cases.yaml`, `tests: file://cases.csv`, or string shorthand; parent - suite context applies because raw cases do not carry their own suite context. -- A **wrapper eval** is eval YAML that imports one or more suites with - `imports.suites` and binds run controls with top-level `target`, `threshold`, - `timeout_seconds`, and `evaluate_options`. - Wrapper evals can live anywhere in the repo. A wrapper that imports suites - with `imports.suites` must not define parent `workspace`; imported suites own - task environment. Machine-local existing workspace paths belong in CLI flags - or `config.local.yaml`, not eval YAML. + directory, or glob of cases. Reference it with `tests: ./cases.yaml`, + `tests: file://cases.csv`, or string entries inside `tests`; parent suite + context applies because raw cases do not carry their own suite context. +- Use multiple eval files, tags, and CLI selection when you want to run several + suites together. Each eval file owns its own task environment and run policy. For example, a reusable task suite can keep the task contract in one file: ```yaml # evals/suites/refunds.eval.yaml suite: refunds -workspace: - repos: - - path: ./support-app +environment: + type: host + workdir: ./support-app + setup: + command: ./scripts/materialize-repo.sh + args: repo: acme/support-app commit: main -input: Answer using the refund policy in the workspace. +prompts: + - Answer using the refund policy in {{ repo_path }}. +default_test: + vars: + repo_path: ./support-app assert: - Applies the refund policy correctly tests: - id: missing-receipt - input: Can this customer get a refund without a receipt? + vars: + question: Can this customer get a refund without a receipt? ``` Raw cases are just case data: @@ -81,10 +82,10 @@ Raw cases are just case data: expected_output: Offer a replacement or refund path. ``` -A wrapper eval stays ordinary eval YAML while choosing a target and run controls: +A run-focused eval stays ordinary eval YAML while choosing a target and run controls: ```yaml -# experiments/refunds-codex.eval.yaml +# evals/refunds-codex.eval.yaml name: refunds-codex target: codex-gpt5 evaluate_options: @@ -93,28 +94,21 @@ evaluate_options: count: 2 strategy: pass_any -imports: - suites: - - path: ../evals/suites/refunds.eval.yaml - tests: - - path: ../evals/cases/refund-smoke.cases.yaml - tests: + - file://../evals/cases/refund-smoke.cases.yaml - id: local-edge-case vars: question: Can a final-sale item be refunded after damage in transit? expected_output: Explain the final-sale exception for damaged transit. ``` -The `experiments/` directory in that example is optional and user-owned. AgentV -does not infer behavior from the path; the wrapper runs because it is eval YAML -with tests or imports. The wrapper owns target selection and run controls. Put -workspace setup in imported child suites. Parent workspace-affecting fields, -including top-level `workspace`, are for parent-owned raw cases, including -cases imported with `imports.tests`. Runtime workspace path overrides belong in -CLI flags or `.agentv/config.local.yaml`; repos, hooks, templates, Docker -config, env checks, and workspace scope belong in top-level or case-level -`workspace`. +The `experiments/` directory is optional and user-owned. AgentV does not infer +behavior from the path; files run because they are eval YAML with tests or +scenarios. Runtime workspace path overrides belong in CLI flags or +`.agentv/config.local.yaml`; coding-agent testbeds, workdirs, Docker config, +repository setup, and reset policy belong in top-level or case-level +`environment`. Provider environment overrides belong in `env`; lifecycle hooks +belong in `extensions`. ## YAML Format @@ -153,20 +147,20 @@ tests: | `evaluate_options` | Optional evaluation runtime options such as `budget_usd`, `repeat`, and `max_concurrency` | | `timeout_seconds` | Optional per-case timeout | | `threshold` | Optional suite quality threshold | -| `workspace` | Suite-level task environment — inline object or string path to an external workspace file. Repo entries declare identity and checkout pins; acquisition is covered in [Workspace Architecture](/docs/guides/workspace-architecture/#repo-provenance-vs-acquisition). | -| `extensions` | Lifecycle hooks: `file://path/to/hooks.mjs:beforeAll`, `beforeEach`, `afterEach`, `afterAll`, plus the built-in `agentv:agent-rules`. Hooks run after `workspace.repos` materializes. | -| `imports` | Optional import groups. `imports.suites` imports full child eval suites with their task context. `imports.tests` imports raw test rows into this file's context. Import entries may use scoped `run:` overrides for `threshold`, `repeat`, `timeout_seconds`, and `budget_usd`. | -| `tests` | Inline raw tests or a string path to an external raw-case file or directory. Legacy `tests[].include` entries still load with a migration warning; prefer `imports.suites` or `imports.tests`. | +| `environment` | Suite-level coding-agent testbed — inline object or `file://` reference for host/Docker setup, workdir, repositories, fixtures, services, and reset policy. | +| `env` | Promptfoo-compatible provider/eval environment-variable overrides and load-time template inputs. | +| `extensions` | Lifecycle hooks: `file://path/to/hooks.mjs:beforeAll`, `beforeEach`, `afterEach`, `afterAll`, plus the built-in `agentv:agent-rules`. | +| `tests` | Inline raw tests, a string path to an external raw-case file or directory, or a list mixing inline tests with raw-case file refs. | | `assert` | Suite-level graders appended to each test unless `execution.skip_defaults: true` is set on the test | -`workspace` is what the agent can inspect or modify through tools, not prompt +`environment` is what the agent can inspect or modify through tools, not prompt input. Put task instructions and chat/system/user messages in `prompts`; put -repos, templates, Docker config, env checks, scope, and repo provenance in -`workspace`. Put lifecycle setup that does not acquire repos in `extensions`. +coding-agent testbed setup, workdir, Docker config, services, and repo +provenance in `environment`. Put lifecycle hooks in `extensions`. For historical or repo-state evals, put the checkout under -`workspace.repos[].commit`. A commit SHA in the prompt or metadata is useful -context, but it does not materialize a repo for the agent to inspect. +`environment` with a pinned setup recipe. A commit SHA in the prompt or metadata +is useful context, but it does not materialize a repo for the agent to inspect. ### Prompts, Vars, and Target Expansion @@ -247,9 +241,9 @@ tests: input: Summarize the July release notes. ``` -External raw-case files imported through `tests: file://...` or -`imports.tests` may still contain raw internal `input` rows for compatibility. -Keep that compatibility out of normal eval YAML authoring. +External raw-case files referenced through `tests: file://...` may still contain +raw internal `input` rows for compatibility. Keep that compatibility out of +normal eval YAML authoring. ### Lifecycle Extensions @@ -270,7 +264,7 @@ Setup hook failures (`beforeAll`, `beforeEach`) fail the affected run; teardown hook failures (`afterEach`, `afterAll`) are non-fatal. `agentv:agent-rules` is the only built-in extension in this slice. It runs after -workspace materialization and exposes staged rule paths to providers and result +environment materialization and exposes staged rule paths to providers and result metadata as `agent_rules_paths`: ```yaml @@ -285,7 +279,7 @@ extensions: If `agentv:agent-rules` is authored as a string, it defaults to `beforeAll` and discovers conventional rule locations already present in the materialized -workspace. It does not clone repositories or replace `workspace.repos`. +environment. It does not clone repositories or replace `environment`. ### Metadata Fields @@ -506,24 +500,18 @@ validation instead of being skipped at runtime. assertion `min_score`. Ordinary columns become `vars`, so CSV rows can rely on suite-level `input` that interpolates those variables. -String shorthand is raw-case-only. Import reusable task suites through -`imports.suites`; use `imports.tests` when you want to drop suite context and -import only raw cases into the parent context: +String shorthand is raw-case-only. Use a direct raw-case path or file ref when +you want case data to run in the parent suite context: ```yaml -imports: - suites: - - path: ./suites/*.eval.yaml - tests: - - path: ./cases/regression.jsonl - tests: + - file://./cases/regression.jsonl - id: local-edge-case input: ... ``` -Legacy `tests[].include` entries still load with a migration warning for older -eval files, but new evals should use `imports.suites` or `imports.tests`. +Run multiple eval files directly from the CLI, and use tags to group suites that +should be selected together. ### Raw Cases as Directory Paths diff --git a/apps/web/src/content/docs/docs/next/evaluation/experiments.mdx b/apps/web/src/content/docs/docs/next/evaluation/experiments.mdx index a58fe984f..2d224e8a9 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/experiments.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/experiments.mdx @@ -57,8 +57,8 @@ experiments/ In that layout, `evals/suites/refunds.eval.yaml` is a reusable task suite, `evals/cases/refund-smoke.cases.yaml` is raw case data, and -`experiments/refunds-codex.eval.yaml` is a wrapper eval. The wrapper still runs -only because it is eval YAML: +`experiments/refunds-codex.eval.yaml` is a normal eval file with its own run +controls: ```yaml # experiments/refunds-codex.eval.yaml @@ -66,91 +66,41 @@ name: refunds-codex target: codex-gpt5 tests: + - file://../evals/cases/refund-smoke.cases.yaml - id: local-edge-case input: Check a damaged final-sale refund. - -imports: - suites: - - path: ../evals/suites/refunds.eval.yaml - tests: - - path: ../evals/cases/refund-smoke.cases.yaml ``` The `experiments/` folder is optional and user-owned. AgentV does not scan it -for special files or infer runtime behavior from the path; the same wrapper eval -could live under `evals/wrappers/`, `benchmarks/`, or beside the suite it runs. +for special files or infer runtime behavior from the path; the same eval could +live under `evals/`, `benchmarks/`, or beside the suite it runs. -## Suite And Test Imports +## Multiple Suites And Raw Case Files -Use `imports.suites` for full child suites and `imports.tests` for raw test -rows. Inline `tests` remain raw cases owned by the current file. +Use direct eval files for suites that own task context. Reference reusable raw +case files from the local `tests` field when those rows should run in the +current suite context. ```yaml -imports: - suites: - - path: evals/support/*.eval.yaml - select: - test_ids: - - refund-* - - missing-order-date - tags: regression - metadata: - priority: high - run: - threshold: 1.0 - timeout_seconds: 300 - tests: - - path: cases/*.cases.yaml - - path: cases/regression.jsonl - tests: - - cases/smoke/*.cases.yaml + - file://cases/*.cases.yaml + - file://cases/regression.jsonl + - file://cases/smoke/*.cases.yaml ``` -`imports.suites` preserves the imported suite's task contract: metadata, -`workspace`, shared `input`, shared `assert`, and tests. The parent eval -still owns the single run bundle and run controls. Use parent `target` and -top-level run controls for the overall run, and import `run:` for scoped -threshold, timeout, or budget overrides. - -A parent eval that imports any `imports.suites` entry must not define top-level -`workspace`. Imported suites own task environment. If the parent should provide -workspace context, import raw cases with `imports.tests` or shorthand paths -instead of importing an eval suite. - -`imports.tests` imports only raw test entries. It intentionally drops shared -context from an imported eval suite, so parent suite fields apply to those raw -cases. - -Import `select.test_ids` filters imported test IDs with glob patterns. -Import `select.tags` filters each imported case's effective `metadata.tags`. -Effective case tags are suite-first and deduped: -`suite.tags + suite.metadata.tags + test.metadata.tags`. Top-level suite `tags` -still remain suite identity metadata for discovery and reporting; selection reads -the merged case metadata view. Import `select.metadata` filters case metadata by -key/value, where selector values may be scalars or lists. Globbed include paths -are resolved in deterministic path order, then test order. - String-valued `tests` and string entries inside `tests[]` are raw-case import -shorthand. They are equivalent to `imports.tests` and may point at -raw case files, directories, or globs. Importing another eval suite must use -`imports.suites`. - -Suite imports are resolved as a deterministic include graph. Circular -`imports.suites` imports fail validation with the import chain; raw-case shorthand does -not recursively load suite runtime blocks. - -Imported suite rows keep their source suite metadata in `index.jsonl`. Use each -row's `result_dir` as the authoritative path to generated artifacts inside the -run directory; do not infer layout from suite names. +shorthand and may point at raw case files, directories, or globs. They do not +load another eval suite's runtime blocks. Run multiple eval suites directly with +the CLI and use tags such as `tags.experiment` or domain-specific tags for +grouping and filtering. ## Scoped Run Overrides -Use scoped `run:` blocks for result interpretation and scheduling policies that -vary by include group or test case. Precedence is: +Use test-level `run:` blocks for result interpretation and scheduling policies +that vary by case. Precedence is: ```text -test.run > import run > parent top-level run controls +test.run > parent top-level run controls ``` ```yaml @@ -160,21 +110,9 @@ evaluate_options: repeat: count: 3 strategy: pass_any - -imports: - suites: - - path: ./evals/flaky-agentic/**/*.eval.yaml - select: - tags: [agentic] - run: - timeout_seconds: 300 - - - path: ./evals/regression/**/*.eval.yaml - select: - tags: [must-pass] - run: - threshold: 1.0 - timeout_seconds: 300 +timeout_seconds: 300 +tags: + area: agentic tests: - id: critical-case @@ -189,8 +127,9 @@ Scoped `run:` supports `threshold`, `repeat`, `timeout_seconds`, and per-case `budget_usd` overrides. Parent suite budgets should use `evaluate_options.budget_usd` for public eval authoring. Use `evaluate_options.max_concurrency` for authored concurrency. -Candidate-changing fields stay parent-level. Executable workspace setup belongs -in top-level lifecycle extensions, and provider-specific setup belongs in target +Candidate-changing fields stay parent-level. Coding-agent testbed setup belongs +in `environment`, lifecycle hooks belong in top-level `extensions`, and +provider-specific setup belongs in target configuration. ## Lifecycle Ownership @@ -202,7 +141,7 @@ target-specific runner state. | --- | --- | | Install dependencies, build the repo, seed files | `extensions: ["file://scripts/setup.mjs:beforeAll"]` | | Apply per-case state | `extensions: ["file://scripts/setup.mjs:beforeEach"]` | -| Reset file state after each case | `workspace.hooks.after_each.reset` | +| Reset file state after each case | `environment` reset policy | | Configure an agent runner or provider variant | `target` object or `targets.yaml` | | Choose the target | top-level `target` | | Override the target's default model | `target.model` | @@ -227,8 +166,9 @@ evaluate_options: Existing local workspace paths are machine-local bindings: pass `--workspace-path` for a one-off run or put `execution.workspace_path` in `.agentv/config.local.yaml`. -Put repos, templates, hooks, Docker config, env checks, and isolation under -top-level or case-level `workspace`. +Put coding-agent testbeds, workdirs, Docker config, repository setup, services, +and reset policy under top-level or case-level `environment`; put lifecycle +hooks in `extensions` and provider environment overrides in `env`. ## Repeat Runs diff --git a/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx b/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx index 31943e120..784482b6a 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx @@ -248,10 +248,10 @@ workspace: ``` Notes: -- Temp workspace materialization is the default for harness-managed workspaces with repos. +- Temp environment materialization is the default for harness-managed testbeds. - `--workspace-path` uses an existing machine-local directory as-is. -- Runtime workspace paths are incompatible with `workspace.scope: attempt`. -- `workspace.hooks.after_each.reset` resets file state after each case. +- Runtime workspace paths are incompatible with per-attempt environment isolation. +- Environment reset policy resets file state after each case. ### Resume an Interrupted Run @@ -519,8 +519,8 @@ Do not wrap referenced field files in another object. For example, not `{ targets: [...] }`. Authored suite concurrency belongs in eval files under -`evaluate_options.max_concurrency`, next to the suite or wrapper eval it -controls. `agentv eval --workers N` remains the operator-side override. +`evaluate_options.max_concurrency`, next to the suite it controls. `agentv eval +--workers N` remains the operator-side override. Other project defaults can live beside the graph: diff --git a/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx b/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx index 35dec6029..24a2ffc39 100644 --- a/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx +++ b/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx @@ -32,7 +32,7 @@ implements equivalent semantics directly. | Surface | Promptfoo shape | AgentV shape | Decision | Notes | | --- | --- | --- | --- | --- | | Prompt matrix | Top-level `prompts` rendered with each test's `vars`. | Top-level `prompts` rendered with `tests[].vars` and `default_test.vars`. | Align with Promptfoo | This is the canonical Promptfoo-compatible input shape in AgentV. Prompt entries can be inline strings, chat arrays, files, or generated prompt functions. | -| Test rows | `tests` can be inline rows or a case-file reference; rows carry `vars`, `assert`, metadata, prompt/provider filters, and expected data. | `tests` can be inline rows or a raw-case path; rows carry `vars`, `assert`, `expected_output`, metadata, workspace overrides, and run overrides. | Align with Promptfoo | AgentV also supports `imports.suites` and `imports.tests` for explicit composition. Raw cases do not own suite context. | +| Test rows | `tests` can be inline rows or a case-file reference; rows carry `vars`, `assert`, metadata, prompt/provider filters, and expected data. | `tests` can be inline rows or a raw-case path; rows carry `vars`, `assert`, `expected_output`, metadata, workspace overrides, and run overrides. | Align with Promptfoo | AgentV uses field-local file refs such as `tests: file://...`, `prompts: file://...`, and `default_test: file://...`; coding-agent testbeds use `environment: file://...`. There is no separate imports table. | | Variables | `tests[].vars` plus `defaultTest.vars`; prompt templates can reference top-level var names. | `tests[].vars` plus `default_test.vars`; templates can use `{{ name }}` or `{{ vars.name }}`. | Align with Promptfoo | Per-test vars override default vars by key. | | Default test | `defaultTest`, inline object or `file://` reference. | `default_test`, inline object or `file://` / `ref://` reference. | Align with Promptfoo | AgentV uses `snake_case` for YAML. Shared prompt matrix defaults belong in `default_test.vars`. | | Output transform | `defaultTest.options.transform`, `tests[].options.transform`, and assertion-level `transform`. | `default_test.options.transform`, `tests[].options.transform`, and assertion-level `transform`. | Align with Promptfoo | Use `transform` to shape provider output before grading, including file-output conversions such as `.xlsx` to text. `tests[].options.transform` overrides the inherited default transform; assertion-level `transform` is scoped to one grader. | @@ -50,7 +50,7 @@ implements equivalent semantics directly. | Script/custom grader terminology | Promptfoo custom code assertions are still assertion types. | `defineScriptGrader()` powers command-backed graders referenced with `type: script` and `command:`. | Keep AgentV divergence | Use script grader wording only for command-backed or LLM-backed scoring components that need explicit score and assertion-result control. | | Tool and trace assertions | Promptfoo includes `trajectory:tool-used`, `trajectory:tool-sequence`, `trajectory:tool-args-match`, `trajectory:step-count`, `trajectory:goal-success`, `tool-call-f1`, `skill-used`, `trace-span-count`, `trace-span-duration`, and `trace-error-spans`. | AgentV rejects those names until their semantics are implemented directly. | Defer/future-scope | These names are not aliases for AgentV's `tool-trajectory` grader. | | Tool trajectory grader | No direct Promptfoo alias for AgentV-normalized transcript semantics. | `type: tool-trajectory`. | Keep AgentV extension | This is AgentV-specific and operates over AgentV-normalized transcripts and trace summaries. | -| Repo-native workspace fields | Promptfoo normal evals do not own AgentV workspace materialization. | `workspace`, `workspace.repos`, `workspace.scope`, `workspace.docker`, `extensions`, and per-test `workspace`. | Keep AgentV extension | AgentV evaluates real repositories and agent workspaces, so workspace provenance is first-class authored config. | +| Coding-agent testbeds | Promptfoo normal evals do not own AgentV repository or Docker materialization. | `environment`, `environment: file://...`, `env`, and `extensions`. | Keep AgentV extension | AgentV evaluates real repositories and agent testbeds. Use `environment` for workdir/Docker/repository/setup/reset policy, top-level `env` for provider environment overrides, and `extensions` for lifecycle hooks. | | Run artifacts and inspection | Promptfoo owns its own result viewer and output formats. | AgentV writes `.agentv/results//` bundles with `summary.json`, `.internal/index.jsonl`, sidecars, and local Dashboard support. | Keep AgentV extension | AgentV-owned bundles are the source of truth for compare, Dashboard, CI, and adapters. Phoenix is link-out correlation only through safe external trace metadata. | | Compare command | Promptfoo has its own result comparison surfaces. | `agentv results compare `. | Keep AgentV extension | Compare consumes completed AgentV run indexes such as `.agentv/results//.internal/index.jsonl`. | | CLI runtime filters | Promptfoo exposes filters such as prompt/provider/test subset flags. | AgentV supports its current CLI filters and selection fields; full Promptfoo runtime-filter parity is future work. | Defer/future-scope | Prefer authored `select`/`imports` or current AgentV CLI flags until runtime-filter parity lands. | diff --git a/apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-cases.mdx b/apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-cases.mdx index 533ebbac6..5d29059cd 100644 --- a/apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-cases.mdx +++ b/apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-cases.mdx @@ -166,7 +166,8 @@ tests: ``` The `metadata` field is included in the stdin JSON passed to lifecycle commands as `case_metadata`. -Operational checkout state belongs under `workspace.repos[].base_commit`; matching metadata fields such as `source_commit` are informational only. +Operational checkout state belongs in `environment`; matching metadata fields +such as `source_commit` are informational only. For benchmark task packs with source pins, patches, generated rows, and supporting files, see [Benchmark Provenance](/docs/v4.42.4/guides/benchmark-provenance/). @@ -411,7 +412,7 @@ tests: input: Write a function to sort a list ``` -`metadata` is passed to workspace lifecycle hooks as `case_metadata`, preserved +`metadata` is passed to lifecycle hooks as `case_metadata`, preserved in result records, and available to in-process custom assertions. AgentV does -not interpret arbitrary metadata keys itself; use `workspace`, `execution`, -`input`, `expected_output`, and `assertions` for operational behavior. +not interpret arbitrary metadata keys itself; use `environment`, `extensions`, +`prompts`, `expected_output`, and `assertions` for operational behavior. diff --git a/apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-files.mdx b/apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-files.mdx index dcdb95114..05998642c 100644 --- a/apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-files.mdx +++ b/apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-files.mdx @@ -8,18 +8,17 @@ editUrl: false pagefind: false --- -Evaluation files define the test cases, graders, workspace lifecycle, and run +Evaluation files define the test cases, graders, environment, lifecycle hooks, and run controls for an evaluation run. The reserved `tags.experiment` key is the run/result grouping label, top-level `target` identifies the system under test, and fields such as `evaluate_options.repeat`, `threshold`, `timeout_seconds`, `evaluate_options.budget_usd`, and `evaluate_options.max_concurrency` control repeated -attempts and gates. Workspace lifetime belongs under `workspace.scope`; -repository provenance belongs under `workspace.repos`; Docker/container binding -belongs under `workspace.docker`. Non-provisioning setup commands belong in -top-level `extensions`; reset policy stays under -`workspace.hooks.after_each.reset`; runner-specific setup belongs in the -`target` object, in `targets`, or in project config. AgentV supports two eval -data formats: YAML and JSONL. +attempts and gates. Coding-agent testbeds, workdirs, Docker config, repository +materialization, setup, and reset policy belong in `environment`. Provider +environment-variable overrides belong in top-level `env`. Lifecycle hooks belong +in `extensions`; runner-specific setup belongs in the `target` object, in +`targets`, or in project config. AgentV supports two eval data formats: YAML and +JSONL. YAML is the canonical portable model. TypeScript helpers, generated fixtures, and Python scripts should lower to the same YAML/JSONL shapes rather than inventing a separate eval contract. Eval files describe the task, target binding, and run controls. Use @@ -31,41 +30,43 @@ override concurrency with `agentv eval --workers N`; do not author legacy Eval YAML is AgentV's composable and runnable authoring primitive. It is a focused, shareable slice of the same config graph as `.agentv/config.yaml`. -Use ordinary `*.eval.yaml` files for direct task suites and for wrapper evals -that compose other suites. Raw case files are reusable data inputs, not a -second runnable experiment format. +Use ordinary `*.eval.yaml` files for direct task suites. Raw case files are +reusable data inputs, not a second runnable experiment format. -- A **task suite** is eval YAML that owns task context: `workspace`, shared - `input`, shared `assert`, fixtures, graders, and test cases. It can run - directly or be imported through `imports.suites`. +- A **task suite** is eval YAML that owns task context: `environment`, shared + `prompts`, shared `assert`, fixtures, graders, and test cases. It can run + directly. - A **raw case file** is a YAML, JSON, JSONL, CSV, script-backed dataset, - directory, or glob of cases. Import it with `imports.tests`, - `tests: ./cases.yaml`, `tests: file://cases.csv`, or string shorthand; parent - suite context applies because raw cases do not carry their own suite context. -- A **wrapper eval** is eval YAML that imports one or more suites with - `imports.suites` and binds run controls with top-level `target`, `threshold`, - `timeout_seconds`, and `evaluate_options`. - Wrapper evals can live anywhere in the repo. A wrapper that imports suites - with `imports.suites` must not define parent `workspace`; imported suites own - task environment. Machine-local existing workspace paths belong in CLI flags - or `config.local.yaml`, not eval YAML. + directory, or glob of cases. Reference it with `tests: ./cases.yaml`, + `tests: file://cases.csv`, or string entries inside `tests`; parent suite + context applies because raw cases do not carry their own suite context. +- Use multiple eval files, tags, and CLI selection when you want to run several + suites together. Each eval file owns its own task environment and run policy. For example, a reusable task suite can keep the task contract in one file: ```yaml # evals/suites/refunds.eval.yaml suite: refunds -workspace: - repos: - - path: ./support-app +environment: + type: host + workdir: ./support-app + setup: + command: ./scripts/materialize-repo.sh + args: repo: acme/support-app commit: main -input: Answer using the refund policy in the workspace. +prompts: + - Answer using the refund policy in {{ repo_path }}. +default_test: + vars: + repo_path: ./support-app assert: - Applies the refund policy correctly tests: - id: missing-receipt - input: Can this customer get a refund without a receipt? + vars: + question: Can this customer get a refund without a receipt? ``` Raw cases are just case data: @@ -77,10 +78,10 @@ Raw cases are just case data: expected_output: Offer a replacement or refund path. ``` -A wrapper eval stays ordinary eval YAML while choosing a target and run controls: +A run-focused eval stays ordinary eval YAML while choosing a target and run controls: ```yaml -# experiments/refunds-codex.eval.yaml +# evals/refunds-codex.eval.yaml name: refunds-codex target: codex-gpt5 evaluate_options: @@ -89,27 +90,20 @@ evaluate_options: count: 2 strategy: pass_any -imports: - suites: - - path: ../evals/suites/refunds.eval.yaml - tests: - - path: ../evals/cases/refund-smoke.cases.yaml - tests: + - file://../evals/cases/refund-smoke.cases.yaml - id: local-edge-case input: Can a final-sale item be refunded after damage in transit? expected_output: Explain the final-sale exception for damaged transit. ``` The `experiments/` directory in that example is optional and user-owned. AgentV -does not infer behavior from the path; the wrapper runs because it is eval YAML -with tests or imports. The wrapper owns target selection and run controls. Put -workspace setup in imported child suites. Parent workspace-affecting fields, -including top-level `workspace`, are for parent-owned raw cases, including -cases imported with `imports.tests`. Runtime workspace path overrides belong in -CLI flags or `.agentv/config.local.yaml`; repos, hooks, templates, Docker -config, env checks, and workspace scope belong in top-level or case-level -`workspace`. +does not infer behavior from the path; files run because they are eval YAML with +tests or scenarios. Runtime workspace path overrides belong in CLI flags or +`.agentv/config.local.yaml`; coding-agent testbeds, workdirs, Docker config, +repository setup, and reset policy belong in top-level or case-level +`environment`. Provider environment overrides belong in `env`; lifecycle hooks +belong in `extensions`. ## YAML Format @@ -144,21 +138,21 @@ tests: | `timeout_seconds` | Optional per-case timeout | | `evaluate_options` | Optional evaluation runtime options such as `budget_usd`, `repeat`, and `max_concurrency` | | `threshold` | Optional suite quality threshold | -| `workspace` | Suite-level task environment — inline object or string path to an external workspace file. Repo entries declare identity and checkout pins; acquisition is covered in [Workspace Architecture](/docs/guides/workspace-architecture/#repo-provenance-vs-acquisition). | -| `extensions` | Promptfoo-style lifecycle hooks: `file://path/to/hooks.mjs:beforeAll`, `beforeEach`, `afterEach`, `afterAll`, plus the built-in `agentv:agent-rules`. Hooks run after `workspace.repos` materializes. | -| `imports` | Optional import groups. `imports.suites` imports full child eval suites with their task context. `imports.tests` imports raw test rows into this file's context. Import entries may use scoped `run:` overrides for `threshold`, `repeat`, `timeout_seconds`, and `budget_usd`. | -| `tests` | Inline raw tests or a string path to an external raw-case file or directory. Legacy `tests[].include` entries still load with a migration warning; prefer `imports.suites` or `imports.tests`. | +| `environment` | Suite-level coding-agent testbed — inline object or `file://` reference for host/Docker setup, workdir, repositories, fixtures, services, and reset policy. | +| `env` | Promptfoo-compatible provider/eval environment-variable overrides and load-time template inputs. | +| `extensions` | Promptfoo-style lifecycle hooks: `file://path/to/hooks.mjs:beforeAll`, `beforeEach`, `afterEach`, `afterAll`, plus the built-in `agentv:agent-rules`. | +| `tests` | Inline raw tests, a string path to an external raw-case file or directory, or a list mixing inline tests with raw-case file refs. | | `assert` | Suite-level graders appended to each test unless `execution.skip_defaults: true` is set on the test | | `input` | Suite-level input messages prepended to each test's input unless `execution.skip_defaults: true` is set on the test | -`workspace` is what the agent can inspect or modify through tools, not prompt -input. Put instructions in `input`; put repos, templates, Docker config, env -checks, scope, and repo provenance in `workspace`. Put lifecycle setup that -does not acquire repos in `extensions`. +`environment` is what the agent can inspect or modify through tools, not prompt +input. Put task instructions and chat/system/user messages in `prompts`; put +coding-agent testbed setup, workdir, Docker config, services, and repo +provenance in `environment`. Put lifecycle hooks in `extensions`. For historical or repo-state evals, put the checkout under -`workspace.repos[].commit`. A commit SHA in the prompt or metadata is useful -context, but it does not materialize a repo for the agent to inspect. +`environment` with a pinned setup recipe. A commit SHA in the prompt or metadata +is useful context, but it does not materialize a repo for the agent to inspect. ### Prompts, Vars, and Target Expansion @@ -228,7 +222,7 @@ Setup hook failures (`beforeAll`, `beforeEach`) fail the affected run; teardown hook failures (`afterEach`, `afterAll`) are non-fatal. `agentv:agent-rules` is the only built-in extension in this slice. It runs after -workspace materialization and exposes staged rule paths to providers and result +environment materialization and exposes staged rule paths to providers and result metadata as `agent_rules_paths`: ```yaml @@ -243,7 +237,7 @@ extensions: If `agentv:agent-rules` is authored as a string, it defaults to `beforeAll` and discovers conventional rule locations already present in the materialized -workspace. It does not clone repositories or replace `workspace.repos`. +environment. It does not clone repositories or replace `environment`. ### Metadata Fields @@ -471,24 +465,18 @@ during validation instead of being skipped at runtime. assertion `min_score`. Ordinary columns become `vars`, so CSV rows can rely on suite-level `input` that interpolates those variables. -String shorthand is raw-case-only. Import reusable task suites through -`imports.suites`; use `imports.tests` when you want to drop suite context and -import only raw cases into the parent context: +String shorthand is raw-case-only. Use a direct raw-case path or file ref when +you want case data to run in the parent suite context: ```yaml -imports: - suites: - - path: ./suites/*.eval.yaml - tests: - - path: ./cases/regression.jsonl - tests: + - file://./cases/regression.jsonl - id: local-edge-case input: ... ``` -Legacy `tests[].include` entries still load with a migration warning for older -eval files, but new evals should use `imports.suites` or `imports.tests`. +Run multiple eval files directly from the CLI, and use tags to group suites that +should be selected together. ### Raw Cases as Directory Paths diff --git a/apps/web/src/content/docs/docs/v4.42.4/evaluation/running-evals.mdx b/apps/web/src/content/docs/docs/v4.42.4/evaluation/running-evals.mdx index 3ef800c34..4edb69f59 100644 --- a/apps/web/src/content/docs/docs/v4.42.4/evaluation/running-evals.mdx +++ b/apps/web/src/content/docs/docs/v4.42.4/evaluation/running-evals.mdx @@ -250,10 +250,10 @@ workspace: ``` Notes: -- Temp workspace materialization is the default for harness-managed workspaces with repos. +- Temp environment materialization is the default for harness-managed testbeds. - `--workspace-path` uses an existing machine-local directory as-is. -- Runtime workspace paths are incompatible with `workspace.scope: attempt`. -- `workspace.hooks.after_each.reset` resets file state after each case. +- Runtime workspace paths are incompatible with per-attempt environment isolation. +- Environment reset policy resets file state after each case. ### Resume an Interrupted Run diff --git a/docs/adr/0006-separate-experiments-from-eval-definitions.md b/docs/adr/0006-separate-experiments-from-eval-definitions.md index b0b805a0b..e31ad2509 100644 --- a/docs/adr/0006-separate-experiments-from-eval-definitions.md +++ b/docs/adr/0006-separate-experiments-from-eval-definitions.md @@ -22,6 +22,14 @@ Superseded for the current eval authoring contract by `prompts`/`vars`, `assert`, direct `input`, `target`/`targets`, and `evaluate_options` are the current promptfoo-aligned authoring surface. +The "Suite And Test Import Surface" section below is historical. Its +`imports.suites` / `imports.tests` design is superseded by av-kfik.43 and the +current Promptfoo-aligned field-local ref contract: raw case files use +`tests: file://...` or string entries under `tests`, reusable prompt/default +config uses `prompts: file://...` and `default_test: file://...`, coding-agent +testbeds use `environment: file://...`, and grouping/running multiple evals uses +tags plus CLI multi-file selection. Top-level `imports` is rejected. + ## Context AgentV needs a stable authoring contract for repo-native evals, run-time knobs, diff --git a/examples/features/external-datasets/README.md b/examples/features/external-datasets/README.md index a013ef80a..f1a0e7eb2 100644 --- a/examples/features/external-datasets/README.md +++ b/examples/features/external-datasets/README.md @@ -1,14 +1,14 @@ # External Datasets Example -Demonstrates loading raw test cases from external files using `imports.tests`. +Demonstrates loading raw test cases from external files using field-local `tests` file refs. ## What This Shows -- Loading tests from external YAML files (`imports.tests[].path: cases/accuracy.yaml`) -- Loading tests from external JSONL files (`imports.tests[].path: cases/regression.jsonl`) -- Loading tests from promptfoo-compatible CSV files (`imports.tests[].path: cases/magic.csv`) -- Mixing inline `tests` with imported raw test rows -- Glob patterns for loading multiple files (`imports.tests[].path: cases/**/*.yaml`) +- Loading tests from external YAML files (`tests: [file://cases/accuracy.yaml]`) +- Loading tests from external JSONL files (`tests: [file://cases/regression.jsonl]`) +- Loading tests from promptfoo-compatible CSV files (`tests: [file://cases/magic.csv]`) +- Mixing inline `tests` with external raw test rows +- Glob patterns for loading multiple files (`tests: [file://cases/**/*.yaml]`) ## Running @@ -19,7 +19,7 @@ bun agentv eval examples/features/external-datasets/evals/suite.yaml ## Key Files -- `evals/suite.yaml` — Main eval with inline tests and `imports.tests` references +- `evals/suite.yaml` — Main eval with inline tests and `tests` file references - `evals/cases/accuracy.yaml` — YAML array of test cases - `evals/cases/regression.jsonl` — JSONL test data (one test per line) - `evals/cases/magic.csv` — CSV test data with promptfoo-style magic columns @@ -66,8 +66,7 @@ columns such as `locale` become `vars` and can be interpolated by suite-level Use glob patterns to load from multiple files: ```yaml -imports: - tests: - - path: cases/**/*.yaml # All YAML files recursively - - path: cases/*.jsonl # All JSONL files in cases/ +tests: + - file://cases/**/*.yaml # All YAML files recursively + - file://cases/*.jsonl # All JSONL files in cases/ ``` diff --git a/examples/features/external-datasets/evals/suite.yaml b/examples/features/external-datasets/evals/suite.yaml index 9dc48a136..a914f2706 100644 --- a/examples/features/external-datasets/evals/suite.yaml +++ b/examples/features/external-datasets/evals/suite.yaml @@ -3,12 +3,10 @@ version: "1.0" target: llm prompts: - "{{ input }}" -imports: - tests: - - path: cases/accuracy.yaml - - path: cases/regression.jsonl - - path: cases/magic.csv tests: + - file://cases/accuracy.yaml + - file://cases/regression.jsonl + - file://cases/magic.csv - id: inline-test assert: - The agent should greet the user politely diff --git a/packages/core/src/evaluation/validation/eval-file.schema.ts b/packages/core/src/evaluation/validation/eval-file.schema.ts index 1d93239c0..f2bcf462e 100644 --- a/packages/core/src/evaluation/validation/eval-file.schema.ts +++ b/packages/core/src/evaluation/validation/eval-file.schema.ts @@ -670,27 +670,6 @@ const TestIncludeSchema = z }) .strict(); -const ImportEntrySchema = z - .object({ - path: z.string().min(1), - select: z.union([SelectPatternSchema, TestIncludeSelectSchema]).optional(), - run: RunOverrideSchema.optional(), - }) - .strict(); - -const ImportGroupSchema = z.union([ - z.array(z.union([ImportEntrySchema, z.string().min(1)])), - z.string().min(1), - ImportEntrySchema, -]); - -const ImportsSchema = z - .object({ - suites: ImportGroupSchema.optional(), - tests: ImportGroupSchema.optional(), - }) - .strict(); - const TestsSchema = z.union([ z.array(z.union([EvalTestSchema, TestIncludeSchema, z.string().min(1)])), z.string().min(1), @@ -766,6 +745,9 @@ const TagsSchema = z.union([ z.record(z.union([z.string(), z.number(), z.boolean()])), ]); +const TOP_LEVEL_IMPORTS_MESSAGE = + "Top-level 'imports' is not supported. Run eval files directly with CLI multi-file selection and tags for grouping. For raw case files, use tests: file://... or string entries under tests. For reusable config, use prompts: file://..., default_test: file://..., and environment: file://... for coding-agent testbeds."; + // --------------------------------------------------------------------------- // Top-level eval file // --------------------------------------------------------------------------- @@ -791,8 +773,7 @@ export const EvalFileSchema: z.ZodType = z prompts: PromptsSchema.optional(), // Suite-level input_files shorthand input_files: z.array(z.string()).optional(), - // Imports: suites preserve child context; tests import raw rows into parent context - imports: ImportsSchema.optional(), + imports: z.never({ invalid_type_error: TOP_LEVEL_IMPORTS_MESSAGE }).optional(), // Tests (inline raw cases, legacy include entries, or external raw-case path) tests: TestsSchema.optional(), // Shared composable config graph fields @@ -833,9 +814,6 @@ export const EvalFileSchema: z.ZodType = z }) .refine( (value) => - value.tests !== undefined || - value.eval_cases !== undefined || - value.imports !== undefined || - value.scenarios !== undefined, - { message: "Eval files must define 'tests', 'imports', or 'scenarios'." }, + value.tests !== undefined || value.eval_cases !== undefined || value.scenarios !== undefined, + { message: "Eval files must define 'tests' or 'scenarios'." }, ); diff --git a/packages/core/src/evaluation/validation/eval-validator.ts b/packages/core/src/evaluation/validation/eval-validator.ts index 364cb839c..175149055 100644 --- a/packages/core/src/evaluation/validation/eval-validator.ts +++ b/packages/core/src/evaluation/validation/eval-validator.ts @@ -19,10 +19,9 @@ type SuiteImportStackEntry = { readonly displayPath: string; readonly filePath: string; }; -type ImportEntryType = 'suite' | 'tests'; type NormalizedImportEntry = { readonly path: string; - readonly type: ImportEntryType; + readonly type: 'suite' | 'tests'; readonly location: string; readonly select?: JsonValue; readonly run?: JsonValue; @@ -147,7 +146,6 @@ const KNOWN_TOP_LEVEL_FIELDS = new Set([ 'input', 'input_files', 'prompts', - 'imports', 'tests', 'graders', 'defaults', @@ -180,7 +178,6 @@ const KNOWN_TOP_LEVEL_FIELDS = new Set([ /** Known fields on legacy tests[] include entries. */ const KNOWN_INCLUDE_FIELDS = new Set(['include', 'type', 'select', 'run']); -const KNOWN_IMPORT_FIELDS = new Set(['path', 'select', 'run']); const KNOWN_RUN_OVERRIDE_FIELDS = new Set(['threshold', 'repeat', 'timeout_seconds', 'budget_usd']); const KNOWN_DEFAULT_TEST_FIELDS = new Set([ 'vars', @@ -222,6 +219,10 @@ const KNOWN_TEST_EXECUTION_FIELDS = new Set([ /** Removed top-level fields with migration hints. */ const REMOVED_TOP_LEVEL_FIELDS = new Map([ + [ + 'imports', + "Top-level 'imports' is not supported. Run eval files directly with CLI multi-file selection and tags for grouping. For raw case files, use tests: file://... or string entries under tests. For reusable config, use prompts: file://..., default_test: file://..., and environment: file://... for coding-agent testbeds.", + ], [ 'expected_output', "Top-level 'expected_output' has been removed from authored eval YAML. Put reference answers in default_test.vars.expected_output or tests[].vars.expected_output and consume them with an explicit assertion such as { type: 'llm-rubric', value: 'Matches the reference answer: {{ expected_output }}' }.", @@ -484,8 +485,6 @@ export async function validateEvalFile(filePath: string): Promise 0; const hasScenarios = Array.isArray(parsed.scenarios); @@ -521,7 +520,7 @@ export async function validateEvalFile(filePath: string): Promise 0) { - return value.trim(); - } - if (!isObject(value)) { - return undefined; - } - return typeof value.path === 'string' && value.path.trim().length > 0 - ? value.path.trim() - : undefined; -} - -async function validateImportsField( - imports: JsonValue | undefined, - filePath: string, - errors: ValidationError[], -): Promise { - if (imports === undefined) { - return; - } - if (!isObject(imports)) { - errors.push({ - severity: 'error', - filePath, - location: 'imports', - message: "Invalid 'imports' field. Use imports.suites and/or imports.tests.", - }); - return; - } - - for (const key of Object.keys(imports)) { - if (key !== 'suites' && key !== 'tests') { - errors.push({ - severity: 'warning', - filePath, - location: `imports.${key}`, - message: `Unknown imports field '${key}'. Use imports.suites or imports.tests.`, - }); - } - } - - await validateImportGroup(imports.suites, 'suite', 'imports.suites', filePath, errors); - await validateImportGroup(imports.tests, 'tests', 'imports.tests', filePath, errors); -} - -async function validateImportGroup( - group: JsonValue | undefined, - type: ImportEntryType, - location: string, - filePath: string, - errors: ValidationError[], -): Promise { - if (group === undefined) { - return; - } - const entries = Array.isArray(group) ? group : [group]; - for (let i = 0; i < entries.length; i++) { - const entry = entries[i]; - const entryLocation = `${location}[${i}]`; - const entryPath = importEntryPath(entry); - if (!entryPath) { - errors.push({ - severity: 'error', - filePath, - location: entryLocation, - message: "Invalid import entry. Use a path string or an object with a non-empty 'path'.", - }); - continue; - } - if (type === 'tests' && !/\.eval\.ya?ml$/i.test(entryPath)) { - await validateRawCaseImportPath(entryPath, filePath, `${entryLocation}.path`, errors); - } - if (isObject(entry)) { - for (const key of Object.keys(entry)) { - if (!KNOWN_IMPORT_FIELDS.has(key)) { - errors.push({ - severity: 'warning', - filePath, - location: `${entryLocation}.${key}`, - message: `Unknown field '${key}'. This field will be ignored.`, - }); - } - } - validateIncludeSelect(entry.select, `${entryLocation}.select`, filePath, errors); - validateRunOverride(entry.run, `${entryLocation}.run`, filePath, errors); - } - } -} - function collectImportEntries(parsed: JsonObject): readonly NormalizedImportEntry[] { const entries: NormalizedImportEntry[] = []; - if (isObject(parsed.imports)) { - entries.push(...collectImportGroup(parsed.imports.suites, 'suite', 'imports.suites')); - entries.push(...collectImportGroup(parsed.imports.tests, 'tests', 'imports.tests')); - } const tests = parsed.tests; if (Array.isArray(tests)) { for (let i = 0; i < tests.length; i++) { @@ -1054,43 +960,18 @@ function collectImportEntries(parsed: JsonObject): readonly NormalizedImportEntr return entries; } -function collectImportGroup( - group: JsonValue | undefined, - type: ImportEntryType, - location: string, -): readonly NormalizedImportEntry[] { - if (group === undefined) { - return []; - } - const entries = Array.isArray(group) ? group : [group]; - return entries.flatMap((entry, index) => { - const pathValue = importEntryPath(entry); - return pathValue - ? [ - { - path: pathValue, - type, - location: `${location}[${index}].path`, - ...(isObject(entry) && entry.select !== undefined ? { select: entry.select } : {}), - ...(isObject(entry) && entry.run !== undefined ? { run: entry.run } : {}), - }, - ] - : []; - }); -} - async function validateCompositionDiagnostics( filePath: string, parsed: JsonObject, errors: ValidationError[], ): Promise { - const imports = collectImportEntries(parsed); - if (imports.length === 0) { + const includeEntries = collectImportEntries(parsed); + if (includeEntries.length === 0) { return; } const parentHasRuntime = hasWrapperRuntimeControls(parsed); - const hasSuiteImport = imports.some((entry) => entry.type === 'suite'); + const hasSuiteImport = includeEntries.some((entry) => entry.type === 'suite'); if (hasSuiteImport) { for (const location of parentWorkspaceLocations(parsed)) { @@ -1100,13 +981,13 @@ async function validateCompositionDiagnostics( location, message: location === 'environment' - ? 'Parent environment is not allowed when an eval imports suites with type: suite. Imported suites own task environment. Move environment into the child suite, or import raw cases with type: tests when you intentionally want parent environment context.' - : 'Parent workspace is not allowed when an eval imports suites with type: suite. A wrapper eval owns target and run controls, while imported suites own task environment. Move workspace into the child suite, or import raw cases with type: tests when you intentionally want parent workspace context.', + ? 'Parent environment is not allowed with legacy tests[].include suite entries. Run eval files directly, or use tests: file://... for raw cases that should use the parent environment.' + : 'Parent workspace is not allowed with legacy tests[].include suite entries. Run eval files directly, or use tests: file://... for raw cases that should use the parent workspace.', }); } } - for (const entry of imports) { + for (const entry of includeEntries) { const resolvedSuites = await resolveSuiteIncludePaths(entry.path, path.dirname(filePath)); if (entry.type === 'suite') { @@ -1125,8 +1006,8 @@ async function validateCompositionDiagnostics( filePath, location: entry.location, message: parentHasRuntime - ? `Imported suite '${resolvedSuite.displayPath}' defines ${runtimeFields.join(', ')}, but child target and run controls are ignored for imports.suites. The parent eval owns wrapper target and run controls; move them to the parent eval or use import run overrides for per-case thresholds, timeouts, and budgets.` - : `Imported suite '${resolvedSuite.displayPath}' defines ${runtimeFields.join(', ')}, but child target and run controls are ignored for imports.suites. The parent eval owns wrapper target and run controls, and this parent has none, so no child target or run controls are applied. Add parent target/run controls or use import run overrides for per-case thresholds, timeouts, and budgets.`, + ? `Imported suite '${resolvedSuite.displayPath}' defines ${runtimeFields.join(', ')}, but child target and run controls are ignored by legacy tests[].include suite imports. Prefer running eval files directly with CLI multi-file selection and tags.` + : `Imported suite '${resolvedSuite.displayPath}' defines ${runtimeFields.join(', ')}, but child target and run controls are ignored by legacy tests[].include suite imports. Prefer running eval files directly with CLI multi-file selection and tags.`, }); } continue; @@ -1141,7 +1022,7 @@ async function validateCompositionDiagnostics( severity: 'warning', filePath, location: entry.location, - message: `imports.tests imports raw cases from eval suite '${resolvedSuite.displayPath}' and drops suite context, including child environment, workspace, input, assertions, metadata, target, and run controls. Parent suite context applies. Use imports.suites to preserve child test and environment semantics.`, + message: `Legacy tests[].include with type: tests imports raw cases from eval suite '${resolvedSuite.displayPath}' and drops suite context, including child environment, workspace, input, assertions, metadata, target, and run controls. Prefer tests: file://... for raw case files.`, }); } } @@ -1208,14 +1089,14 @@ function validateIncludeEntry( filePath: string, errors: ValidationError[], ): void { - const mode = entry.type === 'suite' ? 'suites' : entry.type === 'tests' ? 'tests' : undefined; errors.push({ severity: 'warning', filePath, location, - message: mode - ? `tests[].include is deprecated. Use imports.${mode} entries with path instead.` - : 'tests[].include is deprecated. Use imports.suites or imports.tests entries with path instead.', + message: + entry.type === 'suite' + ? 'tests[].include with type: suite is deprecated. Run eval files directly with CLI multi-file selection and tags instead.' + : 'tests[].include is deprecated. Use tests: file://... or string entries inside tests: for raw case files.', }); for (const key of Object.keys(entry)) { @@ -2267,12 +2148,12 @@ async function validateSuiteImportCyclesFromParsed( stack: readonly SuiteImportStackEntry[], errors: ValidationError[], ): Promise { - const imports = collectImportEntries(parsed); - if (imports.length === 0) { + const includeEntries = collectImportEntries(parsed); + if (includeEntries.length === 0) { return; } - for (const entry of imports) { + for (const entry of includeEntries) { if (entry.type !== 'suite') { continue; } diff --git a/packages/core/src/evaluation/yaml-parser.ts b/packages/core/src/evaluation/yaml-parser.ts index 39fd8dd79..d7f338459 100644 --- a/packages/core/src/evaluation/yaml-parser.ts +++ b/packages/core/src/evaluation/yaml-parser.ts @@ -191,7 +191,6 @@ function formatCircularImportChain( } type RawTestSuite = JsonObject & { - readonly imports?: JsonValue; readonly tests?: JsonValue; /** @deprecated Use `tests` instead */ readonly eval_cases?: JsonValue; @@ -636,6 +635,14 @@ function rejectAuthoredPostprocess(suite: RawTestSuite): void { }); } +function rejectTopLevelImports(suite: JsonObject): void { + if (suite.imports !== undefined) { + throw new Error( + "Top-level 'imports' is not supported. Run eval files directly with CLI multi-file selection and tags for grouping. For raw case files, use tests: file://... or string entries under tests. For reusable config, use prompts: file://..., default_test: file://..., and environment: file://... for coding-agent testbeds.", + ); + } +} + function isChatPromptArray(value: readonly JsonValue[]): boolean { return value.length > 0 && value.every((entry) => isJsonObject(entry) && isTestMessage(entry)); } @@ -1264,6 +1271,7 @@ async function loadTestsFromParsedYamlValue( ...(interpolated as RawTestSuite), default_test: resolvedDefaultTest.value, } as RawTestSuite; + rejectTopLevelImports(suite); rejectAuthoredPostprocess(suite); const defaultTestReferences = resolvedDefaultTest.references; const suiteNameFromFile = asString(suite.name)?.trim(); @@ -1295,25 +1303,10 @@ async function loadTestsFromParsedYamlValue( const nunjucksFilters = await loadNunjucksFilters(suite.nunjucks_filters, evalFileDir); const parentWorkspace = parentWorkspaceLocation(suite); const parentEnvironment = parentEnvironmentLocation(suite); - const importEntries = readImports(suite.imports); - const expandedImports = await expandImportEntries({ - entries: importEntries, - evalFileDir, - repoRoot, - suiteMetadataPayload, - parentWorkspaceLocation: parentWorkspace, - parentEnvironmentLocation: parentEnvironment, - options, - }); - importedSuiteTests.push(...expandedImports.importedSuiteTests); - // Resolve tests: string path to external file/directory, inline array, legacy include entries, or error. let expandedTestCases: readonly JsonValue[]; if (typeof rawTestCases === 'string') { - expandedTestCases = [ - ...expandedImports.rawCases, - ...(await loadRawCasesFromShorthand(rawTestCases, evalFileDir)), - ]; + expandedTestCases = await loadRawCasesFromShorthand(rawTestCases, evalFileDir); } else if (Array.isArray(rawTestCases)) { const expanded = await expandInlineTestEntries({ entries: rawTestCases, @@ -1324,10 +1317,8 @@ async function loadTestsFromParsedYamlValue( parentEnvironmentLocation: parentEnvironment, options, }); - expandedTestCases = [...expandedImports.rawCases, ...expanded.rawCases]; + expandedTestCases = expanded.rawCases; importedSuiteTests.push(...expanded.importedSuiteTests); - } else if (rawTestCases === undefined && importEntries.length > 0) { - expandedTestCases = expandedImports.rawCases; } else { throw new Error(`Invalid test file format: ${evalFilePath} - missing 'tests' field`); } @@ -2034,19 +2025,6 @@ function isIncludeEntry(value: JsonValue): value is JsonObject & { include: stri ); } -function importEntryPath(value: JsonValue): string | undefined { - if (typeof value === 'string' && value.trim().length > 0) { - return value.trim(); - } - if (!isJsonObject(value)) { - return undefined; - } - const pathValue = value.path ?? value.include; - return typeof pathValue === 'string' && pathValue.trim().length > 0 - ? pathValue.trim() - : undefined; -} - function hasGlobMagic(value: string): boolean { return /[*?[\]{}()!+@]/.test(value); } @@ -2191,55 +2169,6 @@ function rawCaseMatchesSelect( ); } -function readImports(rawImports: JsonValue | undefined): readonly NormalizedImportEntry[] { - if (rawImports === undefined) { - return []; - } - if (!isJsonObject(rawImports)) { - throw new Error("Invalid 'imports' field. Use imports.suites and/or imports.tests."); - } - const entries: NormalizedImportEntry[] = []; - entries.push(...readImportGroup(rawImports.suites, 'suite', 'imports.suites')); - entries.push(...readImportGroup(rawImports.tests, 'tests', 'imports.tests')); - return entries; -} - -function readImportGroup( - rawGroup: JsonValue | undefined, - mode: IncludeEntryType, - location: string, -): readonly NormalizedImportEntry[] { - if (rawGroup === undefined) { - return []; - } - const values = Array.isArray(rawGroup) ? rawGroup : [rawGroup]; - return values.map((entry, index) => normalizeImportEntry(entry, mode, `${location}[${index}]`)); -} - -function normalizeImportEntry( - entry: JsonValue, - mode: IncludeEntryType, - location: string, -): NormalizedImportEntry { - const includePath = importEntryPath(entry); - if (!includePath) { - throw new Error(`Invalid ${location}. Use a path string or an object with a non-empty path.`); - } - const select = isJsonObject(entry) - ? readSelectPatterns(entry.select, `${location}.select for path '${includePath}'`) - : undefined; - const includeRun = isJsonObject(entry) - ? normalizeRunOverride(entry.run, `${location}.run for path '${includePath}'`) - : undefined; - return { - path: includePath, - mode, - ...(select !== undefined && { select }), - ...(includeRun !== undefined && { run: includeRun }), - location, - }; -} - function normalizeLegacyIncludeEntry( entry: JsonObject & { include: string }, ): NormalizedImportEntry { @@ -2248,7 +2177,9 @@ function normalizeLegacyIncludeEntry( const select = readSelectPatterns(entry.select, `tests[].select for include '${includePath}'`); const includeRun = normalizeRunOverride(entry.run, `tests[].run for include '${includePath}'`); logWarning( - `tests[].include is deprecated. Use imports.${mode === 'suite' ? 'suites' : 'tests'} with path: ${includePath}`, + mode === 'suite' + ? `tests[].include with type: suite is deprecated. Run eval files directly instead: ${includePath}` + : `tests[].include is deprecated. Use tests: file://... or a tests list file reference instead: ${includePath}`, ); return { path: includePath, @@ -2279,12 +2210,12 @@ async function expandImportEntries(params: { if (entry.mode === 'suite') { if (params.parentWorkspaceLocation) { throw new Error( - `Parent workspace is not allowed when importing eval suites (${params.parentWorkspaceLocation}): ${entry.path}. Move workspace into the child suite, or import raw cases with imports.tests when you intentionally want parent workspace context.`, + `Parent workspace is not allowed with legacy tests[].include suite entries (${params.parentWorkspaceLocation}): ${entry.path}. Run eval files directly, or use tests: file://... for raw cases that should use the parent workspace.`, ); } if (params.parentEnvironmentLocation) { throw new Error( - `Parent environment is not allowed when importing eval suites (${params.parentEnvironmentLocation}): ${entry.path}. Imported suites own task environment. Move environment into the child suite, or import raw cases with imports.tests when you intentionally want parent environment context.`, + `Parent environment is not allowed with legacy tests[].include suite entries (${params.parentEnvironmentLocation}): ${entry.path}. Run eval files directly, or use tests: file://... for raw cases that should use the parent environment.`, ); } const suite = await loadTestSuite(resolvedPath, params.repoRoot, { diff --git a/packages/core/test/evaluation/eval-inline-experiment.test.ts b/packages/core/test/evaluation/eval-inline-experiment.test.ts index d0bdbb45c..acb0bd638 100644 --- a/packages/core/test/evaluation/eval-inline-experiment.test.ts +++ b/packages/core/test/evaluation/eval-inline-experiment.test.ts @@ -1430,75 +1430,21 @@ describe('eval.yaml flat runtime controls and tests imports', () => { expect(suite.tests.every((test) => test.run === undefined)).toBe(true); }); - it('imports suites through imports.suites while preserving child task context', async () => { - await writeFile( - path.join(tempDir, 'child.eval.yaml'), - [ - 'name: child-suite', - 'environment:', - ' type: host', - ' workdir: ./child-workspace', - 'assert:', - ' - type: contains', - ' value: child', - 'threshold: 0.2', - 'prompts:', - ' - "{{ input }}"', - 'tests:', - ' - id: child-case', - ' criteria: ok', - ' vars:', - ' input:', - ' - role: user', - ' content: child shared input', - ' - role: user', - ' content: child case input', - ].join('\n'), - ); + it('rejects top-level imports during suite loading', async () => { const parentPath = path.join(tempDir, 'parent.eval.yaml'); await writeFile( parentPath, - [ - 'name: parent-suite', - 'target: codex-gpt5', - 'threshold: 0.8', - 'prompts:', - ' - "{{ input }}"', - 'imports:', - ' suites:', - ' - path: child.eval.yaml', - ' run:', - ' timeout_seconds: 60', - 'tests:', - ' - id: local-edge', - ' criteria: local ok', - ' vars:', - ' input: local input', - ].join('\n'), + ['name: parent-suite', 'imports:', ' tests:', ' - path: cases.yaml', 'tests: []'].join( + '\n', + ), ); - const suite = await loadTestSuite(parentPath, tempDir); - const byId = new Map(suite.tests.map((test) => [test.id, test])); - - expect(suite.experimentConfig).toMatchObject({ target: 'codex-gpt5', threshold: 0.8 }); - expect(byId.get('child-case')?.suite).toBe('child-suite'); - expect(byId.get('child-case')?.source?.importedSuiteName).toBe('child-suite'); - expect(byId.get('child-case')?.environment?.workdir).toBe( - path.join(tempDir, 'child-workspace'), + await expect(loadTestSuite(parentPath, tempDir)).rejects.toThrow( + /Top-level 'imports' is not supported.*Run eval files directly.*tests: file:\/\/\.\.\..*prompts: file:\/\/\.\.\..*default_test: file:\/\/\.\.\..*environment: file:\/\/\.\.\./, ); - expect(byId.get('child-case')?.input.map((message) => message.content)).toEqual([ - 'child shared input', - 'child case input', - ]); - expect(byId.get('child-case')?.assertions?.[0]).toMatchObject({ - type: 'contains', - value: 'child', - }); - expect(byId.get('child-case')?.run).toEqual({ timeoutSeconds: 60 }); - expect(byId.get('local-edge')?.suite).toBe('parent-suite'); }); - it('imports raw rows through imports.tests and evaluates them in parent context', async () => { + it('loads raw rows through tests file refs in parent context', async () => { await writeFile( path.join(tempDir, 'smoke.jsonl'), '{"id":"jsonl-case","input":"jsonl input","criteria":"ok"}\n', @@ -1523,15 +1469,14 @@ describe('eval.yaml flat runtime controls and tests imports', () => { ' content: parent shared input', ' - role: user', ' content: "{{ input }}"', - 'imports:', - ' tests:', - ' - path: smoke.jsonl', - ' - path: regressions.yaml', 'tests:', + ' - file://smoke.jsonl', + ' - file://regressions.yaml', ' - id: inline-case', ' criteria: ok', ' vars:', ' input: inline input', + '', ].join('\n'), ); @@ -1550,76 +1495,6 @@ describe('eval.yaml flat runtime controls and tests imports', () => { ]); }); - it('combines imports.tests with tests path shorthand in parent context', async () => { - await writeFile( - path.join(tempDir, 'imported.jsonl'), - '{"id":"imported-case","input":"imported input","criteria":"ok"}\n', - ); - await writeFile( - path.join(tempDir, 'local.yaml'), - '- id: local-case\n input: local input\n criteria: ok\n', - ); - const parentPath = path.join(tempDir, 'parent.eval.yaml'); - await writeFile( - parentPath, - [ - 'name: parent-suite', - 'environment:', - ' type: host', - ' workdir: ./parent-workspace', - 'imports:', - ' tests:', - ' - path: imported.jsonl', - 'tests: local.yaml', - '', - ].join('\n'), - ); - - const suite = await loadTestSuite(parentPath, tempDir); - const byId = new Map(suite.tests.map((test) => [test.id, test])); - - expect(suite.tests.map((test) => test.id)).toEqual(['imported-case', 'local-case']); - expect(byId.get('imported-case')?.suite).toBe('parent-suite'); - expect(byId.get('local-case')?.suite).toBe('parent-suite'); - expect(byId.get('imported-case')?.environment?.workdir).toBe( - path.join(tempDir, 'parent-workspace'), - ); - }); - - it('rejects parent environment when imports.suites preserves child environments', async () => { - await writeFile( - path.join(tempDir, 'child.eval.yaml'), - [ - 'name: child-suite', - 'prompts:', - ' - "{{ input }}"', - 'tests:', - ' - id: child-case', - ' criteria: ok', - ' vars:', - ' input: child', - ].join('\n'), - ); - const parentPath = path.join(tempDir, 'parent.eval.yaml'); - await writeFile( - parentPath, - [ - 'name: parent-suite', - 'environment:', - ' type: host', - ' workdir: ./parent-workspace', - 'imports:', - ' suites:', - ' - path: child.eval.yaml', - '', - ].join('\n'), - ); - - await expect(loadTestSuite(parentPath, tempDir)).rejects.toThrow( - /Parent environment is not allowed/, - ); - }); - it('warns but supports legacy tests include entries during migration', async () => { await writeFile( path.join(tempDir, 'child.eval.yaml'), @@ -1654,12 +1529,16 @@ describe('eval.yaml flat runtime controls and tests imports', () => { console.warn = warn; } - expect(warnings.some((message) => message.includes('tests[].include is deprecated'))).toBe( - true, - ); + expect( + warnings.some( + (message) => + message.includes('tests[].include with type: suite is deprecated') && + message.includes('Run eval files directly'), + ), + ).toBe(true); }); - it('validates imports.suites and warns for legacy/confusing imports', async () => { + it('warns for legacy/confusing tests include entries', async () => { await writeFile( path.join(tempDir, 'child.eval.yaml'), [ @@ -1679,11 +1558,6 @@ describe('eval.yaml flat runtime controls and tests imports', () => { [ 'name: parent-suite', 'target: parent-target', - 'imports:', - ' suites:', - ' - path: child.eval.yaml', - ' tests:', - ' - path: child.eval.yaml', 'tests:', ' - include: child.eval.yaml', ' type: suite', @@ -1695,14 +1569,16 @@ describe('eval.yaml flat runtime controls and tests imports', () => { expect(result.valid).toBe(true); const warnings = result.errors.filter((error) => error.severity === 'warning'); - expect(warnings.some((error) => error.message.includes('tests[].include is deprecated'))).toBe( - true, - ); expect( - warnings.some((error) => error.message.includes('child target and run controls are ignored')), + warnings.some( + (error) => + error.message.includes('tests[].include with type: suite is deprecated') && + error.message.includes('CLI multi-file selection') && + error.message.includes('tags'), + ), ).toBe(true); expect( - warnings.some((error) => error.message.includes('imports.tests imports raw cases')), + warnings.some((error) => error.message.includes('child target and run controls are ignored')), ).toBe(true); }); diff --git a/packages/core/test/evaluation/loaders/case-file-loader.test.ts b/packages/core/test/evaluation/loaders/case-file-loader.test.ts index 4efc5ab7a..ae6cf2ee5 100644 --- a/packages/core/test/evaluation/loaders/case-file-loader.test.ts +++ b/packages/core/test/evaluation/loaders/case-file-loader.test.ts @@ -493,21 +493,13 @@ tests: file://file-url-cases.json expect(tests[0].id).toBe('file-url-json'); }); - it('keeps imports.tests select working beside file-backed tests', async () => { + it('loads multiple external case files from field-local tests list refs', async () => { await writeFile( - path.join(tempDir, 'import-cases.yaml'), - `- id: imported-keep - criteria: Imported keep - metadata: - group: keep + path.join(tempDir, 'yaml-cases.yaml'), + `- id: yaml-case + criteria: YAML goal vars: - input: Imported keep input -- id: imported-drop - criteria: Imported drop - metadata: - group: drop - vars: - input: Imported drop input + input: YAML input `, ); await writeFile( @@ -515,22 +507,19 @@ tests: file://file-url-cases.json '{"id": "direct-case", "criteria": "Direct goal", "input": "Direct input"}\n', ); await writeFile( - path.join(tempDir, 'imports-and-file-tests.yaml'), - `imports: - tests: - - path: import-cases.yaml - select: - metadata: - group: keep + path.join(tempDir, 'field-local-file-tests.yaml'), + `name: field-local-file-tests prompts: - "{{ input }}" -tests: file://direct-cases.jsonl +tests: + - file://yaml-cases.yaml + - file://direct-cases.jsonl `, ); - const tests = await loadTests(path.join(tempDir, 'imports-and-file-tests.yaml'), tempDir); + const tests = await loadTests(path.join(tempDir, 'field-local-file-tests.yaml'), tempDir); - expect(tests.map((test) => test.id)).toEqual(['imported-keep', 'direct-case']); + expect(tests.map((test) => test.id)).toEqual(['yaml-case', 'direct-case']); }); it('loads promptfoo CSV magic columns through the full suite parser', async () => { diff --git a/packages/core/test/evaluation/validation/eval-file-schema.test.ts b/packages/core/test/evaluation/validation/eval-file-schema.test.ts index 94c61bd31..a01bcd49c 100644 --- a/packages/core/test/evaluation/validation/eval-file-schema.test.ts +++ b/packages/core/test/evaluation/validation/eval-file-schema.test.ts @@ -488,7 +488,7 @@ describe('EvalFileSchema input shorthand', () => { expect(result.success).toBe(true); }); - it('accepts flatter imports with optional inline tests', () => { + it('rejects top-level imports', () => { const result = EvalFileSchema.safeParse({ name: 'wrapper', imports: { @@ -512,10 +512,19 @@ describe('EvalFileSchema input shorthand', () => { tests: [baseTest], }); - expect(result.success).toBe(true); - }); - - it('accepts import-only wrapper evals', () => { + expect(result.success).toBe(false); + expect(result.error?.issues[0]?.path).toEqual(['imports']); + expect(result.error?.issues[0]?.message).toContain("Top-level 'imports' is not supported"); + expect(result.error?.issues[0]?.message).toContain('Run eval files directly'); + expect(result.error?.issues[0]?.message).toContain('tests: file://...'); + expect(result.error?.issues[0]?.message).toContain('prompts: file://...'); + expect(result.error?.issues[0]?.message).toContain('default_test: file://...'); + expect(result.error?.issues[0]?.message).toContain('environment: file://...'); + expect(result.error?.issues[0]?.message).toContain('tags'); + expect(result.error?.issues[0]?.message).toContain('CLI multi-file selection'); + }); + + it('rejects import-only wrapper evals', () => { const result = EvalFileSchema.safeParse({ name: 'wrapper', target: 'codex', @@ -525,7 +534,8 @@ describe('EvalFileSchema input shorthand', () => { }, }); - expect(result.success).toBe(true); + expect(result.success).toBe(false); + expect(result.error?.issues.some((issue) => issue.path.join('.') === 'imports')).toBe(true); }); it('rejects removed experiment authoring blocks', () => { diff --git a/packages/core/test/evaluation/validation/eval-validator.test.ts b/packages/core/test/evaluation/validation/eval-validator.test.ts index 98a34b231..feb77bee1 100644 --- a/packages/core/test/evaluation/validation/eval-validator.test.ts +++ b/packages/core/test/evaluation/validation/eval-validator.test.ts @@ -95,11 +95,11 @@ tests: expect(result.errors).toHaveLength(0); }); - it('validates top-level target and run controls with flatter import entries', async () => { - const filePath = path.join(tempDir, 'run-controls-include.yaml'); + it('validates top-level target and run controls with field-local tests', async () => { + const filePath = path.join(tempDir, 'run-controls-field-local-tests.yaml'); await writeFile( filePath, - `name: wrapper + `name: direct-suite prompts: - "{{ prompt }}" target: codex @@ -115,25 +115,6 @@ tests: - id: local-case vars: prompt: "Hello" -imports: - suites: - - path: ./evals/**/*.eval.yaml - select: - test_ids: [pr50857-*] - tags: [sql-migration] - metadata: - type: [e2e, regression] - priority: high - run: - threshold: 1.0 - repeat: - count: 2 - strategy: pass_all - early_exit: true - timeout_seconds: 120 - budget_usd: 2 - tests: - - path: ./cases/**/*.cases.yaml `, ); @@ -680,7 +661,9 @@ tests: error.severity === 'error' && error.location === 'workspace' && error.message.includes('Parent workspace is not allowed') && - error.message.includes('type: suite'), + error.message.includes('legacy tests[].include suite entries') && + error.message.includes('Run eval files directly') && + error.message.includes('tests: file://...'), ), ).toBe(true); }); @@ -749,7 +732,9 @@ tests: error.severity === 'error' && error.location === 'execution.workspace' && error.message.includes('Parent workspace is not allowed') && - error.message.includes('type: suite'), + error.message.includes('legacy tests[].include suite entries') && + error.message.includes('Run eval files directly') && + error.message.includes('tests: file://...'), ), ).toBe(true); }); @@ -934,12 +919,14 @@ tests: error.severity === 'warning' && error.location === 'tests[0].include' && error.message.includes('child target and run controls are ignored') && - error.message.includes('parent eval owns wrapper target and run controls'), + error.message.includes('legacy tests[].include suite imports') && + error.message.includes('CLI multi-file selection') && + error.message.includes('tags'), ), ).toBe(true); }); - it('warns when imports.tests-style raw imports drop eval suite context', async () => { + it('warns when legacy raw suite includes drop eval suite context', async () => { await writeFile( path.join(tempDir, 'composition-child-tests-import.eval.yaml'), `environment: @@ -975,33 +962,37 @@ tests: (error) => error.severity === 'warning' && error.location === 'tests[0].include' && - error.message.includes('imports.tests imports raw cases') && + error.message.includes('Legacy tests[].include with type: tests') && error.message.includes('drops suite context'), ), ).toBe(true); }); - it('rejects missing raw case files under imports.tests', async () => { - const filePath = path.join(tempDir, 'missing-imports-tests-path.eval.yaml'); + it('rejects top-level imports', async () => { + const filePath = path.join(tempDir, 'top-level-imports.eval.yaml'); await writeFile( filePath, `imports: tests: - path: ./missing-cases.yaml +tests: [] `, ); const result = await validateEvalFile(filePath); expect(result.valid).toBe(false); - expect( - result.errors.some( - (error) => - error.severity === 'error' && - error.location === 'imports.tests[0].path' && - error.message.includes('Cannot read external test file'), - ), - ).toBe(true); + const importsError = result.errors.find( + (error) => error.severity === 'error' && error.location === 'imports', + ); + expect(importsError?.message).toContain("Top-level 'imports' is not supported"); + expect(importsError?.message).toContain('Run eval files directly'); + expect(importsError?.message).toContain('tests: file://...'); + expect(importsError?.message).toContain('prompts: file://...'); + expect(importsError?.message).toContain('default_test: file://...'); + expect(importsError?.message).toContain('environment: file://...'); + expect(importsError?.message).toContain('tags'); + expect(importsError?.message).toContain('CLI multi-file selection'); }); it('rejects removed execution blocks when experiment label is present', async () => { @@ -2057,13 +2048,11 @@ tests: "./cases-shorthand-workspace.yaml" filePath, `prompts: - "{{ prompt }}" -imports: - tests: - - path: file://cases.csv - - path: cases.json - - path: cases.mjs:createTests - - path: cases.py:create_tests tests: + - file://cases.csv + - cases.json + - cases.mjs:createTests + - cases.py:create_tests - id: inline criteria: Goal vars: diff --git a/skills-data/agentv-eval-migrations/references/breaking-changes.md b/skills-data/agentv-eval-migrations/references/breaking-changes.md index 0768efa9e..b5881efbe 100644 --- a/skills-data/agentv-eval-migrations/references/breaking-changes.md +++ b/skills-data/agentv-eval-migrations/references/breaking-changes.md @@ -40,19 +40,19 @@ For a v4.42.4-era eval: `execution.max_concurrency` to `evaluate_options.max_concurrency`, or leave it to `--workers` / project config if it is operator policy. 10. Remove top-level `execution`; current eval YAML rejects it. -11. Replace `workspace.isolation: shared|per_test` with - `workspace.scope: suite|attempt`. +11. Move authored coding-agent testbed setup from public `workspace` fields to + `environment`. 12. Remove `workspace.mode` and `workspace.path` from committed eval YAML. Use `--workspace-path` or `.agentv/config.local.yaml` for local static directories. -13. Replace workspace hook `script:` with `command:`. -14. For executable setup, prefer top-level `extensions`; keep - `workspace.hooks.after_each.reset` for reset policy. +13. Move lifecycle hooks to top-level `extensions`. +14. Put reset policy and portable testbed setup under `environment`; keep + provider environment overrides under top-level `env`. 15. Replace authored `preprocessors` and deprecated Promptfoo `postprocess` with `transform` at `default_test.options`, `tests[].options`, or the assertion that needs the shaped output. -16. Keep raw cases under `tests` or `imports.tests`; import full eval suites - with `imports.suites`. +16. Keep raw cases under `tests` / `tests: file://...`; run full eval suites + directly with CLI multi-file selection and tags. 17. Validate with `bun apps/cli/src/cli.ts validate `. ## Assertions Renamed To `assert` @@ -266,7 +266,7 @@ tests: contains `{type: file, value: "{{ file_path }}"}` and a text block. Store file paths in `default_test.vars` or `tests[].vars`. - Keep `input` only in external raw-case files imported through - `tests: file://...` or `imports.tests` when preserving existing raw datasets. + `tests: file://...` when preserving existing raw datasets. Do not copy that compatibility shape back into normal eval YAML. ### Verification @@ -945,19 +945,18 @@ The external file contained raw case rows. ### Current Shape -Current eval YAML still accepts inline `tests` and `tests: ./cases.yaml`, but -adds explicit imports for composition: +Current eval YAML accepts inline `tests`, `tests: ./cases.yaml`, and field-local +file refs for raw case data. Run multiple full eval suites directly with CLI +selection and tags: ```yaml -imports: - suites: - - path: ../suites/refunds.eval.yaml - tests: - - path: ../cases/refund-smoke.cases.yaml - +prompts: file://../prompts/refund.yaml +default_test: file://../defaults/refund.yaml tests: + - file://../cases/refund-smoke.cases.yaml - id: local-edge-case - input: Can a final-sale item be refunded after damage in transit? + vars: + question: Can a final-sale item be refunded after damage in transit? assert: - Explains the final-sale exception ``` @@ -966,21 +965,21 @@ tests: - Keep `tests: ./cases.yaml` when the file is a raw case array, JSONL, CSV, directory, glob, or script-backed dataset. -- Use `imports.tests` when importing raw rows into the parent suite context. -- Use `imports.suites` when importing full child eval suites that own their - own `workspace`, input, assertions, and task environment. -- Do not define a parent `workspace` in a wrapper eval that imports child - suites through `imports.suites`; child suites own their environments. -- Replace legacy `tests[].include` entries with `imports.suites` or - `imports.tests` where possible. -- Use `run:` on import entries only for scoped overrides: - `threshold`, `repeat`, `timeout_seconds`, and `budget_usd`. +- Use `tests: file://...` or string entries inside `tests` for raw rows that run + in the parent suite context. +- Run full eval suites directly with CLI multi-file selection and tags. Do not + add wrapper-suite import semantics. +- Use `prompts: file://...`, `default_test: file://...`, and + `environment: file://...` to share reusable config locally at the field that + consumes it. +- Use `run:` on individual tests only for scoped overrides: `threshold`, + `repeat`, `timeout_seconds`, and `budget_usd`. ### Verification ```bash bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml -rg -n "include:|imports:|tests:" path/to/evals +rg -n "include:|tests:" path/to/evals ``` ### Compatibility Notes diff --git a/skills-data/agentv-eval-writer/SKILL.md b/skills-data/agentv-eval-writer/SKILL.md index 17c7e4f10..d33653939 100644 --- a/skills-data/agentv-eval-writer/SKILL.md +++ b/skills-data/agentv-eval-writer/SKILL.md @@ -20,20 +20,19 @@ Promptfoo parity matrix: https://agentv.dev/docs/reference/promptfoo-parity/ Treat YAML as the canonical portable model. Prefer authoring `.eval.yaml` / `EVAL.yaml` first, then use TypeScript helpers, Python scripts, or executable graders only when they lower to the same fields or when the evaluation logic must actually run code. Eval files define what is tested and how it runs: prompts, datasets, assertions, -task fixtures, top-level `target`, and suite run controls. Use `imports.suites` -for full child suites that preserve their workspace, shared input, `assert`, -fixtures, and graders. Use `imports.tests` for raw case rows that should run in -the parent file's context. Inline `tests` are also parent-owned raw cases. -String-valued `tests` and string entries inside `tests[]` are raw-case import -shorthand for direct paths, directories, and globs. Legacy `tests[].include` -entries still load with a migration warning, but new evals should use -`imports.suites` or `imports.tests`. Use scoped `run:` on import entries or -individual tests only for `threshold`, `repeat`, `timeout_seconds`, and -legacy `budget_usd`; keep target selection at top-level `target` or CLI `--target`, -put suite budget caps under `evaluate_options.budget_usd`, authored concurrency -under `evaluate_options.max_concurrency`, suite repeat policy under -`evaluate_options.repeat`, -and keep setup and workspace mutation under `workspace`. +task fixtures, top-level `target`, and suite run controls. Use field-local file +refs such as `tests: file://...`, `prompts: file://...`, `default_test: +file://...`, and `environment: file://...`. String-valued `tests` and string +entries inside `tests[]` are raw-case refs for direct paths, directories, and +globs. Run several full eval suites directly with CLI multi-file selection and +tags. Use scoped `run:` on individual tests only for `threshold`, `repeat`, +`timeout_seconds`, and legacy `budget_usd`; keep target selection at top-level +`target` or CLI `--target`, put suite budget caps under +`evaluate_options.budget_usd`, authored concurrency under +`evaluate_options.max_concurrency`, suite repeat policy under +`evaluate_options.repeat`, coding-agent testbed setup under `environment`, +provider environment overrides under `env`, and lifecycle hooks under +`extensions`. Use `@agentv/sdk` for TypeScript helper imports. Do not use `@agentv/eval` for new evals, examples, scaffolds, or skill guidance; it was a deprecated compatibility package and has been removed from this repository. @@ -42,7 +41,7 @@ Use `@agentv/sdk` for TypeScript helper imports. Do not use `@agentv/eval` for n - Put grading criteria in `assert`, not in test-level `criteria`. Plain assertion strings become an `llm-rubric` grader. - Prefer plain assertion strings for semantic checks when the default rubric grader can judge them. Use `type: llm-rubric` for structured criteria, custom prompts, custom grader targets, or assertion-level transforms, and `type: script` when grading must execute code. - Write `expected_output` as a golden/reference answer the target could have produced. Do not write criteria, scoring instructions, or "the agent should..." rubric prose there. -- For historical or repo-state evals, materialize the repo under `workspace.repos[]` pinned to the commit under test. Mentioning a SHA only in prompt prose is not enough because the agent needs an actual checkout to inspect. +- For historical or repo-state evals, materialize the repo through a pinned `environment` setup recipe. Mentioning a SHA only in prompt prose is not enough because the agent needs an actual checkout to inspect. ## Evaluation Types @@ -132,8 +131,8 @@ tests: ## Eval File Structure -**Required:** `tests` (array or string raw-case path) or `imports` -**Optional:** `name`, `description`, `experiment`, `version`, `author`, `tags`, `license`, `requires`, `target`, `targets`, `prompts`, `default_test`, `timeout_seconds`, `evaluate_options`, `threshold`, `suite`, `workspace`, `assert` +**Required:** `tests` (array or string raw-case path) or `scenarios` +**Optional:** `name`, `description`, `experiment`, `version`, `author`, `tags`, `license`, `requires`, `target`, `targets`, `prompts`, `default_test`, `timeout_seconds`, `evaluate_options`, `threshold`, `suite`, `environment`, `env`, `extensions`, `assert` **Test fields:** @@ -144,7 +143,7 @@ tests: | `expected_output` | no | Gold-standard reference answer (string shorthand or full message array) | | `assert` | yes | Graders: deterministic checks, `llm-rubric` checks, script graders, or plain string rubric criteria | | `execution` | no | Per-case grader/default overrides such as `skip_defaults`; target selection belongs in top-level `target` or CLI `--target` | -| `workspace` | no | Per-case workspace config (overrides suite-level) | +| `environment` | no | Per-case coding-agent testbed config (overrides suite-level) | | `metadata` | no | Arbitrary key-value pairs passed to setup/teardown scripts | | `conversation_id` | no | Thread grouping | @@ -215,7 +214,7 @@ then render those vars from the prompt template next to the input. **JSONL format:** One test per line as JSON. Optional `.yaml` sidecar for shared defaults. See `examples/features/basic-jsonl/`. -**Environment variables:** Use `{{ env.VAR }}` templates in authored config. Missing vars resolve to empty string. Works in eval files, external case files, and workspace configs. `.env` files are loaded automatically. +**Environment variables:** Use `{{ env.VAR }}` templates in authored config. Missing vars resolve to empty string. Works in eval files, external case files, and environment configs. `.env` files are loaded automatically. ## Output Transforms @@ -431,25 +430,21 @@ assert: If a required grader scores below its threshold, the overall verdict is forced to `fail`. -## Workspace Setup/Teardown +## Environment Setup/Teardown Run scripts before/after each test. Define at suite level or override per case: ```yaml -workspace: - template: ./workspace-templates/my-project - repos: - - path: ./repo +environment: + type: host + workdir: ./repo + setup: + command: ["bun", "run", "setup.ts"] + args: repo: sympy/sympy commit: "abc123" - hooks: - before_all: - command: ["bun", "run", "setup.ts"] - timeout_ms: 120000 - after_each: - reset: fast - after_all: - command: ["bun", "run", "teardown.ts"] +extensions: + - file://scripts/teardown.mjs:afterAll tests: - id: case-1 @@ -459,42 +454,41 @@ tests: source_commit: "abc123" ``` -**Lifecycle:** template copy → repo materialization → workspace before_all → target before_all → git baseline → before_each hooks → agent → file changes → after_each hooks → after_all hooks → cleanup -**Merge:** Case-level fields replace suite-level fields. +**Lifecycle:** environment setup → lifecycle extensions → target setup → agent → grading → teardown extensions → cleanup +**Merge:** Case-level environment fields replace suite-level fields. **Commands receive stdin JSON:** `{workspace_path, test_id, eval_run_id, case_input, case_metadata}` **Setup failure:** aborts case. **Teardown failure:** non-fatal (warning). For SWE-bench-style evals, put operational checkout state under -`workspace.repos[].commit`; treat `metadata.source_commit` as informational -only. A SHA in the prompt or metadata without a matching workspace repo pin is +`environment` setup args; treat `metadata.source_commit` as informational only. +A SHA in the prompt or metadata without a matching environment setup recipe is not an operational checkout. -### Repository Lifecycle +### Environment Lifecycle -Materialize repos into the eval workspace automatically. Repo entries declare identity and checkout pins only; AgentV resolves acquisition from configured `repo_resolvers`, then registered projects, `git_cache.mirrors`, its mirror cache, and remote clone. `repo_resolvers` and `git_cache.mirrors` may be defined in `$AGENTV_HOME/config.yaml`, the project's committed `.agentv/config.yaml`, or a gitignored `.agentv/config.override.yaml` (highest precedence) — use the override for machine-specific local bindings without editing tracked or user-global config. Shared repo workspaces use fresh temp materialization by default: +Describe coding-agent testbeds with `environment`. Reusable recipes should live +in field-local files and be loaded with `environment: file://...`: ```yaml -workspace: - repos: - - path: ./repo - repo: https://github.com/org/repo.git - commit: main - ancestor: 1 # parent commit - hooks: - after_each: - reset: fast # none | fast | strict - scope: suite # suite | attempt -``` - -- `repo`: full clone URL or GitHub `org/name` shorthand -- `commit`: branch, tag, or SHA to check out -- `ancestor`: walk N commits back from the checked-out ref -- `sparse`: sparse checkout paths array -- Harness-managed repo workspaces use temp materialization by default; use `workspace.scope: suite | attempt` for portable lifetime -- Existing local workspace directories are machine-local bindings; use `--workspace-path` or `.agentv/config.local.yaml` with `execution.workspace_path` -- `hooks.enabled`: boolean (default `true`); set `false` to skip all lifecycle hooks -- `agentv workspace deps ` scans eval files and outputs a JSON manifest of required git repos (useful for CI pre-cloning) - -See https://agentv.dev/targets/configuration/#repository-lifecycle +environment: file://.agentv/environments/repo.yaml +``` + +```yaml +# .agentv/environments/repo.yaml +type: host +workdir: ./repo +setup: + command: ./scripts/materialize-repo.sh + args: + repo: https://github.com/org/repo.git + commit: main + ancestor: 1 +``` + +- `type`: `host` or `docker` +- `workdir`: path the target and graders should use +- `setup`: command and args for repository/testbed materialization +- Top-level `env`: provider/eval environment overrides +- `extensions`: lifecycle hooks such as `beforeAll`, `beforeEach`, `afterEach`, and `afterAll` ## Grader Types diff --git a/skills-data/agentv-eval-writer/references/eval.schema.json b/skills-data/agentv-eval-writer/references/eval.schema.json index b510193cd..73cdda008 100644 --- a/skills-data/agentv-eval-writer/references/eval.schema.json +++ b/skills-data/agentv-eval-writer/references/eval.schema.json @@ -205,588 +205,7 @@ } }, "imports": { - "type": "object", - "properties": { - "suites": { - "anyOf": [ - { - "type": "array", - "items": { - "anyOf": [ - { - "type": "object", - "properties": { - "path": { - "type": "string", - "minLength": 1 - }, - "select": { - "anyOf": [ - { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - { - "type": "object", - "properties": { - "test_ids": { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - "tags": { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - "metadata": { - "type": "object", - "additionalProperties": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "number" - }, - { - "type": "boolean" - }, - { - "type": "array", - "items": { - "type": ["string", "number", "boolean"] - }, - "minItems": 1 - } - ] - } - } - }, - "additionalProperties": false - } - ] - }, - "run": { - "type": "object", - "properties": { - "threshold": { - "type": "number", - "minimum": 0, - "maximum": 1 - }, - "repeat": { - "type": "object", - "properties": { - "count": { - "type": "integer", - "minimum": 1 - }, - "strategy": { - "type": "string", - "enum": ["pass_any", "pass_all", "mean", "confidence_interval"] - }, - "early_exit": { - "type": "boolean" - }, - "cost_limit_usd": { - "type": "number", - "minimum": 0 - } - }, - "required": ["count"], - "additionalProperties": false - }, - "timeout_seconds": { - "type": "number", - "exclusiveMinimum": true, - "minimum": 0 - }, - "budget_usd": { - "type": "number", - "exclusiveMinimum": true, - "minimum": 0 - } - }, - "additionalProperties": false - } - }, - "required": ["path"], - "additionalProperties": false - }, - { - "type": "string", - "minLength": 1 - } - ] - } - }, - { - "type": "string", - "minLength": 1 - }, - { - "type": "object", - "properties": { - "path": { - "type": "string", - "minLength": 1 - }, - "select": { - "anyOf": [ - { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - { - "type": "object", - "properties": { - "test_ids": { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - "tags": { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - "metadata": { - "type": "object", - "additionalProperties": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "number" - }, - { - "type": "boolean" - }, - { - "type": "array", - "items": { - "type": ["string", "number", "boolean"] - }, - "minItems": 1 - } - ] - } - } - }, - "additionalProperties": false - } - ] - }, - "run": { - "type": "object", - "properties": { - "threshold": { - "type": "number", - "minimum": 0, - "maximum": 1 - }, - "repeat": { - "type": "object", - "properties": { - "count": { - "type": "integer", - "minimum": 1 - }, - "strategy": { - "type": "string", - "enum": ["pass_any", "pass_all", "mean", "confidence_interval"] - }, - "early_exit": { - "type": "boolean" - }, - "cost_limit_usd": { - "type": "number", - "minimum": 0 - } - }, - "required": ["count"], - "additionalProperties": false - }, - "timeout_seconds": { - "type": "number", - "exclusiveMinimum": true, - "minimum": 0 - }, - "budget_usd": { - "type": "number", - "exclusiveMinimum": true, - "minimum": 0 - } - }, - "additionalProperties": false - } - }, - "required": ["path"], - "additionalProperties": false - } - ] - }, - "tests": { - "anyOf": [ - { - "type": "array", - "items": { - "anyOf": [ - { - "type": "object", - "properties": { - "path": { - "type": "string", - "minLength": 1 - }, - "select": { - "anyOf": [ - { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - { - "type": "object", - "properties": { - "test_ids": { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - "tags": { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - "metadata": { - "type": "object", - "additionalProperties": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "number" - }, - { - "type": "boolean" - }, - { - "type": "array", - "items": { - "type": ["string", "number", "boolean"] - }, - "minItems": 1 - } - ] - } - } - }, - "additionalProperties": false - } - ] - }, - "run": { - "type": "object", - "properties": { - "threshold": { - "type": "number", - "minimum": 0, - "maximum": 1 - }, - "repeat": { - "type": "object", - "properties": { - "count": { - "type": "integer", - "minimum": 1 - }, - "strategy": { - "type": "string", - "enum": ["pass_any", "pass_all", "mean", "confidence_interval"] - }, - "early_exit": { - "type": "boolean" - }, - "cost_limit_usd": { - "type": "number", - "minimum": 0 - } - }, - "required": ["count"], - "additionalProperties": false - }, - "timeout_seconds": { - "type": "number", - "exclusiveMinimum": true, - "minimum": 0 - }, - "budget_usd": { - "type": "number", - "exclusiveMinimum": true, - "minimum": 0 - } - }, - "additionalProperties": false - } - }, - "required": ["path"], - "additionalProperties": false - }, - { - "type": "string", - "minLength": 1 - } - ] - } - }, - { - "type": "string", - "minLength": 1 - }, - { - "type": "object", - "properties": { - "path": { - "type": "string", - "minLength": 1 - }, - "select": { - "anyOf": [ - { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - { - "type": "object", - "properties": { - "test_ids": { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - "tags": { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - }, - "minItems": 1 - } - ] - }, - "metadata": { - "type": "object", - "additionalProperties": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "number" - }, - { - "type": "boolean" - }, - { - "type": "array", - "items": { - "type": ["string", "number", "boolean"] - }, - "minItems": 1 - } - ] - } - } - }, - "additionalProperties": false - } - ] - }, - "run": { - "type": "object", - "properties": { - "threshold": { - "type": "number", - "minimum": 0, - "maximum": 1 - }, - "repeat": { - "type": "object", - "properties": { - "count": { - "type": "integer", - "minimum": 1 - }, - "strategy": { - "type": "string", - "enum": ["pass_any", "pass_all", "mean", "confidence_interval"] - }, - "early_exit": { - "type": "boolean" - }, - "cost_limit_usd": { - "type": "number", - "minimum": 0 - } - }, - "required": ["count"], - "additionalProperties": false - }, - "timeout_seconds": { - "type": "number", - "exclusiveMinimum": true, - "minimum": 0 - }, - "budget_usd": { - "type": "number", - "exclusiveMinimum": true, - "minimum": 0 - } - }, - "additionalProperties": false - } - }, - "required": ["path"], - "additionalProperties": false - } - ] - } - }, - "additionalProperties": false + "not": {} }, "tests": { "anyOf": [