Skip to content

fix(config): reject top-level eval imports #1678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
christso merged 1 commit into from
Jul 5, 2026
Merged

fix(config): reject top-level eval imports #1678

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions CONCEPTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,13 @@ Shared domain vocabulary for this project — entities, named processes, and sta

**Matrix authoring** — The Promptfoo-compatible shape AgentV adopts where useful: `prompts x tests/vars x targets`, with repeat samples and retries applied as run policy after the authored matrix is resolved. AgentV uses this matrix model without copying Promptfoo wholesale. AgentV-native boundaries remain: `targets` identify systems under test, `provider` names the backend/adapter kind inside a target, `environment` recipes prepare coding-agent testbeds, `env` carries provider/eval variables, `extensions` are lifecycle hooks, reusable content uses field-local `file://` refs, and grouping uses tags plus run-bundle metadata.

**Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `environment`, assertions, expected references, and judge criteria. It can run directly or share reusable parts through field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, `defaults: file://...`, and `environment: file://...`.
**Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `environment`, assertions, expected references, and judge criteria. It runs directly or shares reusable parts through field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, `default_test: file://...`, and `environment: file://...`.

**Raw case file** — YAML, JSONL, or directory case data loaded with `tests: file://./cases.yaml`, string shorthand, or another supported field-local tests reference. Raw cases are reusable data inputs; they do not carry imported suite context such as shared `environment`, shared `input`, or shared `assertions`.
**Raw case file** — YAML, JSONL, or directory case data loaded with `tests: file://./cases.yaml`, string shorthand, or another supported field-local tests reference. Raw cases are reusable data inputs; they do not carry suite context such as shared `environment`, shared `prompts`, or shared `assertions`.

**Policy eval** — Eval YAML whose main job is to bind top-level runtime policy such as target selection, repeat count, timeout, budget, thresholds, and tags around explicit prompts/tests/targets refs. Policy evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. Use tags and run-bundle metadata for grouping rather than experiment path buckets, Vercel path layout, or model-as-experiment grouping.
**Policy eval** — Eval YAML whose main job is to bind top-level runtime policy such as target selection, repeat count, timeout, budget, thresholds, and tags around explicit prompts/tests/targets refs. Policy evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. Use tags and run-bundle metadata for grouping rather than experiment path buckets, Vercel path layout, model-as-experiment grouping, or wrapper-suite import semantics.

**Multi-file run** — A CLI-selected set of eval YAML files. Use multiple eval files, tags, and CLI run selection to group suites under one run intent without adding wrapper-suite semantics to YAML.

**Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Experiment is expressed as the reserved `tags.experiment` key (see **Tags**); there is no top-level `experiment` field. Runtime policy belongs in top-level eval fields or target objects; the experiment label is recorded in `summary.json` and `.internal/index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `extensions` or target hooks, not in a separate experiment artifact.

Expand Down
22 changes: 13 additions & 9 deletions apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,14 +197,18 @@ tests:
```

The `metadata` field is included in the stdin JSON passed to lifecycle commands as `case_metadata`.
Operational checkout state belongs under `workspace.repos[].commit`; matching metadata fields such as `source_commit` are informational only.
For historical repo-state evals, pin the checkout under `workspace.repos[]`
instead of only mentioning the SHA in prompt prose:
Operational checkout state belongs in `environment`; matching metadata fields
such as `source_commit` are informational only. For historical repo-state evals,
pin the checkout in an environment setup recipe instead of only mentioning the
SHA in prompt prose:

```yaml
workspace:
repos:
- path: ./agentv
environment:
type: host
workdir: ./agentv
setup:
command: ./scripts/materialize-repo.sh
args:
repo: https://github.com/EntityProcess/agentv.git
commit: 5e3c8f46d80fe66b1a75659e4fd94e38a7e09215
```
Expand Down Expand Up @@ -503,7 +507,7 @@ tests:
- Generates valid Python
```

`metadata` is passed to workspace lifecycle hooks as `case_metadata`, preserved
`metadata` is passed to lifecycle hooks as `case_metadata`, preserved
in result records, and available to in-process custom assertions. AgentV does
not interpret arbitrary metadata keys itself; use `workspace`, `execution`,
`input`, `expected_output`, and `assert` for operational behavior.
not interpret arbitrary metadata keys itself; use `environment`, `extensions`,
`prompts`, `expected_output`, and `assert` for operational behavior.
130 changes: 59 additions & 71 deletions apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,17 @@ sidebar:
slug: docs/evaluation/eval-files
---

Evaluation files define the test cases, graders, workspace lifecycle, and run
Evaluation files define the test cases, graders, environment, lifecycle hooks, and run
controls for an evaluation run. The reserved `tags.experiment` key is the
run/result grouping label, top-level `target` identifies the system under test,
and fields such as `evaluate_options.repeat`, `threshold`, `timeout_seconds`,
`evaluate_options.budget_usd`, and `evaluate_options.max_concurrency` control repeated
attempts and gates. Workspace lifetime belongs under `workspace.scope`;
repository provenance belongs under `workspace.repos`; Docker/container binding
belongs under `workspace.docker`. Non-provisioning setup commands belong in
top-level `extensions`; reset policy stays under
`workspace.hooks.after_each.reset`; runner-specific setup belongs in the
`target` object, in `targets`, or in project config. AgentV supports two eval
data formats: YAML and JSONL.
attempts and gates. Coding-agent testbeds, workdirs, Docker config, repository
materialization, setup, and reset policy belong in `environment`. Provider
environment-variable overrides belong in top-level `env`. Lifecycle hooks belong
in `extensions`; runner-specific setup belongs in the `target` object, in
`targets`, or in project config. AgentV supports two eval data formats: YAML and
JSONL.

YAML is the canonical portable model. TypeScript helpers, generated fixtures, and Python scripts should lower to the same YAML/JSONL shapes rather than inventing a separate eval contract.
Eval files describe the task, target binding, and run controls. Use
Expand All @@ -35,41 +34,43 @@ surfaces.

Eval YAML is AgentV's composable and runnable authoring primitive. It is a
focused, shareable slice of the same config graph as `.agentv/config.yaml`.
Use ordinary `*.eval.yaml` files for direct task suites and for wrapper evals
that compose other suites. Raw case files are reusable data inputs, not a
second runnable experiment format.
Use ordinary `*.eval.yaml` files for direct task suites. Raw case files are
reusable data inputs, not a second runnable experiment format.

- A **task suite** is eval YAML that owns task context: `workspace`, shared
`input`, shared `assert`, fixtures, graders, and test cases. It can run
directly or be imported through `imports.suites`.
- A **task suite** is eval YAML that owns task context: `environment`, shared
`prompts`, shared `assert`, fixtures, graders, and test cases. It can run
directly.
- A **raw case file** is a YAML, JSON, JSONL, CSV, script-backed dataset,
directory, or glob of cases. Import it with `imports.tests`,
`tests: ./cases.yaml`, `tests: file://cases.csv`, or string shorthand; parent
suite context applies because raw cases do not carry their own suite context.
- A **wrapper eval** is eval YAML that imports one or more suites with
`imports.suites` and binds run controls with top-level `target`, `threshold`,
`timeout_seconds`, and `evaluate_options`.
Wrapper evals can live anywhere in the repo. A wrapper that imports suites
with `imports.suites` must not define parent `workspace`; imported suites own
task environment. Machine-local existing workspace paths belong in CLI flags
or `config.local.yaml`, not eval YAML.
directory, or glob of cases. Reference it with `tests: ./cases.yaml`,
`tests: file://cases.csv`, or string entries inside `tests`; parent suite
context applies because raw cases do not carry their own suite context.
- Use multiple eval files, tags, and CLI selection when you want to run several
suites together. Each eval file owns its own task environment and run policy.

For example, a reusable task suite can keep the task contract in one file:

```yaml
# evals/suites/refunds.eval.yaml
suite: refunds
workspace:
repos:
- path: ./support-app
environment:
type: host
workdir: ./support-app
setup:
command: ./scripts/materialize-repo.sh
args:
repo: acme/support-app
commit: main
input: Answer using the refund policy in the workspace.
prompts:
- Answer using the refund policy in {{ repo_path }}.
default_test:
vars:
repo_path: ./support-app
assert:
- Applies the refund policy correctly
tests:
- id: missing-receipt
input: Can this customer get a refund without a receipt?
vars:
question: Can this customer get a refund without a receipt?
```

Raw cases are just case data:
Expand All @@ -81,10 +82,10 @@ Raw cases are just case data:
expected_output: Offer a replacement or refund path.
```

A wrapper eval stays ordinary eval YAML while choosing a target and run controls:
A run-focused eval stays ordinary eval YAML while choosing a target and run controls:

```yaml
# experiments/refunds-codex.eval.yaml
# evals/refunds-codex.eval.yaml
name: refunds-codex
target: codex-gpt5
evaluate_options:
Expand All @@ -93,28 +94,21 @@ evaluate_options:
count: 2
strategy: pass_any

imports:
suites:
- path: ../evals/suites/refunds.eval.yaml
tests:
- path: ../evals/cases/refund-smoke.cases.yaml

tests:
- file://../evals/cases/refund-smoke.cases.yaml
- id: local-edge-case
vars:
question: Can a final-sale item be refunded after damage in transit?
expected_output: Explain the final-sale exception for damaged transit.
```

The `experiments/` directory in that example is optional and user-owned. AgentV
does not infer behavior from the path; the wrapper runs because it is eval YAML
with tests or imports. The wrapper owns target selection and run controls. Put
workspace setup in imported child suites. Parent workspace-affecting fields,
including top-level `workspace`, are for parent-owned raw cases, including
cases imported with `imports.tests`. Runtime workspace path overrides belong in
CLI flags or `.agentv/config.local.yaml`; repos, hooks, templates, Docker
config, env checks, and workspace scope belong in top-level or case-level
`workspace`.
The `experiments/` directory is optional and user-owned. AgentV does not infer
behavior from the path; files run because they are eval YAML with tests or
scenarios. Runtime workspace path overrides belong in CLI flags or
`.agentv/config.local.yaml`; coding-agent testbeds, workdirs, Docker config,
repository setup, and reset policy belong in top-level or case-level
`environment`. Provider environment overrides belong in `env`; lifecycle hooks
belong in `extensions`.

## YAML Format

Expand Down Expand Up @@ -153,20 +147,20 @@ tests:
| `evaluate_options` | Optional evaluation runtime options such as `budget_usd`, `repeat`, and `max_concurrency` |
| `timeout_seconds` | Optional per-case timeout |
| `threshold` | Optional suite quality threshold |
| `workspace` | Suite-level task environment — inline object or string path to an external workspace file. Repo entries declare identity and checkout pins; acquisition is covered in [Workspace Architecture](/docs/guides/workspace-architecture/#repo-provenance-vs-acquisition). |
| `extensions` | Lifecycle hooks: `file://path/to/hooks.mjs:beforeAll`, `beforeEach`, `afterEach`, `afterAll`, plus the built-in `agentv:agent-rules`. Hooks run after `workspace.repos` materializes. |
| `imports` | Optional import groups. `imports.suites` imports full child eval suites with their task context. `imports.tests` imports raw test rows into this file's context. Import entries may use scoped `run:` overrides for `threshold`, `repeat`, `timeout_seconds`, and `budget_usd`. |
| `tests` | Inline raw tests or a string path to an external raw-case file or directory. Legacy `tests[].include` entries still load with a migration warning; prefer `imports.suites` or `imports.tests`. |
| `environment` | Suite-level coding-agent testbed — inline object or `file://` reference for host/Docker setup, workdir, repositories, fixtures, services, and reset policy. |
| `env` | Promptfoo-compatible provider/eval environment-variable overrides and load-time template inputs. |
| `extensions` | Lifecycle hooks: `file://path/to/hooks.mjs:beforeAll`, `beforeEach`, `afterEach`, `afterAll`, plus the built-in `agentv:agent-rules`. |
| `tests` | Inline raw tests, a string path to an external raw-case file or directory, or a list mixing inline tests with raw-case file refs. |
| `assert` | Suite-level graders appended to each test unless `execution.skip_defaults: true` is set on the test |

`workspace` is what the agent can inspect or modify through tools, not prompt
`environment` is what the agent can inspect or modify through tools, not prompt
input. Put task instructions and chat/system/user messages in `prompts`; put
repos, templates, Docker config, env checks, scope, and repo provenance in
`workspace`. Put lifecycle setup that does not acquire repos in `extensions`.
coding-agent testbed setup, workdir, Docker config, services, and repo
provenance in `environment`. Put lifecycle hooks in `extensions`.

For historical or repo-state evals, put the checkout under
`workspace.repos[].commit`. A commit SHA in the prompt or metadata is useful
context, but it does not materialize a repo for the agent to inspect.
`environment` with a pinned setup recipe. A commit SHA in the prompt or metadata
is useful context, but it does not materialize a repo for the agent to inspect.

### Prompts, Vars, and Target Expansion

Expand Down Expand Up @@ -247,9 +241,9 @@ tests:
input: Summarize the July release notes.
```

External raw-case files imported through `tests: file://...` or
`imports.tests` may still contain raw internal `input` rows for compatibility.
Keep that compatibility out of normal eval YAML authoring.
External raw-case files referenced through `tests: file://...` may still contain
raw internal `input` rows for compatibility. Keep that compatibility out of
normal eval YAML authoring.

### Lifecycle Extensions

Expand All @@ -270,7 +264,7 @@ Setup hook failures (`beforeAll`, `beforeEach`) fail the affected run; teardown
hook failures (`afterEach`, `afterAll`) are non-fatal.

`agentv:agent-rules` is the only built-in extension in this slice. It runs after
workspace materialization and exposes staged rule paths to providers and result
environment materialization and exposes staged rule paths to providers and result
metadata as `agent_rules_paths`:

```yaml
Expand All @@ -285,7 +279,7 @@ extensions:

If `agentv:agent-rules` is authored as a string, it defaults to `beforeAll` and
discovers conventional rule locations already present in the materialized
workspace. It does not clone repositories or replace `workspace.repos`.
environment. It does not clone repositories or replace `environment`.

### Metadata Fields

Expand Down Expand Up @@ -506,24 +500,18 @@ validation instead of being skipped at runtime.
assertion `min_score`. Ordinary columns become `vars`, so CSV rows can rely on
suite-level `input` that interpolates those variables.

String shorthand is raw-case-only. Import reusable task suites through
`imports.suites`; use `imports.tests` when you want to drop suite context and
import only raw cases into the parent context:
String shorthand is raw-case-only. Use a direct raw-case path or file ref when
you want case data to run in the parent suite context:

```yaml
imports:
suites:
- path: ./suites/*.eval.yaml
tests:
- path: ./cases/regression.jsonl

tests:
- file://./cases/regression.jsonl
- id: local-edge-case
input: ...
```

Legacy `tests[].include` entries still load with a migration warning for older
eval files, but new evals should use `imports.suites` or `imports.tests`.
Run multiple eval files directly from the CLI, and use tags to group suites that
should be selected together.

### Raw Cases as Directory Paths

Expand Down
Loading
Loading