diff --git a/README.md b/README.md index 62b01b1b2..29b91ba85 100644 --- a/README.md +++ b/README.md @@ -61,13 +61,13 @@ targets: - name: copilot-sdk provider: anthropic model: claude-sonnet-4.6 - max_budget_usd: 0.50 ``` **3. Create an eval** in `evals/`: ```yaml description: Code generation quality -experiment: with-skills +tags: + experiment: with-skills target: copilot-sdk evaluate_options: repeat: @@ -103,7 +103,8 @@ The target can be an eval-local object when this eval needs target settings of i ```yaml description: Code generation quality with Copilot target settings -experiment: with-skills +tags: + experiment: with-skills target: extends: copilot-sdk model: claude-sonnet-4.6 @@ -120,7 +121,7 @@ tests: input: Write FizzBuzz in Python ``` -`target: copilot-sdk` resolves the named target from `.agentv/targets.yaml` or `targets.yaml` and uses its default provider, model, hooks, and provider settings. The object form above starts from `copilot-sdk`, then applies the eval-local fields for this eval. If `extends` is omitted, the object defines the full target inline and must include enough provider configuration to run. AgentV records the resolved target information in run artifacts so results can be audited and replayed. The experiment label stays `with-skills` because the condition is unchanged; the model/provider variation belongs to the resolved target metadata. +`target: copilot-sdk` resolves the named target from `.agentv/targets.yaml` or `targets.yaml` and uses its default provider, model, hooks, and provider settings. The object form above starts from `copilot-sdk`, then applies the eval-local fields for this eval. If `extends` is omitted, the object defines the full target inline and must include enough provider configuration to run. AgentV records the resolved target information in run artifacts so results can be audited and replayed. The `tags.experiment` label stays `with-skills` because the condition is unchanged; the model/provider variation belongs to the resolved target metadata. Use `default_test.threshold` for the inherited per-test pass cutoff. Existing eval files with a top-level `threshold` still load during migration, and `--threshold` on the CLI still overrides YAML thresholds for a run. @@ -136,7 +137,7 @@ agentv compare .agentv/results//index.jsonl .agentv/results//`. In this example, `experiment: with-skills` names the condition being measured and `target: copilot-sdk` selects the system under test from `targets.yaml`; both are recorded as metadata, not path segments. The root `index.jsonl` manifest is the portable row index used by scripts, CI, and `agentv compare`; per-case sidecars include the resolved eval and target configuration used for the run. +Each run writes a portable bundle directly under `.agentv/results//`. In this example, `tags.experiment: with-skills` names the condition being measured and `target: copilot-sdk` selects the system under test from `targets.yaml`; both are recorded as metadata, not path segments. The root `index.jsonl` manifest is the portable row index used by scripts, CI, and `agentv compare`; per-case sidecars include the resolved eval and target configuration used for the run. ```bash agentv eval evals/my-eval.yaml @@ -203,7 +204,7 @@ import { defineEval } from '@agentv/sdk'; export default defineEval({ description: 'Code generation quality', - experiment: 'with-skills', + tags: { experiment: 'with-skills' }, target: { extends: 'copilot-sdk', model: 'claude-sonnet-4.6', diff --git a/apps/web/src/content/docs/docs/evaluation/eval-files.mdx b/apps/web/src/content/docs/docs/evaluation/eval-files.mdx index 87ce57f03..9dc940b97 100644 --- a/apps/web/src/content/docs/docs/evaluation/eval-files.mdx +++ b/apps/web/src/content/docs/docs/evaluation/eval-files.mdx @@ -5,7 +5,7 @@ sidebar: order: 1 --- -Evaluation files define the test cases, graders, workspace lifecycle, and run controls for an evaluation run. Top-level `experiment` is the run/result grouping label, top-level `target` identifies the system under test, and fields such as `repeat`, `threshold`, `timeout_seconds`, `evaluate_options.budget_usd`, and `evaluate_options.max_concurrency` control repeated attempts and gates. Workspace reuse belongs under `workspace.isolation`; Docker/container binding belongs under `workspace.docker`. Install, build, and reset commands belong under `workspace.hooks`; runner-specific setup belongs in the `target` object or `targets.yaml`. AgentV supports two eval data formats: YAML and JSONL. +Evaluation files define the test cases, graders, workspace lifecycle, and run controls for an evaluation run. The reserved `tags.experiment` key is the run/result grouping label, top-level `target` identifies the system under test, and fields such as `repeat`, `threshold`, `timeout_seconds`, `evaluate_options.budget_usd`, and `evaluate_options.max_concurrency` control repeated attempts and gates. Workspace reuse belongs under `workspace.isolation`; Docker/container binding belongs under `workspace.docker`. Install, build, and reset commands belong under `workspace.hooks`; runner-specific setup belongs in the `target` object or `targets.yaml`. AgentV supports two eval data formats: YAML and JSONL. YAML is the canonical portable model. TypeScript helpers, generated fixtures, and Python scripts should lower to the same YAML/JSONL shapes rather than inventing a separate eval contract. Eval files describe the task, target binding, and run controls. Use `evaluate_options.max_concurrency` for authored suite concurrency. Operators can still override concurrency with `--workers` or set defaults with `execution.workers` in `agentv.config.*` / `.agentv/config.yaml`; do not author legacy `workers` fields in eval YAML. diff --git a/apps/web/src/content/docs/docs/evaluation/experiments.mdx b/apps/web/src/content/docs/docs/evaluation/experiments.mdx index 6647ce90e..aab8f78f6 100644 --- a/apps/web/src/content/docs/docs/evaluation/experiments.mdx +++ b/apps/web/src/content/docs/docs/evaluation/experiments.mdx @@ -6,7 +6,7 @@ sidebar: --- AgentV eval files are the runnable authoring artifact. Use top-level -`description` for display metadata, `experiment` as the run/result grouping +`description` for display metadata, `tags.experiment` as the run/result grouping label, `target` for the system under test, and flat top-level run controls such as `repeat`, `timeout_seconds`, and `threshold`. Use `evaluate_options` for evaluation runtime options such as `budget_usd` and `max_concurrency`. @@ -17,7 +17,8 @@ overrides. ```yaml name: support-regression description: Support regression suite -experiment: support-codex +tags: + experiment: support-codex target: extends: codex-gpt5 model: gpt-5.1 @@ -252,11 +253,11 @@ Eval runs write to a direct run bundle: ``` CLI `--experiment` sets the experiment label explicitly. Without that flag, AgentV -uses a reserved `tags.experiment` key (see below), then top-level `experiment`, -then top-level `name`, then the eval filename. The precedence is -`--experiment` > `tags.experiment` > default. The Dashboard uses "Experiment" for -the comparison and result grouping concept; folder names are only storage -allocation and must not define result semantics. +uses the reserved `tags.experiment` key (see below), then the suite `name`, then +the eval filename. The precedence is `--experiment` > `tags.experiment` > default. +There is no top-level `experiment` field — a run is labeled with `tags.experiment`. +The Dashboard uses "Experiment" for the comparison and result grouping concept; +folder names are only storage allocation and must not define result semantics. ### Tags as run metadata (`tags.experiment`) diff --git a/packages/sdk/src/eval.ts b/packages/sdk/src/eval.ts index 1a925a8d0..0a04d49cc 100644 --- a/packages/sdk/src/eval.ts +++ b/packages/sdk/src/eval.ts @@ -204,12 +204,23 @@ export interface EvalDefinition { readonly category?: string; readonly version?: string; readonly author?: string; - readonly tags?: readonly string[]; + /** + * Suite tags. Either the selection list form (`string[]`, drives + * `select.tags` / `--tag name` filtering) or the promptfoo-shaped + * `Record` map. In the map form the reserved `experiment` key + * labels the run/experiment (grouped by the Dashboard), matching + * `tags.experiment` in YAML evals. + */ + readonly tags?: readonly string[] | Readonly>; readonly license?: string; readonly requires?: EvalRequires; readonly input?: string | readonly EvalMessage[]; readonly inputFiles?: readonly string[]; readonly tests: readonly EvalTest[] | string; + /** + * @deprecated A top-level `experiment` label no longer sets the run's + * experiment namespace. Use `tags: { experiment: '' }` instead. + */ readonly experiment?: string; readonly target?: string | EvalTargetConfig; readonly repeat?: EvalRepeat; diff --git a/packages/sdk/test/eval-authoring.test.ts b/packages/sdk/test/eval-authoring.test.ts index 4fd98f546..96e463308 100644 --- a/packages/sdk/test/eval-authoring.test.ts +++ b/packages/sdk/test/eval-authoring.test.ts @@ -194,6 +194,30 @@ describe('YAML-aligned eval authoring helpers', () => { expect(yaml).not.toContain('inputFiles'); }); + it('preserves a promptfoo-shaped tags map (tags.experiment) without mangling keys', () => { + const suite = defineEval({ + name: 'sdk-tags-map', + tags: { experiment: 'sdk-baseline', team: 'compliance' }, + target: 'mock-target', + tests: [{ id: 'hello', input: 'Say hello', assertions: [{ type: 'contains', value: 'hi' }] }], + }); + + const lowered = toEvalYamlObject(suite); + expect(lowered.tags).toEqual({ experiment: 'sdk-baseline', team: 'compliance' }); + }); + + it('keeps the list form of tags for selection', () => { + const suite = defineEval({ + name: 'sdk-tags-list', + tags: ['smoke', 'regression'], + target: 'mock-target', + tests: [{ id: 'hello', input: 'Say hello', assertions: [{ type: 'contains', value: 'hi' }] }], + }); + + const lowered = toEvalYamlObject(suite); + expect(lowered.tags).toEqual(['smoke', 'regression']); + }); + it('rejects removed experiment authoring blocks', () => { expect(() => defineEval({