docs(experiment): label runs via tags.experiment, drop top-level experiment

README.md

@@ -61,13 +61,13 @@ targets:
   - name: copilot-sdk
     provider: anthropic
     model: claude-sonnet-4.6
-    max_budget_usd: 0.50
 ```
 
 **3. Create an eval** in `evals/`:
 ```yaml
 description: Code generation quality
-experiment: with-skills
+tags:
+  experiment: with-skills
 target: copilot-sdk
 evaluate_options:
   repeat:
@@ -103,7 +103,8 @@ The target can be an eval-local object when this eval needs target settings of i
 
 ```yaml
 description: Code generation quality with Copilot target settings
-experiment: with-skills
+tags:
+  experiment: with-skills
 target:
   extends: copilot-sdk
   model: claude-sonnet-4.6
@@ -120,7 +121,7 @@ tests:
     input: Write FizzBuzz in Python
 ```
 
-`target: copilot-sdk` resolves the named target from `.agentv/targets.yaml` or `targets.yaml` and uses its default provider, model, hooks, and provider settings. The object form above starts from `copilot-sdk`, then applies the eval-local fields for this eval. If `extends` is omitted, the object defines the full target inline and must include enough provider configuration to run. AgentV records the resolved target information in run artifacts so results can be audited and replayed. The experiment label stays `with-skills` because the condition is unchanged; the model/provider variation belongs to the resolved target metadata.
+`target: copilot-sdk` resolves the named target from `.agentv/targets.yaml` or `targets.yaml` and uses its default provider, model, hooks, and provider settings. The object form above starts from `copilot-sdk`, then applies the eval-local fields for this eval. If `extends` is omitted, the object defines the full target inline and must include enough provider configuration to run. AgentV records the resolved target information in run artifacts so results can be audited and replayed. The `tags.experiment` label stays `with-skills` because the condition is unchanged; the model/provider variation belongs to the resolved target metadata.
 
 Use `default_test.threshold` for the inherited per-test pass cutoff. Existing eval files with a top-level `threshold` still load during migration, and `--threshold` on the CLI still overrides YAML thresholds for a run.
 
@@ -136,7 +137,7 @@ agentv compare .agentv/results/<baseline-run-id>/index.jsonl .agentv/results/<ca
 
 ## Results
 
-Each run writes a portable bundle directly under `.agentv/results/<run_id>/`. In this example, `experiment: with-skills` names the condition being measured and `target: copilot-sdk` selects the system under test from `targets.yaml`; both are recorded as metadata, not path segments. The root `index.jsonl` manifest is the portable row index used by scripts, CI, and `agentv compare`; per-case sidecars include the resolved eval and target configuration used for the run.
+Each run writes a portable bundle directly under `.agentv/results/<run_id>/`. In this example, `tags.experiment: with-skills` names the condition being measured and `target: copilot-sdk` selects the system under test from `targets.yaml`; both are recorded as metadata, not path segments. The root `index.jsonl` manifest is the portable row index used by scripts, CI, and `agentv compare`; per-case sidecars include the resolved eval and target configuration used for the run.
 
 ```bash
 agentv eval evals/my-eval.yaml
@@ -203,7 +204,7 @@ import { defineEval } from '@agentv/sdk';
 
 export default defineEval({
   description: 'Code generation quality',
-  experiment: 'with-skills',
+  tags: { experiment: 'with-skills' },
   target: {
     extends: 'copilot-sdk',
     model: 'claude-sonnet-4.6',

apps/web/src/content/docs/docs/evaluation/eval-files.mdx

@@ -5,7 +5,7 @@ sidebar:
   order: 1
 ---
 
-Evaluation files define the test cases, graders, workspace lifecycle, and run controls for an evaluation run. Top-level `experiment` is the run/result grouping label, top-level `target` identifies the system under test, and fields such as `repeat`, `threshold`, `timeout_seconds`, `evaluate_options.budget_usd`, and `evaluate_options.max_concurrency` control repeated attempts and gates. Workspace reuse belongs under `workspace.isolation`; Docker/container binding belongs under `workspace.docker`. Install, build, and reset commands belong under `workspace.hooks`; runner-specific setup belongs in the `target` object or `targets.yaml`. AgentV supports two eval data formats: YAML and JSONL.
+Evaluation files define the test cases, graders, workspace lifecycle, and run controls for an evaluation run. The reserved `tags.experiment` key is the run/result grouping label, top-level `target` identifies the system under test, and fields such as `repeat`, `threshold`, `timeout_seconds`, `evaluate_options.budget_usd`, and `evaluate_options.max_concurrency` control repeated attempts and gates. Workspace reuse belongs under `workspace.isolation`; Docker/container binding belongs under `workspace.docker`. Install, build, and reset commands belong under `workspace.hooks`; runner-specific setup belongs in the `target` object or `targets.yaml`. AgentV supports two eval data formats: YAML and JSONL.
 
 YAML is the canonical portable model. TypeScript helpers, generated fixtures, and Python scripts should lower to the same YAML/JSONL shapes rather than inventing a separate eval contract.
 Eval files describe the task, target binding, and run controls. Use `evaluate_options.max_concurrency` for authored suite concurrency. Operators can still override concurrency with `--workers` or set defaults with `execution.workers` in `agentv.config.*` / `.agentv/config.yaml`; do not author legacy `workers` fields in eval YAML.

apps/web/src/content/docs/docs/evaluation/experiments.mdx

@@ -6,7 +6,7 @@ sidebar:
 ---
 
 AgentV eval files are the runnable authoring artifact. Use top-level
-`description` for display metadata, `experiment` as the run/result grouping
+`description` for display metadata, `tags.experiment` as the run/result grouping
 label, `target` for the system under test, and flat top-level run controls such
 as `repeat`, `timeout_seconds`, and `threshold`. Use `evaluate_options` for
 evaluation runtime options such as `budget_usd` and `max_concurrency`.
@@ -17,7 +17,8 @@ overrides.
 ```yaml
 name: support-regression
 description: Support regression suite
-experiment: support-codex
+tags:
+  experiment: support-codex
 target:
   extends: codex-gpt5
   model: gpt-5.1
@@ -252,11 +253,11 @@ Eval runs write to a direct run bundle:
 ```
 
 CLI `--experiment` sets the experiment label explicitly. Without that flag, AgentV
-uses a reserved `tags.experiment` key (see below), then top-level `experiment`,
-then top-level `name`, then the eval filename. The precedence is
-`--experiment` > `tags.experiment` > default. The Dashboard uses "Experiment" for
-the comparison and result grouping concept; folder names are only storage
-allocation and must not define result semantics.
+uses the reserved `tags.experiment` key (see below), then the suite `name`, then
+the eval filename. The precedence is `--experiment` > `tags.experiment` > default.
+There is no top-level `experiment` field — a run is labeled with `tags.experiment`.
+The Dashboard uses "Experiment" for the comparison and result grouping concept;
+folder names are only storage allocation and must not define result semantics.
 
 ### Tags as run metadata (`tags.experiment`)
 

packages/sdk/src/eval.ts

@@ -204,12 +204,23 @@ export interface EvalDefinition {
   readonly category?: string;
   readonly version?: string;
   readonly author?: string;
-  readonly tags?: readonly string[];
+  /**
+   * Suite tags. Either the selection list form (`string[]`, drives
+   * `select.tags` / `--tag name` filtering) or the promptfoo-shaped
+   * `Record<string,string>` map. In the map form the reserved `experiment` key
+   * labels the run/experiment (grouped by the Dashboard), matching
+   * `tags.experiment` in YAML evals.
+   */
+  readonly tags?: readonly string[] | Readonly<Record<string, string>>;
   readonly license?: string;
   readonly requires?: EvalRequires;
   readonly input?: string | readonly EvalMessage[];
   readonly inputFiles?: readonly string[];
   readonly tests: readonly EvalTest[] | string;
+  /**
+   * @deprecated A top-level `experiment` label no longer sets the run's
+   * experiment namespace. Use `tags: { experiment: '<name>' }` instead.
+   */
   readonly experiment?: string;
   readonly target?: string | EvalTargetConfig;
   readonly repeat?: EvalRepeat;

packages/sdk/test/eval-authoring.test.ts

@@ -194,6 +194,30 @@ describe('YAML-aligned eval authoring helpers', () => {
     expect(yaml).not.toContain('inputFiles');
   });
 
+  it('preserves a promptfoo-shaped tags map (tags.experiment) without mangling keys', () => {
+    const suite = defineEval({
+      name: 'sdk-tags-map',
+      tags: { experiment: 'sdk-baseline', team: 'compliance' },
+      target: 'mock-target',
+      tests: [{ id: 'hello', input: 'Say hello', assertions: [{ type: 'contains', value: 'hi' }] }],
+    });
+
+    const lowered = toEvalYamlObject(suite);
+    expect(lowered.tags).toEqual({ experiment: 'sdk-baseline', team: 'compliance' });
+  });
+
+  it('keeps the list form of tags for selection', () => {
+    const suite = defineEval({
+      name: 'sdk-tags-list',
+      tags: ['smoke', 'regression'],
+      target: 'mock-target',
+      tests: [{ id: 'hello', input: 'Say hello', assertions: [{ type: 'contains', value: 'hi' }] }],
+    });
+
+    const lowered = toEvalYamlObject(suite);
+    expect(lowered.tags).toEqual(['smoke', 'regression']);
+  });
+
   it('rejects removed experiment authoring blocks', () => {
     expect(() =>
       defineEval({