Separate experiments from eval definitions

.agents/verification.md

@@ -120,6 +120,18 @@ bun apps/cli/src/cli.ts eval examples/features/rubric/evals/dataset.eval.yaml --
 4. Update baseline files if output format changes. Baselines live next to eval YAML files as `*.baseline.jsonl`.
 5. `--dry-run` returns schema-valid mock responses, but the scores are not meaningful. Use it only for plumbing and harness checks.
 
+## Live Dogfood for Eval and Experiment Changes
+
+Use live dogfood before marking PRs ready when they affect eval execution, experiments, repeat runs, targets, providers, graders, or artifact provenance.
+
+- Live means both sides are real: a live agent/provider target and a live grader target. Do not count `mock`, `--dry-run`, or deterministic-only assertions as dogfood for these changes.
+- Prefer the smallest realistic eval: one or two cases, bounded timeouts, and `workers: 1` for heavyweight agent providers.
+- For native experiment changes, run through `agentv eval run ... --experiment <experiment.yaml|ts>` so resolution, setup, scripts, target selection, run knobs, and artifact metadata are exercised together.
+- For repeat-run changes, use an experiment-level repeat config with `count >= 2`, `early_exit: false` when validating all attempts are persisted. Inspect root `index.jsonl`, root `benchmark.json`, and the repeated case folder. The repeated case folder should carry aggregate `summary.json` with flattened snake_case timing fields plus AgentV aggregate `grading.json`; attempt-specific outputs and transcripts live under `run-N/`. Each `run-N/` folder should contain `result.json`, `grading.json`, `transcript.json`, `transcript-raw.jsonl`, and `outputs/answer.md`. Do not write per-run `metrics.json`; timing and o11y fields belong in `result.json`, and `result.json` points at `./grading.json` through `grading_path`.
+- For local OpenAI-compatible grading through the OAuth proxy, use `endpoint: http://127.0.0.1:10531/v1`, but still route `api_key` and `model` through environment references such as `${{ LOCAL_OPENAI_PROXY_API_KEY }}` and `${{ LOCAL_OPENAI_PROXY_MODEL }}`. Literal secrets and literal model values are intentionally rejected by target validation unless a resolver explicitly allows them.
+- Preserve review evidence in `agentv-private` on an `evidence/<bead-or-feature-slug>` branch. Include the run bundle, source eval/experiment/targets files, a short README, an artifact tree, and screenshots when folder structure or UI behavior is under review.
+- If comparing against an external convention such as Vercel `agent-eval`, verify both semantic provenance and the physical `run-N` artifact layout for repeat runs.
+
 ## Checking Grader Score Ranges
 
 Use `scripts/check-grader-scores.ts` as a post-processor after an eval run.

AGENTS.md

@@ -43,6 +43,7 @@ Read the full rationale and examples in [.agents/product-boundary.md](.agents/pr
 - Non-trivial work needs a plan or task list. If the implementation surface starts to balloon, stop and re-plan.
 - Large or high-risk PRs need meaningful, reviewable commits for each coherent change. Rewrite only the PR branch with `git push --force-with-lease` when needed to replace WIP or accidental squashed history before review.
 - Manual red/green UAT is blocking before a branch is ready for review. GitHub Actions is the authoritative merge gate.
+- For eval execution, experiments, repeat runs, providers, graders, or artifact-layout changes, dogfood with a live provider and a real LLM grader before marking ready. Mock graders, dry-run, and deterministic-only smoke tests are useful plumbing checks, but they are not live dogfood. Use canonical `.agentv/results/<experiment>/<timestamp>` output and publish private evidence. See [.agents/verification.md](.agents/verification.md).
 - For browser or screenshot UAT, keep evidence out of the public repo and publish reviewable artifacts to an `agentv-private` evidence branch. See [.agents/verification.md](.agents/verification.md).
 - Wire formats are `snake_case`; internal TypeScript is `camelCase`. Translate only at the boundary.
 - In AgentV, a `project` holds runs, traces, and experiments; a `benchmark` is a curated eval suite. Do not collapse those terms.

apps/cli/src/commands/eval/artifact-writer.ts

@@ -6,6 +6,7 @@ import {
   type BenchmarkArtifact,
   type EvalTest,
   type EvaluationResult,
+  type ExperimentArtifactMetadata,
   type ExportDuplicatePolicy,
   type GradingArtifact,
   type IndexArtifactEntry,
@@ -228,6 +229,7 @@ export async function writeArtifactsFromResults(
   options?: {
     evalFile?: string;
     experiment?: string;
+    experimentMetadata?: ExperimentArtifactMetadata;
     plannedTestCount?: number;
     runId?: string;
     duplicatePolicy?: ExportDuplicatePolicy;
@@ -245,6 +247,7 @@ export async function writeArtifactsFromResults(
   return writeCoreArtifactsFromResults(results, outputDir, {
     evalFile: options?.evalFile,
     experiment: options?.experiment,
+    experimentMetadata: options?.experimentMetadata,
     plannedTestCount: options?.plannedTestCount,
     runId: options?.runId,
     duplicatePolicy: options?.duplicatePolicy,

apps/cli/src/commands/eval/commands/run.ts

@@ -11,7 +11,6 @@ import {
 } from 'cmd-ts';
 
 import { runEvalCommand } from '../run-eval.js';
-import { resolveEvalPaths } from '../shared.js';
 
 export const evalRunCommand = command({
   name: 'eval',
@@ -264,14 +263,6 @@ export const evalRunCommand = command({
     }),
   },
   handler: async (args) => {
-    // Launch interactive wizard when no eval paths and stdin is a TTY
-    if (args.evalPaths.length === 0 && process.stdin.isTTY) {
-      const { launchInteractiveWizard } = await import('../interactive.js');
-      await launchInteractiveWizard();
-      return;
-    }
-
-    const resolvedPaths = await resolveEvalPaths(args.evalPaths, process.cwd());
     if (args.budgetUsd !== undefined && args.budgetUsd <= 0) {
       console.error('Error: --budget-usd must be a positive number.');
       process.exit(2);
@@ -330,7 +321,7 @@ export const evalRunCommand = command({
       recordReplay: args.recordReplay,
       recordReplayVariant: args.recordReplayVariant,
     };
-    const result = await runEvalCommand({ testFiles: resolvedPaths, rawOptions });
+    const result = await runEvalCommand({ testFiles: args.evalPaths, rawOptions });
     if (result?.allExecutionErrors) {
       process.exit(2);
     }