feat(artifacts): write eval runs at results root

.agents/verification.md

@@ -104,7 +104,7 @@ wait
 
 Unit tests alone are not enough for grader changes.
 
-1. If you are in a git worktree, copy `.env` into the worktree root before claiming E2E or grader verification:
+1. If you are in a git worktree, copy the ignored `.env` from the primary/main checkout into the worktree root before claiming E2E or grader verification:
 
 ```bash
 cp /path/to/main/.env .env
@@ -114,6 +114,10 @@ cp /path/to/main/.env .env
 Copy-Item D:/path/to/main/.env .env
 ```
 
+Do not use `.env.example` as a credential substitute. If the primary/main
+checkout has no `.env`, record the missing credentials as the exact live
+provider or grader blocker.
+
 2. Run a real eval with a real example file:
 
 ```bash
@@ -167,12 +171,16 @@ bun scripts/check-grader-scores.ts
 
 Before marking a branch ready for review:
 
-1. Preflight: if in a git worktree, ensure `.env` exists in the worktree root.
+1. Preflight: if in a git worktree, ensure `.env` exists in the worktree root by copying it from the primary/main checkout.
 
 ```bash
 cp "$(git worktree list --porcelain | head -1 | sed 's/worktree //')/.env" .env
 ```
 
+The copied file is local-only and must remain uncommitted. If there is no
+primary/main `.env`, record that as a live-provider blocker before claiming
+grader dogfood is unavailable.
+
 2. Run unit tests with `bun run test`.
 3. Blocking manual red and green UAT:
 

.agents/workflow.md

@@ -42,6 +42,10 @@ bd where
 
 - These setup steps are required before running builds, tests, evals, or tracker
   updates in the worktree.
+- The `.env` copy must come from the primary/main checkout and stays ignored in
+  the worktree. Do not copy `.env.example` as a credential substitute. If the
+  primary checkout has no `.env`, record the missing credentials as the exact
+  blocker for live provider/grader verification.
 - If you discover you are on a stale base or have uncoordinated dirty files, stop and fix that before changing code.
 - Whenever you `git checkout`, `gh pr checkout`, `git pull`, or otherwise switch to a ref that may have changed `package.json` or `bun.lock`, run `bun install` before building or testing.
 
@@ -102,6 +106,10 @@ bun install
 cp "$(git worktree list --porcelain | head -1 | sed 's/worktree //')/.env" .env
 ```
 
+If the primary/main checkout does not have `.env`, stop before live eval or
+grader verification and record the missing local credentials/proxy settings as
+the blocker.
+
 After the first meaningful commit, push and open a draft PR unless the user directs a different PR lifecycle:
 
 ```bash

AGENTS.md

@@ -40,10 +40,11 @@ Read the full rationale and examples in [.agents/product-boundary.md](.agents/pr
 - Do not use `git stash` on shared checkouts. Stage explicit paths only, and never push directly to `main`.
 - Every merge to `main` requires a GitHub pull request with passing GitHub Actions. Do not locally merge feature or integration branches into `main` as a substitute for opening a PR.
 - Prefer the primary checkout only for small, clean, bounded work. Use a dedicated worktree from the latest `origin/main` for non-trivial, risky, long-running, or parallel changes.
+- When working from a dedicated worktree, copy the ignored `.env` from the primary/main checkout into the worktree before running evals, provider dogfood, grader verification, or local OpenAI OAuth proxy checks. Keep copied env files local and uncommitted; if the primary checkout has no `.env`, record that exact blocker instead of using `.env.example` as credentials.
 - Non-trivial work needs a plan or task list. If the implementation surface starts to balloon, stop and re-plan.
 - Large or high-risk PRs need meaningful, reviewable commits for each coherent change. Rewrite only the PR branch with `git push --force-with-lease` when needed to replace WIP or accidental squashed history before review.
 - Manual red/green UAT is blocking before a branch is ready for review. GitHub Actions is the authoritative merge gate.
-- For eval execution, experiments, repeat runs, providers, graders, or artifact-layout changes, dogfood with a live provider and a real LLM grader before marking ready. `agentv validate`, mock targets, replay/frozen transcript runs, and deterministic-only smoke tests are useful checks, but they are not live dogfood. Use canonical `.agentv/results/<experiment>/<timestamp>` output and publish private evidence. See [.agents/verification.md](.agents/verification.md).
+- For eval execution, experiments, repeat runs, providers, graders, or artifact-layout changes, dogfood with a live provider and a real LLM grader before marking ready. `agentv validate`, mock targets, replay/frozen transcript runs, and deterministic-only smoke tests are useful checks, but they are not live dogfood. Use canonical `.agentv/results/<run_id>/` output and publish private evidence. See [.agents/verification.md](.agents/verification.md).
 - For browser or screenshot UAT, keep evidence out of the public repo and publish reviewable artifacts to an `agentv-private` evidence branch. See [.agents/verification.md](.agents/verification.md).
 - When dogfood or review reveals a durable workflow lesson, capture it in this guide or the relevant `.agents/*.md` guide before merge; do not leave durable agent instructions only in PR comments, Bead comments, or private evidence. Use `docs/solutions/` for fuller reusable writeups.
 - Research-only workers must not run `bun install`, `bun run build`, tests, or evals unless the assigned work explicitly needs that command and the worker records why.

CONCEPTS.md

@@ -12,29 +12,35 @@ Shared domain vocabulary for this project — entities, named processes, and sta
 
 ## Evaluation Model
 
-**Eval / Eval YAML** — The only composable and runnable AgentV authoring primitive. An eval YAML file can be a reusable task suite that owns task context, a wrapper eval that imports suites and carries an inline `experiment:` block, or a sidecar around raw JSONL cases. AgentV does not have a separate runnable `experiment.yaml` artifact.
+**Eval / Eval YAML** — The only composable and runnable AgentV authoring primitive. An eval YAML file can be a reusable task suite that owns task context, a wrapper eval that imports suites and binds top-level runtime policy, or a sidecar around raw JSONL cases. AgentV does not have a separate runnable `experiment.yaml` artifact.
 
 **Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `workspace`, assertions, expected references, and judge criteria. It can run directly or be imported by another eval with `tests[].include` and `type: suite`.
 
 **Raw case file** — YAML, JSONL, or directory case data imported with `tests: ./cases.yaml`, string shorthand, or `type: tests`. Raw cases are reusable data inputs; they do not carry imported suite context such as shared `workspace`, shared `input`, or shared `assertions`.
 
-**Wrapper eval** — Eval YAML whose main job is to import task suites and bind runtime policy with an inline `experiment:` block. Wrapper evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. A wrapper that imports suites with `type: suite` does not define parent `workspace`; imported suites own task environment.
+**Wrapper eval** — Eval YAML whose main job is to import task suites and bind top-level runtime policy such as target selection, repeat count, timeout, budget, and thresholds. Wrapper evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. A wrapper that imports suites with `type: suite` does not define parent `workspace`; imported suites own task environment.
 
-**Experiment** — The run-policy namespace for how evals are executed: target or target matrix, eval filters, repeat counts, timeouts, workers, budgets, thresholds, and related run knobs. In authored files it lives as inline `experiment:` inside eval YAML; CLI `--experiment` and `experiment.name` choose the result bucket. Lifecycle setup belongs in `workspace.hooks` or `targets[].hooks`, not in a separate experiment artifact.
+**Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Runtime policy belongs in top-level eval fields or target objects; the experiment label is recorded in `summary.json` and `index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `workspace.hooks` or `targets[].hooks`, not in a separate experiment artifact.
 
 **Workspace** — The task environment an eval prepares for the agent: repositories, templates, fixture files, and lifecycle hooks. It is not prompt input; use `input` for instructions and `workspace.repos[]` for multi-repo workspaces the agent can inspect or modify through tools.
 
-**Run manifest** — The root `index.jsonl` file in a run bundle. It is the dashboard and tooling loading contract for per-case result rows and artifact locations, including fields such as `result_dir`, `test_dir`, `summary_path`, and `grading_path`.
+**Run bundle** — A committed local result directory at `.agentv/results/<run_id>/`. `summary.json` records run metadata such as `run_id` and `experiment`; `index.jsonl` records per-case rows.
+
+**Run manifest** — The root `summary.json` file in a run bundle. It owns aggregate run metadata and rollups such as `run_id`, `experiment`, timestamps, planned/completed counts, pass rate, score summaries, duration, tokens, and cost.
+
+**Result index** — The root `index.jsonl` file in a run bundle. It is the dashboard and tooling loading contract for per-case result rows and artifact locations, including fields such as `result_dir`, `test_dir`, `summary_path`, and `grading_path`.
 
 **Result source identity** — The stable source identity for a result row: repo-relative `eval_path`, `test_id`, and `target`. `suite` and `name` are display metadata, not storage or routing identity.
 
 **Result directory** — The `result_dir` field in a `index.jsonl` row. It is a run-local directory allocation for that row's sidecars and outputs, usually a readable test-id or slug prefix plus a UUID/hash-like suffix. Consumers discover it from `index.jsonl` and must not infer it from suite names, display names, test IDs, targets, models, or folder position.
 
 **Artifact sidecar** — A file beside or below a result directory that provides evidence for a result, such as `summary.json`, `grading.json`, `result.json`, transcripts, logs, or outputs. Sidecars are evidence, not the primary discovery mechanism for a run.
 
+**Artifact attempt folder** — A per-case `run-N/` folder under a result directory. It stores one materialized execution's sidecars and outputs. It is not the primary comparison dimension: stochastic samples and infrastructure retries should be represented with explicit sample/retry metadata rather than inferred from `run-1`, `run-2`, and so on.
+
 ## Evaluation Reliability
 
-**Repeat run** — A configured request to execute the same eval case and target more than once in the same timestamped run bundle. Repeat runs measure stochastic reliability, verifier stability, and drift; they are not the default CI path.
+**Repeat run** — A configured request to execute the same eval case and target more than once in the same run bundle. Repeat runs measure stochastic reliability, verifier stability, and drift; they are not the default CI path.
 
 **Attempt** — One concrete execution inside a repeat run. Attempts keep their own score, status, timing, trace, transcript, logs, and artifacts so aggregate results never hide individual evidence.
 

apps/cli/src/commands/eval/result-layout.ts

@@ -8,7 +8,7 @@ export const DEFAULT_EXPERIMENT_NAME = 'default';
 export const RESERVED_RESULTS_NAMESPACES = new Set(['export', 'metadata', 'runs']);
 
 export function isReservedResultsNamespace(value: string | undefined): boolean {
-  return value !== undefined && RESERVED_RESULTS_NAMESPACES.has(value);
+  return value !== undefined && (value.startsWith('.') || RESERVED_RESULTS_NAMESPACES.has(value));
 }
 
 export function normalizeExperimentName(experiment?: string): string {
@@ -32,8 +32,10 @@ export function createRunDirName(timestamp = new Date()): string {
 }
 
 function defaultRunPathSegments(experiment: string | undefined, runDirName: string): string[] {
-  const normalizedExperiment = normalizeExperimentName(experiment);
-  return [normalizedExperiment, runDirName];
+  if (experiment !== undefined) {
+    normalizeExperimentName(experiment);
+  }
+  return [runDirName];
 }
 
 export function buildResultsRootDir(cwd: string): string {
@@ -159,7 +161,7 @@ export function relativeRunPathFromCwd(cwd: string, runDir: string): string | un
   }
 
   const parts = relative.split(path.sep).filter(Boolean);
-  if (parts.length < 2 || isReservedResultsNamespace(parts[0])) {
+  if (parts.length !== 1 || isReservedResultsNamespace(parts[0])) {
     return undefined;
   }
 

apps/cli/src/commands/eval/run-cache.ts

@@ -17,7 +17,7 @@ const CACHE_FILENAME = 'cache.json';
  * Keys use camelCase (not snake_case) for backward compat with pre-existing cache files.
  */
 export interface RunCache {
-  /** Directory path for the run workspace (e.g. .agentv/results/default/<ts>/) */
+  /** Directory path for the run workspace (e.g. .agentv/results/<run_id>/) */
   readonly lastRunDir?: string;
   /** @deprecated Legacy flat-file pointer from old cache files. Ignored on read. */
   readonly lastResultFile?: string;

apps/cli/src/commands/eval/run-eval.ts

@@ -1764,15 +1764,21 @@ export async function runEvalCommand(
   const explicitDir = options.outputDir;
   let runDir: string;
   let outputPath: string;
-  const runDirName = process.env.AGENTV_RUN_TIMESTAMP?.trim() || createRunDirName();
+  let runDirName = process.env.AGENTV_RUN_TIMESTAMP?.trim() || createRunDirName();
 
   if (explicitDir) {
     runDir = path.resolve(explicitDir);
     mkdirSync(runDir, { recursive: true });
     outputPath = runDir;
   } else {
-    // Default: .agentv/results/<eval-name>/<timestamp>/.
+    // Default: .agentv/results/<run_id>/.
     runDir = buildDefaultRunDirFromName(cwd, resultGroupName, runDirName);
+    let suffix = 1;
+    while (existsSync(runDir)) {
+      runDirName = `${process.env.AGENTV_RUN_TIMESTAMP?.trim() || createRunDirName()}-${suffix}`;
+      runDir = buildDefaultRunDirFromName(cwd, resultGroupName, runDirName);
+      suffix++;
+    }
     mkdirSync(runDir, { recursive: true });
     outputPath = runDir;
   }
@@ -2111,6 +2117,7 @@ export async function runEvalCommand(
       evalFile,
       plannedTestCount: totalEvalCount,
       experiment: normalizeExperimentName(options.experiment),
+      runId: path.basename(runDir),
       experimentMetadata: runExperimentMetadata,
       runtimeSource: runtimeSourceMetadata,
     });
@@ -2382,6 +2389,7 @@ export async function runEvalCommand(
         const { writePerTestArtifacts } = await import('./artifact-writer.js');
         await writePerTestArtifacts(allResults, runDir, {
           experiment: normalizeExperimentName(options.experiment),
+          runId: path.basename(runDir),
           resultGroup: resultGroupName,
           cwd,
           repoRoot,
@@ -2392,6 +2400,7 @@ export async function runEvalCommand(
         const { summaryPath } = await aggregateRunDir(runDir, {
           evalFile,
           experiment: normalizeExperimentName(options.experiment),
+          runId: path.basename(runDir),
           experimentMetadata: runExperimentMetadata,
           runtimeSource: runtimeSourceMetadata,
         });
@@ -2407,6 +2416,7 @@ export async function runEvalCommand(
           {
             evalFile,
             experiment: normalizeExperimentName(options.experiment),
+            runId: path.basename(runDir),
             experimentMetadata: runExperimentMetadata,
             resultGroup: resultGroupName,
             cwd,

apps/cli/src/commands/eval/wip-checkpoint.ts

@@ -15,8 +15,8 @@
  * Manual recovery from a WIP branch:
  *   git clone <results-repo> /tmp/recovery
  *   cd /tmp/recovery && git checkout agentv/wip/<hostname>/<run-dir>
- *   cp -r <experiment>/<run-dir> <project>/.agentv/results/<experiment>/
- *   agentv eval <eval-file> --output <project>/.agentv/results/<experiment>/<run-dir> --resume
+ *   cp -r <run-id> <project>/.agentv/results/
+ *   agentv eval <eval-file> --output <project>/.agentv/results/<run-id> --resume
  *
  * All checkpoint operations are best-effort: failures are logged as warnings
  * and never propagate to the eval run.

apps/cli/src/commands/inspect/utils.ts

@@ -564,15 +564,7 @@ export interface ResultFileMeta {
 function buildRunId(relativeRunPath: string): string {
   const normalized = relativeRunPath.split(path.sep).join('/');
   const segments = normalized.split('/').filter(Boolean);
-  if (segments.length >= 2) {
-    const experiment = segments.slice(0, -1).join('/');
-    const timestamp = segments.at(-1);
-    if (experiment === 'default') {
-      return timestamp ?? normalized;
-    }
-    return `${experiment}::${timestamp}`;
-  }
-  return segments[0];
+  return segments[0] ?? normalized;
 }
 
 function readRunDisplayName(runDir: string): string | undefined {
@@ -604,13 +596,6 @@ function collectRunManifestPaths(
     });
     return;
   }
-
-  const entries = readdirSync(currentDir, { withFileTypes: true });
-  for (const entry of entries) {
-    if (entry.isDirectory()) {
-      collectRunManifestPaths(runsDir, path.join(currentDir, entry.name), files);
-    }
-  }
 }
 
 function listResultFilesFromRoot(
@@ -675,13 +660,15 @@ function listResultFilesFromRoot(
 /**
  * Enumerate canonical run manifests in `.agentv/results/`.
  *
- * Reserved local namespaces such as `.agentv/results/runs/`,
+ * Reserved local namespaces such as `.agentv/results/.indexes/`,
+ * `.agentv/results/.cache/`, `.agentv/results/runs/`,
  * `.agentv/results/metadata/`, and `.agentv/results/export/` are intentionally
- * skipped by default discovery.
+ * skipped by default discovery. Discovery only treats direct children of
+ * `.agentv/results/` as committed run bundles.
  */
 export function listResultFiles(cwd: string, limit?: number): ResultFileMeta[] {
   const metas = listResultFilesFromRoot(buildResultsRootDir(cwd), {
-    skipTopLevelDirs: new Set(['export', 'metadata', 'runs']),
+    skipTopLevelDirs: new Set(['export', 'metadata', 'runs', '.indexes', '.cache']),
   }).sort((a, b) => {
     const byTimestamp = b.timestamp.localeCompare(a.timestamp);
     return byTimestamp !== 0 ? byTimestamp : b.displayName.localeCompare(a.displayName);
@@ -692,7 +679,7 @@ export function listResultFiles(cwd: string, limit?: number): ResultFileMeta[] {
 export function listResultFilesFromRunsDir(runsDir: string, limit?: number): ResultFileMeta[] {
   return listResultFilesFromRoot(runsDir, {
     limit,
-    skipTopLevelDirs: new Set(['export', 'metadata', 'runs']),
+    skipTopLevelDirs: new Set(['export', 'metadata', 'runs', '.indexes', '.cache']),
   });
 }
 

apps/cli/src/commands/pipeline/input.ts

@@ -57,8 +57,7 @@ export const evalInputCommand = command({
     out: option({
       type: optional(string),
       long: 'out',
-      description:
-        'Output directory for extracted inputs (default: .agentv/results/<experiment>/<timestamp>)',
+      description: 'Output directory for extracted inputs (default: .agentv/results/<run_id>)',
     }),
     experiment: option({
       type: optional(string),

apps/cli/src/commands/pipeline/run.ts

@@ -74,8 +74,7 @@ export const evalRunCommand = command({
     out: option({
       type: optional(string),
       long: 'out',
-      description:
-        'Output directory for results (default: .agentv/results/<experiment>/<timestamp>)',
+      description: 'Output directory for results (default: .agentv/results/<run_id>)',
     }),
     workers: option({
       type: optional(number),