docs(environment): teach environment recipe vocabulary

README.md

@@ -5,7 +5,7 @@ Test AI targets on real repo tasks and measure what actually works.
 ## Why?
 
 - **Local-first** — runs on your machine, no cloud accounts or API keys for eval infrastructure
-- **Repo-backed workspaces** — reuse real repos, setup scripts, and existing harnesses instead of rebuilding synthetic tasks
+- **Repo-backed environments** — reuse real repos, setup scripts, Docker images, and existing harnesses instead of rebuilding synthetic tasks
 - **Portable artifacts** — results, traces, and reports are saved in a durable format other tools can consume
 - **Version-controlled** — evals, judges, and results all live in Git
 - **Hybrid graders** — deterministic code checks + LLM-based subjective scoring
@@ -14,9 +14,9 @@ Test AI targets on real repo tasks and measure what actually works.
 
 ## Core Concepts
 
-- **Eval suite / imports / tests** are the task corpus: the prompts, cases, datasets, and imported benchmarks you want to evaluate.
+- **Eval suite / tests** are the task corpus: the prompts, cases, datasets, and reusable field-local files you want to evaluate.
 - **Category** is derived from where the eval lives, such as folder path and file name. Use paths to organize the corpus instead of repeating category labels in every eval.
-- **Workspace / fixtures / graders** are task-owned context: repos, setup scripts, files, fixtures, isolation, deterministic checks, and LLM grading prompts.
+- **Environment / fixtures / graders** are task-owned context: host or Docker setup, repos, setup scripts, files, fixtures, deterministic checks, and LLM grading prompts.
 - **Target** is the system under test: an agent, provider, gateway, replay target, CLI wrapper, transcript provider, or future app/service wrapper. Each eval selects one `target` by configured target `id` or with an eval-local target object.
 - **Tags** are run/result grouping labels. `tags.experiment` is the default experiment namespace, such as `with-skills` or `without-skills`; keep suite/category and target/model names out of that tag.
 - **Evaluate options** configure eval run behavior such as `max_concurrency`, repeat policy, and budgets.
@@ -259,15 +259,19 @@ export default defineEval({
   },
   threshold: 0.8,
   prompts: ['{{ input }}'],
-  workspace: {
-    scope: 'attempt',
-    repos: [
-      {
-        path: './fixture',
-        repo: 'EntityProcess/agentv-contract-fixture',
-        commit: '21a34daed7ebcfe36cbed053607622a55e5e94cb',
-      },
-    ],
+  environment: {
+    type: 'host',
+    workdir: './fixture',
+    setup: {
+      command: [
+        'bash',
+        './scripts/materialize-repo.sh',
+        './fixture',
+        'EntityProcess/agentv-contract-fixture',
+        '21a34daed7ebcfe36cbed053607622a55e5e94cb',
+      ],
+      cwd: '.',
+    },
   },
   tests: [
     {

apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx

@@ -30,8 +30,8 @@ tests:
 | `expected_output` | No | Passive gold/reference data available to graders (string, object, or message array) |
 | `assert` | Yes | Per-test graders; plain strings become `llm-rubric` rubric checks |
 | `execution` | No | Per-case grader/default overrides such as `skip_defaults`; target selection belongs in top-level `target` or CLI `--target` |
-| `workspace` | No | Per-case workspace config (overrides suite-level) |
-| `metadata` | No | Arbitrary key-value pairs passed to graders and workspace scripts |
+| `environment` | No | Per-case environment recipe (overrides suite-level) |
+| `metadata` | No | Arbitrary key-value pairs passed to graders, setup commands, and lifecycle extensions |
 
 ## Input
 
@@ -145,34 +145,40 @@ tests:
     # Does NOT get latency_check
 ```
 
-## Per-Case Workspace Config
+## Per-Case Environment Config
 
-Override the suite-level workspace config for individual tests. Test-level fields replace suite-level fields:
+Override the suite-level environment recipe for individual tests. Test-level
+environment fields replace suite-level fields:
 
 ```yaml
-workspace:
-  hooks:
-    before_all:
-      command: ["bun", "run", "default-setup.ts"]
+environment:
+  type: host
+  workdir: ./workspaces/default
+  setup:
+    command: ["bash", "-lc", "bun install && bun run build"]
+    cwd: "."
 
 tests:
   - id: case-1
     input: Do something
     assert:
       - Completes the requested task
-    workspace:
-      hooks:
-        before_all:
-          command: ["bun", "run", "custom-setup.ts"]
+    environment:
+      type: host
+      workdir: ./workspaces/case-1
+      setup:
+        command: ["bash", "-lc", "bun install && bun run build && bun run setup:case-1"]
+        cwd: "."
 
   - id: case-2
     input: Do something else
     assert:
       - Completes the requested task
-    # Inherits suite-level hooks.before_all
+    # Inherits suite-level environment
 ```
 
-See [Workspace Lifecycle Hooks](/docs/targets/configuration/#workspace-lifecycle-hooks) for the full workspace config reference.
+See [Environment Recipes](/docs/guides/workspace-architecture/) for the full
+environment config reference.
 
 ## Per-Case Metadata
 
@@ -186,17 +192,25 @@ tests:
       source_repo: sympy/sympy
       source_commit: "abc123def"
       test_patch: cases/sympy-20590/test.patch
-    workspace:
-      repos:
-        - path: ./repo
-          repo: sympy/sympy
-          commit: "abc123def"
-      hooks:
-        before_each:
-          command: ["python", "apply_test_patch.py"]
 ```
 
-The `metadata` field is included in the stdin JSON passed to lifecycle commands as `case_metadata`.
+The `metadata` field is included in the stdin JSON passed to lifecycle
+extensions as `case_metadata`. Use an `environment` recipe to materialize the
+checkout, then use a `beforeEach` extension when per-case metadata needs to
+drive patch application or fixture selection:
+
+```yaml
+environment:
+  type: host
+  workdir: ./repo
+  setup:
+    command: ["bash", "./scripts/materialize-repo.sh", "./repo", "sympy/sympy", "abc123def"]
+    cwd: "."
+
+extensions:
+  - file://scripts/apply-test-patch.py:beforeEach
+```
+
 Operational checkout state belongs in `environment`; matching metadata fields
 such as `source_commit` are informational only. For historical repo-state evals,
 pin the checkout in an environment setup recipe instead of only mentioning the

apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx

@@ -547,11 +547,13 @@ vars:
 
 - **Directory name as `id`:** If `case.yaml` doesn't specify an `id`, the directory name is used (e.g., `fix-null-check`)
 - **Alphabetical ordering:** Subdirectories are sorted alphabetically for deterministic order
-- **Per-case workspace:** A `workspace/` subdirectory inside the case directory automatically sets `workspace.template` to that path, unless the case already defines a `workspace` field
+- **Per-case files:** Put bulky case-local prompts, patches, fixtures, and oracle files beside `case.yaml`; reference them with `type: file` blocks or read them from lifecycle extensions via `case_metadata`
 - **Skipped directories:** Subdirectories without `case.yaml` are skipped with a warning
-- **Suite-level config applies:** Suite-level `assert`, `prompts`, `workspace`, `target`, and top-level run controls still apply to directory-discovered cases
+- **Suite-level config applies:** Suite-level `assert`, `prompts`, `environment`, `target`, and top-level run controls still apply to directory-discovered cases
 
-This pattern is useful for benchmarks with many cases, where each case benefits from its own directory for workspace templates, supporting files, or documentation.
+This pattern is useful for benchmarks with many cases, where each case benefits
+from its own directory for supporting files, patches, fixtures, or
+documentation.
 For guidance on keeping provenance metadata, patches, oracle files, and generated
 dataset rows out of oversized inline YAML, see [Benchmark Provenance](/docs/guides/benchmark-provenance/).
 
@@ -560,11 +562,20 @@ dataset rows out of oversized inline YAML, see [Benchmark Provenance](/docs/guid
 All string fields in eval files support `{{ env.VAR }}` syntax for environment variable interpolation. This enables portable eval configs that work across machines and CI environments without hardcoded paths.
 
 ```yaml
-workspace:
-  repos:
-    - path: ./RepoA
-      repo: "{{ env.REPO_A_URL }}"
-      commit: "{{ env.REPO_A_COMMIT }}"
+environment:
+  type: host
+  workdir: ./workspaces/repo-a
+  setup:
+    command:
+      - bash
+      - ./scripts/materialize-repo.sh
+      - ./workspaces/repo-a
+      - "{{ env.REPO_A_URL }}"
+      - "{{ env.REPO_A_COMMIT }}"
+    cwd: "."
+
+env:
+  OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}"
 
 prompts:
   - "{{ prompt }}"
@@ -583,17 +594,23 @@ tests:
 - **Partial interpolation** is supported: `{{ env.HOME }}/repos/{{ env.PROJECT }}` becomes `/home/user/repos/myproject`
 - **Non-string values** (numbers, booleans) are not affected
 - Interpolation is applied recursively to all nested objects and arrays
-- Works in YAML eval files, external YAML/JSONL case files, and external workspace config files
+- Works in YAML eval files, external YAML/JSONL case files, and external environment recipe files
 - `.env` files in the directory hierarchy are loaded automatically before interpolation
 
-### Example: Portable Workspace Config
+### Example: Portable Environment Recipe
 
 ```yaml
-# workspace.yaml — works on any machine
-repos:
-  - path: ./my-repo
-    repo: "{{ env.MY_REPO_URL }}"
-    commit: "{{ env.MY_REPO_COMMIT }}"
+# .agentv/environments/repo.yaml — works on any machine
+type: host
+workdir: ./workspaces/my-repo
+setup:
+  command:
+    - bash
+    - ./scripts/materialize-repo.sh
+    - ./workspaces/my-repo
+    - "{{ env.MY_REPO_URL }}"
+    - "{{ env.MY_REPO_COMMIT }}"
+  cwd: "."
 ```
 
 ```bash

apps/web/src/content/docs/docs/next/evaluation/experiments.mdx

@@ -30,10 +30,12 @@ evaluate_options:
   budget_usd: 2.00
   max_concurrency: 3
 
-workspace:
-  hooks:
-    before_all:
-      command: ["bash", "-lc", "bun install && bun run build"]
+environment:
+  type: host
+  workdir: ./workspaces/support-codex
+  setup:
+    command: ["bash", "-lc", "bun install && bun run build"]
+    cwd: "."
 
 tests:
   - id: refund-eligibility

apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx

@@ -240,11 +240,12 @@ env, Docker, and folder isolation. Use `extensions` for lifecycle function hooks
 extensions:
   - file://scripts/setup.mjs:beforeAll
 
-workspace:
-  scope: suite           # suite | attempt
-  hooks:
-    after_each:
-      reset: fast        # none | fast | strict
+environment:
+  type: host
+  workdir: ./workspaces/my-eval
+  setup:
+    command: ["bash", "-lc", "bun install && bun run build"]
+    cwd: "."
 ```
 
 Notes:

apps/web/src/content/docs/docs/next/graders/script-graders.mdx

@@ -385,7 +385,8 @@ Use `expected_output` for reference answers and `output` for the actual final an
 
 ## Workspace Access
 
-When `workspace` is configured in the eval YAML (via `workspace.template`, `workspace.repos`, or lifecycle hooks), script graders receive the prepared workspace path in two ways:
+When an `environment` recipe prepares a coding-agent testbed, script graders
+receive the prepared workspace path in two ways:
 
 1. **JSON payload**: `workspace_path` field in the stdin input
 2. **Environment variable**: `AGENTV_WORKSPACE_PATH`
@@ -396,7 +397,7 @@ This enables **functional grading** — running commands like `npm test`, `pytes
 
 `file_changes` is a unified diff built from two sources, merged in order:
 
-1. **Git baseline**: `git diff` against a baseline commit taken before the agent ran. Captures edits, new files at workspace root, and changes inside any nested git repos materialized via `workspace.repos` or set up via a `before_all` hook.
+1. **Git baseline**: `git diff` against a baseline commit taken before the agent ran. Captures edits, new files at workspace root, and changes inside any nested git repos materialized by `environment.setup.command` or set up via a lifecycle hook.
 2. **Provider-reported artifacts**: Copilot providers scan their session-state `files/` directory after each run and append those as synthetic diffs. This surfaces files the agent wrote *outside* `workspace_path` entirely (e.g. `~/.copilot/session-state/<uuid>/files/`).
 
 ### Example: Deploy-and-Test Pattern
@@ -440,8 +441,9 @@ console.log(JSON.stringify({
 
 ```yaml
 # suite.yaml
-workspace:
-  template: ./workspace-template   # copied into a temp dir before each run
+environment:
+  type: host
+  workdir: ./workspace-template
 
 target: my_agent