docs(agent): refresh restructure references

.agents/conventions.md

@@ -51,7 +51,7 @@ These terms are distinct and not interchangeable.
 - Project: the top-level container Dashboard organizes around, backed by a registered workspace directory with `.agentv/`, run artifacts, traces, and experiments. The registry lives in `~/.agentv/projects.yaml` and is modeled by `ProjectEntry` and `ProjectRegistry` in `packages/core/src/projects.ts`.
 - Benchmark: a curated eval suite designed to measure something specific, in the academic ML sense. Example directories using this meaning are correctly named and should not be renamed.
 
-The legacy `~/.agentv/benchmarks.yaml` file is auto-migrated to `projects.yaml` by `migrateLegacyBenchmarksFile()`. Run-level results metadata lives in `summary.json`, with `index.jsonl` as the discovery anchor.
+The legacy `~/.agentv/benchmarks.yaml` file is auto-migrated to `projects.yaml` by `migrateLegacyBenchmarksFile()`. Run-level results metadata lives in `summary.json`, with `.internal/index.jsonl` as the per-run discovery anchor.
 
 Rule of thumb:
 
@@ -128,17 +128,18 @@ If you spot a camelCase key already on disk or in a response, treat it as a bug
 
 `artifact_pointers` are for offloading large detached payload bytes from the results metadata/control plane. They describe where payloads such as transcript files live when a run is projected to `agentv/artifacts/v1` or a future object store, including `key`, `object_version`, `sha256`, `size`, `media_type`, and `schema_version`.
 
-Do not add an `artifact_pointers.*` entry just because a new per-case artifact exists. Normal sidecars that stay in the run tree should be discoverable through explicit path fields on `index.jsonl` or manifests, for example `metrics_path` for `outputs/metrics.json`.
+Do not add an `artifact_pointers.*` entry just because a new per-case artifact exists. Normal sidecars that stay in the run tree should be discoverable through explicit path fields on `.internal/index.jsonl` rows or manifests, for example `metrics_path` for `metrics.json`.
 
 Before adding a new pointer family, verify that the artifact is large enough or detached enough to benefit from offloading and that published result repos should avoid carrying those payload bytes on the primary results branch.
 
 ## Grader Type System
 
 Grader types use kebab-case everywhere.
 
-- Authored YAML config: `type: llm-rubric`, `type: script`, `type: is-json`
-- Internal TypeScript still has the shared LLM grader implementation and registry key (`'llm-grader'`), but new authored evals should use `llm-rubric` for semantic LLM grading.
-- Output `scores[].type`: use the authored grader type when available, such as `"llm-rubric"` or `"is-json"`.
+- Authored YAML config: `type: llm-rubric`, `type: g-eval`, `type: script`, `type: is-json`
+- Use `llm-rubric` for promptfoo-compatible free-form rubric checks, and `g-eval` for structured or multi-criteria rubric judging when the eval needs itemized criteria semantics.
+- Internal TypeScript may keep shared LLM grader implementation names, but new authored evals should not use `llm-grader` as a public type.
+- Output `scores[].type`: use the authored grader type when available, such as `"llm-rubric"`, `"g-eval"`, or `"is-json"`.
 - Registry keys: `registry.register('llm-rubric', ...)` for the authored rubric surface, with `llm-grader` retained as internal/shared implementation plumbing.
 
 Source of truth: `GRADER_KIND_VALUES` in `packages/core/src/evaluation/types.ts`.

.agents/product-boundary.md

@@ -41,7 +41,9 @@ Prefer these extension points before adding a built-in:
 
 - `script` graders for custom executable evaluation logic
 - plain assertion strings for simple semantic rubric checks
-- `llm-rubric` for promptfoo-compatible free-form rubrics, structured rubric criteria, custom prompts, custom grader targets, or preprocessing
+- `llm-rubric` for promptfoo-compatible free-form rubric checks
+- `g-eval` for structured or multi-criteria rubric judging
+- config-level grader targets selected through `defaults.grader` or assertion-level target selection, not target-level grader configuration
 - CLI wrappers that consume AgentV JSON or JSONL output for post-processing such as aggregation, comparison, or reporting
 
 Ask: can this be achieved with existing primitives plus a plugin or wrapper? If yes, it should not be a built-in. That includes niche config overrides for existing graders.

.agents/verification.md

@@ -124,11 +124,11 @@ provider or grader blocker.
 bun apps/cli/src/cli.ts eval examples/features/rubric/evals/dataset.eval.yaml --test-id <test-id>
 ```
 
-3. Inspect the results JSONL and verify:
+3. Inspect the result bundle and verify:
 
 - the correct grader type ran by checking `scores[].type`
 - scores are calculated as expected
-- the `assertions` array reflects the evaluation logic
+- `grading.json.assertion_results` reflects the evaluation logic, score, verdict, and evidence
 
 4. Update baseline files if output format changes. Baselines live next to eval YAML files as `*.baseline.jsonl`.
 5. `agentv validate` is the cheap schema/config check. For no-live-provider quality validation, run graders against a real reference/oracle target or frozen transcript/replay fixture.
@@ -142,12 +142,12 @@ Use live dogfood before marking PRs ready when they affect eval execution, exper
 - Prefer the smallest realistic eval: one or two cases, bounded timeouts, and `workers: 1` for heavyweight agent providers.
 - For artifact/result contract changes, prefer letting AgentV choose the canonical run directory and capture the printed `Artifact workspace written to:` and `Results written to:` paths for evidence. Do not precompute `--output` unless the test specifically needs a fixed path.
 - For native experiment changes, run through `agentv eval run ... --experiment <experiment.yaml|ts>` so resolution, setup, scripts, target selection, run knobs, and artifact metadata are exercised together.
-- For repeat-run changes, use `evaluate_options.repeat.count >= 2` when validating repeated attempts. Inspect root `index.jsonl`, root `summary.json`, and the repeated case folder. Use `repeat` for authored configuration and `attempts[]` for produced executions. The repeated case folder should carry aggregate `summary.json` with flattened snake_case timing fields; attempt-specific outputs, transcripts, and metrics live under `attempt-N/`. Each `attempt-N/` folder should contain `result.json`, `grading.json`, `metrics.json`, `transcript.json`, `transcript-raw.jsonl`, and `outputs/answer.md` when answer output is available. `result.json` should point at `./grading.json`, `./metrics.json`, `./transcript.json`, and `./transcript-raw.jsonl` through the corresponding path fields.
-- For local OpenAI-compatible grading through the OAuth proxy, use `endpoint: http://127.0.0.1:10531/v1`, but still route `api_key` and `model` through environment references such as `${{ LOCAL_OPENAI_PROXY_API_KEY }}` and `${{ LOCAL_OPENAI_PROXY_MODEL }}`. Literal secrets and literal model values are intentionally rejected by target validation unless a resolver explicitly allows them.
-- For `codex`/Codex SDK live dogfood through the same local proxy, configure the agent target with `provider: codex`, `base_url: ${{ LOCAL_OPENAI_PROXY_BASE_URL }}`, `api_key: ${{ LOCAL_OPENAI_PROXY_API_KEY }}`, `model: ${{ LOCAL_OPENAI_PROXY_MODEL }}`, `api_format: responses`, `grader_target: <local-openai-grader>`, `workers: 1`, and a bounded `timeout_seconds`. Configure the grader target as `provider: openai` with the same local proxy env references; OpenAI-compatible targets default to chat completions unless `api_format: responses` is explicitly required. A minimal run should use `bun apps/cli/src/cli.ts eval run <eval.yaml> --targets <targets.yaml> --target <codex-target> --workers 1`.
+- For repeat-run changes, use `evaluate_options.repeat.count >= 2` when validating repeated executions. Inspect `.internal/index.jsonl`, root `summary.json`, and the repeated case folder. Use `repeat` for authored configuration and `sample_index`/`retry_index` for produced executions. The repeated case folder should carry aggregate `summary.json`; sample-specific outputs, transcripts, grading, and metrics live under `sample-N/`. Each `sample-N/` folder should contain `result.json`, `grading.json`, `metrics.json`, `transcript.json`, `transcript-raw.jsonl`, and `outputs/answer.md` when answer output is available. `result.json` should point at `./grading.json`, `./metrics.json`, `./transcript.json`, and `./transcript-raw.jsonl` through the corresponding path fields.
+- For local OpenAI-compatible grading through the OAuth proxy, use `base_url: http://127.0.0.1:10531/v1`, but still route `api_key` and `model` through environment references such as `{{ env.LOCAL_OPENAI_PROXY_API_KEY }}` and `{{ env.LOCAL_OPENAI_PROXY_MODEL }}`. Literal secrets and literal model values are intentionally rejected by target validation unless a resolver explicitly allows them.
+- For `codex`/Codex SDK live dogfood through the same local proxy, configure the agent target with the current target graph (`id`, `provider`, `runtime`, `config`) and put provider settings such as `base_url`, `api_key`, `model`, and `api_format` under `config`. Configure the reusable grader under `graders`, then select it with `defaults.grader` or assertion-level target selection; do not put a grader selector on the system-under-test target. A minimal run should use `bun apps/cli/src/cli.ts eval run <eval.yaml> --targets <targets.yaml> --target <codex-target> --workers 1`.
 - If the local proxy returns `401 token_expired`, the blocker is stale Codex OAuth, not AgentV target configuration. Refresh from a trusted local terminal with `codex logout`, `codex login --device-auth`, then restart `openai-oauth` and rerun the same eval command.
 - Preserve review evidence in `agentv-private` on an orphan `evidence/<bead-or-feature-slug>` branch. Include the run bundle, source eval/experiment/targets files, a short README, an artifact tree, contract checks, and screenshots when folder structure or UI behavior is under review.
-- If comparing against an external convention such as Vercel `agent-eval`, verify both semantic provenance and the physical `attempt-N` artifact layout for repeat runs.
+- If comparing against an external convention such as Vercel `agent-eval`, verify both semantic provenance and the physical `sample-N` artifact layout for repeat runs.
 - For transcript/result artifact contract changes, try the same provider spread before merging: `pi-cli`, `codex-sdk`, and `copilot-sdk` through the local OpenAI-compatible endpoint when available. If a provider cannot run live, record the exact blocker, the run bundle or command output, and whether coverage moved to fixture/regression tests.
 - If dogfood or review changes the durable verification playbook, update this file or `AGENTS.md` in the same PR. Use `docs/solutions/` for longer reusable lessons rather than relying on PR comments or private evidence as the only source.
 

CONCEPTS.md

@@ -6,7 +6,7 @@ Shared domain vocabulary for this project — entities, named processes, and sta
 
 **Provider** — an adapter plugin that connects AgentV's evaluation engine to a specific AI system (e.g., copilot CLI, copilot SDK, Claude API, pi). Each provider implements the request/response contract: given a test case, invoke the AI system and return its output. Providers are selected per-target in eval YAML and can be extended via the provider registry.
 
-**Target** — the eval YAML declaration that activates a specific provider for an evaluation run. A target names the provider, supplies configuration (model, API keys, timeouts, passthrough args), and scopes to a subset of test cases when needed. A single eval file can declare multiple targets to compare AI systems side by side.
+**Target** — The eval YAML or config declaration that activates a specific provider for an evaluation run. A target has a stable `id`, a `provider` backend kind, an optional `runtime`, and provider settings under `config`; field-level `file://` references can load prompts, defaults, or other config fragments at the boundary. A single eval file can declare multiple targets to compare AI systems side by side.
 
 **Provider runtime boundary** — the process boundary between AgentV's evaluation orchestrator and the agent runtime a provider invokes. CLI-backed providers place the agent runtime outside the orchestrator; in-process SDK providers share the orchestrator process and need either a targeted transport fix or subprocess-style isolation when runtime teardown can threaten run artifact finalization.
 
@@ -20,11 +20,11 @@ Shared domain vocabulary for this project — entities, named processes, and sta
 
 **Wrapper eval** — Eval YAML whose main job is to import task suites and bind top-level runtime policy such as target selection, repeat count, timeout, budget, and thresholds. Wrapper evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. A wrapper that imports suites with `type: suite` does not define parent `workspace`; imported suites own task environment.
 
-**Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Experiment is expressed as the reserved `tags.experiment` key (see **Tags**); there is no top-level `experiment` field. Runtime policy belongs in top-level eval fields or target objects; the experiment label is recorded in `summary.json` and `index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `workspace.hooks` or `targets[].hooks`, not in a separate experiment artifact.
+**Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Experiment is expressed as the reserved `tags.experiment` key (see **Tags**); there is no top-level `experiment` field. Runtime policy belongs in top-level eval fields or target objects; the experiment label is recorded in `summary.json` and `.internal/index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `extensions` or target hooks, not in a separate experiment artifact.
 
-**Tags** — A promptfoo-shaped `Record<string,string>` map authored on an eval (or project config / `--tag key=value`) that labels a run with structured facets such as `experiment`, `team`, or `env`. The reserved `experiment` key feeds the experiment namespace. The resolved map is recorded in `summary.json` `metadata.tags` and every `index.jsonl` row, and the Dashboard "Tags" tab groups and compares runs by any tag key. This is the only "tags" concept: the earlier free-form manual per-run tag chips have been removed. (Suite-level `tags` may still be authored as a string list, which is a selection construct for `select.tags` / `--tag name` filtering rather than run metadata.)
+**Tags** — A promptfoo-shaped `Record<string,string>` map authored on an eval (or project config / `--tag key=value`) that labels a run with structured facets such as `experiment`, `team`, or `env`. The reserved `experiment` key feeds the experiment namespace. The resolved map is recorded in `summary.json` `metadata.tags` and every `.internal/index.jsonl` row, and the Dashboard "Tags" tab groups and compares runs by any tag key. This is the only "tags" concept: the earlier free-form manual per-run tag chips have been removed. (Suite-level `tags` may still be authored as a string list, which is a selection construct for `select.tags` / `--tag name` filtering rather than run metadata.)
 
-**Workspace** — The task environment an eval prepares for the agent: repositories, templates, fixture files, and lifecycle hooks. It is not prompt input; use `input` for instructions and `workspace.repos[]` for multi-repo workspaces the agent can inspect or modify through tools.
+**Workspace** — The task environment an eval prepares for the agent: repositories, templates, fixture files, and post-materialization extensions. It is not prompt input; use `input` for instructions and `workspace.repos[]` for multi-repo workspaces the agent can inspect or modify through tools. `workspace.repos[]` is first-class declarative provenance, materializes before extensions run, and `workspace.scope` is `suite` or `attempt`.
 
 **Run bundle** — A committed local result directory at `.agentv/results/<run_id>/`. `summary.json` records run metadata such as `run_id` and `experiment`; `.internal/index.jsonl` records per-case rows.
 
@@ -34,7 +34,7 @@ Shared domain vocabulary for this project — entities, named processes, and sta
 
 **Result source identity** — The stable source identity for a result row: repo-relative `eval_path`, `test_id`, and `target`. `suite` and `name` are display metadata, not storage or routing identity.
 
-**Result directory** — The `result_dir` field in a `index.jsonl` row. It is a run-local directory allocation for that row's sidecars and outputs, usually a readable test-id or slug prefix plus a UUID/hash-like suffix. Consumers discover it from `index.jsonl` and must not infer it from suite names, display names, test IDs, targets, models, or folder position.
+**Result directory** — The `result_dir` field in a `.internal/index.jsonl` row. It is a run-local directory allocation for that row's sidecars and outputs, usually a readable test-id or slug prefix plus a UUID/hash-like suffix. Consumers discover it from `.internal/index.jsonl` and must not infer it from suite names, display names, test IDs, targets, models, or folder position.
 
 **Artifact sidecar** — A file beside or below a result directory that provides evidence for a result, such as `summary.json`, `grading.json`, `result.json`, transcripts, logs, or outputs. Sidecars are evidence, not the primary discovery mechanism for a run.
 
@@ -44,7 +44,7 @@ Shared domain vocabulary for this project — entities, named processes, and sta
 
 **Repeat run** — A configured request to execute the same eval case and target more than once in the same run bundle. Repeat runs measure stochastic reliability, verifier stability, and drift; they are not the default CI path.
 
-**Attempt** — One concrete execution inside a repeat run. Attempts keep their own score, status, timing, trace, transcript, logs, and artifacts so aggregate results never hide individual evidence.
+**Attempt** — One concrete execution inside a repeat run. Attempts keep their own score, status, metrics, trace, transcript, logs, and artifacts so aggregate results never hide individual evidence.
 
 **Pass rate** — Assertion or expectation pass rate inside a grading result: passed assertions or expectations divided by total assertions or expectations. AgentV does not use `pass_rate` for repeat-attempt success frequency.