From 7b8d627cb99f3788db5f216d0b4d827a4674b356 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Sat, 4 Jul 2026 05:29:35 +0200 Subject: [PATCH] docs(web): align authoring contract docs --- .../docs/docs/next/evaluation/eval-files.mdx | 27 ++++++++--------- .../docs/docs/next/evaluation/experiments.mdx | 15 +++++----- .../docs/docs/next/evaluation/rubrics.mdx | 2 +- .../docs/next/evaluation/running-evals.mdx | 17 ++++------- .../content/docs/docs/next/evaluation/sdk.mdx | 4 +-- .../docs/docs/next/graders/composite.mdx | 2 +- .../docs/next/graders/custom-assertions.mdx | 6 ++-- .../docs/docs/next/guides/autoresearch.mdx | 2 +- .../docs/next/guides/benchmark-provenance.mdx | 4 +-- .../guides/skill-improvement-workflow.mdx | 15 +++++----- .../integrations/autoevals-integration.mdx | 2 +- .../docs/next/reference/result-artifacts.mdx | 9 +++--- .../docs/docs/next/targets/configuration.mdx | 9 +++--- .../content/docs/docs/next/tools/results.mdx | 29 +++++++++---------- 14 files changed, 66 insertions(+), 77 deletions(-) diff --git a/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx b/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx index db8a64b41..ce20a5dfb 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx @@ -10,7 +10,7 @@ Evaluation files define the test cases, graders, workspace lifecycle, and run controls for an evaluation run. The reserved `tags.experiment` key is the run/result grouping label, top-level `target` identifies the system under test, and fields such as `evaluate_options.repeat`, `threshold`, `timeout_seconds`, -`evaluate_options.budget_usd`, and `execution.max_concurrency` control repeated +`evaluate_options.budget_usd`, and `evaluate_options.max_concurrency` control repeated attempts and gates. Workspace lifetime belongs under `workspace.scope`; repository provenance belongs under `workspace.repos`; Docker/container binding belongs under `workspace.docker`. Non-provisioning setup commands belong in @@ -21,7 +21,7 @@ data formats: YAML and JSONL. YAML is the canonical portable model. TypeScript helpers, generated fixtures, and Python scripts should lower to the same YAML/JSONL shapes rather than inventing a separate eval contract. Eval files describe the task, target binding, and run controls. Use -`execution.max_concurrency` for authored suite concurrency. Operators can still +`evaluate_options.max_concurrency` for authored suite concurrency. Operators can still override concurrency with `agentv eval --workers N`; do not author legacy `workers` fields in eval YAML. @@ -81,10 +81,8 @@ A wrapper eval stays ordinary eval YAML while choosing a target and run controls # experiments/refunds-codex.eval.yaml name: refunds-codex target: codex-gpt5 -execution: - max_concurrency: 3 - evaluate_options: + max_concurrency: 3 repeat: count: 2 strategy: pass_any @@ -137,16 +135,15 @@ tests: | `suite` | Optional suite identifier | | `category` | Optional slash-delimited analytics taxonomy path. Overrides the category derived from the eval file path. | | `target` | System under test by configured target `id` or inline target object | -| `tags` | Optional promptfoo-style metadata map. Use `tags.experiment` as the run/result grouping label. | +| `tags` | Optional metadata map. Use `tags.experiment` as the run/result grouping label. | | `prompts` | Optional top-level prompt matrix. Entries can be strings, chat message arrays, files, or generated prompt functions. | | `targets` | Optional target matrix. Entries reference target ids or inline target objects. | | `evaluate_options.repeat` | Optional repeat policy as a positive integer shorthand or object with `count`, `strategy`, `early_exit`, and `cost_limit_usd` | +| `evaluate_options` | Optional evaluation runtime options such as `budget_usd`, `repeat`, and `max_concurrency` | | `timeout_seconds` | Optional per-case timeout | -| `evaluate_options` | Optional evaluation runtime options such as `budget_usd` and `repeat` | -| `execution.max_concurrency` | Optional general eval parallelism for this suite | | `threshold` | Optional suite quality threshold | | `workspace` | Suite-level task environment — inline object or string path to an external workspace file. Repo entries declare identity and checkout pins; acquisition is covered in [Workspace Architecture](/docs/guides/workspace-architecture/#repo-provenance-vs-acquisition). | -| `extensions` | Promptfoo-style lifecycle hooks: `file://path/to/hooks.mjs:beforeAll`, `beforeEach`, `afterEach`, `afterAll`, plus the built-in `agentv:agent-rules`. Hooks run after `workspace.repos` materializes. | +| `extensions` | Lifecycle hooks: `file://path/to/hooks.mjs:beforeAll`, `beforeEach`, `afterEach`, `afterAll`, plus the built-in `agentv:agent-rules`. Hooks run after `workspace.repos` materializes. | | `imports` | Optional import groups. `imports.suites` imports full child eval suites with their task context. `imports.tests` imports raw test rows into this file's context. Import entries may use scoped `run:` overrides for `threshold`, `repeat`, `timeout_seconds`, and `budget_usd`. | | `tests` | Inline raw tests or a string path to an external raw-case file or directory. Legacy `tests[].include` entries still load with a migration warning; prefer `imports.suites` or `imports.tests`. | | `assert` | Suite-level graders appended to each test unless `execution.skip_defaults: true` is set on the test | @@ -163,7 +160,7 @@ context, but it does not materialize a repo for the agent to inspect. ### Prompts, Vars, and Target Expansion -Use top-level `prompts` when you want promptfoo-style prompt variants. AgentV +Use top-level `prompts` when you want a prompt matrix. AgentV renders each prompt with each test's `vars`, then expands the run as `prompts x targets x tests x repeat` before execution. Each expanded row keeps the original `test_id` plus prompt and target identity for Dashboard filtering, @@ -212,7 +209,7 @@ for the target task and can be a string, object, or message array. Use ### Lifecycle Extensions -`extensions` uses Promptfoo-compatible lifecycle names. File hooks are local +`extensions` uses AgentV lifecycle names. File hooks are local JavaScript or TypeScript modules resolved relative to the eval file: ```yaml @@ -446,7 +443,7 @@ tests: ./cases.yaml The path is resolved relative to the eval file's directory. The external raw case file can be a YAML or JSON array of test objects, a JSONL file with one -test per line, a promptfoo-compatible CSV file, or an explicit JavaScript or +test per line, a CSV file with AgentV expected columns, or an explicit JavaScript or Python dataset function such as `file://generate-tests.mjs:createTests` or `file://generate_tests.py:create_tests`. String entries inside a `tests:` list work the same way and may use direct paths, `file://` paths, directories, or @@ -457,15 +454,15 @@ tests: - ./cases/*.cases.yaml ``` -CSV datasets support promptfoo-style magic columns. `__expected` and +CSV datasets support magic columns. `__expected` and `__expectedN` create AgentV assertions using the supported expected-column mini-DSL (`contains:*`, `icontains:*`, `contains-any:*`, `contains-all:*`, `icontains-any:*`, `icontains-all:*`, `starts-with:*`, `ends-with:*`, `regex:*`, `equals:*`, `is-json`, `latency()`, `cost()`, `grade:*`, `llm-rubric:*`, `javascript:*`, `fn:*`, `eval:*`, `python:*`, and `file://*.py`; file paths inside CSV cells are resolved relative to the CSV -file). Unsupported promptfoo assertion forms such as `similar:*` are rejected -during validation instead of being skipped at runtime. +file). Unsupported assertion forms such as `similar:*` are rejected during +validation instead of being skipped at runtime. `__provider_output` becomes first-class `expected_output` reference data, `__metric` names the generated assertions, `__threshold` sets the test threshold, `__metadata:` adds metadata, and `__config:__expectedN:threshold` sets an diff --git a/apps/web/src/content/docs/docs/next/evaluation/experiments.mdx b/apps/web/src/content/docs/docs/next/evaluation/experiments.mdx index 8740ab7e4..a58fe984f 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/experiments.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/experiments.mdx @@ -11,9 +11,7 @@ AgentV eval files are the runnable authoring artifact. Use top-level label, `target` for the system under test, and flat top-level run controls such as `timeout_seconds` and `threshold`. Use `evaluate_options` for evaluation runtime options such as `repeat`, `budget_usd`, and `max_concurrency`. -Use `execution.max_concurrency` in eval YAML or `.agentv/config.yaml` for the -AgentV config graph concurrency field. `agentv eval --workers N` remains an -operator-side override. +`agentv eval --workers N` remains an operator-side override. ```yaml name: support-regression @@ -190,9 +188,10 @@ tests: Scoped `run:` supports `threshold`, `repeat`, `timeout_seconds`, and per-case `budget_usd` overrides. Parent suite budgets should use `evaluate_options.budget_usd` for public eval authoring. Use -`evaluate_options.max_concurrency` for authored concurrency. Candidate-changing fields stay -parent-level. Executable workspace setup belongs in top-level lifecycle extensions, and -provider-specific setup belongs in target configuration. +`evaluate_options.max_concurrency` for authored concurrency. +Candidate-changing fields stay parent-level. Executable workspace setup belongs +in top-level lifecycle extensions, and provider-specific setup belongs in target +configuration. ## Lifecycle Ownership @@ -278,8 +277,8 @@ folder names are only storage allocation and must not define result semantics. ### Tags as run metadata (`tags.experiment`) Suite-level `tags` accepts either the existing selection form (a string or list of -strings that drives `select.tags` / `--tag name` filtering) **or** a -promptfoo-shaped map: +strings that drives `select.tags` / `--tag name` filtering) **or** a metadata +map: ```yaml tags: diff --git a/apps/web/src/content/docs/docs/next/evaluation/rubrics.mdx b/apps/web/src/content/docs/docs/next/evaluation/rubrics.mdx index bfba4da4e..a76cbb2d7 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/rubrics.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/rubrics.mdx @@ -27,7 +27,7 @@ All strings are collected into a single llm-rubric grader automatically. ### Full form for advanced options -Use `type: llm-rubric` explicitly when you need weights, required flags, or score ranges. Put structured rubric items in `value` so the assertion stays compatible with promptfoo's `llm-rubric.value` object/array field: +Use `type: llm-rubric` explicitly when you need weights, required flags, or score ranges. Put structured rubric items in `value`: ```yaml tests: diff --git a/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx b/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx index fe4a211aa..31943e120 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx @@ -191,7 +191,7 @@ OpenTelemetry/OpenInference spans directly to that backend during execution. agentv inspect stats .agentv/results//.internal/index.jsonl # Inspect AgentV-owned per-case artifacts and transcript sidecars -agentv inspect show .agentv/results//index.jsonl --tree +agentv inspect show .agentv/results//.internal/index.jsonl --tree ``` `index.jsonl` contains aggregate metrics such as score, latency, cost, token @@ -217,7 +217,7 @@ agentv eval evals/my-eval.yaml --target gpt --target claude --workers 4 # The target matrix shares the same 4-worker budget ``` -This matches the standard model used by eval frameworks (promptfoo, deepeval, OpenAI Evals) and avoids cross-file workspace races without any special configuration. +This avoids cross-file workspace races without requiring per-file worker pools. ### Workspace Path and Finish Policy @@ -461,7 +461,7 @@ Use `config.yaml` for portable defaults and shared eval-definition fields that c ### YAML config (`config.yaml` plus optional `config.local.yaml`) Project config and eval YAML share the same eval-definition graph for targets, -graders, tests, defaults, and execution policy. Small projects can keep that +graders, tests, defaults, and supported execution defaults. Small projects can keep that graph inline: ```yaml @@ -486,9 +486,6 @@ tests: defaults: target: codex-local grader: openai-grader - -execution: - max_concurrency: 3 ``` Larger projects can decompose any supported top-level field with a direct @@ -500,7 +497,6 @@ targets: file://targets.yaml graders: file://graders.yaml tests: file://tests.yaml defaults: file://defaults.yaml -execution: file://execution.yaml ``` ```yaml @@ -522,9 +518,9 @@ Do not wrap referenced field files in another object. For example, `targets: file://targets.yaml` expects `targets.yaml` to contain a bare array, not `{ targets: [...] }`. -`execution.max_concurrency` is AgentV's general eval parallelism field for this -config graph. It is AgentV's run-policy shape, aligned with the general -max-concurrency concept in eval runners, not a copied Promptfoo YAML path. +Authored suite concurrency belongs in eval files under +`evaluate_options.max_concurrency`, next to the suite or wrapper eval it +controls. `agentv eval --workers N` remains the operator-side override. Other project defaults can live beside the graph: @@ -550,7 +546,6 @@ eval_patterns: | Field | CLI equivalent | Type | Default | Description | |-------|---------------|------|---------|-------------| | `verbose` | `--verbose` | boolean | `false` | Enable verbose logging | -| `max_concurrency` | `--workers` | integer | none | Default eval parallelism for the composable config graph | | `keep_workspaces` | `--keep-workspaces` | boolean | `false` | Always keep temp workspaces after eval | | `workspace_path` | `--workspace-path` | string | none | Machine-local existing workspace directory | | `refs` | none | object | none | Project-defined named references for fields that support `ref://name`, such as shared `default_test` files | diff --git a/apps/web/src/content/docs/docs/next/evaluation/sdk.mdx b/apps/web/src/content/docs/docs/next/evaluation/sdk.mdx index 725e3694e..2e55aca0f 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/sdk.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/sdk.mdx @@ -220,7 +220,7 @@ export default defineAssertion(({ output }) => { const pass = wordCount >= 3; return { pass, - assert: [{ text: `Output has ${wordCount} words`, passed: pass }], + assertions: [{ text: `Output has ${wordCount} words`, passed: pass }], }; }); ``` @@ -272,7 +272,7 @@ import { defineScriptGrader } from '@agentv/sdk'; export default defineScriptGrader(({ output, traceSummary }) => ({ score: (output ?? '').length > 0 && (traceSummary?.eventCount ?? 0) <= 5 ? 1.0 : 0.5, - assert: [ + assertions: [ { text: 'Answer is not empty', passed: (output ?? '').length > 0 }, { text: 'Efficient tool usage', passed: (traceSummary?.eventCount ?? 0) <= 5 }, ], diff --git a/apps/web/src/content/docs/docs/next/graders/composite.mdx b/apps/web/src/content/docs/docs/next/graders/composite.mdx index a6c6014d5..6f202b264 100644 --- a/apps/web/src/content/docs/docs/next/graders/composite.mdx +++ b/apps/web/src/content/docs/docs/next/graders/composite.mdx @@ -134,7 +134,7 @@ console.log( JSON.stringify({ score: anyPassed ? 1 : 0, verdict: anyPassed ? 'pass' : 'fail', - assert: [{ text: `Any-or gate: ${anyPassed ? 'passed' : 'failed'}`, passed: anyPassed }], + assertions: [{ text: `Any-or gate: ${anyPassed ? 'passed' : 'failed'}`, passed: anyPassed }], }), ); ``` diff --git a/apps/web/src/content/docs/docs/next/graders/custom-assertions.mdx b/apps/web/src/content/docs/docs/next/graders/custom-assertions.mdx index 43fb35500..fcd12376c 100644 --- a/apps/web/src/content/docs/docs/next/graders/custom-assertions.mdx +++ b/apps/web/src/content/docs/docs/next/graders/custom-assertions.mdx @@ -69,7 +69,7 @@ export default defineAssertion(({ output }) => { const pass = wordCount >= 3; return { pass, - assert: [{ text: `Output has ${wordCount} words`, passed: pass }], + assertions: [{ text: `Output has ${wordCount} words`, passed: pass }], }; }); ``` @@ -89,7 +89,7 @@ export default defineAssertion(({ output, traceSummary }) => { const isEfficient = (traceSummary?.eventCount ?? 0) <= 5 ? 0.5 : 0; return { score: hasContent + isEfficient, - assert: [ + assertions: [ { text: 'Has content', passed: hasContent > 0 }, { text: 'Efficient', passed: isEfficient > 0 }, ], @@ -202,7 +202,7 @@ export default defineAssertion(({ output }) => { return { pass, score: pass ? 1.0 : Math.min(wordCount / minWords, 0.9), - assert: [ + assertions: [ { text: pass ? `Output has ${wordCount} words (>= ${minWords} required)` diff --git a/apps/web/src/content/docs/docs/next/guides/autoresearch.mdx b/apps/web/src/content/docs/docs/next/guides/autoresearch.mdx index 5a1bfe4cf..993806856 100644 --- a/apps/web/src/content/docs/docs/next/guides/autoresearch.mdx +++ b/apps/web/src/content/docs/docs/next/guides/autoresearch.mdx @@ -102,7 +102,7 @@ Review the mutation history with `git log` after the run completes. After each eval cycle, autoresearch runs `agentv results compare` between the current candidate and the best baseline: ```bash -agentv results compare /index.jsonl /index.jsonl --json +agentv results compare /.internal/index.jsonl /.internal/index.jsonl --json ``` The decision rule: diff --git a/apps/web/src/content/docs/docs/next/guides/benchmark-provenance.mdx b/apps/web/src/content/docs/docs/next/guides/benchmark-provenance.mdx index a648e1429..61126f307 100644 --- a/apps/web/src/content/docs/docs/next/guides/benchmark-provenance.mdx +++ b/apps/web/src/content/docs/docs/next/guides/benchmark-provenance.mdx @@ -28,10 +28,10 @@ Use this split when deciding where a benchmark key belongs: |------------|--------------|------------------| | `workspace.repos[]` | Yes | Declares repo identity and checkout refs; AgentV resolves acquisition and materializes the checkout. | | `workspace.template` | Yes | Copies a workspace template into the run workspace. | -| `extensions` | Yes | Runs Promptfoo-style lifecycle setup after `workspace.template` and `workspace.repos` materialize. | +| `extensions` | Yes | Runs lifecycle setup after `workspace.template` and `workspace.repos` materialize. | | `workspace.hooks.after_each.reset` | Yes | Controls workspace reset policy after each case. | | `workspace.scope` | Yes | Controls suite vs per-attempt workspace lifetime. Runtime workspace paths are machine-local config/CLI bindings, not benchmark provenance. | -| `experiment` | Yes | Selects targets, thresholds, repeat policy, budgets, and default grader behavior. Concurrency is an operator/run setting from `--workers` or project config. | +| `experiment` | Yes | Selects targets, thresholds, repeat policy, budgets, and default grader behavior. Authored concurrency uses `evaluate_options.max_concurrency`; `--workers` is the operator override. | | `input`, `input_files`, `expected_output` | Yes | Builds the target prompt and carries passive gold/reference data for graders. | | `assert` | Yes | Runs deterministic, LLM, composite, or script graders. | | Top-level `name`, `version`, `tags`, `license`, `requires` | Informational | Identifies and categorizes the suite. | diff --git a/apps/web/src/content/docs/docs/next/guides/skill-improvement-workflow.mdx b/apps/web/src/content/docs/docs/next/guides/skill-improvement-workflow.mdx index 611542d12..d767cc410 100644 --- a/apps/web/src/content/docs/docs/next/guides/skill-improvement-workflow.mdx +++ b/apps/web/src/content/docs/docs/next/guides/skill-improvement-workflow.mdx @@ -94,10 +94,11 @@ Run the evaluation **without** the skill loaded to establish a baseline: agentv eval evals.json --target baseline agentv convert evals.json --out EVAL.yaml -agentv eval EVAL.yaml --target baseline +agentv eval EVAL.yaml --target baseline --output .agentv/results/baseline ``` -This produces a results file (e.g., `results-baseline.jsonl`) showing how the agent performs on its own. +This produces a run bundle under `.agentv/results/baseline` showing how the +agent performs on its own. ### Baseline isolation @@ -112,7 +113,7 @@ drafts/ SKILL.md # Baseline run won't pick it up -agentv eval EVAL.yaml --target baseline +agentv eval EVAL.yaml --target baseline --output .agentv/results/baseline ``` ## Step 3: Run Candidate Evaluation @@ -120,7 +121,7 @@ agentv eval EVAL.yaml --target baseline Run the same evaluation **with** the skill loaded: ```bash -agentv eval EVAL.yaml --target candidate +agentv eval EVAL.yaml --target candidate --output .agentv/results/candidate ``` Or grade existing sessions offline (no API keys required): @@ -141,7 +142,7 @@ Offline grading is useful when you want to evaluate skills with agents that don' Compare the baseline and candidate runs: ```bash -agentv results compare results-baseline.jsonl results-candidate.jsonl +agentv results compare .agentv/results/baseline/.internal/index.jsonl .agentv/results/candidate/.internal/index.jsonl ``` The comparison output shows: @@ -199,10 +200,10 @@ Loop back to Step 3 with the improved skill: ```bash # Run the improved candidate -agentv eval EVAL.yaml --target candidate +agentv eval EVAL.yaml --target candidate --output .agentv/results/candidate # Compare against the previous baseline -agentv results compare results-baseline.jsonl results-candidate.jsonl +agentv results compare .agentv/results/baseline/.internal/index.jsonl .agentv/results/candidate/.internal/index.jsonl ``` Each iteration should show: diff --git a/apps/web/src/content/docs/docs/next/integrations/autoevals-integration.mdx b/apps/web/src/content/docs/docs/next/integrations/autoevals-integration.mdx index 0d74a6fa7..65fb10197 100644 --- a/apps/web/src/content/docs/docs/next/integrations/autoevals-integration.mdx +++ b/apps/web/src/content/docs/docs/next/integrations/autoevals-integration.mdx @@ -96,7 +96,7 @@ const rationale = result.metadata?.rationale ?? "No rationale provided"; console.log( JSON.stringify({ score, - assert: [{ text: rationale, passed: score >= 0.5 }], + assertions: [{ text: rationale, passed: score >= 0.5 }], reasoning: rationale, }) ); diff --git a/apps/web/src/content/docs/docs/next/reference/result-artifacts.mdx b/apps/web/src/content/docs/docs/next/reference/result-artifacts.mdx index 735cb75d8..20ef96e9f 100644 --- a/apps/web/src/content/docs/docs/next/reference/result-artifacts.mdx +++ b/apps/web/src/content/docs/docs/next/reference/result-artifacts.mdx @@ -236,11 +236,10 @@ manual `prepare`/`grade` attempts, or imported provider sessions. That is why source metadata belong in `index.jsonl`: tools can filter dynamically without requiring every run to be pre-split into semantic folders. -When a run resolves a promptfoo-shaped tags map (from suite `tags`, project -config `tags`, or `--tag key=value`), the resolved map is emitted as `tags` on -each row and as `summary.json.metadata.tags`. Its reserved `experiment` key -matches the row `experiment` field, so trend/compare views can group by -`tags.experiment`. +When a run resolves a tags metadata map (from suite `tags`, project config +`tags`, or `--tag key=value`), the resolved map is emitted as `tags` on each row +and as `summary.json.metadata.tags`. Its reserved `experiment` key matches the +row `experiment` field, so trend/compare views can group by `tags.experiment`. Use `repeat` for authoring configuration and `samples` for produced executions. The `sample-1/`, `sample-2/`, and later folders under a result diff --git a/apps/web/src/content/docs/docs/next/targets/configuration.mdx b/apps/web/src/content/docs/docs/next/targets/configuration.mdx index 01a2dc15a..21468005a 100644 --- a/apps/web/src/content/docs/docs/next/targets/configuration.mdx +++ b/apps/web/src/content/docs/docs/next/targets/configuration.mdx @@ -250,8 +250,8 @@ still choose its own grader target when the evaluator supports that override. ### Lifecycle Extensions -Run non-provisioning setup at Promptfoo-compatible lifecycle points using -top-level `extensions`. The harness materializes `workspace.template` and +Run non-provisioning setup at AgentV lifecycle points using top-level +`extensions`. The harness materializes `workspace.template` and `workspace.repos` first, then runs `beforeAll` extensions. Use extensions for dependency installs, builds, fixture generation, and agent-rule staging. Use target hooks for runner-specific setup. Keep repo identity and checkout pins in @@ -302,9 +302,8 @@ case context: } ``` -`workspace.hooks` remains the reset-policy home for `after_each.reset`. Legacy -command hooks still parse for existing local suites, but new portable evals -should use `extensions` for executable setup. +`workspace.hooks` is the reset-policy home for `after_each.reset`. Use +`extensions` for executable setup. ### Repository Lifecycle diff --git a/apps/web/src/content/docs/docs/next/tools/results.mdx b/apps/web/src/content/docs/docs/next/tools/results.mdx index 445f299cb..ab4ae12b6 100644 --- a/apps/web/src/content/docs/docs/next/tools/results.mdx +++ b/apps/web/src/content/docs/docs/next/tools/results.mdx @@ -10,7 +10,7 @@ import { Image } from 'astro:assets'; import resultsReportOverview from '../../../../../assets/screenshots/results-report-overview.png'; import resultsReportDetails from '../../../../../assets/screenshots/results-report-details.png'; -The `results` command family works on existing local AgentV run workspaces and `index.jsonl` manifests. Use it after an eval run to inspect failures, validate manifests, export artifact layouts, combine/delete local run workspaces, or generate a shareable HTML report. +The `results` command family works on existing local AgentV run workspaces and `.internal/index.jsonl` manifests. Use it after an eval run to inspect failures, validate manifests, export artifact layouts, combine/delete local run workspaces, or generate a shareable HTML report. Remote result repository exchange is intentionally not part of `agentv results`. New eval runs publish completed artifacts to a configured results repo or branch; `auto_push: true` additionally pushes that branch to the remote. Manual remote status and sync are Dashboard/API workflows. See [Dashboard Remote Results](/docs/tools/dashboard/#remote-results) for configuration and sync behavior, and [WIP checkpoints](/docs/tools/wip-checkpoints/) for recovering in-progress runs before final publish. @@ -30,16 +30,16 @@ start with [Result Artifact Contract](/docs/reference/result-artifacts/). | `results show` | Display case-level rows from a run workspace | | `results validate` | Validate that a workspace or manifest resolves correctly | -`results combine` writes a new direct run workspace under `.agentv/results//` and records the selected experiment label in `summary.json` and `index.jsonl` metadata. If the source runs span multiple experiments, pass `--experiment ` for the new combined run; AgentV does not silently invent a mixed-experiment label. +`results combine` writes a new direct run workspace under `.agentv/results//` and records the selected experiment label in `summary.json` and `.internal/index.jsonl` metadata. If the source runs span multiple experiments, pass `--experiment ` for the new combined run; AgentV does not silently invent a mixed-experiment label. ## `results report` -The `results report` command turns an existing run workspace or `index.jsonl` manifest into a self-contained HTML report for sharing, inspection, and human review. +The `results report` command turns an existing run workspace or `.internal/index.jsonl` manifest into a self-contained HTML report for sharing, inspection, and human review. AgentV results report overview showing 11 tests across 2 eval files with pass, fail, pass rate, duration, and cost summary cards ```bash -agentv results report +agentv results report ``` Examples: @@ -97,10 +97,10 @@ Use `--out docs/.html` when a repository should publish multiple runs. Lin Use `results export` when you need the artifact workspace layout itself rather than a rendered report. ```bash -agentv results export [--out ] [--duplicate-policy update] +agentv results export [--out ] [--duplicate-policy update] ``` -This is useful when a manifest needs to be materialized into a predictable artifact tree for other tooling, review, or archiving. The run workspace is also where generated test bundles live: `index.jsonl` rows may point to per-result `test_dir`, `eval_path`, `targets_path`, `files_path`, and `graders_path` entries. Keep those generated artifacts with the run when sharing or auditing results. +This is useful when a manifest needs to be materialized into a predictable artifact tree for other tooling, review, or archiving. The run workspace is also where generated test bundles live: `.internal/index.jsonl` rows may point to per-result `test_dir`, `eval_path`, `targets_path`, `files_path`, and `graders_path` entries. Keep those generated artifacts with the run when sharing or auditing results. The export source is still the canonical run bundle described in the [Result Artifact Contract](/docs/reference/result-artifacts/): `summary.json` @@ -125,8 +125,7 @@ Each attempt directory includes `metrics.json` (`schema_version: "agentv.metrics.v1"`). This is an AgentV-owned derived projection over the attempt trace/transcript, result row, and `grading.json`. It is the compact executor behavior summary for dashboards, comparison exports, -and metric-style graders; it is not canonical trace storage and does not carry -token/cost usage. +and metric-style graders; it is not canonical trace storage. Every case uses aggregate `summary.json`, then stores execution artifact details under `sample-N/`. Each `sample-N/` contains a compact per-attempt manifest @@ -145,13 +144,13 @@ conversation transcript with canonical `tool_name` values, joined `tool_use.result` blocks, and a precomputed `transcript_summary`. AgentV does not persist a public `trace.json` sidecar in run bundles; external observability systems can be linked through safe `external_trace` metadata when available. -`summary.json` remains the run-level aggregate summary. `index.jsonl` is the +`summary.json` remains the run-level aggregate summary. `.internal/index.jsonl` is the canonical row index for the run: one row per result, attempt, or case, carrying lightweight explicit paths such as `transcript_path`, `transcript_raw_path`, `file_changes_path`, and `metrics_path` plus artifact pointers only when detached payload publishing needs them. Dashboard search indexes, SQLite indexes, and other read models are derived projections over these run artifacts, -not replacements for `index.jsonl`. +not replacements for `.internal/index.jsonl`. Duration, token, and cost usage remains in `metrics.json`, including source labels such as `provider_reported`, `token_estimated`, `aggregate`, or `unavailable`. @@ -182,11 +181,11 @@ Agent Skills eval artifacts map into AgentV like this: | Agent Skills pattern | AgentV field | Artifact location | |----------------------|--------------|-------------------| -| Converted Agent Skills cases | AgentV eval cases and test bundle paths | Converted EVAL YAML plus optional `test_dir`, `eval_path`, `targets_path`, `files_path`, and `graders_path` in `index.jsonl` | +| Converted Agent Skills cases | AgentV eval cases and test bundle paths | Converted EVAL YAML plus optional `test_dir`, `eval_path`, `targets_path`, `files_path`, and `graders_path` in `.internal/index.jsonl` | | Per-case answer | Generated target output artifact | `sample-N/outputs/answer.md` | | Per-attempt sidecars | Normalized transcript, metrics, and raw provider evidence | `sample-N/transcript.json`, `sample-N/transcript-raw.jsonl`, `sample-N/metrics.json` | | Per-sample `metrics.json` | Duration, token totals, cost, execution, trajectory, and usage source labels | `sample-N/metrics.json` | -| Per-attempt `grading.json` | Assertions, graders, execution metrics, workspace changes | `sample-N/grading.json`; summary fields can reference the same trace/result facts | +| Per-attempt `grading.json` | `assertion_results`, graders, rubric evidence, and workspace-change grading facts | `sample-N/grading.json`; summary fields can reference the same trace/result facts | | Iteration-level `summary.json` | Pass rate, time, tokens, tool calls, cost aggregates | Run-level `summary.json` | | Transcript/log outlier analysis | Normalized transcript, raw evidence, metrics, and optional external trace link | `transcript.json` for portable review; `transcript-raw.jsonl` for native evidence; `metrics.json` for behavior summaries; `external_trace` for link-out correlation | | Aggregate pass rate/time/tokens/delta | Run summaries and comparison tooling | `summary.json`, result comparisons, and projection bundles | @@ -197,7 +196,7 @@ Use the additive projection bundle path when an external adapter needs a backend-neutral handoff instead of AgentV's full artifact tree: ```bash -agentv results export --projection-bundle +agentv results export --projection-bundle ``` This writes `projection_bundle.json` next to the exported artifacts. The bundle @@ -214,7 +213,7 @@ transcripts, datasets, experiments, or indexes into Phoenix. For adapter development and CI snapshots, use dry-run mode: ```bash -agentv results export --dry-run > projection_bundle.json +agentv results export --dry-run > projection_bundle.json ``` Dry-run prints deterministic JSON and does not write export artifacts. Vendor @@ -230,7 +229,7 @@ default, and raw-bearing artifact refs such as `grading_path`, `input_path`, include raw payloads and raw-bearing refs in the bundle, opt in explicitly: ```bash -agentv results export --dry-run --include-raw-content +agentv results export --dry-run --include-raw-content ``` Keep backend-specific anonymization in the adapter layer. For example, an Opik