diff --git a/.agents/conventions.md b/.agents/conventions.md index d20c2c676..dab30d4d9 100644 --- a/.agents/conventions.md +++ b/.agents/conventions.md @@ -51,7 +51,7 @@ These terms are distinct and not interchangeable. - Project: the top-level container Dashboard organizes around, backed by a registered workspace directory with `.agentv/`, run artifacts, traces, and experiments. The registry lives in `~/.agentv/projects.yaml` and is modeled by `ProjectEntry` and `ProjectRegistry` in `packages/core/src/projects.ts`. - Benchmark: a curated eval suite designed to measure something specific, in the academic ML sense. Example directories using this meaning are correctly named and should not be renamed. -The legacy `~/.agentv/benchmarks.yaml` file is auto-migrated to `projects.yaml` by `migrateLegacyBenchmarksFile()`. Run-level results metadata lives in `summary.json`, with `index.jsonl` as the discovery anchor. +The legacy `~/.agentv/benchmarks.yaml` file is auto-migrated to `projects.yaml` by `migrateLegacyBenchmarksFile()`. Run-level results metadata lives in `summary.json`, with `.internal/index.jsonl` as the per-run discovery anchor. Rule of thumb: @@ -128,7 +128,7 @@ If you spot a camelCase key already on disk or in a response, treat it as a bug `artifact_pointers` are for offloading large detached payload bytes from the results metadata/control plane. They describe where payloads such as transcript files live when a run is projected to `agentv/artifacts/v1` or a future object store, including `key`, `object_version`, `sha256`, `size`, `media_type`, and `schema_version`. -Do not add an `artifact_pointers.*` entry just because a new per-case artifact exists. Normal sidecars that stay in the run tree should be discoverable through explicit path fields on `index.jsonl` or manifests, for example `metrics_path` for `outputs/metrics.json`. +Do not add an `artifact_pointers.*` entry just because a new per-case artifact exists. Normal sidecars that stay in the run tree should be discoverable through explicit path fields on `.internal/index.jsonl` rows or manifests, for example `metrics_path` for `metrics.json`. Before adding a new pointer family, verify that the artifact is large enough or detached enough to benefit from offloading and that published result repos should avoid carrying those payload bytes on the primary results branch. @@ -136,9 +136,10 @@ Before adding a new pointer family, verify that the artifact is large enough or Grader types use kebab-case everywhere. -- Authored YAML config: `type: llm-rubric`, `type: script`, `type: is-json` -- Internal TypeScript still has the shared LLM grader implementation and registry key (`'llm-grader'`), but new authored evals should use `llm-rubric` for semantic LLM grading. -- Output `scores[].type`: use the authored grader type when available, such as `"llm-rubric"` or `"is-json"`. +- Authored YAML config: `type: llm-rubric`, `type: g-eval`, `type: script`, `type: is-json` +- Use `llm-rubric` for promptfoo-compatible free-form rubric checks, and `g-eval` for structured or multi-criteria rubric judging when the eval needs itemized criteria semantics. +- Internal TypeScript may keep shared LLM grader implementation names, but new authored evals should not use `llm-grader` as a public type. +- Output `scores[].type`: use the authored grader type when available, such as `"llm-rubric"`, `"g-eval"`, or `"is-json"`. - Registry keys: `registry.register('llm-rubric', ...)` for the authored rubric surface, with `llm-grader` retained as internal/shared implementation plumbing. Source of truth: `GRADER_KIND_VALUES` in `packages/core/src/evaluation/types.ts`. diff --git a/.agents/product-boundary.md b/.agents/product-boundary.md index ec959166c..ba2884291 100644 --- a/.agents/product-boundary.md +++ b/.agents/product-boundary.md @@ -41,7 +41,9 @@ Prefer these extension points before adding a built-in: - `script` graders for custom executable evaluation logic - plain assertion strings for simple semantic rubric checks -- `llm-rubric` for promptfoo-compatible free-form rubrics, structured rubric criteria, custom prompts, custom grader targets, or preprocessing +- `llm-rubric` for promptfoo-compatible free-form rubric checks +- `g-eval` for structured or multi-criteria rubric judging +- config-level grader targets selected through `defaults.grader` or assertion-level target selection, not target-level grader configuration - CLI wrappers that consume AgentV JSON or JSONL output for post-processing such as aggregation, comparison, or reporting Ask: can this be achieved with existing primitives plus a plugin or wrapper? If yes, it should not be a built-in. That includes niche config overrides for existing graders. diff --git a/.agents/verification.md b/.agents/verification.md index a6d5d4ec8..482267d9f 100644 --- a/.agents/verification.md +++ b/.agents/verification.md @@ -124,11 +124,11 @@ provider or grader blocker. bun apps/cli/src/cli.ts eval examples/features/rubric/evals/dataset.eval.yaml --test-id ``` -3. Inspect the results JSONL and verify: +3. Inspect the result bundle and verify: - the correct grader type ran by checking `scores[].type` - scores are calculated as expected -- the `assertions` array reflects the evaluation logic +- `grading.json.assertion_results` reflects the evaluation logic, score, verdict, and evidence 4. Update baseline files if output format changes. Baselines live next to eval YAML files as `*.baseline.jsonl`. 5. `agentv validate` is the cheap schema/config check. For no-live-provider quality validation, run graders against a real reference/oracle target or frozen transcript/replay fixture. @@ -142,12 +142,12 @@ Use live dogfood before marking PRs ready when they affect eval execution, exper - Prefer the smallest realistic eval: one or two cases, bounded timeouts, and `workers: 1` for heavyweight agent providers. - For artifact/result contract changes, prefer letting AgentV choose the canonical run directory and capture the printed `Artifact workspace written to:` and `Results written to:` paths for evidence. Do not precompute `--output` unless the test specifically needs a fixed path. - For native experiment changes, run through `agentv eval run ... --experiment ` so resolution, setup, scripts, target selection, run knobs, and artifact metadata are exercised together. -- For repeat-run changes, use `evaluate_options.repeat.count >= 2` when validating repeated attempts. Inspect root `index.jsonl`, root `summary.json`, and the repeated case folder. Use `repeat` for authored configuration and `attempts[]` for produced executions. The repeated case folder should carry aggregate `summary.json` with flattened snake_case timing fields; attempt-specific outputs, transcripts, and metrics live under `attempt-N/`. Each `attempt-N/` folder should contain `result.json`, `grading.json`, `metrics.json`, `transcript.json`, `transcript-raw.jsonl`, and `outputs/answer.md` when answer output is available. `result.json` should point at `./grading.json`, `./metrics.json`, `./transcript.json`, and `./transcript-raw.jsonl` through the corresponding path fields. -- For local OpenAI-compatible grading through the OAuth proxy, use `endpoint: http://127.0.0.1:10531/v1`, but still route `api_key` and `model` through environment references such as `${{ LOCAL_OPENAI_PROXY_API_KEY }}` and `${{ LOCAL_OPENAI_PROXY_MODEL }}`. Literal secrets and literal model values are intentionally rejected by target validation unless a resolver explicitly allows them. -- For `codex`/Codex SDK live dogfood through the same local proxy, configure the agent target with `provider: codex`, `base_url: ${{ LOCAL_OPENAI_PROXY_BASE_URL }}`, `api_key: ${{ LOCAL_OPENAI_PROXY_API_KEY }}`, `model: ${{ LOCAL_OPENAI_PROXY_MODEL }}`, `api_format: responses`, `grader_target: `, `workers: 1`, and a bounded `timeout_seconds`. Configure the grader target as `provider: openai` with the same local proxy env references; OpenAI-compatible targets default to chat completions unless `api_format: responses` is explicitly required. A minimal run should use `bun apps/cli/src/cli.ts eval run --targets --target --workers 1`. +- For repeat-run changes, use `evaluate_options.repeat.count >= 2` when validating repeated executions. Inspect `.internal/index.jsonl`, root `summary.json`, and the repeated case folder. Use `repeat` for authored configuration and `sample_index`/`retry_index` for produced executions. The repeated case folder should carry aggregate `summary.json`; sample-specific outputs, transcripts, grading, and metrics live under `sample-N/`. Each `sample-N/` folder should contain `result.json`, `grading.json`, `metrics.json`, `transcript.json`, `transcript-raw.jsonl`, and `outputs/answer.md` when answer output is available. `result.json` should point at `./grading.json`, `./metrics.json`, `./transcript.json`, and `./transcript-raw.jsonl` through the corresponding path fields. +- For local OpenAI-compatible grading through the OAuth proxy, use `base_url: http://127.0.0.1:10531/v1`, but still route `api_key` and `model` through environment references such as `{{ env.LOCAL_OPENAI_PROXY_API_KEY }}` and `{{ env.LOCAL_OPENAI_PROXY_MODEL }}`. Literal secrets and literal model values are intentionally rejected by target validation unless a resolver explicitly allows them. +- For `codex`/Codex SDK live dogfood through the same local proxy, configure the agent target with the current target graph (`id`, `provider`, `runtime`, `config`) and put provider settings such as `base_url`, `api_key`, `model`, and `api_format` under `config`. Configure the reusable grader under `graders`, then select it with `defaults.grader` or assertion-level target selection; do not put a grader selector on the system-under-test target. A minimal run should use `bun apps/cli/src/cli.ts eval run --targets --target --workers 1`. - If the local proxy returns `401 token_expired`, the blocker is stale Codex OAuth, not AgentV target configuration. Refresh from a trusted local terminal with `codex logout`, `codex login --device-auth`, then restart `openai-oauth` and rerun the same eval command. - Preserve review evidence in `agentv-private` on an orphan `evidence/` branch. Include the run bundle, source eval/experiment/targets files, a short README, an artifact tree, contract checks, and screenshots when folder structure or UI behavior is under review. -- If comparing against an external convention such as Vercel `agent-eval`, verify both semantic provenance and the physical `attempt-N` artifact layout for repeat runs. +- If comparing against an external convention such as Vercel `agent-eval`, verify both semantic provenance and the physical `sample-N` artifact layout for repeat runs. - For transcript/result artifact contract changes, try the same provider spread before merging: `pi-cli`, `codex-sdk`, and `copilot-sdk` through the local OpenAI-compatible endpoint when available. If a provider cannot run live, record the exact blocker, the run bundle or command output, and whether coverage moved to fixture/regression tests. - If dogfood or review changes the durable verification playbook, update this file or `AGENTS.md` in the same PR. Use `docs/solutions/` for longer reusable lessons rather than relying on PR comments or private evidence as the only source. diff --git a/CONCEPTS.md b/CONCEPTS.md index cb3c73a8e..2aafb03bf 100644 --- a/CONCEPTS.md +++ b/CONCEPTS.md @@ -6,7 +6,7 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Provider** — an adapter plugin that connects AgentV's evaluation engine to a specific AI system (e.g., copilot CLI, copilot SDK, Claude API, pi). Each provider implements the request/response contract: given a test case, invoke the AI system and return its output. Providers are selected per-target in eval YAML and can be extended via the provider registry. -**Target** — the eval YAML declaration that activates a specific provider for an evaluation run. A target names the provider, supplies configuration (model, API keys, timeouts, passthrough args), and scopes to a subset of test cases when needed. A single eval file can declare multiple targets to compare AI systems side by side. +**Target** — The eval YAML or config declaration that activates a specific provider for an evaluation run. A target has a stable `id`, a `provider` backend kind, an optional `runtime`, and provider settings under `config`; field-level `file://` references can load prompts, defaults, or other config fragments at the boundary. A single eval file can declare multiple targets to compare AI systems side by side. **Provider runtime boundary** — the process boundary between AgentV's evaluation orchestrator and the agent runtime a provider invokes. CLI-backed providers place the agent runtime outside the orchestrator; in-process SDK providers share the orchestrator process and need either a targeted transport fix or subprocess-style isolation when runtime teardown can threaten run artifact finalization. @@ -20,11 +20,11 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Wrapper eval** — Eval YAML whose main job is to import task suites and bind top-level runtime policy such as target selection, repeat count, timeout, budget, and thresholds. Wrapper evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. A wrapper that imports suites with `type: suite` does not define parent `workspace`; imported suites own task environment. -**Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Experiment is expressed as the reserved `tags.experiment` key (see **Tags**); there is no top-level `experiment` field. Runtime policy belongs in top-level eval fields or target objects; the experiment label is recorded in `summary.json` and `index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `workspace.hooks` or `targets[].hooks`, not in a separate experiment artifact. +**Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Experiment is expressed as the reserved `tags.experiment` key (see **Tags**); there is no top-level `experiment` field. Runtime policy belongs in top-level eval fields or target objects; the experiment label is recorded in `summary.json` and `.internal/index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `extensions` or target hooks, not in a separate experiment artifact. -**Tags** — A promptfoo-shaped `Record` map authored on an eval (or project config / `--tag key=value`) that labels a run with structured facets such as `experiment`, `team`, or `env`. The reserved `experiment` key feeds the experiment namespace. The resolved map is recorded in `summary.json` `metadata.tags` and every `index.jsonl` row, and the Dashboard "Tags" tab groups and compares runs by any tag key. This is the only "tags" concept: the earlier free-form manual per-run tag chips have been removed. (Suite-level `tags` may still be authored as a string list, which is a selection construct for `select.tags` / `--tag name` filtering rather than run metadata.) +**Tags** — A promptfoo-shaped `Record` map authored on an eval (or project config / `--tag key=value`) that labels a run with structured facets such as `experiment`, `team`, or `env`. The reserved `experiment` key feeds the experiment namespace. The resolved map is recorded in `summary.json` `metadata.tags` and every `.internal/index.jsonl` row, and the Dashboard "Tags" tab groups and compares runs by any tag key. This is the only "tags" concept: the earlier free-form manual per-run tag chips have been removed. (Suite-level `tags` may still be authored as a string list, which is a selection construct for `select.tags` / `--tag name` filtering rather than run metadata.) -**Workspace** — The task environment an eval prepares for the agent: repositories, templates, fixture files, and lifecycle hooks. It is not prompt input; use `input` for instructions and `workspace.repos[]` for multi-repo workspaces the agent can inspect or modify through tools. +**Workspace** — The task environment an eval prepares for the agent: repositories, templates, fixture files, and post-materialization extensions. It is not prompt input; use `input` for instructions and `workspace.repos[]` for multi-repo workspaces the agent can inspect or modify through tools. `workspace.repos[]` is first-class declarative provenance, materializes before extensions run, and `workspace.scope` is `suite` or `attempt`. **Run bundle** — A committed local result directory at `.agentv/results//`. `summary.json` records run metadata such as `run_id` and `experiment`; `.internal/index.jsonl` records per-case rows. @@ -34,7 +34,7 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Result source identity** — The stable source identity for a result row: repo-relative `eval_path`, `test_id`, and `target`. `suite` and `name` are display metadata, not storage or routing identity. -**Result directory** — The `result_dir` field in a `index.jsonl` row. It is a run-local directory allocation for that row's sidecars and outputs, usually a readable test-id or slug prefix plus a UUID/hash-like suffix. Consumers discover it from `index.jsonl` and must not infer it from suite names, display names, test IDs, targets, models, or folder position. +**Result directory** — The `result_dir` field in a `.internal/index.jsonl` row. It is a run-local directory allocation for that row's sidecars and outputs, usually a readable test-id or slug prefix plus a UUID/hash-like suffix. Consumers discover it from `.internal/index.jsonl` and must not infer it from suite names, display names, test IDs, targets, models, or folder position. **Artifact sidecar** — A file beside or below a result directory that provides evidence for a result, such as `summary.json`, `grading.json`, `result.json`, transcripts, logs, or outputs. Sidecars are evidence, not the primary discovery mechanism for a run. @@ -44,7 +44,7 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Repeat run** — A configured request to execute the same eval case and target more than once in the same run bundle. Repeat runs measure stochastic reliability, verifier stability, and drift; they are not the default CI path. -**Attempt** — One concrete execution inside a repeat run. Attempts keep their own score, status, timing, trace, transcript, logs, and artifacts so aggregate results never hide individual evidence. +**Attempt** — One concrete execution inside a repeat run. Attempts keep their own score, status, metrics, trace, transcript, logs, and artifacts so aggregate results never hide individual evidence. **Pass rate** — Assertion or expectation pass rate inside a grading result: passed assertions or expectations divided by total assertions or expectations. AgentV does not use `pass_rate` for repeat-attempt success frequency. diff --git a/README.md b/README.md index 1a435d096..e08710e75 100644 --- a/README.md +++ b/README.md @@ -164,18 +164,18 @@ The checked-in version of this quickstart lives in [`examples/features/readme-qu agentv eval evals/my-eval.eval.yaml ``` -**6. Compare two runs** (pass two `index.jsonl` manifests — e.g. before and after a change): +**6. Compare two runs** (pass two run indexes — e.g. before and after a change): ```bash -agentv results compare .agentv/results//index.jsonl .agentv/results//index.jsonl +agentv results compare .agentv/results//.internal/index.jsonl .agentv/results//.internal/index.jsonl ``` ## Results -Each run writes a portable bundle directly under `.agentv/results//`. In this example, `tags.experiment: with-skills` names the condition being measured and `target: local-openai` selects the system under test from `.agentv/config.yaml`; both are recorded as metadata, not path segments. The root `index.jsonl` manifest is the portable row index used by scripts, CI, and `agentv results compare`; per-case sidecars include the resolved eval and target configuration used for the run. +Each run writes a portable bundle directly under `.agentv/results//`. In this example, `tags.experiment: with-skills` names the condition being measured and `target: local-openai` selects the system under test from `.agentv/config.yaml`; both are recorded as metadata, not path segments. The `.internal/index.jsonl` file is the portable row index used by scripts, CI, and `agentv results compare`; per-case sidecars include the resolved eval and target configuration used for the run. ```bash agentv eval evals/my-eval.eval.yaml -cat .agentv/results//index.jsonl +cat .agentv/results//.internal/index.jsonl ``` Run bundle layout: @@ -183,22 +183,22 @@ Run bundle layout: ``` .agentv/results/ ├── 2026-06-30T08-30-00-000Z/ # — one committed run bundle -│ ├── index.jsonl # row index for scripts/CI and `agentv results compare` │ ├── summary.json # run rollup: metadata, pass rate, counts, cost -│ └── fizzbuzz--a1b2c3d4/ # for one test/target row -│ ├── summary.json # optional per-case rollup across attempts -│ ├── test/ # generated test bundle: frozen inputs for reproducibility -│ │ ├── EVAL.yaml # resolved eval spec -│ │ ├── targets.yaml # resolved target config -│ │ └── graders/ # grader files used -│ └── attempt-1/ # one materialized attempt -│ ├── result.json # compact attempt manifest -│ ├── grading.json # assertion_results and grader evidence -│ ├── metrics.json # tool calls, transcript stats, behavior metrics -│ ├── timing.json # duration, token usage, cost -│ ├── transcript.json # normalized agent transcript -│ ├── transcript-raw.jsonl # raw agent output (debugging) -│ └── outputs/ # captured stdout and grader outputs +│ ├── fizzbuzz--a1b2c3d4/ # for one test/target row +│ │ ├── summary.json # optional per-case rollup across attempts +│ │ ├── test/ # generated test bundle: frozen inputs for reproducibility +│ │ │ ├── EVAL.yaml # resolved eval spec +│ │ │ ├── targets.yaml # resolved target config +│ │ │ └── graders/ # grader files used +│ │ └── sample-1/ # one materialized sample +│ │ ├── result.json # compact attempt manifest +│ │ ├── grading.json # assertion_results and grader evidence +│ │ ├── metrics.json # tool calls, transcript stats, behavior metrics +│ │ ├── transcript.json # normalized agent transcript +│ │ ├── transcript-raw.jsonl # raw agent output (debugging) +│ │ └── outputs/ # captured stdout and grader outputs +│ └── .internal/ +│ └── index.jsonl # row index for scripts/CI and `agentv results compare` ├── .indexes/ # reserved local/rebuildable indexes └── .cache/ # reserved local cache ``` diff --git a/docs/adr/0011-consolidate-case-conversion-boundary.md b/docs/adr/0011-consolidate-case-conversion-boundary.md index aa97162a1..85e71b034 100644 --- a/docs/adr/0011-consolidate-case-conversion-boundary.md +++ b/docs/adr/0011-consolidate-case-conversion-boundary.md @@ -37,7 +37,7 @@ Current conversion call sites include: - core result parsing and artifact writing in `packages/core/src/evaluation/run-artifacts.ts` - core grader and prompt-template stdin payloads in - `packages/core/src/evaluation/graders/code-grader.ts` and + `packages/core/src/evaluation/graders/script-grader.ts` and `packages/core/src/evaluation/graders/prompt-resolution.ts` - SDK stdin boundaries in `packages/sdk/src/runtime.ts`, `packages/sdk/src/assertion.ts`, and `packages/sdk/src/prompt-template.ts` diff --git a/docs/adr/0011-result-output-artifact-contract.md b/docs/adr/0011-result-output-artifact-contract.md index 7fbde0e81..7dd172b72 100644 --- a/docs/adr/0011-result-output-artifact-contract.md +++ b/docs/adr/0011-result-output-artifact-contract.md @@ -20,6 +20,12 @@ Updated by [ADR 0012](0012-finalize-run-artifact-layout.md), which makes artifact-format v2 runs direct children of `.agentv/results/` and treats `experiment` as run metadata rather than path identity. +Refined and superseded for current output layout by +[ADR 0017](0017-output-artifact-and-workspace-resolver-contract.md). Keep this +ADR as historical context; current run bundles use `.internal/index.jsonl`, +`sample-N/`, `metrics.json`, and `grading.json.assertion_results`, not root +`index.jsonl`, `run-N/`, `timing_path`, or `timing.json` as the active contract. + ## Context AgentV needs a result output contract that works for local runs, CI gates, diff --git a/docs/adr/0012-finalize-run-artifact-layout.md b/docs/adr/0012-finalize-run-artifact-layout.md index 82dd7ef49..69d110bfd 100644 --- a/docs/adr/0012-finalize-run-artifact-layout.md +++ b/docs/adr/0012-finalize-run-artifact-layout.md @@ -11,6 +11,12 @@ Supersedes the experiment-parent result layout portions of [ADR 0006](0006-separate-experiments-from-eval-definitions.md), and [ADR 0011](0011-result-output-artifact-contract.md). +Refined and superseded for current output layout by +[ADR 0017](0017-output-artifact-and-workspace-resolver-contract.md). Keep this +ADR as historical context for the results-root move; current run bundles place +the per-run index at `.internal/index.jsonl`, store repeated executions under +`sample-N/`, and merge timing data into `metrics.json`. + ## Context AgentV run bundles are the portable source of truth for Dashboard, reports, diff --git a/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md b/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md index 2d1fa4e13..786cb6fbc 100644 --- a/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md +++ b/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md @@ -11,6 +11,14 @@ and [ADR 0013 (experiment as tags.experiment)](0013-experiment-is-metadata-expre multi-turn is carved out to [ADR 0015](0015-multi-turn-conversation-execution-vs-evaluation.md); the output/artifact contract to [ADR 0017](0017-output-artifact-and-workspace-resolver-contract.md). +Status note (2026-07-04): implementation settled the grader vocabulary after +this ADR was accepted. Current authored executable graders use `type: script`. +`llm-rubric` is the promptfoo-compatible free-form rubric judge. Structured and +multi-criteria rubric judging uses `g-eval` where itemized rubric semantics are +needed. `grading.json` exposes `assertion_results` plus `score`, `verdict`, and +`evidence`; do not teach `grading.json.assertions[]` as the current artifact +contract. + ## Context AgentV's eval-authoring surface diverged from industry primitives. We are re-basing @@ -30,24 +38,23 @@ keep AgentV's only where its semantics are genuinely better.** Promptfoo type names adopted (`contains`/`equals`/`regex`/`is-json`/`icontains`/ `contains-all|any`/`starts-with`/`similar`/`latency`/`cost`/`webhook`/`javascript`/ `python`/`assert-set`). `composite` removed → `assert-set`. -2. **LLM judge vocabulary follows semantics.** `llm-rubric` is the criteria/rubric - scoring type. Bare-string `assert` entries desugar to grouped `llm-rubric` - assertions as an AgentV superset extension. Structured AgentV rubric criteria - are authored directly in promptfoo's permissive `llm-rubric.value` field; old - `rubric`/`rubrics` assertion type names are removed. `llm-rubric` also remains - the promptfoo-compatible free-form rubric judge. Agentic evidence-gathering - judges stay an AgentV extension rather than being forced into `llm-rubric`. +2. **LLM judge vocabulary follows semantics.** `llm-rubric` is the + promptfoo-compatible free-form rubric judge. Bare-string `assert` entries and + structured AgentV rubric criteria desugar to grouped `g-eval` assertions when + AgentV needs itemized, multi-criteria rubric semantics. Old `rubric`/`rubrics` + assertion type names are removed. Agentic evidence-gathering judges stay an + AgentV extension rather than being forced into `llm-rubric`. Structured AgentV rubric criteria are preserved, not flattened into a single text blob: criteria objects keep `weight`, `operator`, `required`, `score_ranges`, and `min_score`. Artifact assertion rows are the generic - AgentV grader contract, not a `llm-rubric` special case: each grader returns - `assertions[]`, the orchestrator flattens those rows into - `grading.json.assertions[]`, and `grading.json.graders[].assertions[]` keeps - the per-grader breakdown. Deterministic graders usually emit one row, while - multi-aspect graders emit one row per authored check or result unit. Structured - `llm-rubric` criteria therefore populate one assertion row per criterion so the - Dashboard can show criterion-level evidence, using the same mechanism as code - graders, field accuracy, execution metrics, and tool trajectory. + AgentV grader contract: `grading.json.assertion_results[]` holds flattened + assertion evidence, and nested grader entries keep their own + `assertion_results[]` breakdown with score, verdict, and evidence. + Deterministic graders usually emit one row, while multi-aspect graders emit + one row per authored check or result unit. Structured rubric criteria + therefore populate one assertion row per criterion so the Dashboard can show + criterion-level evidence, using the same mechanism as script graders, field + accuracy, execution metrics, and tool trajectory. 3. **Grader execution**: `javascript` in-process (Bun `import`), `python` subprocess, `script` = the subprocess power tool (workspace-`cwd`, arbitrary language). `javascript` is NOT desugared to `script`. @@ -97,8 +104,9 @@ keep AgentV's only where its semantics are genuinely better.** (`beforeAll`/`afterAll`/`beforeEach`/`afterEach`), running *after* materialization — e.g. `agentv:agent-rules` (stage skills/hooks/agents) and custom `file://` hooks. Removed: `on_run_complete`, `preprocessors` (→ `extensions`). -11. **Scope**: `similar` ships with a configured embeddings provider, and `llm-rubric` ships - as the structured criteria/rubric judge. Exotic promptfoo assertions +11. **Scope**: `similar` ships with a configured embeddings provider, `llm-rubric` ships + as the free-form rubric judge, and `g-eval` covers structured or multi-criteria + rubric judging. Exotic promptfoo assertions (`context-*`/`moderation`/…) and `redteam` are **future scope** — treated as unrecognized fields, not stubbed. Superset holds over the *implemented* surface. diff --git a/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md b/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md index 3ee90effa..2d7389229 100644 --- a/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md +++ b/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md @@ -26,7 +26,7 @@ bundle, and how a workspace is acquired. 2. **Queryable aggregate ← margin-lab**: run-root `summary.json` is a rich `jq`-queryable `Summary` (run_id, status breakdown, per-case **pass@k**, per-instance summaries, usage, infra-failure taxonomy) — widen AgentV's current thin summary to this. Plus - `index.jsonl` (one row per case) for streaming/line queries. No database. + `.internal/index.jsonl` (one row per case) for streaming/line queries. No database. 3. **Transcript + metrics ← vercel**: two-layer transcript (raw + normalized) with a canonical cross-agent `tool_name` enum and precomputed `transcript_summary`, the summary **inlined into each result row** for cheap trajectory/metrics assertions; @@ -39,8 +39,9 @@ bundle, and how a workspace is acquired. while multi-aspect graders return one row per distinct criterion/aspect. The artifact preserves both the flattened rows and each grader's nested rows. Default judge = skeptical evidence-by-path (opt-out via - explicit `prompt`); judge pinning via `grader_target`. Evidence stays in - `grading.json`. + explicit `prompt`); grader target selection flows through the config graph + (`defaults.grader`) or assertion-level target selection, not a system-under-test + target field. Evidence stays in `grading.json`. 5. **Bundle layout / naming**: machine files move under per-run **`.internal/`** (`index.jsonl`, `progress.json`, `events.jsonl`, `bundle.json`); run root stays clean (`summary.json` + per-case dirs). Rename the reference field `manifest_path` → diff --git a/plugins/agentv-self/skills/image-compress-and-docs/SKILL.md b/plugins/agentv-self/skills/image-compress-and-docs/SKILL.md index e70fe79ff..b21a8372e 100644 --- a/plugins/agentv-self/skills/image-compress-and-docs/SKILL.md +++ b/plugins/agentv-self/skills/image-compress-and-docs/SKILL.md @@ -46,11 +46,11 @@ agent-browser --session docs-shots close kill $(lsof -ti:14800) 2>/dev/null ``` -**Screenshots with realistic data:** Dashboard screenshots must have populated data — multiple runs with varying pass rates and real targets. If results are sparse, create synthetic JSONL files in `.agentv/results///index.jsonl` with realistic fields before launching Dashboard. +**Screenshots with realistic data:** Dashboard screenshots must have populated data — multiple runs with varying pass rates and real targets. If results are sparse, create synthetic run bundles under `.agentv/results//` with `summary.json` plus `.internal/index.jsonl` rows before launching Dashboard. Synthetic JSONL record format: ```json -{"test_id": "my-test", "score": 0.95, "target": "claude-sonnet", "experiment": "default", "timestamp": "2026-04-08T09:15:44.003Z", "execution_status": "success", "suite": "my-suite", "category": "default", "duration_ms": 3500, "token_usage": {"input_tokens": 1200, "output_tokens": 400}, "scores": [{"type": "llm-grader", "score": 0.95, "passed": true}], "error": null} +{"test_id": "my-test", "score": 0.95, "target": "claude-sonnet", "experiment": "default", "timestamp": "2026-04-08T09:15:44.003Z", "execution_status": "success", "suite": "my-suite", "duration_ms": 3500, "token_usage": {"input_tokens": 1200, "output_tokens": 400}, "scores": [{"type": "llm-rubric", "score": 0.95, "verdict": "pass"}], "grading_path": "my-test--demo/sample-1/grading.json", "metrics_path": "my-test--demo/sample-1/metrics.json", "error": null} ``` ## Step 2 — Optimize