From 079acafd7fe0ea707af429fb37bd8769fe22d7c7 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Sat, 4 Jul 2026 08:38:41 +0200 Subject: [PATCH] docs: add Promptfoo parity matrix --- .../docs/docs/next/evaluation/eval-files.mdx | 6 + apps/web/src/content/docs/docs/next/index.mdx | 1 + .../docs/next/reference/promptfoo-parity.mdx | 123 ++++++++++++++++++ examples/README.md | 6 + skills-data/agentv-eval-writer/SKILL.md | 1 + 5 files changed, 137 insertions(+) create mode 100644 apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx diff --git a/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx b/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx index f9a897415..be899d2fa 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx @@ -25,6 +25,12 @@ Eval files describe the task, target binding, and run controls. Use override concurrency with `agentv eval --workers N`; do not author legacy `workers` fields in eval YAML. +For Promptfoo-style authoring, AgentV uses the same broad prompt, test, vars, +default test, evaluate-options, and assertion contract with `snake_case` wire +fields. See the [Promptfoo parity matrix](/docs/reference/promptfoo-parity/) for +the exact alignments, intentional AgentV extensions, and future-scope Promptfoo +surfaces. + ## Authoring Shapes Eval YAML is AgentV's composable and runnable authoring primitive. It is a diff --git a/apps/web/src/content/docs/docs/next/index.mdx b/apps/web/src/content/docs/docs/next/index.mdx index a05e988e5..e7e69c59a 100644 --- a/apps/web/src/content/docs/docs/next/index.mdx +++ b/apps/web/src/content/docs/docs/next/index.mdx @@ -50,6 +50,7 @@ Use this topic map when you are an AI agent trying to decide which primitive or | Goal | Start here | Why | | --- | --- | --- | | Create a first eval | [Quickstart](/docs/getting-started/quickstart/) → [Eval files](/docs/evaluation/eval-files/) | Defines the smallest runnable YAML shape before adding advanced fields. | +| Translate Promptfoo-style evals | [Promptfoo parity matrix](/docs/reference/promptfoo-parity/) → [Eval files](/docs/evaluation/eval-files/) | Shows which prompt, test, vars, target, assertion, and runtime-option fields align directly and which AgentV fields are intentional extensions. | | Choose graders | [Rubrics](/docs/evaluation/rubrics/) → [Script graders](/docs/graders/script-graders/) → [LLM graders](/docs/graders/llm-rubrics/) | Keeps deterministic checks, rubric scoring, and LLM judgment separate. | | Evaluate tool use or agents | [Tool trajectory](/docs/graders/tool-trajectory/) → [Coding agents](/docs/targets/coding-agents/) → [CLI provider](/docs/targets/cli-provider/) | Shows how targets, transcripts, and tool-call assertions compose. | | Share and inspect results | [Result artifact contract](/docs/reference/result-artifacts/) → [Results](/docs/tools/results/) → [Dashboard](/docs/tools/dashboard/) | Explains canonical run bundles, local artifacts, reports, remote result repositories, and Dashboard review flows. | diff --git a/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx b/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx new file mode 100644 index 000000000..c395d8822 --- /dev/null +++ b/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx @@ -0,0 +1,123 @@ +--- +title: Promptfoo Parity Matrix +description: Promptfoo-aligned AgentV eval authoring fields, intentional divergences, and future-scope surfaces. +sidebar: + order: 2 +slug: docs/reference/promptfoo-parity +--- + +AgentV uses a similar eval config contract to Promptfoo for ordinary authored +evals: prompt matrices, test rows, vars, default test data, assertions, and +target matrices all use the same broad shape. AgentV keeps the wire format +`snake_case`, keeps target identity separate from provider/backend selection, +and adds repo-native workspace and artifact fields for agent evaluation. + +Use this matrix when translating a Promptfoo-style normal eval into AgentV YAML. +It documents which surfaces align directly, which AgentV surfaces are cleaner +greenfield extensions, and which Promptfoo surfaces are deferred until AgentV +implements equivalent semantics directly. + +## Decision Terms + +| Decision | Meaning | +| --- | --- | +| Align with Promptfoo | AgentV accepts the same concept, with `snake_case` where the field crosses the YAML boundary. | +| Keep AgentV divergence | AgentV intentionally uses a different shape because it is clearer for repo-native agent evals. | +| Keep AgentV extension | AgentV adds a capability that does not try to be Promptfoo-compatible. | +| Defer/future-scope | AgentV does not accept the Promptfoo surface yet. Use an AgentV primitive or wait for direct implementation. | + +## Authored Config Matrix + +| Surface | Promptfoo shape | AgentV shape | Decision | Notes | +| --- | --- | --- | --- | --- | +| Prompt matrix | Top-level `prompts` rendered with each test's `vars`. | Top-level `prompts` rendered with `tests[].vars` and `default_test.vars`. | Align with Promptfoo | This is the canonical Promptfoo-compatible input shape in AgentV. Prompt entries can be inline strings, chat arrays, files, or generated prompt functions. | +| Test rows | `tests` can be inline rows or a case-file reference; rows carry `vars`, `assert`, metadata, prompt/provider filters, and expected data. | `tests` can be inline rows or a raw-case path; rows carry `vars`, `assert`, `expected_output`, metadata, workspace overrides, and run overrides. | Align with Promptfoo | AgentV also supports `imports.suites` and `imports.tests` for explicit composition. Raw cases do not own suite context. | +| Variables | `tests[].vars` plus `defaultTest.vars`; prompt templates can reference top-level var names. | `tests[].vars` plus `default_test.vars`; templates can use `{{ name }}` or `{{ vars.name }}`. | Align with Promptfoo | Per-test vars override default vars by key. | +| Default test | `defaultTest`, inline object or `file://` reference. | `default_test`, inline object or `file://` / `ref://` reference. | Align with Promptfoo | AgentV uses `snake_case` for YAML. Shared prompt matrix defaults belong in `default_test.vars`. | +| Evaluate options | `evaluateOptions` for runtime controls. | `evaluate_options` for runtime controls. | Align with Promptfoo | AgentV uses `evaluate_options.repeat`, `evaluate_options.budget_usd`, and `evaluate_options.max_concurrency`. | +| Authored concurrency | Common Promptfoo usage includes runtime options such as `maxConcurrency`. | `evaluate_options.max_concurrency`. | Keep AgentV divergence | Do not author `execution.max_concurrency` or top-level `workers` in eval YAML. CLI `--workers` remains an operator override. | +| Target selection | Promptfoo normal evals use `providers`; `targets` can alias providers in unified config. | Use top-level `target` for one system under test or top-level `targets` for a target matrix. | Keep AgentV divergence | AgentV reserves `provider` for the backend/adapter kind inside a target object. Top-level `providers` is rejected to avoid overloading that term. | +| Target object identity | Provider options often use `id` for backend/provider spec and optional `label` for display or matching. | Target objects use stable `id` for target identity, `provider` for backend kind, optional `runtime`, and `config` for provider settings. | Keep AgentV divergence | AgentV does not copy Promptfoo's `label`/`id` baggage because `provider` already names the backend boundary. | +| Direct input suites | Promptfoo prompt authoring normally goes through `prompts` plus vars. | `tests[].input`, top-level `input`, `tests[].input_files`, and top-level `input_files` are direct-input conveniences. | Keep AgentV extension | Do not mix these fields with top-level `prompts`. Use `tests[].vars` with `prompts`, or remove `prompts` for a direct-input AgentV suite. | +| Suite assertions | `assert` entries can be strings or typed assertion objects. | `assert` entries can be strings, typed assertion objects, script graders, or AgentV extension graders. | Align with Promptfoo | Plain strings become semantic rubric checks. Use `assert`, not `assertions`, in current authored eval YAML. | +| Assertion grouping | `type: assert-set` with child `assert` entries. | `type: assert-set` with child `assert`, weights, and parent threshold. | Align with Promptfoo | `type: composite` is rejected; use `assert-set`. | +| Deterministic assertion vocabulary | Common Promptfoo types include `contains`, `icontains`, `contains-any`, `contains-all`, `starts-with`, `regex`, `is-json`, `equals`, `latency`, `cost`, `javascript`, `python`, `webhook`, `similar`, and `llm-rubric`. | AgentV accepts the implemented overlap, including `contains`, `icontains`, `contains-any`, `contains-all`, `starts-with`, `regex`, `is-json`, `equals`, `latency`, `cost`, `javascript`, `python`, `webhook`, `similar`, and `llm-rubric`. | Align with Promptfoo | Unsupported Promptfoo assertion names error instead of silently becoming custom assertion names. | +| Custom assertion terminology | Promptfoo calls normal eval custom logic assertions, with fixed code assertion types such as `javascript`, `python`, `ruby`, and `webhook`. | `defineAssertion()` files in `.agentv/assertions/` become reusable assertion type names. | Keep AgentV extension | AgentV keeps assertion terminology and extends discovery to arbitrary assertion type names such as `has-citation`. | +| Script/custom grader terminology | Promptfoo custom code assertions are still assertion types. | `defineScriptGrader()` powers command-backed graders referenced with `type: script` and `command:`. | Keep AgentV divergence | Use script grader wording only for command-backed or LLM-backed scoring components that need explicit score and assertion-result control. | +| Tool and trace assertions | Promptfoo includes `trajectory:tool-used`, `trajectory:tool-sequence`, `trajectory:tool-args-match`, `trajectory:step-count`, `trajectory:goal-success`, `tool-call-f1`, `skill-used`, `trace-span-count`, `trace-span-duration`, and `trace-error-spans`. | AgentV rejects those names until their semantics are implemented directly. | Defer/future-scope | These names are not aliases for AgentV's `tool-trajectory` grader. | +| Tool trajectory grader | No direct Promptfoo alias for AgentV-normalized transcript semantics. | `type: tool-trajectory`. | Keep AgentV extension | This is AgentV-specific and operates over AgentV-normalized transcripts and trace summaries. | +| Repo-native workspace fields | Promptfoo normal evals do not own AgentV workspace materialization. | `workspace`, `workspace.repos`, `workspace.scope`, `workspace.docker`, `extensions`, and per-test `workspace`. | Keep AgentV extension | AgentV evaluates real repositories and agent workspaces, so workspace provenance is first-class authored config. | +| Run artifacts and inspection | Promptfoo owns its own result viewer and output formats. | AgentV writes `.agentv/results//` bundles with `summary.json`, `.internal/index.jsonl`, sidecars, and local Dashboard support. | Keep AgentV extension | AgentV-owned bundles are the source of truth for compare, Dashboard, CI, and adapters. Phoenix is link-out correlation only through safe external trace metadata. | +| Compare command | Promptfoo has its own result comparison surfaces. | `agentv results compare `. | Keep AgentV extension | Compare consumes completed AgentV run indexes such as `.agentv/results//.internal/index.jsonl`. | +| CLI runtime filters | Promptfoo exposes filters such as prompt/provider/test subset flags. | AgentV supports its current CLI filters and selection fields; full Promptfoo runtime-filter parity is future work. | Defer/future-scope | Prefer authored `select`/`imports` or current AgentV CLI flags until runtime-filter parity lands. | +| Wire-format casing | Promptfoo config uses camelCase fields such as `defaultTest` and `evaluateOptions`. | AgentV YAML, JSONL, artifacts, and CLI JSON use `snake_case`; internal TypeScript uses `camelCase`. | Keep AgentV divergence | Translate only at process boundaries. New public wire fields should be `snake_case`. | +| Hard-rejected stale AgentV fields | Not applicable to Promptfoo. | Removed AgentV-era fields such as top-level `execution`, `execution.target`, `execution.targets`, top-level `budget_usd`, top-level `repeat`/`runs`, and `composite` are rejected. | Keep AgentV divergence | Use top-level `target`/`targets`, `evaluate_options`, `evaluate_options.repeat`, and `assert-set`. Migration guidance lives in the eval migration skill reference. | + +## Canonical Prompt-Compatible Shape + +```yaml +description: Release-note summarization +target: local-mini + +prompts: + - id: direct + label: Direct + prompt: "Summarize {{ topic }} for {{ audience }}." + +default_test: + vars: + audience: engineers + +evaluate_options: + max_concurrency: 2 + +tests: + - id: release-notes + vars: + topic: the July release notes + expected_output: concise release-note summary + assert: + - Identifies the most important change + - type: assert-set + metric: release_gate + threshold: 0.8 + assert: + - type: contains + value: July + - type: llm-rubric + value: The answer is concise and accurate. +``` + +## Canonical AgentV Extension Shape + +```yaml +description: Repo-native direct task suite +target: + id: codex-local + provider: codex-app-server + runtime: host + config: + command: ["codex", "app-server"] + +workspace: + repos: + - path: ./app + repo: acme/support-app + commit: main + scope: attempt + +input_files: + - ./instructions.md + +tests: + - id: refund-policy + input: Update the refund policy handler. + expected_output: The handler supports the damaged-item exception. + assert: + - type: tool-trajectory + mode: any_order + minimums: + shell: 1 + - type: script + command: [bun, run, graders/check-refund-policy.ts] +``` diff --git a/examples/README.md b/examples/README.md index 88600a13b..640c4faa8 100644 --- a/examples/README.md +++ b/examples/README.md @@ -2,6 +2,12 @@ This directory contains working examples demonstrating AgentV's evaluation capabilities. +For the authored eval contract behind these examples, start with the +[Eval files](https://agentv.dev/docs/evaluation/eval-files/) reference. The +[Promptfoo parity matrix](https://agentv.dev/docs/reference/promptfoo-parity/) +calls out which fields align with Promptfoo-style evals and which AgentV fields +are repo-native extensions. + ## Setup Examples are self-contained packages with their own dependencies. Before running any example, install dependencies from the repository root: diff --git a/skills-data/agentv-eval-writer/SKILL.md b/skills-data/agentv-eval-writer/SKILL.md index 7ec4b13bb..2cf1de883 100644 --- a/skills-data/agentv-eval-writer/SKILL.md +++ b/skills-data/agentv-eval-writer/SKILL.md @@ -13,6 +13,7 @@ description: >- # AgentV Eval Writer Comprehensive docs: https://agentv.dev +Promptfoo parity matrix: https://agentv.dev/docs/reference/promptfoo-parity/ ## Authoring Principle