diff --git a/.agents/product-boundary.md b/.agents/product-boundary.md index 5aa4079ce..fdb9a0ab1 100644 --- a/.agents/product-boundary.md +++ b/.agents/product-boundary.md @@ -39,15 +39,16 @@ AgentV transcript artifacts are not Phoenix-native conversation inputs. Model-ca ## Promptfoo-Compatible Authoring Boundary -AgentV should adopt Promptfoo-compatible eval matrix authoring where it strengthens repo-native evaluation, but Promptfoo is reference evidence rather than schema authority. The core mental model is `prompts x tests/vars x targets`, with repeat samples and retries applied as run policy after the authored matrix is resolved. +AgentV adopts Promptfoo-compatible eval matrix authoring by default where Promptfoo has matching semantics. The core mental model is `prompts x tests/vars x providers`, with repeat samples and retries applied as run policy after the authored matrix is resolved. Keep these AgentV-native boundaries explicit: -- `targets` are systems under test. A target `id` is stable AgentV identity; `provider` inside the target names the backend or adapter kind. -- Top-level Promptfoo `providers` is not the canonical AgentV authoring key. +- `providers` are systems under test. AgentV accepts Promptfoo-shaped provider declarations: strings like `openai:gpt-4.1-mini`, complete package provider strings like `package:@agentv/promptfoo-providers:CodexCliProvider`, provider option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, and provider maps like `{ "openai:gpt-4": { label, config } }`. In AgentV, `id` names the backend/spec and `label` is the stable AgentV identity used for selection and result grouping. +- AgentV-only provider ids such as `agentv:codex-cli` are first-class AgentV authoring sugar, but they are not directly Promptfoo-runnable. `agentv export promptfoo` must lower supported built-ins to Promptfoo-readable `file://...:callApi` or complete `package:...:Export` provider entries. - Coding-agent testbeds use `environment` recipes for host/Docker substrate, setup, fixtures, services, and cwd. Do not make Promptfoo lifecycle `extensions` or public `workspace` authoring the canonical testbed contract. +- Promptfoo export may lower a filesystem/host subset of `environment` into generated `extensions` plus provider workdir configuration. Docker environments are not part of the initial export subset and must fail with explicit diagnostics rather than silently losing isolation. - Top-level `env` means provider/eval environment variables. `extensions` remain lifecycle hooks. -- Reusable prompts, tests, defaults, and environments use field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, and `environment: file://...`. +- Reusable prompts, tests, defaults, and environments use field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, and `environment: file://...`. Export may rewrite supported AgentV refs into Promptfoo-readable generated files. - Grouping and Dashboard navigation use tags and run-bundle metadata, not experiment path buckets, Vercel path layout, or model-as-experiment grouping. - AgentV run bundles, traces, transcripts, datasets, indexes, and Git-backed artifacts stay AgentV-owned. Do not design an Opik export path or Phoenix projection path for those artifacts. @@ -63,7 +64,7 @@ Prefer these extension points before adding a built-in: - plain assertion strings for simple semantic rubric checks - `llm-rubric` for promptfoo-compatible free-form rubric checks - `g-eval` for structured or multi-criteria rubric judging -- config-level grader targets selected through `defaults.grader` or assertion-level target selection, not target-level grader configuration +- config-level grader providers selected through `defaults.grader` or assertion-level provider selection, not system-under-test provider configuration - CLI wrappers that consume AgentV JSON or JSONL output for post-processing such as aggregation, comparison, or reporting Ask: can this be achieved with existing primitives plus a plugin or wrapper? If yes, it should not be a built-in. That includes niche config overrides for existing graders. @@ -107,7 +108,7 @@ Research those references from local cloned repositories first when a clone is a Treat these as reference inputs, not dependencies. AgentV should adopt the shared lowest common denominator when it fits the repo-native artifact model, and document any intentional divergence in the relevant plan, ADR, or contract docs. -Do not copy another framework's schema baggage just because the framework is credible. When a peer contract carries historical constraints, overloaded field names, or compatibility aliases, prefer a cleaner AgentV contract if it preserves the core user need. Document the reason for diverging so future workers do not "realign" it back to the peer shape. For target/provider contracts, keep identity and backend/control boundary separate: use a stable AgentV `id` for the target registry key when `provider` already names the adapter/backend kind. Promptfoo's `label` is useful evidence but should not be copied as target identity merely because Promptfoo uses `id` for provider/backend specs. +Do not copy another framework's schema baggage just because the framework is credible. When a peer contract carries historical constraints, overloaded field names, or compatibility aliases, prefer a cleaner AgentV contract if it preserves the core user need. Document the reason for diverging so future workers do not "realign" it back to the peer shape. For the provider surface, ADR 0019 is the current exception: AgentV follows Promptfoo's `providers`/`id`/`label` shape so the uncommon AgentV differences stay concentrated in `environment`, refs, built-in AgentV providers, artifacts, and Dashboard behavior. ### 5. YAGNI - You Aren't Gonna Need It diff --git a/AGENTS.md b/AGENTS.md index 6a9844970..eb92e72ae 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,10 +14,11 @@ AgentV aims to be the repo-native, workspace-native evaluation framework for AI Eval authoring mental model: -- AgentV adopts Promptfoo-compatible matrix authoring where it helps: an eval expands `prompts x tests/vars x targets`, then applies repeat samples/retries as run policy. This is compatibility by composition, not a wholesale copy of Promptfoo's schema. -- Author systems under test as `targets`. A target `id` is the stable AgentV identity; `provider` inside the target names the backend or adapter kind. Top-level Promptfoo `providers` is reference evidence, not the canonical AgentV authoring key. -- Use `environment` recipes for coding-agent testbeds, including host/Docker setup, repo materialization, fixtures, services, and cwd. Do not use Promptfoo `extensions` or public `workspace` authoring as the canonical testbed contract. -- Use top-level `env` for provider/eval environment variables. Use `extensions` for lifecycle hooks. Use field-local `file://` refs for reusable prompts, tests, defaults, and environments. +- AgentV adopts Promptfoo-compatible matrix authoring by default: an eval expands `prompts x tests/vars x providers`, then applies repeat samples/retries as run policy. +- Author systems under test as `providers`. A provider entry `id` names the backend/spec string; `label` is the stable AgentV selection and result identity. `id` may contain Promptfoo-style colons such as `openai:responses:gpt-5.4`. +- The intended authored-config differences from Promptfoo are limited to `environment`, AgentV refs, and built-in AgentV providers such as `agentv:codex-cli`. Full Promptfoo compatibility for those AgentV-native pieces is by `agentv export promptfoo`. +- Use `environment` recipes for coding-agent testbeds, including host/Docker setup, repo materialization, fixtures, services, and cwd. Promptfoo export may lower a host/filesystem subset into generated `extensions` plus provider workdir configuration; Docker environments must remain unsupported on export until a faithful runner boundary exists. Do not use Promptfoo `extensions` or public `workspace` authoring as the canonical AgentV testbed contract. +- Use top-level `env` for provider/eval environment variables. Use `extensions` for lifecycle hooks. Use field-local `file://` refs for reusable prompts, tests, defaults, and environments; export may rewrite supported AgentV refs into Promptfoo-readable files. - Use `tags` and run-bundle metadata for grouping and Dashboard navigation. Do not use experiment path buckets, Vercel path layout, or model-as-experiment grouping as canonical AgentV semantics. - AgentV run bundles, traces, transcripts, datasets, indexes, and Git-backed artifacts stay AgentV-owned. Do not design an Opik export path or Phoenix projection path for them; Phoenix correlation is link-out only when `external_trace` metadata already exists. @@ -39,20 +40,21 @@ tests: issue: "Explain the flaky retry behavior" repo: "file://fixtures/retry" -targets: - - id: codex-host - provider: codex-cli +providers: + - id: agentv:codex-cli + label: codex-host runtime: host - - id: claude-docker - provider: claude-cli - runtime: docker + config: + command: codex + - id: anthropic:claude-agent-sdk + label: claude environment: file://.agentv/environments/local-repo.yaml tags: experiment: prompt-compare ``` -This produces eight authored target-case executions before repeat policy: 2 prompts x 2 `tests[].vars` cases x 2 targets. +This produces eight authored provider-case executions before repeat policy: 2 prompts x 2 `tests[].vars` cases x 2 providers. Phoenix boundary after the 2026-06-20 product decision: @@ -68,7 +70,7 @@ Design guardrails: - Document composition patterns before inventing a new feature. - Match industry-standard lowest-common-denominator contracts when possible. - When designing AgentV contracts, check public reference standards such as Claude Skills, Vercel agent-eval, Hugging Face Datasets, and OpenInference before inventing AgentV-specific shapes. Use their shared lowest common denominator where it fits, and document any intentional divergence. -- Treat peer frameworks as evidence, not schema authority. Do not inherit baggage such as overloaded field names, compatibility aliases, or framework-specific historical constraints when AgentV can express a cleaner repo-native contract. Example: prefer `id` for stable AgentV target identity when `provider` already names the backend/control boundary, even if Promptfoo uses `label` because its `id` field is overloaded as a provider spec. +- Treat peer frameworks as evidence, not schema authority. Do not inherit baggage such as overloaded field names, compatibility aliases, or framework-specific historical constraints when AgentV can express a cleaner repo-native contract. For the provider surface, AgentV intentionally follows Promptfoo's `providers`/`id`/`label` shape so the uncommon AgentV differences stay concentrated in `environment`, refs, built-in AgentV providers, artifacts, and Dashboard behavior. - For peer-framework research, use local cloned repositories and DeepWiki MCP before broad web search. In this operator workspace, Promptfoo is cloned at `/home/entity/projects/promptfoo/promptfoo` and DeepEval is cloned at `/home/entity/projects/confident-ai/deepeval`; use DeepWiki repos `promptfoo/promptfoo` and `confident-ai/deepeval` for architecture-level orientation, then verify exact claims with `rg` and `git` in the local clone. If a public contract must be checked for currentness, use official docs and record the source URL or clone commit behind the conclusion. - Apply YAGNI aggressively and solve the current request with the smallest surface that works. - Keep extensions non-breaking unless a same-week unreleased surface should be hard-corrected. diff --git a/CONCEPTS.md b/CONCEPTS.md index a6dc6a30d..86be7a189 100644 --- a/CONCEPTS.md +++ b/CONCEPTS.md @@ -4,27 +4,27 @@ Shared domain vocabulary for this project — entities, named processes, and sta ## Providers and Targets -**Provider** — an adapter plugin that connects AgentV's evaluation engine to a specific AI system (e.g., copilot CLI, copilot SDK, Claude API, pi). Each provider implements the request/response contract: given a test case, invoke the AI system and return its output. Providers are selected inside `targets[]` in eval YAML and can be extended via the provider registry. Top-level Promptfoo `providers` is useful reference evidence, but it is not the canonical AgentV authoring key. +**Provider** — The eval YAML or config declaration for a system under test. AgentV follows Promptfoo's provider declaration layer: entries may be strings such as `openai:responses:gpt-5.4`, complete package provider strings such as `package:@agentv/promptfoo-providers:CodexCliProvider`, provider option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, or provider maps such as `{ "openai:gpt-4": { label, config } }`. In AgentV, `id` names the backend/spec and may contain colons; `label` is the stable AgentV selection and result identity. Built-in AgentV ids such as `agentv:codex-cli` are AgentV-native sugar and require Promptfoo export before Promptfoo can run them. -**Target** — The eval YAML or config declaration for a system under test. A target has a stable AgentV `id`, a `provider` backend or adapter kind, an optional `runtime`, and provider settings under `config`; field-level `file://` references can load prompts, defaults, or other config fragments at the boundary. A single eval file can declare multiple targets to compare AI systems side by side. Targets select agents/providers; they do not own the authored host/Docker testbed recipe. +**Target** — Historical/public-authoring term for a system under test and still an internal/result-artifact vocabulary where existing run bundles, Dashboard facets, and sidecars use `target`. New authored eval YAML uses `providers`. Do not reintroduce public `targets` authoring without a new ADR; keep artifact renames separate from the provider-surface migration. -**Target runtime** — The placement/transport mode for invoking a target provider, such as host execution, sandbox/container placement, CLI subprocess, app-server protocol, RPC, or SDK child runner. Runtime describes how the selected agent is invoked. Advanced home/env/profile-style overlays are provider or runtime configuration details, not the authored testbed recipe. Runtime is separate from the environment that prepares files, services, and cwd. +**Provider runtime** — The placement/transport mode for invoking a provider, such as host execution, sandbox/container placement, CLI subprocess, app-server protocol, RPC, or SDK child runner. Runtime describes how the selected agent is invoked. Advanced home/env/profile-style overlays are provider or runtime configuration details, not the authored testbed recipe. Runtime is separate from the environment that prepares files, services, and cwd. -**Grader (selection)** — Not a distinct entity type. A grader is a regular target, listed under `targets` like any other and selected for a grading role via `defaults.grader`, CLI `--grader-target`, or an assertion-level target override. Authoring a separate `graders:` list is a hard error: move each entry into `targets`. This also means a grader can itself be the target of an ordinary eval (oracle/calibration checks), which a separate schema would block. Do not confuse this with **grader (method)** — an assertion `type` such as `llm-rubric`, `code-grader`, or `g-eval` that names *how* scoring works, independent of *which* target performs it; or with a rubric's `criteria`/`value`, which is the grading *prompt*. AgentV once called grader-selection "judge" (`judge_target`) before a deliberate, full rename to "grader" — do not reintroduce "judge" without a stronger reason than surface-level clarity, since peer frameworks (Margin-Lab/evals, Harbor) don't treat judge/grader as a first-class schema concept either. +**Grader (selection)** — Not a distinct entity type. A grader is a regular provider, listed under `providers` like any other and selected for a grading role via `defaults.grader`, CLI `--grader-provider`, test options, or an assertion-level provider override. Authoring a separate `graders:` list is a hard error: move each entry into `providers`. This also means a grader can itself be the provider under evaluation (oracle/calibration checks), which a separate schema would block. Do not confuse this with **grader (method)** — an assertion `type` such as `llm-rubric`, `script`, or `g-eval` that names *how* scoring works, independent of *which* provider performs it; or with a rubric's `criteria`/`value`, which is the grading *prompt*. AgentV once called grader-selection "judge" before a deliberate rename to "grader" — do not reintroduce "judge" without a stronger reason than surface-level clarity. **Provider runtime boundary** — The process boundary between AgentV's evaluation orchestrator and the agent runtime a provider invokes. CLI-backed providers place the agent runtime outside the orchestrator; SDK providers should run through an AgentV-owned child-runner boundary when runtime teardown can threaten run artifact finalization. This boundary does not own repository/testbed setup by default. ## Evaluation Model -**Eval / Eval YAML** — The composable and runnable AgentV authoring primitive. An eval YAML file describes the prompts, tests, variables, targets, assertions, environments, tags, and run policy for an evaluation. AgentV does not have a separate runnable `experiment.yaml` artifact. +**Eval / Eval YAML** — The composable and runnable AgentV authoring primitive. An eval YAML file describes the prompts, tests, variables, providers, assertions, environments, tags, and run policy for an evaluation. AgentV does not have a separate runnable `experiment.yaml` artifact. -**Matrix authoring** — The Promptfoo-compatible shape AgentV adopts where useful: `prompts x tests/vars x targets`, with repeat samples and retries applied as run policy after the authored matrix is resolved. AgentV uses this matrix model without copying Promptfoo wholesale. AgentV-native boundaries remain: `targets` identify systems under test, `provider` names the backend/adapter kind inside a target, `environment` recipes prepare coding-agent testbeds, `env` carries provider/eval variables, `extensions` are lifecycle hooks, reusable content uses field-local `file://` refs, and grouping uses tags plus run-bundle metadata. +**Matrix authoring** — The Promptfoo-compatible shape AgentV adopts by default: `prompts x tests/vars x providers`, with repeat samples and retries applied as run policy after the authored matrix is resolved. AgentV-native boundaries are intentionally narrow: `environment` recipes prepare coding-agent testbeds, AgentV refs may be rewritten during export, built-in AgentV provider ids such as `agentv:codex-cli` require export for Promptfoo execution, and grouping uses tags plus run-bundle metadata. **Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `environment`, assertions, expected references, and judge criteria. It runs directly or shares reusable parts through field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, `default_test: file://...`, and `environment: file://...`. **Raw case file** — YAML, JSONL, or directory case data loaded with `tests: file://./cases.yaml`, string shorthand, or another supported field-local tests reference. Raw cases are reusable data inputs; they do not carry suite context such as shared `environment`, shared `prompts`, or shared `assertions`. -**Policy eval** — Eval YAML whose main job is to bind top-level runtime policy such as target selection, repeat count, timeout, budget, thresholds, and tags around explicit prompts/tests/targets refs. Policy evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. Use tags and run-bundle metadata for grouping rather than experiment path buckets, Vercel path layout, model-as-experiment grouping, or wrapper-suite import semantics. +**Policy eval** — Eval YAML whose main job is to bind top-level runtime policy such as provider selection, repeat count, timeout, budget, thresholds, and tags around explicit prompts/tests/providers refs. Policy evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. Use tags and run-bundle metadata for grouping rather than experiment path buckets, Vercel path layout, model-as-experiment grouping, or wrapper-suite import semantics. **Multi-file run** — A CLI-selected set of eval YAML files. Use multiple eval files, tags, and CLI run selection to group suites under one run intent without adding wrapper-suite semantics to YAML. @@ -32,9 +32,9 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Tags** — A promptfoo-shaped `Record` map authored on an eval (or project config / `--tag key=value`) that labels a run with structured facets such as `experiment`, `team`, or `env`. The reserved `experiment` key feeds the experiment namespace. The resolved map is recorded in `summary.json` `metadata.tags` and every `.internal/index.jsonl` row, and the Dashboard "Tags" tab groups and compares runs by any tag key. This is the only "tags" concept: the earlier free-form manual per-run tag chips have been removed. (Suite-level `tags` may still be authored as a string list, which is a selection construct for `select.tags` / `--tag name` filtering rather than run metadata.) -**Environment** — The AgentV-authored testbed recipe for coding-agent evals. It prepares the host or Docker state an agent will inspect or modify: repositories, archives, patches, generated fixtures, services, dependency setup, and cwd. `environment` can be inline or loaded with `file://`, with shared `file://` recipes as the canonical reusable form. Initial `environment.type` values are `host` and `docker`. Promptfoo does not define this primitive; it is an AgentV extension to promptfoo-compatible eval authoring. +**Environment** — The AgentV-authored testbed recipe for coding-agent evals. It prepares the host or Docker state an agent will inspect or modify: repositories, archives, patches, generated fixtures, services, dependency setup, and cwd. `environment` can be inline or loaded with `file://`, with shared `file://` recipes as the canonical reusable form. Initial `environment.type` values are `host` and `docker`. Promptfoo does not define this primitive; it is an AgentV extension to promptfoo-compatible eval authoring. Promptfoo export may lower a host/filesystem subset into generated extensions and provider workdir config; Docker environment export is unsupported until a faithful runner boundary exists. -**Workdir** — The current working directory inside an environment. `environment.workdir` is the cwd passed to target providers and graders/test scripts unless a later scoped feature explicitly overrides it. Host workdirs are local paths such as `./workspaces/bottle`; Docker workdirs are container paths such as `/app`. +**Workdir** — The current working directory inside an environment. `environment.workdir` is the cwd passed to providers and graders/test scripts unless a later scoped feature explicitly overrides it. Host workdirs are local paths such as `./workspaces/bottle`; Docker workdirs are container paths such as `/app`. **Top-level `env`** — Promptfoo-compatible provider/eval environment-variable overrides and load-time template inputs such as `OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}"`. Top-level `env` is not the testbed recipe and must not be moved under `environment`. @@ -49,9 +49,11 @@ Shared domain vocabulary for this project — entities, named processes, and sta ```yaml environment: file://.agentv/environments/local-python.yaml -targets: - - id: codex - provider: codex-cli +providers: + - id: agentv:codex-cli + label: codex + config: + command: codex ``` ```yaml diff --git a/ROADMAP.md b/ROADMAP.md index e8d9f1b49..67e4816ce 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ # AgentV Roadmap -Last updated: 2026-06-21 +Last updated: 2026-07-07 This roadmap translates [STRATEGY.md](STRATEGY.md) into the next few product phases. It is intentionally short: the goal is to keep priorities and boundaries visible without turning the roadmap into a plan dump. @@ -10,7 +10,9 @@ This roadmap translates [STRATEGY.md](STRATEGY.md) into the next few product pha - The AgentV Dashboard is the supported zero-infra local cockpit for AgentV-owned runs, traces, sessions, transcripts, and Git-backed artifacts. - Phoenix is optional external trace infrastructure only when Codex, Arize, or another hook already emitted spans independently; AgentV may correlate with those sessions but does not write AgentV artifacts into Phoenix. - Harbor stays an optional benchmark-grade runner boundary, not AgentV core. -- AgentV YAML remains the authoring surface even when execution moves behind another runner; prefer a lightweight translation layer over duplicated specs. +- AgentV YAML remains the authoring surface even when execution moves behind another runner; keep it Promptfoo-compatible by default and prefer a lightweight export/translation layer over duplicated specs. +- The intended authored-config differences from Promptfoo are limited to `environment`, AgentV refs, and built-in AgentV providers. Promptfoo-native declarations should run directly; AgentV-native sugar should become Promptfoo-compatible through `agentv export promptfoo`. +- Promptfoo export may lower host/filesystem `environment` setup into generated extensions and provider workdir configuration. Docker environments remain AgentV-native until a separate runner/export boundary is accepted. - Adapters, workers, and artifact projections are preferred over rebuilding adjacent platforms inside AgentV, except Phoenix: Phoenix integration is link-out correlation only and not an AgentV-to-Phoenix projection path. ## Phase 1: Finish the artifact and local inspection foundation @@ -35,6 +37,7 @@ This roadmap translates [STRATEGY.md](STRATEGY.md) into the next few product pha ## Phase 4: Extend outward through optional boundaries +- Promptfoo: add and maintain an exporter/validation gate that proves AgentV-native provider sugar and supported refs lower into Promptfoo-readable configs. Keep Docker environment export unsupported with explicit diagnostics until a faithful runner boundary exists. - Harbor: move toward Harbor becoming the benchmark-grade runner behind a lightweight translation layer from AgentV YAML, while AgentV stays the authoring, gating, import, and comparison surface. - Harbor: in the near term, launch or import benchmark-grade runs through a runner boundary; over time, converge on Harbor as the execution layer for the suites it already owns. - Opik and similar systems: consume completed AgentV projection bundles as post-run adapters rather than as runtime owners. diff --git a/STRATEGY.md b/STRATEGY.md index b4056eabd..b135003b1 100644 --- a/STRATEGY.md +++ b/STRATEGY.md @@ -1,6 +1,6 @@ --- name: AgentV -last_updated: 2026-06-21 +last_updated: 2026-07-07 --- # AgentV Strategy @@ -9,9 +9,11 @@ last_updated: 2026-06-21 Teams evaluating coding agents and other tool-using workflows need results from the real repositories, fixtures, and harnesses their agents already touch, but that work often gets split away from the actual workspace and development loop it came from. That makes it hard to reproduce failures, compare targets fairly, and keep evaluation evidence close to the code and workflow it came from. -## Our approach +## Our Approach -AgentV stays repo-native and workspace-native: it runs or imports evaluations around the user's existing harness, writes portable run artifacts, and keeps core primitives focused on execution, grading, routing, and results storage. It integrates outward through clear boundaries: Phoenix can be correlated with as an optional external trace database when spans were emitted independently, Harbor can provide benchmark-grade execution, and post-run/export adapters can serve adjacent systems without AgentV trying to own every layer. +AgentV stays repo-native and workspace-native while using Promptfoo-compatible eval authoring as the default public surface. Users author familiar `providers`, `prompts`, `tests`, `vars`, `assert`, `env`, and `extensions`; AgentV adds repo-native `environment` recipes, AgentV-native built-in providers, portable run artifacts, and the local Dashboard. When a config uses AgentV-only pieces, `agentv export promptfoo` is the compatibility boundary: supported refs and built-in providers lower to Promptfoo-readable files/providers, while unsupported environment semantics fail with explicit diagnostics. + +AgentV runs or imports evaluations around the user's existing harness, writes portable run artifacts, and keeps core primitives focused on execution, grading, routing, and results storage. It integrates outward through clear boundaries: Phoenix can be correlated with as an optional external trace database when spans were emitted independently, Harbor can provide benchmark-grade execution, and post-run/export adapters can serve adjacent systems without AgentV trying to own every layer. ## Who it's for @@ -44,6 +46,12 @@ Add Phoenix, Harbor, Opik, Langfuse, and similar systems through narrow correlat _Why it serves the approach:_ This expands AgentV's reach without turning it into a hosted observability stack, benchmark platform, or integration kitchen sink. +### Promptfoo-compatible authoring and export + +Keep AgentV's authored eval surface close to Promptfoo wherever the semantics match. The intentional differences are limited to `environment`, AgentV refs, and built-in AgentV providers. The exporter turns supported AgentV-native sugar into Promptfoo-readable config, with filesystem/host environment setup lowered through generated extensions and provider workdir config where possible. + +_Why it serves the approach:_ This lets users and coding agents reuse the incumbent eval vocabulary while preserving AgentV's repo-native setup, artifacts, and dashboard. + ### Evaluation primitives for real agent workflows Strengthen provider routing, grader composition, trace and trajectory scoring, and CI gates around coding-agent and tool-using workflows. diff --git a/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md b/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md index 2612d70f5..bd4c0d07e 100644 --- a/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md +++ b/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md @@ -39,6 +39,14 @@ identity key for AgentV reruns, trends, result links, or cross-run comparison. This deliberately diverges from Promptfoo's display-first row model while keeping Promptfoo config importable. +Status note (2026-07-07): [ADR 0019](0019-promptfoo-first-provider-authoring-and-export-boundary.md) +supersedes this ADR's target/provider authoring decision. Public systems under +test are now authored as top-level `providers` using Promptfoo-compatible +`id`/`label` semantics. AgentV's intended authored-config differences from +Promptfoo are limited to `environment`, AgentV refs, and built-in AgentV +providers; full Promptfoo compatibility for those differences is by +`agentv export promptfoo`. + ## Context AgentV's eval-authoring surface diverged from industry primitives. We are re-basing @@ -79,10 +87,13 @@ keep AgentV's only where its semantics are genuinely better.** `javascript` is NOT desugared to `script`. 4. **`metric` is the named-score field** (nunjucks-templated); grader `name` becomes display-only. Add `named_scores` + `derived_metrics`. -5. **`targets` is the canonical system-under-test** axis (promptfoo target/`ProviderOptions` - object shape + AgentV extensions). `provider`/`apiId` = the **backend** kind (never a - SUT). No runtime top-level `providers` alias (would overload the backend term); the - codemod/conversion remaps promptfoo `providers:` → `targets:`. +5. **`providers` is the canonical system-under-test** axis. Provider entries + follow Promptfoo-compatible shapes. `providers[].id` is the backend/spec + string, may contain colons, and may be Promptfoo-native or AgentV-native. + `providers[].label` is the stable AgentV selection/result identity and + defaults to `id` when omitted. Old public `targets` authoring is removed + before release; run-bundle artifact fields named `target` remain a separate + artifact-contract concern. 6. **Prompts + vars plus direct `input`**: adopt top-level `prompts` (string/chat-array/file/ fn, nunjucks `{{vars}}`) for prompt matrices, while keeping `input` as the supported direct-task shorthand for one-prompt suites. When `prompts` is @@ -122,23 +133,25 @@ keep AgentV's only where its semantics are genuinely better.** lifecycle extension and not target identity.** AgentV remains promptfoo-compatible where promptfoo has matching primitives: `prompts`, `vars`, `tests`, `default_test`/`defaultTest`, `assert`, transforms, - `targets`/providers, top-level `env`, and lifecycle `extensions`. AgentV + `providers`, top-level `env`, and lifecycle `extensions`. AgentV adds `environment` as an AgentV-specific suite/test/case testbed recipe for repo materialization, fixtures, patches, services, Docker context/image, - setup scripts, and the workdir/cwd handed to targets and graders. The + setup scripts, and the workdir/cwd handed to providers and graders. The recipe may be inline or a `file://` reference; shared `file://` recipes are the canonical reusable form: ```yaml environment: file://.agentv/environments/local-python.yaml - targets: - - id: codex - provider: codex-cli + providers: + - id: agentv:codex-cli + label: codex + config: + command: codex ``` `environment.type` starts with `host` and `docker`. `environment.workdir` - defines the current working directory passed to target providers and + defines the current working directory passed to providers and graders/test scripts unless a later scoped feature explicitly overrides it. Top-level `env` remains promptfoo-compatible provider/eval env overrides rendered from `{{ env.VAR }}` and must not be moved under `environment`. @@ -172,7 +185,8 @@ top-level `budget_usd`, scalar top-level `threshold`, grader `name`-as-metric, t - A one-shot codemod migrates existing eval files and hard-errors on removed keys with a message pointing at the replacement. - promptfoo authors get a near-drop-in contract (snake_case); AgentV keeps repo/agent - differentiation as documented extensions. + differentiation concentrated in `environment`, AgentV refs, built-in AgentV + providers, artifacts, and Dashboard behavior. - FizzBuzz/SWE-bench-style test grading needs no new assertion primitive -- a script grader runs the tests from `environment.workdir` (see ADR 0017 note on SWE-bench `FAIL_TO_PASS`/`PASS_TO_PASS`). diff --git a/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md b/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md index a93a84eea..bf87430f1 100644 --- a/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md +++ b/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md @@ -23,6 +23,14 @@ contract below supersedes both the earlier agentskills-shaped `graders[]`/`checks[]` wording. Native AgentV grading artifacts now use a recursive Promptfoo-style grading result in `snake_case`. +Amended (2026-07-07) by [ADR 0019](0019-promptfoo-first-provider-authoring-and-export-boundary.md): +Promptfoo-compatible authoring now uses top-level `providers` for systems under +test. The artifact contract may keep `target` fields until a separate artifact +migration is accepted. `environment` remains the AgentV-owned testbed recipe; +Promptfoo export may lower a host/filesystem subset into generated extensions +and provider workdir configuration, while Docker environments are unsupported +for export until a faithful runner boundary exists. + Amended (2026-07-06) by tracker `av-cpl5`: decision point 3 below described `transcript_summary` as inlined into each `index.jsonl` result row. That inlining is superseded — see "Summary/index/sidecar boundary is locked" below @@ -285,8 +293,8 @@ contract than this ADR's original `workspace` decision: AgentV combines promptfoo-compatible eval authoring with an AgentV-owned environment recipe informed by Margin, Harbor, and Terminal-Bench evidence: -Promptfoo is the compatibility baseline for prompts, vars, tests, assertions, -targets/providers, top-level `env`, and lifecycle `extensions`; Margin Evals is +Promptfoo is the compatibility baseline for providers, prompts, vars, tests, +assertions, top-level `env`, and lifecycle `extensions`; Margin Evals is the closest reference for local coding-agent UX, filesystem-native suite ergonomics, immutable run bundles, resume/artifact discipline, and per-case image/cwd/test execution; Harbor and Terminal-Bench 2 are the strongest @@ -298,9 +306,11 @@ in ```yaml environment: file://.agentv/environments/local-python.yaml -targets: - - id: codex - provider: codex-cli +providers: + - id: agentv:codex-cli + label: codex + config: + command: codex ``` ```yaml @@ -339,9 +349,11 @@ mechanically run setup code. For example, an eval could author a reusable extensions: - file://.agentv/extensions/setup-workspace.ts:beforeAll -targets: - - id: codex - provider: codex-cli +providers: + - id: agentv:codex-cli + label: codex + config: + command: codex ``` That shape is not a good canonical product contract for coding-agent eval @@ -358,6 +370,12 @@ separate visible concepts. Therefore `extensions` stay lifecycle hooks for customizing eval flow, while `environment` is the explicit AgentV substrate, setup, and `workdir` contract. +For Promptfoo export, AgentV may generate extensions that implement a supported +host/filesystem environment subset and pass the resolved workdir to providers. +That generated shape is an export artifact, not the canonical AgentV authoring +contract. Docker environments must not be flattened into extensions until an +export path can preserve image, mount, service, resource, and provenance +semantics. 1. **`environment` is the authored testbed recipe** at suite/test/case scope. It may be inline or loaded through a field-level `file://` reference. Shared @@ -367,14 +385,14 @@ Therefore `extensions` stay lifecycle hooks for customizing eval flow, while fields such as `context`, `dockerfile`, `image`, `workdir`, and future scoped resource/mount/secrets fields. 3. **`environment.workdir` defines cwd.** AgentV passes the resolved workdir to - target providers and graders/test scripts unless a later scoped feature - explicitly overrides it. Target configs may still expose provider-specific - knobs, but the canonical testbed cwd comes from the environment recipe. + providers and graders/test scripts unless a later scoped feature explicitly + overrides it. Provider configs may still expose provider-specific knobs, but + the canonical testbed cwd comes from the environment recipe. 4. **`environment.setup` materializes testbed state.** Setup is declarative data plus an argv `command`: repos, archives, patches, generated fixtures, installed dependencies, services, and other case state. Shell behavior is explicit by authoring an argv such as `["bash", "-lc", "..."]`. Setup runs - before target execution and before ordinary promptfoo lifecycle hooks. + before provider execution and before ordinary promptfoo lifecycle hooks. 5. **Top-level `env` remains promptfoo-compatible.** It is for provider/eval env overrides and load-time `{{ env.VAR }}` rendering. Do not move it under `environment`. If `environment.env` is implemented, it means variables scoped @@ -382,10 +400,12 @@ Therefore `extensions` stay lifecycle hooks for customizing eval flow, while 6. **Promptfoo `extensions` remain lifecycle hooks.** They can customize eval flow, but they are not the canonical testbed setup contract because hidden hook code is weaker for review, validation, sharing, and cwd semantics. -7. **Targets select agents/providers.** `targets[].id` is stable AgentV target - identity, `targets[].provider` names the adapter/control boundary, and - `targets[].runtime` remains placement/transport. Targets do not own - Docker/testbed setup by default. +7. **Providers select agents/runtimes.** Superseded by ADR 0019 for public + authoring: systems under test are authored under top-level `providers`. + `providers[].id` names the backend/spec string, `providers[].label` is the + stable AgentV selection/result identity, and `providers[].runtime` remains + placement/transport for AgentV-native provider entries. Provider entries do + not own Docker/testbed setup by default. 8. **`workspace` is not the public coding-agent benchmark contract.** The original `workspace.repos`, `workspace.scope`, `workspace.docker`, and `workspace.template` names are superseded where they meant authored testbed @@ -395,10 +415,10 @@ Therefore `extensions` stay lifecycle hooks for customizing eval flow, while authored testbed primitive. The invariants matter more than the mechanism: testbed setup is declared as -data; materialization precedes target execution and normal lifecycle hooks; cwd -is explicit; provider/target identity remains separate from testbed setup; and -run bundles can snapshot the resolved recipe, setup inputs, and resolved -workdir as provenance. +data; materialization precedes provider execution and normal lifecycle hooks; +cwd is explicit; provider identity remains separate from testbed setup; and run +bundles can snapshot the resolved recipe, setup inputs, and resolved workdir as +provenance. ### Note: SWE-bench `FAIL_TO_PASS` / `PASS_TO_PASS` diff --git a/docs/adr/0018-coding-agent-target-runtime-contract.md b/docs/adr/0018-coding-agent-target-runtime-contract.md index 63267b587..407fdfc1a 100644 --- a/docs/adr/0018-coding-agent-target-runtime-contract.md +++ b/docs/adr/0018-coding-agent-target-runtime-contract.md @@ -21,6 +21,12 @@ feature explicitly overrides it. Runtime mode vocabulary may be refined by Bead `av-oi9a`; that separate cleanup does not move testbed setup under targets. +Amended (2026-07-07) by [ADR 0019](0019-promptfoo-first-provider-authoring-and-export-boundary.md): +public coding-agent systems under test are now authored under `providers`, not +`targets`. This ADR's runtime-boundary decisions still apply to AgentV-native +provider entries such as `agentv:codex-cli`. Existing artifact fields named +`target` are not renamed by this amendment. + ## Context AgentV evaluates coding agents in real repositories. Those agents are not @@ -70,15 +76,15 @@ Peer frameworks are evidence, not schema authority: ## Decision -AgentV treats coding-agent targets as external runtimes to orchestrate, not +AgentV treats coding-agent providers as external runtimes to orchestrate, not libraries to call in-process by default. -Authored targets use this shape: +Authored provider entries use this shape: ```yaml -targets: - - id: codex-local - provider: codex-app-server +providers: + - id: agentv:codex-app-server + label: codex-local runtime: host config: command: ["codex", "app-server"] @@ -89,8 +95,8 @@ The fields mean: | Field | Meaning | | --- | --- | -| `id` | Stable AgentV target identity used for CLI selection, artifacts, Dashboard, and comparisons. | -| `provider` | Adapter/control boundary such as `codex-cli`, `codex-app-server`, `pi-rpc`, `claude-cli`, or `copilot-sdk`. | +| `id` | Backend/spec string. For AgentV-native coding-agent providers this may be `agentv:codex-cli`, `agentv:codex-app-server`, or another AgentV resolver id. | +| `label` | Stable AgentV selection/result identity used for CLI selection, artifacts, Dashboard, and comparisons. | | `runtime` | Placement/isolation mode: `host`, `profile`, or `sandbox`; may be a string shorthand or an object with `mode`. | | `config` | Provider-specific knobs such as `command`, `model`, `cwd`, `timeout_seconds`, auth endpoint settings, permission flags, and provider protocol settings. | @@ -98,37 +104,36 @@ Do not add competing top-level fields such as `isolation`, `sandbox`, `profile`, `install`, `container`, `environment`, `executable`, `binary`, `args`, or `arguments` for this contract. Process/protocol providers use `config.command` as a non-empty argv array. Authored eval concurrency belongs -under `evaluate_options.max_concurrency`, not inside a target definition. +under `evaluate_options.max_concurrency`, not inside a provider entry. Grader selection belongs to `defaults.grader`, CLI overrides, or -evaluator-level target selection, not to the system-under-test target. +evaluator-level provider selection, not to the system-under-test provider. -`environment` is valid as an AgentV eval/test/case recipe field outside target +`environment` is valid as an AgentV eval/test/case recipe field outside provider definitions. It prepares the host or Docker testbed and defines -`environment.workdir`; the target runtime then decides how the selected +`environment.workdir`; the provider runtime then decides how the selected provider/agent is invoked against that prepared cwd. `environment` does not -replace `targets[].id`, `targets[].provider`, or `targets[].runtime`. +replace `providers[].id`, `providers[].label`, or `providers[].runtime`. Docker/testbed setup belongs to the environment driver by default, not to each -target provider. +provider entry. ### Provider Boundaries Process and protocol providers are the preferred defaults: -- `codex-app-server`: preferred Codex rich protocol/control boundary. -- `codex-cli`: simple Codex subprocess boundary for host/profile execution and +- `agentv:codex-app-server`: preferred Codex rich protocol/control boundary. +- `agentv:codex-cli`: simple Codex subprocess boundary for host/profile execution and installed user shims. -- `pi-rpc`: preferred Pi rich control boundary over stdio/RPC. -- `pi-cli`: simple Pi subprocess boundary. -- `claude-cli`: default Claude path through the installed Claude CLI. -- `copilot-cli`: active Copilot execution through the installed CLI/protocol +- `agentv:pi-rpc`: preferred Pi rich control boundary over stdio/RPC. +- `agentv:pi-cli`: simple Pi subprocess boundary. +- `agentv:claude-cli`: default Claude path through the installed Claude CLI. +- `agentv:copilot-cli`: active Copilot execution through the installed CLI/protocol path. SDK providers are explicit advanced paths: -- `codex-sdk` -- `pi-sdk` -- `claude-sdk` -- `copilot-sdk` +- Promptfoo-native SDK ids such as `openai:codex-sdk` where available. +- AgentV-native SDK ids such as `agentv:pi-sdk`, `agentv:claude-sdk`, or + `agentv:copilot-sdk` where AgentV owns a provider adapter. SDK transports run behind an AgentV child-runner process on the host. The parent CLI/orchestrator starts the child with the target config and provider request, @@ -154,8 +159,8 @@ and app-server transports run against the host-installed agent/profile. SDK transports also run on the host, but through the AgentV child-runner process described above. -The current implementation supports Docker sandbox execution for generic -`provider: cli`. Sandbox-aware coding-agent providers are future work. When a +The current implementation supports Docker sandbox execution for generic CLI +providers. Sandbox-aware coding-agent providers are future work. When a coding-agent provider is authored with `runtime.mode: sandbox` before a sandbox-aware runner exists, AgentV should return a deliberate `target_execution` error envelope rather than pretending the target ran or @@ -183,16 +188,16 @@ crashes or prevent final run-bundle artifacts from being written. ### Replay and Log Providers -`provider: copilot-log` is removed from the authored live target surface before -beta. AgentV should not add `codex-log`, `claude-log`, `pi-log`, or other -provider-specific log target providers. +Provider-specific log entries such as `copilot-log` are removed from the +authored live provider surface before beta. AgentV should not add `codex-log`, +`claude-log`, `pi-log`, or other provider-specific log providers. Provider-native logs remain useful as raw provenance and import inputs. Copilot `events.jsonl` parsing should feed import/normalization into a provider-agnostic recorded trajectory replay contract. Replay is an eval/orchestrator mode or generic replay target over AgentV trajectory artifacts, -not a live coding-agent runtime provider. Live Copilot targets remain -`copilot-cli` and `copilot-sdk`. +not a live coding-agent runtime provider. Live Copilot providers remain +`agentv:copilot-cli` and `agentv:copilot-sdk`. This aligns with ADR 0008: raw native transcripts are preserved for debugging and parser improvement, while normalized AgentV transcript/trajectory artifacts @@ -206,9 +211,9 @@ are the durable input to grading, Dashboard inspection, and replay. - SDK providers stay available when SDK-native events or controls are worth the extra complexity, but SDK dependency failures do not take down the parent CLI. - `runtime: host` remains lightweight and zero-infra; stronger profile/sandbox - isolation can be added without changing target identity semantics. -- Generic Docker sandbox support through `provider: cli` remains valid, while - sandbox-aware coding-agent adapters are deliberately deferred. + isolation can be added without changing provider identity semantics. +- Generic Docker sandbox support for CLI providers remains valid, while + sandbox-aware coding-agent providers are deliberately deferred. - Offline grading/replay gets one provider-agnostic path instead of one provider-specific `*-log` target surface per backend. @@ -226,16 +231,14 @@ artifact writing. Rejected. SDKs can expose useful events, but the default AgentV path should match the real installed CLI/profile where possible and keep the product -zero-infra. SDK providers are explicit advanced targets. +zero-infra. SDK providers are explicit advanced paths. ### Copy Promptfoo provider naming wholesale -Rejected. Promptfoo is useful evidence for explicit provider IDs and optional -provider dependencies, but AgentV keeps target identity and backend/control -boundary separate: `id` is stable AgentV target identity, while `provider` names -the adapter kind. AgentV does not copy Promptfoo's use of `label` as the target -identity field or carry compatibility aliases where the beta contract can be -cleaner. +Superseded by ADR 0019. AgentV now follows Promptfoo's public +`providers`/`id`/`label` shape where semantics match, while reserving +`agentv:*` ids for AgentV-native built-ins and using export/transpile for full +Promptfoo execution compatibility. ### Put runtime placement under provider-specific config @@ -243,17 +246,17 @@ Rejected. Runtime placement is cross-provider orchestration state. It belongs in `runtime`, not in every provider's `config` with different names and precedence rules. -### Treat provider logs as live target providers +### Treat provider logs as live providers Rejected. Passive logs do not run an agent and should not satisfy live host-runtime dogfood. They are import/replay sources. Keeping them out of -authored live target YAML avoids a family of `*-log` providers and preserves a +authored live provider YAML avoids a family of `*-log` providers and preserves a single normalized replay contract. ## Non-Goals - Implementing or validating the full live provider matrix. - Implementing profile-mode or sandbox-aware coding-agent provider runners. -- Replacing the generic `provider: cli` sandbox path. +- Replacing the generic CLI provider sandbox path. - Designing the full provider-agnostic replay cassette contract. - Adding compatibility aliases for removed beta-only target provider names. diff --git a/docs/adr/0019-promptfoo-first-provider-authoring-and-export-boundary.md b/docs/adr/0019-promptfoo-first-provider-authoring-and-export-boundary.md new file mode 100644 index 000000000..a6a4679f0 --- /dev/null +++ b/docs/adr/0019-promptfoo-first-provider-authoring-and-export-boundary.md @@ -0,0 +1,185 @@ +# 19. Promptfoo-first provider authoring and export boundary + +Date: 2026-07-07 + +## Status + +Accepted (2026-07-07). Supersedes the target/provider authoring portions of +[ADR 0016](0016-promptfoo-superset-eval-authoring-contract.md), +[ADR 0017](0017-output-artifact-and-workspace-resolver-contract.md), and +[ADR 0018](0018-coding-agent-target-runtime-contract.md). It does not rename +existing run-bundle artifact fields such as `target` or Dashboard target facets; +those remain a separate artifact migration decision. + +Tracked by Beads `av-fdco`, `av-ctfu`, `av-uttb`, and `av-lbcv`. + +## Context + +AgentV originally kept `targets` as the canonical public system-under-test axis +and treated Promptfoo `providers` as reference evidence. That made AgentV's +internal identity model explicit, but it also created a large avoidable +translation gap against the incumbent eval authoring surface we want users and +agents to recognize. + +Promptfoo already has broad ecosystem familiarity for normal eval matrices: +`providers`, `prompts`, `tests`, `vars`, `assert`, `env`, `extensions`, provider +option objects, package providers, and colon-delimited provider ids such as +`openai:responses:gpt-5.4`. AgentV's durable product difference is not a +different name for the matrix axis. It is repo-native environment setup, +agent-native providers, portable run artifacts, and the local Dashboard. + +## Decision + +AgentV authored eval YAML is Promptfoo-first by default. Public systems under +test are authored under top-level `providers`, not `targets`. + +AgentV intentionally diverges from directly runnable Promptfoo YAML only for +three authoring categories: + +1. **`environment`** + AgentV owns declarative repo/testbed setup, fixtures, services, workdir, + host/Docker lifecycle, and provenance. Promptfoo has lifecycle `extensions`, + but it has no typed environment primitive. + +2. **AgentV refs** + AgentV may support repo-native references that are more ergonomic than raw + Promptfoo config. Export must inline, copy, or rewrite supported refs into + Promptfoo-readable `file://` or generated files where possible. + +3. **Built-in AgentV providers** + AgentV-native provider ids such as `agentv:codex-cli` are first-class + ergonomic built-ins. They are not directly Promptfoo-runnable ids. Promptfoo + compatibility for those providers is by export/transpile into a + Promptfoo-readable provider reference. + +All other public eval authoring should either match Promptfoo's shape or be +intentionally rejected rather than expanded into a parallel AgentV surface. + +## Provider Contract + +Provider entries use Promptfoo-compatible shapes: + +```yaml +providers: + - openai:responses:gpt-5.4 + - id: openai:codex-sdk:gpt-5.5 + label: codex-sdk + - id: agentv:codex-cli + label: codex-cli + config: + command: codex + model: gpt-5.5 +``` + +Field meanings: + +| Field | Meaning | +| --- | --- | +| `id` | Backend/spec string. It may contain colons and is interpreted by the provider resolver. | +| `label` | Stable AgentV selection/result identity. Defaults to `id` when omitted. | +| `config` | Provider-specific configuration. | +| `env`, `prompts`, `transform`, `delay`, `inputs` | Promptfoo-compatible provider option fields. | + +`providers[].id` is not a safe artifact path segment. Artifact writers should +continue using existing sanitized target/result identity fields unless a +separate artifact migration is accepted. + +Duplicate `label` values are invalid because label is the AgentV selection and +result identity. Duplicate `id` values may be valid when labels differ, such as +two configurations of the same backend. + +## Promptfoo Export Boundary + +Direct compatibility applies to Promptfoo-native declarations. Full Promptfoo +compatibility for AgentV-native YAML is provided by `agentv export promptfoo`. + +The exporter lowers supported AgentV-native provider ids into Promptfoo-readable +providers, preferring cross-platform TypeScript or package providers over shell +wrappers: + +```yaml +providers: + - id: file://.agentv/generated/promptfoo/providers/codex-cli-provider.ts:callApi + label: codex-cli + config: + command: codex + model: gpt-5.5 +``` + +When a published provider package exists and is installed, export may also emit +complete Promptfoo package provider ids: + +```yaml +providers: + - id: package:@agentv/promptfoo-providers:CodexCliProvider + label: codex-cli +``` + +Promptfoo package provider ids must include the exported class/function segment +after the final colon. A bare `package:@agentv/promptfoo-providers/codex-cli` +is invalid at provider-load time. + +## Environment Export Boundary + +Promptfoo export may support a filesystem-isolated host subset of +`environment`. In that subset, AgentV can transpile environment setup into +Promptfoo `extensions` and generated provider configuration: + +- a generated `beforeAll`/`beforeEach` extension materializes the isolated + workspace and records the resolved workdir in a generated variable or + metadata field; +- exported providers receive that workdir through their `config` or vars; +- generated files keep the setup logic visible in the exported Promptfoo + project; +- the exporter records diagnostics explaining which AgentV environment fields + were lowered. + +Docker environments are not part of the initial Promptfoo export subset. +Exporting a Docker `environment` must fail with a clear unsupported-feature +diagnostic instead of silently producing a config that loses isolation, +resources, mounts, or service semantics. + +This means Promptfoo export can be fully runnable for Promptfoo-native evals and +for AgentV evals that use supported filesystem/host environment setup. Docker +remains AgentV-native until a deliberate runner/export boundary is accepted. + +## Consequences + +- Public docs, examples, SDK config, CLI selection flags, and migration tooling + should use `providers`, `id`, and `label`. +- Old public `targets` authoring is a hard deprecation before release. +- Internal code may keep `target` vocabulary where it reflects existing run + artifact contracts or Dashboard grouping. Avoid broad artifact renames in the + provider-surface migration. +- AgentV does not become a Promptfoo fork. It remains a compatible product with + AgentV-owned environment orchestration, artifacts, dashboard, attempts policy, + and agent-native providers. +- Promptfoo compatibility should be guarded by CI that validates exported + Promptfoo configs against Promptfoo's loader or validator. + +## Alternatives Considered + +### Keep `targets` as the public canonical axis + +Rejected. The identity/backend split was clean, but it made every Promptfoo +example look foreign and forced AgentV to document a needless translation for +the most common matrix axis. + +### Make every AgentV YAML file directly Promptfoo-runnable + +Rejected. Direct execution cannot preserve AgentV's `environment` semantics or +AgentV-native providers. Export/transpile is the correct compatibility boundary +because it can generate provider shims, setup extensions, diagnostics, and +generated files intentionally. + +### Emit shell wrappers for built-in providers + +Rejected as the default. Shell scripts are not cross-platform and would make +Windows compatibility worse. Generated TypeScript providers or complete package +provider ids are the preferred Promptfoo export forms. + +### Export Docker environments as Promptfoo extensions + +Rejected for the initial subset. Promptfoo extensions can run commands, but they +do not encode Docker resources, mounts, images, service lifecycle, or provenance +well enough to be a faithful export.