diff --git a/.agents/product-boundary.md b/.agents/product-boundary.md index fdb9a0ab1..002000609 100644 --- a/.agents/product-boundary.md +++ b/.agents/product-boundary.md @@ -39,16 +39,16 @@ AgentV transcript artifacts are not Phoenix-native conversation inputs. Model-ca ## Promptfoo-Compatible Authoring Boundary -AgentV adopts Promptfoo-compatible eval matrix authoring by default where Promptfoo has matching semantics. The core mental model is `prompts x tests/vars x providers`, with repeat samples and retries applied as run policy after the authored matrix is resolved. +AgentV should adopt Promptfoo-compatible eval matrix authoring where it strengthens repo-native evaluation, but Promptfoo is reference evidence rather than schema authority. The core mental model is `prompts x tests/vars x providers`, with repeat samples and retries applied as run policy after the authored matrix is resolved. Compatibility is at the provider declaration layer, not a promise that every AgentV config executes unchanged in Promptfoo. Keep these AgentV-native boundaries explicit: -- `providers` are systems under test. AgentV accepts Promptfoo-shaped provider declarations: strings like `openai:gpt-4.1-mini`, complete package provider strings like `package:@agentv/promptfoo-providers:CodexCliProvider`, provider option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, and provider maps like `{ "openai:gpt-4": { label, config } }`. In AgentV, `id` names the backend/spec and `label` is the stable AgentV identity used for selection and result grouping. -- AgentV-only provider ids such as `agentv:codex-cli` are first-class AgentV authoring sugar, but they are not directly Promptfoo-runnable. `agentv export promptfoo` must lower supported built-ins to Promptfoo-readable `file://...:callApi` or complete `package:...:Export` provider entries. +- `providers` are systems under test. AgentV accepts Promptfoo-shaped provider declarations: strings like `openai:gpt-4.1-mini`, complete package provider strings like `package:@agentv/promptfoo-providers:CodexCliProvider` or `package:@agentv/promptfoo-providers/codex-cli:Provider`, provider option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, and provider maps like `{ "openai:gpt-4": { label, config } }`. Package provider strings must include the exported class/function segment after the final colon. In AgentV, `id` names the backend/spec and `label` is the stable AgentV identity used for selection and result grouping. +- AgentV-only fields such as top-level `environment` and provider-local runtime/testbed overlays are AgentV semantics. Promptfoo may ignore or strip unknown keys and will not execute AgentV environment setup without a transpiler or wrapper. +- Internal runtime, artifacts, and Dashboard grouping may keep target-named fields until a separate artifact migration. - Coding-agent testbeds use `environment` recipes for host/Docker substrate, setup, fixtures, services, and cwd. Do not make Promptfoo lifecycle `extensions` or public `workspace` authoring the canonical testbed contract. -- Promptfoo export may lower a filesystem/host subset of `environment` into generated `extensions` plus provider workdir configuration. Docker environments are not part of the initial export subset and must fail with explicit diagnostics rather than silently losing isolation. - Top-level `env` means provider/eval environment variables. `extensions` remain lifecycle hooks. -- Reusable prompts, tests, defaults, and environments use field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, and `environment: file://...`. Export may rewrite supported AgentV refs into Promptfoo-readable generated files. +- Reusable prompts, tests, defaults, and environments use field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, and `environment: file://...`. - Grouping and Dashboard navigation use tags and run-bundle metadata, not experiment path buckets, Vercel path layout, or model-as-experiment grouping. - AgentV run bundles, traces, transcripts, datasets, indexes, and Git-backed artifacts stay AgentV-owned. Do not design an Opik export path or Phoenix projection path for those artifacts. @@ -64,7 +64,7 @@ Prefer these extension points before adding a built-in: - plain assertion strings for simple semantic rubric checks - `llm-rubric` for promptfoo-compatible free-form rubric checks - `g-eval` for structured or multi-criteria rubric judging -- config-level grader providers selected through `defaults.grader` or assertion-level provider selection, not system-under-test provider configuration +- config-level grader providers selected through `defaults.grader` or assertion-level `provider`, not provider-local grader configuration - CLI wrappers that consume AgentV JSON or JSONL output for post-processing such as aggregation, comparison, or reporting Ask: can this be achieved with existing primitives plus a plugin or wrapper? If yes, it should not be a built-in. That includes niche config overrides for existing graders. @@ -86,9 +86,9 @@ Aim for the maximum feature surface with the minimum primitives. Before proposing a new feature, enumerate which existing primitives could achieve the same outcome when composed. -- Oracle validation is a `cli` provider target that runs a reference solution through the same evaluators. +- Oracle validation is a `cli` provider that runs a reference solution through the same evaluators. - Snapshot MCP for benchmarks is frozen data in the environment recipe plus `before_all` and `after_all` hooks. -- Harness variant comparison is target hooks with different `before_each` setup scripts. +- Harness variant comparison is provider hooks with different `before_each` setup scripts. - Skill evaluation is `tool-trajectory` plus `execution-metrics` plus `rubric` composed via `assert-set`. If existing primitives cover the need, document the pattern instead of building a new feature. New primitives are justified only when composition is impossible, not merely undocumented. @@ -108,7 +108,7 @@ Research those references from local cloned repositories first when a clone is a Treat these as reference inputs, not dependencies. AgentV should adopt the shared lowest common denominator when it fits the repo-native artifact model, and document any intentional divergence in the relevant plan, ADR, or contract docs. -Do not copy another framework's schema baggage just because the framework is credible. When a peer contract carries historical constraints, overloaded field names, or compatibility aliases, prefer a cleaner AgentV contract if it preserves the core user need. Document the reason for diverging so future workers do not "realign" it back to the peer shape. For the provider surface, ADR 0019 is the current exception: AgentV follows Promptfoo's `providers`/`id`/`label` shape so the uncommon AgentV differences stay concentrated in `environment`, refs, built-in AgentV providers, artifacts, and Dashboard behavior. +Do not copy another framework's schema baggage just because the framework is credible. When a peer contract carries historical constraints, overloaded field names, or compatibility aliases, prefer a cleaner AgentV contract if it preserves the core user need. Document the reason for diverging so future workers do not "realign" it back to the peer shape. For provider declarations, AgentV follows the Promptfoo-compatible split: `id` is the backend/spec and `label` is the stable AgentV identity. ### 5. YAGNI - You Aren't Gonna Need It diff --git a/.agents/verification.md b/.agents/verification.md index aa78e9dfd..29eec2386 100644 --- a/.agents/verification.md +++ b/.agents/verification.md @@ -87,19 +87,19 @@ agent-browser --cdp 9222 screenshot output.png ## Agent Provider Eval Concurrency -- When running evals against agent-provider targets such as `claude`, `claude-sdk`, `codex`, `copilot`, `copilot-sdk`, `pi`, or `pi-cli`, limit concurrency to 3 targets at a time. +- When running evals against heavyweight agent providers such as `claude`, `claude-sdk`, `codex`, `copilot`, `copilot-sdk`, `pi`, or `pi-cli`, limit concurrency to 3 providers at a time. - These providers spawn heavyweight subprocesses and can exhaust system resources if you run too many in parallel. ```bash -bun apps/cli/src/cli.ts eval my.EVAL.yaml --target claude & -bun apps/cli/src/cli.ts eval my.EVAL.yaml --target codex & +bun apps/cli/src/cli.ts eval my.EVAL.yaml --provider claude & +bun apps/cli/src/cli.ts eval my.EVAL.yaml --provider codex & wait -bun apps/cli/src/cli.ts eval my.EVAL.yaml --target copilot & -bun apps/cli/src/cli.ts eval my.EVAL.yaml --target pi & +bun apps/cli/src/cli.ts eval my.EVAL.yaml --provider copilot & +bun apps/cli/src/cli.ts eval my.EVAL.yaml --provider pi & wait ``` -- This limit does not apply to lightweight LLM-only targets such as `azure`, `openai`, `gemini`, or `openrouter`. +- This limit does not apply to lightweight LLM-only providers such as `azure`, `openai`, `gemini`, or `openrouter`. ## Writing Tests @@ -142,22 +142,22 @@ bun apps/cli/src/cli.ts eval examples/features/rubric/evals/dataset.eval.yaml -- - `grading.json.assertion_results` reflects the evaluation logic, pass boolean, score, and evidence 4. Update baseline files if output format changes. Baselines live next to eval YAML files as `*.baseline.jsonl`. -5. `agentv validate` is the cheap schema/config check. For no-live-provider quality validation, run graders against a real reference/oracle target or frozen transcript/replay fixture. +5. `agentv validate` is the cheap schema/config check. For no-live-provider quality validation, run graders against a real reference/oracle provider or frozen transcript/replay fixture. ## Live Dogfood for Eval and Experiment Changes -Use live dogfood before marking PRs ready when they affect eval execution, experiments, repeat runs, targets, providers, graders, or artifact provenance. +Use live dogfood before marking PRs ready when they affect eval execution, experiments, repeat runs, providers, graders, or artifact provenance. -- Live means both sides are real: a live agent/provider target and a live grader target. Do not count `mock`, replay/frozen transcript runs, or deterministic-only assertions as dogfood for these changes. +- Live means both sides are real: a live agent provider and a live grader provider. Do not count `mock`, replay/frozen transcript runs, or deterministic-only assertions as dogfood for these changes. - Dogfood must use a threshold high enough to prove correctness for the eval under test. A threshold of `0` is only an execution smoke check and does not count as dogfood evidence. - Prefer the smallest realistic eval: one or two cases, bounded timeouts, and `workers: 1` for heavyweight agent providers. - For artifact/result contract changes, prefer letting AgentV choose the canonical run directory and capture the printed `Artifact workspace written to:` and `Results written to:` paths for evidence. Do not precompute `--output` unless the test specifically needs a fixed path. -- For native experiment changes, run through `agentv eval run ... --experiment ` so resolution, setup, scripts, target selection, run knobs, and artifact metadata are exercised together. +- For native experiment changes, run through `agentv eval run ... --experiment ` so resolution, setup, scripts, provider selection, run knobs, and artifact metadata are exercised together. - For repeat-run changes, use `evaluate_options.repeat.count >= 2` when validating repeated executions. Inspect `.internal/index.jsonl`, root `summary.json`, and the repeated case folder. Use `repeat` for authored configuration and `sample_index`/`retry_index` for produced executions. The repeated case folder should carry aggregate `summary.json`; sample-specific outputs, transcripts, grading, and metrics live under `sample-N/`. Each `sample-N/` folder should contain `result.json`, `grading.json`, `metrics.json`, `transcript.json`, `transcript-raw.jsonl`, and `outputs/answer.md` when answer output is available. `result.json` should point at `./grading.json`, `./metrics.json`, `./transcript.json`, and `./transcript-raw.jsonl` through the corresponding path fields. -- For local OpenAI-compatible grading through the OAuth proxy, use `base_url: http://127.0.0.1:10531/v1`, but still route `api_key` and `model` through environment references such as `{{ env.LOCAL_OPENAI_PROXY_API_KEY }}` and `{{ env.LOCAL_OPENAI_PROXY_MODEL }}`. Literal secrets and literal model values are intentionally rejected by target validation unless a resolver explicitly allows them. -- For `codex`/Codex SDK live dogfood through the same local proxy, configure the agent target with the current target graph (`id`, `provider`, `runtime`, `config`) and put provider settings such as `base_url`, `api_key`, `model`, and `api_format` under `config`. Configure the reusable grader under `graders`, then select it with `defaults.grader` or assertion-level target selection; do not put a grader selector on the system-under-test target. A minimal run should use `bun apps/cli/src/cli.ts eval run --targets --target --workers 1`. +- For local OpenAI-compatible grading through the OAuth proxy, use `base_url: http://127.0.0.1:10531/v1`, but still route `api_key` and `model` through environment references such as `{{ env.LOCAL_OPENAI_PROXY_API_KEY }}` and `{{ env.LOCAL_OPENAI_PROXY_MODEL }}`. Literal secrets and literal model values are intentionally rejected by provider validation unless a resolver explicitly allows them. +- For `codex`/Codex SDK live dogfood through the same local proxy, configure the agent provider with `id`, `label`, `runtime`, and `config`, and put backend settings such as `base_url`, `api_key`, `model`, and `api_format` under `config`. Configure the reusable grader in the same `providers` catalog, then select it with `defaults.grader` or assertion-level `provider`; do not put a grader selector on the system-under-test provider. A minimal run should use `bun apps/cli/src/cli.ts eval run --providers --provider --workers 1`. - If the local proxy returns `401 token_expired`, the blocker is stale Codex OAuth, not AgentV target configuration. Refresh from a trusted local terminal with `codex logout`, `codex login --device-auth`, then restart `openai-oauth` and rerun the same eval command. -- Preserve review evidence in `agentv-private` on an orphan `evidence/` branch. Include the run bundle, source eval/experiment/targets files, a short README, an artifact tree, contract checks, and screenshots when folder structure or UI behavior is under review. +- Preserve review evidence in `agentv-private` on an orphan `evidence/` branch. Include the run bundle, source eval/experiment/providers files, a short README, an artifact tree, contract checks, and screenshots when folder structure or UI behavior is under review. - If comparing against an external convention such as Vercel `agent-eval`, verify both semantic provenance and the physical `sample-N` artifact layout for repeat runs. - For transcript/result artifact contract changes, try the same provider spread before merging: `pi-cli`, `codex-sdk`, and `copilot-sdk` through the local OpenAI-compatible endpoint when available. If a provider cannot run live, record the exact blocker, the run bundle or command output, and whether coverage moved to fixture/regression tests. - If dogfood or review changes the durable verification playbook, update this file or `AGENTS.md` in the same PR. Use `docs/solutions/` for longer reusable lessons rather than relying on PR comments or private evidence as the only source. @@ -169,7 +169,7 @@ Use `scripts/check-grader-scores.ts` as a post-processor after an eval run. Workflow: ```bash -bun apps/cli/src/cli.ts eval examples/path/to/suite.eval.yaml --target azure \ +bun apps/cli/src/cli.ts eval examples/path/to/suite.eval.yaml --provider azure \ --output examples/path/to/suite.run bun scripts/check-grader-scores.ts diff --git a/AGENTS.md b/AGENTS.md index eb92e72ae..f120e5a96 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,11 +14,11 @@ AgentV aims to be the repo-native, workspace-native evaluation framework for AI Eval authoring mental model: -- AgentV adopts Promptfoo-compatible matrix authoring by default: an eval expands `prompts x tests/vars x providers`, then applies repeat samples/retries as run policy. -- Author systems under test as `providers`. A provider entry `id` names the backend/spec string; `label` is the stable AgentV selection and result identity. `id` may contain Promptfoo-style colons such as `openai:responses:gpt-5.4`. -- The intended authored-config differences from Promptfoo are limited to `environment`, AgentV refs, and built-in AgentV providers such as `agentv:codex-cli`. Full Promptfoo compatibility for those AgentV-native pieces is by `agentv export promptfoo`. -- Use `environment` recipes for coding-agent testbeds, including host/Docker setup, repo materialization, fixtures, services, and cwd. Promptfoo export may lower a host/filesystem subset into generated `extensions` plus provider workdir configuration; Docker environments must remain unsupported on export until a faithful runner boundary exists. Do not use Promptfoo `extensions` or public `workspace` authoring as the canonical AgentV testbed contract. -- Use top-level `env` for provider/eval environment variables. Use `extensions` for lifecycle hooks. Use field-local `file://` refs for reusable prompts, tests, defaults, and environments; export may rewrite supported AgentV refs into Promptfoo-readable files. +- AgentV adopts Promptfoo-compatible matrix authoring where it helps: an eval expands `prompts x tests/vars x providers`, then applies repeat samples/retries as run policy. Compatibility is at the provider declaration layer, not a claim that every AgentV config runs in Promptfoo unchanged. +- Author systems under test as `providers`. A provider entry `id` names the backend/spec; `label` is the stable AgentV selection and result identity. +- Treat AgentV as a Promptfoo-compatible superset for provider declarations. Promptfoo-shaped provider entries can be strings like `openai:gpt-4.1-mini`, complete package provider strings like `package:@agentv/promptfoo-providers:CodexCliProvider` or `package:@agentv/promptfoo-providers/codex-cli:Provider`, option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, or provider maps like `{ "openai:gpt-4": { label, config } }`. Package provider strings must include the exported class/function segment after the final colon. AgentV-only fields such as top-level `environment` and provider-local runtime/testbed overlays remain AgentV semantics; Promptfoo may ignore or strip unknown keys and will not execute AgentV environment setup without a transpiler or wrapper. +- Use `environment` recipes for coding-agent testbeds, including host/Docker setup, repo materialization, fixtures, services, and cwd. Do not use Promptfoo `extensions` or public `workspace` authoring as the canonical testbed contract. +- Use top-level `env` for provider/eval environment variables. Use `extensions` for lifecycle hooks. Use field-local `file://` refs for reusable prompts, tests, defaults, and environments. - Use `tags` and run-bundle metadata for grouping and Dashboard navigation. Do not use experiment path buckets, Vercel path layout, or model-as-experiment grouping as canonical AgentV semantics. - AgentV run bundles, traces, transcripts, datasets, indexes, and Git-backed artifacts stay AgentV-owned. Do not design an Opik export path or Phoenix projection path for them; Phoenix correlation is link-out only when `external_trace` metadata already exists. @@ -41,13 +41,12 @@ tests: repo: "file://fixtures/retry" providers: - - id: agentv:codex-cli + - id: codex-cli label: codex-host runtime: host - config: - command: codex - - id: anthropic:claude-agent-sdk - label: claude + - id: claude-cli + label: claude-docker + runtime: docker environment: file://.agentv/environments/local-repo.yaml tags: @@ -70,7 +69,7 @@ Design guardrails: - Document composition patterns before inventing a new feature. - Match industry-standard lowest-common-denominator contracts when possible. - When designing AgentV contracts, check public reference standards such as Claude Skills, Vercel agent-eval, Hugging Face Datasets, and OpenInference before inventing AgentV-specific shapes. Use their shared lowest common denominator where it fits, and document any intentional divergence. -- Treat peer frameworks as evidence, not schema authority. Do not inherit baggage such as overloaded field names, compatibility aliases, or framework-specific historical constraints when AgentV can express a cleaner repo-native contract. For the provider surface, AgentV intentionally follows Promptfoo's `providers`/`id`/`label` shape so the uncommon AgentV differences stay concentrated in `environment`, refs, built-in AgentV providers, artifacts, and Dashboard behavior. +- Treat peer frameworks as evidence, not schema authority. Do not inherit baggage such as overloaded field names, compatibility aliases, or framework-specific historical constraints when AgentV can express a cleaner repo-native contract. For provider declarations, AgentV intentionally follows Promptfoo's `providers` shapes: `id` is the backend/spec and `label` is the stable AgentV identity. - For peer-framework research, use local cloned repositories and DeepWiki MCP before broad web search. In this operator workspace, Promptfoo is cloned at `/home/entity/projects/promptfoo/promptfoo` and DeepEval is cloned at `/home/entity/projects/confident-ai/deepeval`; use DeepWiki repos `promptfoo/promptfoo` and `confident-ai/deepeval` for architecture-level orientation, then verify exact claims with `rg` and `git` in the local clone. If a public contract must be checked for currentness, use official docs and record the source URL or clone commit behind the conclusion. - Apply YAGNI aggressively and solve the current request with the smallest surface that works. - Keep extensions non-breaking unless a same-week unreleased surface should be hard-corrected. diff --git a/CONCEPTS.md b/CONCEPTS.md index 86be7a189..cad8b6392 100644 --- a/CONCEPTS.md +++ b/CONCEPTS.md @@ -4,13 +4,15 @@ Shared domain vocabulary for this project — entities, named processes, and sta ## Providers and Targets -**Provider** — The eval YAML or config declaration for a system under test. AgentV follows Promptfoo's provider declaration layer: entries may be strings such as `openai:responses:gpt-5.4`, complete package provider strings such as `package:@agentv/promptfoo-providers:CodexCliProvider`, provider option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, or provider maps such as `{ "openai:gpt-4": { label, config } }`. In AgentV, `id` names the backend/spec and may contain colons; `label` is the stable AgentV selection and result identity. Built-in AgentV ids such as `agentv:codex-cli` are AgentV-native sugar and require Promptfoo export before Promptfoo can run them. +**Provider** — The eval YAML or config declaration for a system under test. AgentV is Promptfoo-compatible at this provider declaration layer: entries may be strings such as `openai:gpt-4.1-mini`, complete package provider strings such as `package:@agentv/promptfoo-providers:CodexCliProvider` or `package:@agentv/promptfoo-providers/codex-cli:Provider`, provider option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, or provider maps such as `{ "openai:gpt-4": { label, config } }`. Package provider strings must include the exported class/function segment after the final colon. In AgentV, `id` names the backend/spec and `label` is the stable AgentV selection identity. AgentV-only provider fields such as `runtime` or provider-local testbed/runtime overlays are AgentV semantics, not full Promptfoo execution compatibility. A single eval file can declare multiple providers to compare AI systems side by side. Providers select agents/adapters; they do not own the authored host/Docker testbed recipe. -**Target** — Historical/public-authoring term for a system under test and still an internal/result-artifact vocabulary where existing run bundles, Dashboard facets, and sidecars use `target`. New authored eval YAML uses `providers`. Do not reintroduce public `targets` authoring without a new ADR; keep artifact renames separate from the provider-surface migration. +**Provider adapter** — The plugin/runtime implementation that connects AgentV's evaluation engine to a specific AI system, such as Copilot CLI, Copilot SDK, Claude API, or Pi. Provider adapters implement the request/response contract: given a test case, invoke the AI system and return its output. Public `providers[]` entries select and configure these adapters by `id`. + +**Target** — Internal/runtime and artifact vocabulary for the stable comparison dimension produced by a selected provider. Result rows and run-bundle indexes keep the `target` field as the stable source identity until a separate artifact migration changes that contract. **Provider runtime** — The placement/transport mode for invoking a provider, such as host execution, sandbox/container placement, CLI subprocess, app-server protocol, RPC, or SDK child runner. Runtime describes how the selected agent is invoked. Advanced home/env/profile-style overlays are provider or runtime configuration details, not the authored testbed recipe. Runtime is separate from the environment that prepares files, services, and cwd. -**Grader (selection)** — Not a distinct entity type. A grader is a regular provider, listed under `providers` like any other and selected for a grading role via `defaults.grader`, CLI `--grader-provider`, test options, or an assertion-level provider override. Authoring a separate `graders:` list is a hard error: move each entry into `providers`. This also means a grader can itself be the provider under evaluation (oracle/calibration checks), which a separate schema would block. Do not confuse this with **grader (method)** — an assertion `type` such as `llm-rubric`, `script`, or `g-eval` that names *how* scoring works, independent of *which* provider performs it; or with a rubric's `criteria`/`value`, which is the grading *prompt*. AgentV once called grader-selection "judge" before a deliberate rename to "grader" — do not reintroduce "judge" without a stronger reason than surface-level clarity. +**Grader (selection)** — Not a distinct entity type. A grader is a regular provider, listed under `providers` like any other and selected for a grading role via `defaults.grader`, CLI `--grader-provider`, `default_test.options.provider`, `tests[].options.provider`, or an assertion-level `provider` override. Authoring a separate `graders:` list is a hard error: move each entry into `providers`. This also means a grader can itself be the provider under evaluation for oracle/calibration checks, which a separate schema would block. Do not confuse this with **grader (method)** — an assertion `type` such as `llm-rubric`, `code-grader`, or `g-eval` that names *how* scoring works, independent of *which* provider performs it; or with a rubric's `criteria`/`value`, which is the grading *prompt*. AgentV once called grader-selection "judge" (`judge_target`) before a deliberate, full rename to "grader" — do not reintroduce "judge" without a stronger reason than surface-level clarity, since peer frameworks (Margin-Lab/evals, Harbor) don't treat judge/grader as a first-class schema concept either. **Provider runtime boundary** — The process boundary between AgentV's evaluation orchestrator and the agent runtime a provider invokes. CLI-backed providers place the agent runtime outside the orchestrator; SDK providers should run through an AgentV-owned child-runner boundary when runtime teardown can threaten run artifact finalization. This boundary does not own repository/testbed setup by default. @@ -18,7 +20,7 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Eval / Eval YAML** — The composable and runnable AgentV authoring primitive. An eval YAML file describes the prompts, tests, variables, providers, assertions, environments, tags, and run policy for an evaluation. AgentV does not have a separate runnable `experiment.yaml` artifact. -**Matrix authoring** — The Promptfoo-compatible shape AgentV adopts by default: `prompts x tests/vars x providers`, with repeat samples and retries applied as run policy after the authored matrix is resolved. AgentV-native boundaries are intentionally narrow: `environment` recipes prepare coding-agent testbeds, AgentV refs may be rewritten during export, built-in AgentV provider ids such as `agentv:codex-cli` require export for Promptfoo execution, and grouping uses tags plus run-bundle metadata. +**Matrix authoring** — The Promptfoo-compatible shape AgentV adopts where useful: `prompts x tests/vars x providers`, with repeat samples and retries applied as run policy after the authored matrix is resolved. AgentV is a Promptfoo-compatible superset at the provider declaration layer, not a fully Promptfoo-executable config format. AgentV-native boundaries remain: provider `id` names the backend/spec, provider `label` is the stable AgentV identity, `environment` recipes prepare coding-agent testbeds, `env` carries provider/eval variables, `extensions` are lifecycle hooks, reusable content uses field-local `file://` refs, and grouping uses tags plus run-bundle metadata. **Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `environment`, assertions, expected references, and judge criteria. It runs directly or shares reusable parts through field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, `default_test: file://...`, and `environment: file://...`. @@ -28,13 +30,13 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Multi-file run** — A CLI-selected set of eval YAML files. Use multiple eval files, tags, and CLI run selection to group suites under one run intent without adding wrapper-suite semantics to YAML. -**Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Experiment is expressed as the reserved `tags.experiment` key (see **Tags**); there is no top-level `experiment` field. Runtime policy belongs in top-level eval fields or target objects; the experiment label is recorded in `summary.json` and `.internal/index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `extensions` or target hooks, not in a separate experiment artifact. +**Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Experiment is expressed as the reserved `tags.experiment` key (see **Tags**); there is no top-level `experiment` field. Runtime policy belongs in top-level eval fields or provider objects; the experiment label is recorded in `summary.json` and `.internal/index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `extensions` or provider hooks, not in a separate experiment artifact. **Tags** — A promptfoo-shaped `Record` map authored on an eval (or project config / `--tag key=value`) that labels a run with structured facets such as `experiment`, `team`, or `env`. The reserved `experiment` key feeds the experiment namespace. The resolved map is recorded in `summary.json` `metadata.tags` and every `.internal/index.jsonl` row, and the Dashboard "Tags" tab groups and compares runs by any tag key. This is the only "tags" concept: the earlier free-form manual per-run tag chips have been removed. (Suite-level `tags` may still be authored as a string list, which is a selection construct for `select.tags` / `--tag name` filtering rather than run metadata.) -**Environment** — The AgentV-authored testbed recipe for coding-agent evals. It prepares the host or Docker state an agent will inspect or modify: repositories, archives, patches, generated fixtures, services, dependency setup, and cwd. `environment` can be inline or loaded with `file://`, with shared `file://` recipes as the canonical reusable form. Initial `environment.type` values are `host` and `docker`. Promptfoo does not define this primitive; it is an AgentV extension to promptfoo-compatible eval authoring. Promptfoo export may lower a host/filesystem subset into generated extensions and provider workdir config; Docker environment export is unsupported until a faithful runner boundary exists. +**Environment** — The AgentV-authored testbed recipe for coding-agent evals. It prepares the host or Docker state an agent will inspect or modify: repositories, archives, patches, generated fixtures, services, dependency setup, and cwd. `environment` can be inline or loaded with `file://`, with shared `file://` recipes as the canonical reusable form. Initial `environment.type` values are `host` and `docker`. Promptfoo does not define this primitive; it is an AgentV extension, and Promptfoo will not execute this setup without a transpiler or wrapper. -**Workdir** — The current working directory inside an environment. `environment.workdir` is the cwd passed to providers and graders/test scripts unless a later scoped feature explicitly overrides it. Host workdirs are local paths such as `./workspaces/bottle`; Docker workdirs are container paths such as `/app`. +**Workdir** — The current working directory inside an environment. `environment.workdir` is the cwd passed to target providers and graders/test scripts unless a later scoped feature explicitly overrides it. Host workdirs are local paths such as `./workspaces/bottle`; Docker workdirs are container paths such as `/app`. **Top-level `env`** — Promptfoo-compatible provider/eval environment-variable overrides and load-time template inputs such as `OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}"`. Top-level `env` is not the testbed recipe and must not be moved under `environment`. @@ -50,10 +52,8 @@ Shared domain vocabulary for this project — entities, named processes, and sta environment: file://.agentv/environments/local-python.yaml providers: - - id: agentv:codex-cli + - id: codex-cli label: codex - config: - command: codex ``` ```yaml diff --git a/apps/cli/src/commands/eval/commands/run.ts b/apps/cli/src/commands/eval/commands/run.ts index c32459c03..134140620 100644 --- a/apps/cli/src/commands/eval/commands/run.ts +++ b/apps/cli/src/commands/eval/commands/run.ts @@ -21,15 +21,26 @@ export const evalRunCommand = command({ displayName: 'eval-paths', description: 'Path(s) or glob(s) to evaluation files (.yaml, .eval.ts)', }), + provider: multioption({ + type: array(string), + long: 'provider', + description: + 'Override provider label(s) from providers.yaml (repeatable for matrix evaluation)', + }), + providers: option({ + type: optional(string), + long: 'providers', + description: 'Path to providers.yaml (overrides discovery)', + }), target: multioption({ type: array(string), long: 'target', - description: 'Override target name(s) from targets.yaml (repeatable for matrix evaluation)', + description: '[Removed: use --provider