feat(provider-surface): migrate SDK CLI docs to providers

.agents/product-boundary.md

@@ -39,16 +39,16 @@ AgentV transcript artifacts are not Phoenix-native conversation inputs. Model-ca
 
 ## Promptfoo-Compatible Authoring Boundary
 
-AgentV adopts Promptfoo-compatible eval matrix authoring by default where Promptfoo has matching semantics. The core mental model is `prompts x tests/vars x providers`, with repeat samples and retries applied as run policy after the authored matrix is resolved.
+AgentV should adopt Promptfoo-compatible eval matrix authoring where it strengthens repo-native evaluation, but Promptfoo is reference evidence rather than schema authority. The core mental model is `prompts x tests/vars x providers`, with repeat samples and retries applied as run policy after the authored matrix is resolved. Compatibility is at the provider declaration layer, not a promise that every AgentV config executes unchanged in Promptfoo.
 
 Keep these AgentV-native boundaries explicit:
 
-- `providers` are systems under test. AgentV accepts Promptfoo-shaped provider declarations: strings like `openai:gpt-4.1-mini`, complete package provider strings like `package:@agentv/promptfoo-providers:CodexCliProvider`, provider option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, and provider maps like `{ "openai:gpt-4": { label, config } }`. In AgentV, `id` names the backend/spec and `label` is the stable AgentV identity used for selection and result grouping.
-- AgentV-only provider ids such as `agentv:codex-cli` are first-class AgentV authoring sugar, but they are not directly Promptfoo-runnable. `agentv export promptfoo` must lower supported built-ins to Promptfoo-readable `file://...:callApi` or complete `package:...:Export` provider entries.
+- `providers` are systems under test. AgentV accepts Promptfoo-shaped provider declarations: strings like `openai:gpt-4.1-mini`, complete package provider strings like `package:@agentv/promptfoo-providers:CodexCliProvider` or `package:@agentv/promptfoo-providers/codex-cli:Provider`, provider option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, and provider maps like `{ "openai:gpt-4": { label, config } }`. Package provider strings must include the exported class/function segment after the final colon. In AgentV, `id` names the backend/spec and `label` is the stable AgentV identity used for selection and result grouping.
+- AgentV-only fields such as top-level `environment` and provider-local runtime/testbed overlays are AgentV semantics. Promptfoo may ignore or strip unknown keys and will not execute AgentV environment setup without a transpiler or wrapper.
+- Internal runtime, artifacts, and Dashboard grouping may keep target-named fields until a separate artifact migration.
 - Coding-agent testbeds use `environment` recipes for host/Docker substrate, setup, fixtures, services, and cwd. Do not make Promptfoo lifecycle `extensions` or public `workspace` authoring the canonical testbed contract.
-- Promptfoo export may lower a filesystem/host subset of `environment` into generated `extensions` plus provider workdir configuration. Docker environments are not part of the initial export subset and must fail with explicit diagnostics rather than silently losing isolation.
 - Top-level `env` means provider/eval environment variables. `extensions` remain lifecycle hooks.
-- Reusable prompts, tests, defaults, and environments use field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, and `environment: file://...`. Export may rewrite supported AgentV refs into Promptfoo-readable generated files.
+- Reusable prompts, tests, defaults, and environments use field-local `file://` refs such as `prompts: file://...`, `tests: file://...`, and `environment: file://...`.
 - Grouping and Dashboard navigation use tags and run-bundle metadata, not experiment path buckets, Vercel path layout, or model-as-experiment grouping.
 - AgentV run bundles, traces, transcripts, datasets, indexes, and Git-backed artifacts stay AgentV-owned. Do not design an Opik export path or Phoenix projection path for those artifacts.
 
@@ -64,7 +64,7 @@ Prefer these extension points before adding a built-in:
 - plain assertion strings for simple semantic rubric checks
 - `llm-rubric` for promptfoo-compatible free-form rubric checks
 - `g-eval` for structured or multi-criteria rubric judging
-- config-level grader providers selected through `defaults.grader` or assertion-level provider selection, not system-under-test provider configuration
+- config-level grader providers selected through `defaults.grader` or assertion-level `provider`, not provider-local grader configuration
 - CLI wrappers that consume AgentV JSON or JSONL output for post-processing such as aggregation, comparison, or reporting
 
 Ask: can this be achieved with existing primitives plus a plugin or wrapper? If yes, it should not be a built-in. That includes niche config overrides for existing graders.
@@ -86,9 +86,9 @@ Aim for the maximum feature surface with the minimum primitives.
 
 Before proposing a new feature, enumerate which existing primitives could achieve the same outcome when composed.
 
-- Oracle validation is a `cli` provider target that runs a reference solution through the same evaluators.
+- Oracle validation is a `cli` provider that runs a reference solution through the same evaluators.
 - Snapshot MCP for benchmarks is frozen data in the environment recipe plus `before_all` and `after_all` hooks.
-- Harness variant comparison is target hooks with different `before_each` setup scripts.
+- Harness variant comparison is provider hooks with different `before_each` setup scripts.
 - Skill evaluation is `tool-trajectory` plus `execution-metrics` plus `rubric` composed via `assert-set`.
 
 If existing primitives cover the need, document the pattern instead of building a new feature. New primitives are justified only when composition is impossible, not merely undocumented.
@@ -108,7 +108,7 @@ Research those references from local cloned repositories first when a clone is a
 
 Treat these as reference inputs, not dependencies. AgentV should adopt the shared lowest common denominator when it fits the repo-native artifact model, and document any intentional divergence in the relevant plan, ADR, or contract docs.
 
-Do not copy another framework's schema baggage just because the framework is credible. When a peer contract carries historical constraints, overloaded field names, or compatibility aliases, prefer a cleaner AgentV contract if it preserves the core user need. Document the reason for diverging so future workers do not "realign" it back to the peer shape. For the provider surface, ADR 0019 is the current exception: AgentV follows Promptfoo's `providers`/`id`/`label` shape so the uncommon AgentV differences stay concentrated in `environment`, refs, built-in AgentV providers, artifacts, and Dashboard behavior.
+Do not copy another framework's schema baggage just because the framework is credible. When a peer contract carries historical constraints, overloaded field names, or compatibility aliases, prefer a cleaner AgentV contract if it preserves the core user need. Document the reason for diverging so future workers do not "realign" it back to the peer shape. For provider declarations, AgentV follows the Promptfoo-compatible split: `id` is the backend/spec and `label` is the stable AgentV identity.
 
 ### 5. YAGNI - You Aren't Gonna Need It
 

.agents/verification.md

@@ -87,19 +87,19 @@ agent-browser --cdp 9222 screenshot output.png
 
 ## Agent Provider Eval Concurrency
 
-- When running evals against agent-provider targets such as `claude`, `claude-sdk`, `codex`, `copilot`, `copilot-sdk`, `pi`, or `pi-cli`, limit concurrency to 3 targets at a time.
+- When running evals against heavyweight agent providers such as `claude`, `claude-sdk`, `codex`, `copilot`, `copilot-sdk`, `pi`, or `pi-cli`, limit concurrency to 3 providers at a time.
 - These providers spawn heavyweight subprocesses and can exhaust system resources if you run too many in parallel.
 
 ```bash
-bun apps/cli/src/cli.ts eval my.EVAL.yaml --target claude &
-bun apps/cli/src/cli.ts eval my.EVAL.yaml --target codex &
+bun apps/cli/src/cli.ts eval my.EVAL.yaml --provider claude &
+bun apps/cli/src/cli.ts eval my.EVAL.yaml --provider codex &
 wait
-bun apps/cli/src/cli.ts eval my.EVAL.yaml --target copilot &
-bun apps/cli/src/cli.ts eval my.EVAL.yaml --target pi &
+bun apps/cli/src/cli.ts eval my.EVAL.yaml --provider copilot &
+bun apps/cli/src/cli.ts eval my.EVAL.yaml --provider pi &
 wait
 ```
 
-- This limit does not apply to lightweight LLM-only targets such as `azure`, `openai`, `gemini`, or `openrouter`.
+- This limit does not apply to lightweight LLM-only providers such as `azure`, `openai`, `gemini`, or `openrouter`.
 
 ## Writing Tests
 
@@ -142,22 +142,22 @@ bun apps/cli/src/cli.ts eval examples/features/rubric/evals/dataset.eval.yaml --
 - `grading.json.assertion_results` reflects the evaluation logic, pass boolean, score, and evidence
 
 4. Update baseline files if output format changes. Baselines live next to eval YAML files as `*.baseline.jsonl`.
-5. `agentv validate` is the cheap schema/config check. For no-live-provider quality validation, run graders against a real reference/oracle target or frozen transcript/replay fixture.
+5. `agentv validate` is the cheap schema/config check. For no-live-provider quality validation, run graders against a real reference/oracle provider or frozen transcript/replay fixture.
 
 ## Live Dogfood for Eval and Experiment Changes
 
-Use live dogfood before marking PRs ready when they affect eval execution, experiments, repeat runs, targets, providers, graders, or artifact provenance.
+Use live dogfood before marking PRs ready when they affect eval execution, experiments, repeat runs, providers, graders, or artifact provenance.
 
-- Live means both sides are real: a live agent/provider target and a live grader target. Do not count `mock`, replay/frozen transcript runs, or deterministic-only assertions as dogfood for these changes.
+- Live means both sides are real: a live agent provider and a live grader provider. Do not count `mock`, replay/frozen transcript runs, or deterministic-only assertions as dogfood for these changes.
 - Dogfood must use a threshold high enough to prove correctness for the eval under test. A threshold of `0` is only an execution smoke check and does not count as dogfood evidence.
 - Prefer the smallest realistic eval: one or two cases, bounded timeouts, and `workers: 1` for heavyweight agent providers.
 - For artifact/result contract changes, prefer letting AgentV choose the canonical run directory and capture the printed `Artifact workspace written to:` and `Results written to:` paths for evidence. Do not precompute `--output` unless the test specifically needs a fixed path.
-- For native experiment changes, run through `agentv eval run ... --experiment <experiment.yaml|ts>` so resolution, setup, scripts, target selection, run knobs, and artifact metadata are exercised together.
+- For native experiment changes, run through `agentv eval run ... --experiment <experiment.yaml|ts>` so resolution, setup, scripts, provider selection, run knobs, and artifact metadata are exercised together.
 - For repeat-run changes, use `evaluate_options.repeat.count >= 2` when validating repeated executions. Inspect `.internal/index.jsonl`, root `summary.json`, and the repeated case folder. Use `repeat` for authored configuration and `sample_index`/`retry_index` for produced executions. The repeated case folder should carry aggregate `summary.json`; sample-specific outputs, transcripts, grading, and metrics live under `sample-N/`. Each `sample-N/` folder should contain `result.json`, `grading.json`, `metrics.json`, `transcript.json`, `transcript-raw.jsonl`, and `outputs/answer.md` when answer output is available. `result.json` should point at `./grading.json`, `./metrics.json`, `./transcript.json`, and `./transcript-raw.jsonl` through the corresponding path fields.
-- For local OpenAI-compatible grading through the OAuth proxy, use `base_url: http://127.0.0.1:10531/v1`, but still route `api_key` and `model` through environment references such as `{{ env.LOCAL_OPENAI_PROXY_API_KEY }}` and `{{ env.LOCAL_OPENAI_PROXY_MODEL }}`. Literal secrets and literal model values are intentionally rejected by target validation unless a resolver explicitly allows them.
-- For `codex`/Codex SDK live dogfood through the same local proxy, configure the agent target with the current target graph (`id`, `provider`, `runtime`, `config`) and put provider settings such as `base_url`, `api_key`, `model`, and `api_format` under `config`. Configure the reusable grader under `graders`, then select it with `defaults.grader` or assertion-level target selection; do not put a grader selector on the system-under-test target. A minimal run should use `bun apps/cli/src/cli.ts eval run <eval.yaml> --targets <targets.yaml> --target <codex-target> --workers 1`.
+- For local OpenAI-compatible grading through the OAuth proxy, use `base_url: http://127.0.0.1:10531/v1`, but still route `api_key` and `model` through environment references such as `{{ env.LOCAL_OPENAI_PROXY_API_KEY }}` and `{{ env.LOCAL_OPENAI_PROXY_MODEL }}`. Literal secrets and literal model values are intentionally rejected by provider validation unless a resolver explicitly allows them.
+- For `codex`/Codex SDK live dogfood through the same local proxy, configure the agent provider with `id`, `label`, `runtime`, and `config`, and put backend settings such as `base_url`, `api_key`, `model`, and `api_format` under `config`. Configure the reusable grader in the same `providers` catalog, then select it with `defaults.grader` or assertion-level `provider`; do not put a grader selector on the system-under-test provider. A minimal run should use `bun apps/cli/src/cli.ts eval run <eval.yaml> --providers <providers.yaml> --provider <codex-label> --workers 1`.
 - If the local proxy returns `401 token_expired`, the blocker is stale Codex OAuth, not AgentV target configuration. Refresh from a trusted local terminal with `codex logout`, `codex login --device-auth`, then restart `openai-oauth` and rerun the same eval command.
-- Preserve review evidence in `agentv-private` on an orphan `evidence/<bead-or-feature-slug>` branch. Include the run bundle, source eval/experiment/targets files, a short README, an artifact tree, contract checks, and screenshots when folder structure or UI behavior is under review.
+- Preserve review evidence in `agentv-private` on an orphan `evidence/<bead-or-feature-slug>` branch. Include the run bundle, source eval/experiment/providers files, a short README, an artifact tree, contract checks, and screenshots when folder structure or UI behavior is under review.
 - If comparing against an external convention such as Vercel `agent-eval`, verify both semantic provenance and the physical `sample-N` artifact layout for repeat runs.
 - For transcript/result artifact contract changes, try the same provider spread before merging: `pi-cli`, `codex-sdk`, and `copilot-sdk` through the local OpenAI-compatible endpoint when available. If a provider cannot run live, record the exact blocker, the run bundle or command output, and whether coverage moved to fixture/regression tests.
 - If dogfood or review changes the durable verification playbook, update this file or `AGENTS.md` in the same PR. Use `docs/solutions/` for longer reusable lessons rather than relying on PR comments or private evidence as the only source.
@@ -169,7 +169,7 @@ Use `scripts/check-grader-scores.ts` as a post-processor after an eval run.
 Workflow:
 
 ```bash
-bun apps/cli/src/cli.ts eval examples/path/to/suite.eval.yaml --target azure \
+bun apps/cli/src/cli.ts eval examples/path/to/suite.eval.yaml --provider azure \
   --output examples/path/to/suite.run
 
 bun scripts/check-grader-scores.ts