feat(eval): wire defaults.grader as a config-level grader fallback

[christso]

Summary

Promptfoo research this session confirmed their grading-provider selection is a single layered override (assertion → test → suite default → env-detected default), fully decoupled from the providers-under-test list — never a per-provider field. This repo's own .agents/product-boundary.md design principles already list "config-level grader targets selected through defaults.grader or assertion-level target selection, not target-level grader configuration" as a preferred extension point, and .agents/verification.md tells engineers configuring codex dogfood to "select it with defaults.grader... do not put a grader selector on the system-under-test target." But defaults.grader was dead: config-graph.ts validated it at config-parse time but nothing consumed it at eval-run time, and orchestrator.ts hard-required an explicit grader_target on every single agent-provider target with no global fallback — contradicting the framework's own documented guidance.

• orchestrator.ts: new defaultGraderTarget option threaded through RunEvaluationOptions/EvaluationRuntimeOptions, inserted into the grader-resolution chain between a target's own grader_target and self-grading: target.graderTarget ?? defaultGraderTarget ?? target.name. Also satisfies the agent-provider "needs a grader" hard-requirement guard.
• run-eval.ts: threads .agentv/config.yaml's defaults.grader into the new option.
• config-graph.ts: defaults.target/defaults.grader validation incorrectly required a match against this same config document's inline targets:/graders: arrays — always empty in the common case where targets live in a separately-discovered .agentv/targets.yaml (true of this repo and every downstream consumer checked this session). Now only validates when those arrays are non-empty; otherwise resolution (and its "not found" errors) happens lazily at eval-run time, the same way CLI --grader-target already works.

Priority order is unchanged/additive: CLI --grader-target > target's own grader_target > defaults.grader (new) > self-grade (LLM-capable providers only).

Test plan

• New unit tests: defaultGraderTarget used as fallback when target has none; target's own grader_target still wins over defaultGraderTarget (priority ordering).
• New regression test: defaults.target/defaults.grader no longer throws when no inline targets:/graders: block exists in .agentv/config.yaml.
• bun test packages/core/test/evaluation/orchestrator.test.ts packages/core/test/evaluation/loaders/config-loader.test.ts — 191/191 pass.
• bun run typecheck (core + cli) — clean.
• bunx biome check on touched files — clean.
• Live dogfood: scratch .agentv/config.yaml with defaults: {target: agent-under-test, grader: grader-llm}, a real copilot-cli agent target with no grader_target, and a real Azure LLM grader-llm target. Before this change: hard error "... is an agent provider ... with no grader_target". After: runs and passes, and grading.json's assertion metadata shows "target": "grader-llm", confirming the config-level fallback actually routed grading to the separate LLM target rather than silently no-op'ing or erroring.

Note on naming

grader_target/defaults.grader/graders: all use the word "grader" to mean which provider judges, distinct from an assertion's type (the grading method, e.g. llm-rubric) and a rubric's criteria/value (the grading prompt). This PR doesn't rename anything — flagging the overload for a possible future naming pass now that defaults.grader is the primary path and per-target grader_target becomes the rarer override.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying agentv with    Cloudflare Pages

Latest commit:	c43888b
Status:	✅  Deploy successful!
Preview URL:	https://710e237a.agentv.pages.dev
Branch Preview URL:	https://feat-defaults-grader-fallbac.agentv.pages.dev

View logs