From 6e7c32267e7b10dba5289b8d957027ef5ab44063 Mon Sep 17 00:00:00 2001 From: Christopher Date: Mon, 6 Jul 2026 23:17:48 +1000 Subject: [PATCH] =?UTF-8?q?feat(config)!:=20hard-deprecate=20the=20graders?= =?UTF-8?q?:=20config=20block=20=E2=80=94=20a=20grader=20is=20just=20a=20t?= =?UTF-8?q?arget?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Peer-framework research this session (Margin-Lab/evals, Harbor) found neither maintains a separate grader/judge schema type distinct from their target/agent config — Margin-Lab's own docs say it outright: "there is no separate grader abstraction." AgentV's own design principles already prefer config-level grader selection through `defaults.grader` over target-level `grader_target`, and the live `.agentv/targets.yaml` execution path already resolves `grader_target`/`defaults.grader` against the same target-id namespace. The only place a separate "grader" entity still existed was the newer, dead `graders:` top-level config-graph schema (`NormalizedGraderConfig`) — zero implementation depth, nothing ever turned it into a Provider. A grader can also need grading itself (oracle/calibration checks), which only works for free if it's addressable the same way as any other target. - config-graph.ts: remove `NormalizedGraderConfig`/`parseGraders`/the `graders` field from `ComposableConfigGraph`; authoring a `graders:` block now throws a clear error pointing at folding entries into `targets`. `defaults.grader` now validates against the same `graph.targets` pool as `defaults.target`, instead of the removed `graph.graders`. - eval-file.schema.ts: same hard-rejection for the Zod schema backing the published eval.schema.json (IDE-autocomplete surface), matching the existing pattern for other removed fields (`eval_cases`, `providerPromptMap`). - config-loader.ts: drop the `graders` pass-through. - Updated the one place in docs, README.md, and the readme-quickstart example that authored a `graders:` block, folding each grader into `targets:`. readme-quickstart's targets.yaml/graders.yaml also still had the legacy `${{ VAR }}` env syntax (pre-existing, unrelated breakage) — fixed to `{{ env.VAR }}` in the same pass since the file needed touching anyway. - Added a CONCEPTS.md entry disambiguating grader-selection (a target) from grader-method (an assertion type) and grader-prompt (criteria/rubric). Verified: full workspace typecheck and lint clean; 935 tests across the touched loader/validation/orchestrator suites pass; all 109 example eval files validate; live CLI check confirms the error fires end-to-end and the folded-into-targets shape validates cleanly. Co-Authored-By: Claude Sonnet 5 --- CONCEPTS.md | 2 + README.md | 2 +- .../docs/next/evaluation/running-evals.mdx | 13 +++- .../docs/docs/next/targets/cli-provider.mdx | 4 +- .../docs/docs/next/targets/coding-agents.mdx | 6 +- .../docs/docs/next/targets/configuration.mdx | 20 ++++-- .../docs/next/targets/custom-providers.mdx | 2 +- .../docs/docs/next/targets/llm-providers.mdx | 3 +- .../readme-quickstart/.agentv/config.yaml | 1 - .../readme-quickstart/.agentv/graders.yaml | 7 -- .../readme-quickstart/.agentv/targets.yaml | 15 +++- examples/features/readme-quickstart/README.md | 5 +- .../src/evaluation/loaders/config-graph.ts | 69 +++++++------------ .../src/evaluation/loaders/config-loader.ts | 1 - .../evaluation/validation/eval-file.schema.ts | 15 ++-- .../evaluation/loaders/config-loader.test.ts | 50 +++++++++++--- .../validation/config-validator.test.ts | 9 ++- .../validation/eval-file-schema.test.ts | 29 +++++++- .../validation/eval-validator.test.ts | 2 +- .../references/eval.schema.json | 29 +------- 20 files changed, 156 insertions(+), 128 deletions(-) delete mode 100644 examples/features/readme-quickstart/.agentv/graders.yaml diff --git a/CONCEPTS.md b/CONCEPTS.md index cf7a7629f..a6dc6a30d 100644 --- a/CONCEPTS.md +++ b/CONCEPTS.md @@ -10,6 +10,8 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Target runtime** — The placement/transport mode for invoking a target provider, such as host execution, sandbox/container placement, CLI subprocess, app-server protocol, RPC, or SDK child runner. Runtime describes how the selected agent is invoked. Advanced home/env/profile-style overlays are provider or runtime configuration details, not the authored testbed recipe. Runtime is separate from the environment that prepares files, services, and cwd. +**Grader (selection)** — Not a distinct entity type. A grader is a regular target, listed under `targets` like any other and selected for a grading role via `defaults.grader`, CLI `--grader-target`, or an assertion-level target override. Authoring a separate `graders:` list is a hard error: move each entry into `targets`. This also means a grader can itself be the target of an ordinary eval (oracle/calibration checks), which a separate schema would block. Do not confuse this with **grader (method)** — an assertion `type` such as `llm-rubric`, `code-grader`, or `g-eval` that names *how* scoring works, independent of *which* target performs it; or with a rubric's `criteria`/`value`, which is the grading *prompt*. AgentV once called grader-selection "judge" (`judge_target`) before a deliberate, full rename to "grader" — do not reintroduce "judge" without a stronger reason than surface-level clarity, since peer frameworks (Margin-Lab/evals, Harbor) don't treat judge/grader as a first-class schema concept either. + **Provider runtime boundary** — The process boundary between AgentV's evaluation orchestrator and the agent runtime a provider invokes. CLI-backed providers place the agent runtime outside the orchestrator; SDK providers should run through an AgentV-owned child-runner boundary when runtime teardown can threaten run artifact finalization. This boundary does not own repository/testbed setup by default. ## Evaluation Model diff --git a/README.md b/README.md index c25f21282..dd171c3d7 100644 --- a/README.md +++ b/README.md @@ -44,9 +44,9 @@ targets: api_key: "{{ env.LOCAL_OPENAI_PROXY_API_KEY }}" model: "{{ env.LOCAL_OPENAI_PROXY_MODEL }}" -graders: - id: local-openai-grader provider: openai + runtime: host config: api_format: chat base_url: "{{ env.LOCAL_OPENAI_PROXY_BASE_URL }}" diff --git a/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx b/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx index 0c68186b5..716e467bc 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx @@ -474,7 +474,9 @@ default. ### YAML config (`config.yaml` plus optional `config.local.yaml`) Project config and eval YAML share the same eval-definition graph for targets, -graders, tests, defaults, and supported execution defaults. Small projects can keep that +tests, defaults, and supported execution defaults. A grader is not a separate +part of that graph — it is a target, listed under `targets` like any other, +selected for a grading role via `defaults.grader`. Small projects can keep the graph inline: ```yaml @@ -486,9 +488,9 @@ targets: command: ["codex", "app-server"] model: gpt-5-codex -graders: - id: openai-grader provider: openai + runtime: host config: model: gpt-5-mini @@ -511,7 +513,6 @@ directly: ```yaml targets: file://targets.yaml -graders: file://graders.yaml tests: file://tests.yaml defaults: file://defaults.yaml ``` @@ -523,6 +524,12 @@ defaults: file://defaults.yaml runtime: host config: command: ["codex", "app-server"] + +- id: openai-grader + provider: openai + runtime: host + config: + model: gpt-5-mini ``` ```yaml diff --git a/apps/web/src/content/docs/docs/next/targets/cli-provider.mdx b/apps/web/src/content/docs/docs/next/targets/cli-provider.mdx index 9f315291e..ce3dc7a11 100644 --- a/apps/web/src/content/docs/docs/next/targets/cli-provider.mdx +++ b/apps/web/src/content/docs/docs/next/targets/cli-provider.mdx @@ -21,9 +21,9 @@ targets: config: command: python agent.py --prompt {PROMPT} --out {OUTPUT_FILE} -graders: - id: azure-grader provider: azure + runtime: host config: endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}" api_key: "{{ env.AZURE_OPENAI_API_KEY }}" @@ -133,9 +133,9 @@ targets: config: command: cp fixtures/{EVAL_ID}.expected.txt {OUTPUT_FILE} -graders: - id: azure-grader provider: azure + runtime: host config: endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}" api_key: "{{ env.AZURE_OPENAI_API_KEY }}" diff --git a/apps/web/src/content/docs/docs/next/targets/coding-agents.mdx b/apps/web/src/content/docs/docs/next/targets/coding-agents.mdx index 70b8efcc4..11874025b 100644 --- a/apps/web/src/content/docs/docs/next/targets/coding-agents.mdx +++ b/apps/web/src/content/docs/docs/next/targets/coding-agents.mdx @@ -15,7 +15,9 @@ settings live under `config`. Use `defaults.grader`, CLI `--grader` / `--grader-target`, or an evaluator-specific target override for LLM-based grading. Grader selection is separate from the coding-agent target, so target definitions do not carry a -grader field. +grader field. A grader is not a distinct kind of entity either — it is a +regular target, listed under `targets` like any other, selected for a grading +role. ```yaml targets: @@ -27,9 +29,9 @@ targets: model: gpt-5-codex reasoning_effort: high -graders: - id: openai-grader provider: openai + runtime: host config: model: gpt-5-mini diff --git a/apps/web/src/content/docs/docs/next/targets/configuration.mdx b/apps/web/src/content/docs/docs/next/targets/configuration.mdx index cd8102389..49700d676 100644 --- a/apps/web/src/content/docs/docs/next/targets/configuration.mdx +++ b/apps/web/src/content/docs/docs/next/targets/configuration.mdx @@ -10,8 +10,8 @@ Targets define which agent or LLM provider to evaluate. AgentV uses one composable config graph across project manifests and eval files: - `.agentv/config.yaml` is the project-local discovery and composition root. It - can hold targets, graders, tests, defaults, execution policy, results - settings, and repo-local project policy. + can hold targets, tests, defaults, execution policy, results settings, and + repo-local project policy. - `$AGENTV_HOME/config.yaml` is the user/operator config. Use it for defaults that apply across projects, project registry data, default result locations, and provider defaults that should not be copied into each repo. @@ -43,9 +43,9 @@ targets: command: ["codex", "app-server"] model: gpt-5-codex -graders: - id: openai-grader provider: openai + runtime: host config: model: gpt-5-mini @@ -61,6 +61,11 @@ the shorthand for the current machine, or object form when you need settings belong under `config`. Process-backed coding-agent providers use `config.command` as a non-empty argv array. +A grader is not a separate kind of entity — it is a target selected for a +grading role, either through `defaults.grader` (shown above) or an +assertion-level target override. There is no separate `graders:` list; +authoring one is a hard error telling you to move each entry into `targets`. + ## Runtime Modes Use `runtime: host` when you want AgentV to run the target exactly as it is @@ -128,9 +133,9 @@ targets: command: ["codex", "app-server"] model: gpt-5-codex -graders: - id: openai-grader provider: openai + runtime: host config: model: gpt-5-mini @@ -151,7 +156,6 @@ can be decomposed like this: ```yaml targets: file://targets.yaml -graders: file://graders.yaml tests: file://tests.yaml defaults: file://defaults.yaml ``` @@ -167,6 +171,12 @@ a bare array, not an object wrapped in `targets:`: config: command: ["codex", "app-server"] model: gpt-5-codex + +- id: openai-grader + provider: openai + runtime: host + config: + model: gpt-5-mini ``` ```yaml diff --git a/apps/web/src/content/docs/docs/next/targets/custom-providers.mdx b/apps/web/src/content/docs/docs/next/targets/custom-providers.mdx index 1d90fc4a8..ac97d5122 100644 --- a/apps/web/src/content/docs/docs/next/targets/custom-providers.mdx +++ b/apps/web/src/content/docs/docs/next/targets/custom-providers.mdx @@ -184,9 +184,9 @@ targets: config: base_url: http://localhost:8080 -graders: - id: azure-grader provider: azure + runtime: host config: endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}" api_key: "{{ env.AZURE_OPENAI_API_KEY }}" diff --git a/apps/web/src/content/docs/docs/next/targets/llm-providers.mdx b/apps/web/src/content/docs/docs/next/targets/llm-providers.mdx index cdd43f262..41ae4e3eb 100644 --- a/apps/web/src/content/docs/docs/next/targets/llm-providers.mdx +++ b/apps/web/src/content/docs/docs/next/targets/llm-providers.mdx @@ -66,9 +66,10 @@ For smoke tests and dogfood runs against a local OpenAI-compatible proxy, keep the endpoint, model, and placeholder key in environment variables: ```yaml -graders: +targets: - id: local-openai-grader provider: openai + runtime: host config: api_format: chat base_url: "{{ env.LOCAL_OPENAI_PROXY_BASE_URL }}" diff --git a/examples/features/readme-quickstart/.agentv/config.yaml b/examples/features/readme-quickstart/.agentv/config.yaml index 15cd0c876..477390705 100644 --- a/examples/features/readme-quickstart/.agentv/config.yaml +++ b/examples/features/readme-quickstart/.agentv/config.yaml @@ -1,4 +1,3 @@ targets: file://targets.yaml -graders: file://graders.yaml tests: file://tests.yaml defaults: file://defaults.yaml diff --git a/examples/features/readme-quickstart/.agentv/graders.yaml b/examples/features/readme-quickstart/.agentv/graders.yaml deleted file mode 100644 index 2411d814d..000000000 --- a/examples/features/readme-quickstart/.agentv/graders.yaml +++ /dev/null @@ -1,7 +0,0 @@ -- id: local-openai-grader - provider: openai - config: - api_format: chat - base_url: ${{ LOCAL_OPENAI_PROXY_BASE_URL }} - api_key: ${{ LOCAL_OPENAI_PROXY_API_KEY }} - model: ${{ LOCAL_OPENAI_PROXY_MODEL }} diff --git a/examples/features/readme-quickstart/.agentv/targets.yaml b/examples/features/readme-quickstart/.agentv/targets.yaml index b5bb93e17..09f039a78 100644 --- a/examples/features/readme-quickstart/.agentv/targets.yaml +++ b/examples/features/readme-quickstart/.agentv/targets.yaml @@ -3,6 +3,15 @@ runtime: host config: api_format: chat - base_url: ${{ LOCAL_OPENAI_PROXY_BASE_URL }} - api_key: ${{ LOCAL_OPENAI_PROXY_API_KEY }} - model: ${{ LOCAL_OPENAI_PROXY_MODEL }} + base_url: "{{ env.LOCAL_OPENAI_PROXY_BASE_URL }}" + api_key: "{{ env.LOCAL_OPENAI_PROXY_API_KEY }}" + model: "{{ env.LOCAL_OPENAI_PROXY_MODEL }}" + +- id: local-openai-grader + provider: openai + runtime: host + config: + api_format: chat + base_url: "{{ env.LOCAL_OPENAI_PROXY_BASE_URL }}" + api_key: "{{ env.LOCAL_OPENAI_PROXY_API_KEY }}" + model: "{{ env.LOCAL_OPENAI_PROXY_MODEL }}" diff --git a/examples/features/readme-quickstart/README.md b/examples/features/readme-quickstart/README.md index 4ebd3a55d..4ba71673f 100644 --- a/examples/features/readme-quickstart/README.md +++ b/examples/features/readme-quickstart/README.md @@ -7,14 +7,15 @@ graph into direct field refs: ```yaml targets: file://targets.yaml -graders: file://graders.yaml tests: file://tests.yaml defaults: file://defaults.yaml ``` Each referenced file contains that field's value directly, such as a bare target array in `.agentv/targets.yaml` and a bare defaults object in -`.agentv/defaults.yaml`. +`.agentv/defaults.yaml`. A grader is not a separate kind of entity — it is a +target listed under `targets` like any other, selected for the grading role +via `defaults.grader`. Run it against a local OpenAI-compatible endpoint: diff --git a/packages/core/src/evaluation/loaders/config-graph.ts b/packages/core/src/evaluation/loaders/config-graph.ts index 50696ac58..a7691294b 100644 --- a/packages/core/src/evaluation/loaders/config-graph.ts +++ b/packages/core/src/evaluation/loaders/config-graph.ts @@ -61,12 +61,6 @@ export type NormalizedTargetConfig = { readonly config: Record; }; -export type NormalizedGraderConfig = { - readonly id: string; - readonly provider: string; - readonly config: Record; -}; - export type ConfigDefaults = { readonly target?: string; readonly grader?: string; @@ -78,7 +72,6 @@ export type ConfigExecution = { export type ComposableConfigGraph = { readonly targets?: readonly NormalizedTargetConfig[]; - readonly graders?: readonly NormalizedGraderConfig[]; readonly tests?: readonly unknown[]; readonly defaults?: ConfigDefaults; readonly execution?: ConfigExecution; @@ -124,13 +117,15 @@ export function normalizeComposableConfigGraph( configPath: string, options: NormalizeOptions = {}, ): ComposableConfigGraph { + if (rawConfig.graders !== undefined) { + throw new Error( + `Field 'graders' in ${configPath} has been removed. A grader is just a target — move each entry into 'targets' and select it via 'defaults.grader' or an assertion's target override, not a separate grader list.`, + ); + } const graph: ComposableConfigGraph = { ...(rawConfig.targets !== undefined ? { targets: parseTargets(rawConfig.targets, `${configPath}:targets`) } : {}), - ...(rawConfig.graders !== undefined - ? { graders: parseGraders(rawConfig.graders, `${configPath}:graders`) } - : {}), ...(rawConfig.tests !== undefined ? { tests: parseArray(rawConfig.tests, `${configPath}:tests`) } : {}), @@ -253,20 +248,6 @@ function parseRuntime(value: unknown, location: string): NormalizedRuntimeConfig throw new Error(`Invalid ${location}: use 'host' or an object with mode: host|profile|sandbox.`); } -function parseGraders(value: unknown, location: string): readonly NormalizedGraderConfig[] { - return parseArray(value, location).map((entry, index) => { - const graderLocation = `${location}[${index}]`; - if (!isPlainConfigObject(entry)) { - throw new Error(`Invalid ${graderLocation}: grader must be an object.`); - } - const id = readRequiredString(entry.id, `${graderLocation}.id`); - const provider = readRequiredString(entry.provider, `${graderLocation}.provider`); - const config = readOptionalObject(entry.config, `${graderLocation}.config`) ?? {}; - validateCommand(config.command, `${graderLocation}.config.command`); - return { id, provider, config }; - }); -} - function parseDefaults(value: unknown, location: string): ConfigDefaults { const defaults = readOptionalObject(value, location); if (!defaults) { @@ -315,27 +296,27 @@ function parseExecution( } function validateDefaultSelections(graph: ComposableConfigGraph, configPath: string): void { - // Only validated against targets/graders defined inline in this same config - // document. `defaults.target`/`defaults.grader` may instead name a target - // defined in a separately-discovered `.agentv/targets.yaml`, which this - // graph has no visibility into — that case is resolved (and, on an unknown - // name, reported) lazily at eval-run time, the same way CLI - // `--grader-target` already is. - if (graph.defaults?.target !== undefined && graph.targets && graph.targets.length > 0) { - const targetIds = new Set(graph.targets.map((target) => target.id)); - if (!targetIds.has(graph.defaults.target)) { - throw new Error( - `Invalid defaults.target in ${configPath}: '${graph.defaults.target}' does not match a configured target id.`, - ); - } + // A grader is just a target selected for a grading role, not a separate + // entity — `defaults.target` and `defaults.grader` both resolve against the + // same `targets` pool. Only validated against targets defined inline in + // this same config document; either may instead name a target defined in a + // separately-discovered `.agentv/targets.yaml`, which this graph has no + // visibility into — that case is resolved (and, on an unknown name, + // reported) lazily at eval-run time, the same way CLI `--grader-target` + // already is. + if (!graph.targets || graph.targets.length === 0) { + return; } - if (graph.defaults?.grader !== undefined && graph.graders && graph.graders.length > 0) { - const graderIds = new Set(graph.graders.map((grader) => grader.id)); - if (!graderIds.has(graph.defaults.grader)) { - throw new Error( - `Invalid defaults.grader in ${configPath}: '${graph.defaults.grader}' does not match a configured grader id.`, - ); - } + const targetIds = new Set(graph.targets.map((target) => target.id)); + if (graph.defaults?.target !== undefined && !targetIds.has(graph.defaults.target)) { + throw new Error( + `Invalid defaults.target in ${configPath}: '${graph.defaults.target}' does not match a configured target id.`, + ); + } + if (graph.defaults?.grader !== undefined && !targetIds.has(graph.defaults.grader)) { + throw new Error( + `Invalid defaults.grader in ${configPath}: '${graph.defaults.grader}' does not match a configured target id.`, + ); } } diff --git a/packages/core/src/evaluation/loaders/config-loader.ts b/packages/core/src/evaluation/loaders/config-loader.ts index 3803af8a2..9aa211a85 100644 --- a/packages/core/src/evaluation/loaders/config-loader.ts +++ b/packages/core/src/evaluation/loaders/config-loader.ts @@ -265,7 +265,6 @@ function parseConfigObject( ...(refs && { refs }), ...(tags && { tags }), ...(graph.targets && { targets: graph.targets }), - ...(graph.graders && { graders: graph.graders }), ...(graph.tests && { tests: graph.tests }), ...(graph.defaults && { defaults: graph.defaults }), configDir: projectDir, diff --git a/packages/core/src/evaluation/validation/eval-file.schema.ts b/packages/core/src/evaluation/validation/eval-file.schema.ts index 6ce12750c..21b59431e 100644 --- a/packages/core/src/evaluation/validation/eval-file.schema.ts +++ b/packages/core/src/evaluation/validation/eval-file.schema.ts @@ -834,14 +834,6 @@ const ConfigTargetSchema = z }) .strict(); -const ConfigGraderSchema = z - .object({ - id: z.string().min(1), - provider: z.string().min(1), - config: JsonRecordSchema.optional(), - }) - .strict(); - const ConfigDefaultsSchema = z .object({ target: z.string().min(1).optional(), @@ -918,7 +910,12 @@ export const EvalFileSchemaInput: z.ZodType = z.object({ }) .optional(), // Shared composable config graph fields - graders: z.union([z.array(ConfigGraderSchema), z.string().min(1)]).optional(), + graders: z + .never({ + invalid_type_error: + "Top-level 'graders' has been removed. A grader is just a target — move each entry into 'targets' and select it via 'defaults.grader' or an assertion's target override.", + }) + .optional(), defaults: z.union([ConfigDefaultsSchema, z.string().min(1)]).optional(), // Removed legacy aliases eval_cases: z diff --git a/packages/core/test/evaluation/loaders/config-loader.test.ts b/packages/core/test/evaluation/loaders/config-loader.test.ts index 9eb99adee..2460b972d 100644 --- a/packages/core/test/evaluation/loaders/config-loader.test.ts +++ b/packages/core/test/evaluation/loaders/config-loader.test.ts @@ -63,9 +63,9 @@ describe('loadConfig', () => { ' config:', ' command: ["codex", "app-server"]', ' model: gpt-5-codex', - 'graders:', ' - id: openai-grader', ' provider: openai', + ' runtime: host', ' config:', ' model: gpt-5-mini', 'tests:', @@ -89,9 +89,12 @@ describe('loadConfig', () => { runtime: { mode: 'host' }, config: { command: ['codex', 'app-server'], model: 'gpt-5-codex' }, }, - ]); - expect(config?.graders).toEqual([ - { id: 'openai-grader', provider: 'openai', config: { model: 'gpt-5-mini' } }, + { + id: 'openai-grader', + provider: 'openai', + runtime: { mode: 'host' }, + config: { model: 'gpt-5-mini' }, + }, ]); expect(config?.tests).toEqual([{ id: 'smoke', input: 'Fix the failing test' }]); expect(config?.defaults).toEqual({ target: 'codex-local', grader: 'openai-grader' }); @@ -117,9 +120,9 @@ describe('loadConfig', () => { ' home: .agentv/profiles/codex-local', ' config:', ' command: ["codex"]', - 'graders:', ' - id: openai-grader', ' provider: openai', + ' runtime: host', ' config: {}', 'tests:', ' - id: smoke', @@ -136,7 +139,6 @@ describe('loadConfig', () => { splitPath, [ 'targets: file://targets.yaml', - 'graders: file://graders.yaml', 'tests: file://tests.yaml', 'defaults: file://defaults.yaml', 'execution: file://execution.yaml', @@ -153,13 +155,13 @@ describe('loadConfig', () => { ' home: .agentv/profiles/codex-local', ' config:', ' command: ["codex"]', + '- id: openai-grader', + ' provider: openai', + ' runtime: host', + ' config: {}', '', ].join('\n'), ); - writeFileSync( - path.join(tempDir, 'graders.yaml'), - ['- id: openai-grader', ' provider: openai', ' config: {}', ''].join('\n'), - ); writeFileSync( path.join(tempDir, 'tests.yaml'), ['- id: smoke', ' input: Fix the failing test', ''].join('\n'), @@ -371,6 +373,34 @@ describe('loadConfig', () => { } }); + it('hard-rejects an authored graders: block — a grader is just a target', async () => { + const tempDir = mkdtempSync(path.join(os.tmpdir(), 'agentv-config-graph-graders-removed-')); + try { + const configPath = path.join(tempDir, 'config.yaml'); + writeFileSync( + configPath, + [ + 'targets:', + ' - id: codex-local', + ' provider: codex-app-server', + ' runtime: host', + ' config: {}', + 'graders:', + ' - id: openai-grader', + ' provider: openai', + ' config: {}', + '', + ].join('\n'), + ); + + await expect(loadComposableConfigGraph(configPath)).rejects.toThrow( + /'graders' in .+ has been removed.*move each entry into 'targets'/, + ); + } finally { + rmSync(tempDir, { recursive: true, force: true }); + } + }); + it('falls back to AGENTV_HOME/config.yaml when no project-local config exists', async () => { const tempDir = mkdtempSync(path.join(os.tmpdir(), 'agentv-global-config-')); try { diff --git a/packages/core/test/evaluation/validation/config-validator.test.ts b/packages/core/test/evaluation/validation/config-validator.test.ts index cdec5eadd..cd536be31 100644 --- a/packages/core/test/evaluation/validation/config-validator.test.ts +++ b/packages/core/test/evaluation/validation/config-validator.test.ts @@ -74,7 +74,6 @@ describe('validateConfigFile', () => { filePath, [ 'targets: file://targets.yaml', - 'graders: file://graders.yaml', 'tests: file://tests.yaml', 'defaults: file://defaults.yaml', 'execution: file://execution.yaml', @@ -89,13 +88,13 @@ describe('validateConfigFile', () => { ' runtime: host', ' config:', ' command: ["codex", "app-server"]', + '- id: openai-grader', + ' provider: openai', + ' runtime: host', + ' config: {}', '', ].join('\n'), ); - await writeFile( - path.join(path.dirname(filePath), 'graders.yaml'), - ['- id: openai-grader', ' provider: openai', ' config: {}', ''].join('\n'), - ); await writeFile( path.join(path.dirname(filePath), 'tests.yaml'), ['- id: smoke', ' input: Fix the failing test', ''].join('\n'), diff --git a/packages/core/test/evaluation/validation/eval-file-schema.test.ts b/packages/core/test/evaluation/validation/eval-file-schema.test.ts index c79cb849d..56521f0d0 100644 --- a/packages/core/test/evaluation/validation/eval-file-schema.test.ts +++ b/packages/core/test/evaluation/validation/eval-file-schema.test.ts @@ -221,11 +221,10 @@ describe('EvalFileSchema input shorthand', () => { runtime: 'host', config: { command: ['codex', 'app-server'] }, }, - ], - graders: [ { id: 'openai-grader', provider: 'openai', + runtime: 'host', config: { model: 'gpt-5-mini' }, }, ], @@ -239,6 +238,32 @@ describe('EvalFileSchema input shorthand', () => { expect(result.success).toBe(true); }); + it('rejects a top-level graders block — a grader is just a target', () => { + const result = EvalFileSchema.safeParse({ + targets: [ + { + id: 'codex-local', + provider: 'codex-app-server', + runtime: 'host', + config: { command: ['codex', 'app-server'] }, + }, + ], + graders: [ + { + id: 'openai-grader', + provider: 'openai', + config: { model: 'gpt-5-mini' }, + }, + ], + tests: [baseTest], + }); + + expect(result.success).toBe(false); + if (result.success) throw new Error('Expected graders to be rejected'); + const messages = collectIssueMessages(result.error.issues); + expect(messages.some((message) => message.includes("'graders' has been removed"))).toBe(true); + }); + it('rejects removed top-level runs and early_exit controls', () => { const result = EvalFileSchema.safeParse({ target: 'codex', diff --git a/packages/core/test/evaluation/validation/eval-validator.test.ts b/packages/core/test/evaluation/validation/eval-validator.test.ts index 15a2fdf94..005039d27 100644 --- a/packages/core/test/evaluation/validation/eval-validator.test.ts +++ b/packages/core/test/evaluation/validation/eval-validator.test.ts @@ -159,9 +159,9 @@ tests: runtime: host config: command: ["codex", "app-server"] -graders: - id: openai-grader provider: openai + runtime: host config: model: gpt-5-mini defaults: diff --git a/skills-data/agentv-eval-writer/references/eval.schema.json b/skills-data/agentv-eval-writer/references/eval.schema.json index 571c0bd38..e99f814ac 100644 --- a/skills-data/agentv-eval-writer/references/eval.schema.json +++ b/skills-data/agentv-eval-writer/references/eval.schema.json @@ -2019,34 +2019,7 @@ "not": {} }, "graders": { - "anyOf": [ - { - "type": "array", - "items": { - "type": "object", - "properties": { - "id": { - "type": "string", - "minLength": 1 - }, - "provider": { - "type": "string", - "minLength": 1 - }, - "config": { - "type": "object", - "additionalProperties": {} - } - }, - "required": ["id", "provider"], - "additionalProperties": false - } - }, - { - "type": "string", - "minLength": 1 - } - ] + "not": {} }, "defaults": { "anyOf": [