diff --git a/apps/web/src/content/docs/docs/next/graders/llm-graders.mdx b/apps/web/src/content/docs/docs/next/graders/llm-graders.mdx index 8ee69b580..959975e47 100644 --- a/apps/web/src/content/docs/docs/next/graders/llm-graders.mdx +++ b/apps/web/src/content/docs/docs/next/graders/llm-graders.mdx @@ -46,6 +46,40 @@ assert: Use `provider:` when you want different `llm-rubric` entries in the same eval to run on different grader models. This is useful for grader panels, majority-vote ensembles, and grader A/B benchmarks. +## Agent Rubrics + +Use `type: agent-rubric` when a Promptfoo-style rubric needs an agentic grader +that can inspect the workspace instead of only judging the final answer text. +AgentV routes `agent-rubric` through the same `llm-rubric` / `llm-grader` +scoring path, so results still appear as normal `EvaluationScore` entries. + +```yaml +assert: + - name: workspace-evidence + type: agent-rubric + provider: codex-grader + value: The answer's claims are backed by concrete files in the workspace. +``` + +The resolved grader provider must be agent-capable, such as a Codex, Claude, +Copilot, Pi, VS Code, or `agentv` provider. If `provider:` resolves to a +plain LLM grader provider, AgentV fails clearly instead of silently downgrading +the check. + +For agent-backed grading, AgentV creates a temporary `verdict.json` path for +each grading call and instructs the agent grader to write exactly one JSON +object there: + +```json +{ "pass": true, "score": 1, "reason": "Evidence found in src/logger.ts." } +``` + +AgentV reads that file before parsing the final assistant message. The `score` +must be a finite number from `0` to `1`; malformed verdict files fail closed as +grader failures. If the file is missing, AgentV may still parse the final +assistant text using the same structured JSON fallback used by other agent +grader modes. + ## Prompt Files The prompt file defines evaluation criteria and scoring guidelines. It can be a markdown text template or a TypeScript/JavaScript dynamic template. diff --git a/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx b/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx index 6be312b02..dc5586d7a 100644 --- a/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx +++ b/apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx @@ -72,7 +72,7 @@ implements equivalent semantics directly. | Authored preprocessors | Not Promptfoo's canonical output-shaping surface. | Rejected in current authored YAML. | Removed/rejected surface | Use `transform` at `default_test.options`, `tests[].options`, or the assertion that needs the shaped output. Historical versioned docs may still show old preprocessor examples. | | Suite assertions | `assert` entries can be strings or typed assertion objects. | `assert` entries can be strings, typed assertion objects, script graders, or AgentV extension graders. | Align with Promptfoo | Plain strings become semantic rubric checks. Use `assert`, not `assertions`, in current authored eval YAML. | | Assertion grouping | `type: assert-set` with child `assert` entries, optional `config`, `metric`, `weight`, and `threshold`. | `type: assert-set` with child `assert`, optional `config`, metric names, weights, and parent threshold. | Align with Promptfoo | Parent `config` is inherited by child assertions; child `config` keys override shared parent keys. Without `threshold`, pass/fail follows nonzero-weight child assertions. With `threshold`, the weighted aggregate score determines pass/fail. `type: composite` is rejected; use `assert-set`. | -| Deterministic assertion vocabulary | Common Promptfoo types include `contains`, `icontains`, `contains-any`, `contains-all`, `starts-with`, `regex`, `is-json`, `equals`, `latency`, `cost`, `javascript`, `python`, `webhook`, `similar`, and `llm-rubric`. | AgentV accepts the implemented overlap, including `contains`, `icontains`, `contains-any`, `contains-all`, `starts-with`, `regex`, `is-json`, `equals`, `latency`, `cost`, `javascript`, `python`, `webhook`, `similar`, and `llm-rubric`. | Align with Promptfoo | Unsupported Promptfoo assertion names error instead of silently becoming custom assertion names. | +| Deterministic and rubric assertion vocabulary | Common Promptfoo types include `contains`, `icontains`, `contains-any`, `contains-all`, `starts-with`, `regex`, `is-json`, `equals`, `latency`, `cost`, `javascript`, `python`, `webhook`, `similar`, `llm-rubric`, and `agent-rubric`. | AgentV accepts the implemented overlap, including `contains`, `icontains`, `contains-any`, `contains-all`, `starts-with`, `regex`, `is-json`, `equals`, `latency`, `cost`, `javascript`, `python`, `webhook`, `similar`, `llm-rubric`, and `agent-rubric`. | Align with Promptfoo | `agent-rubric` runs through AgentV-native agent grader providers and writes a structured verdict file before fallback text parsing. Unsupported Promptfoo assertion names error instead of silently becoming custom assertion names. | | Custom assertion terminology | Promptfoo calls normal eval custom logic assertions, with fixed code assertion types such as `javascript`, `python`, `ruby`, and `webhook`. | `defineAssertion()` files in `.agentv/assertions/` become reusable assertion type names. | Keep AgentV extension | AgentV keeps assertion terminology and extends discovery to arbitrary assertion type names such as `has-citation`. | | Script/custom grader terminology | Promptfoo custom code assertions are still assertion types. | `defineScriptGrader()` powers command-backed graders referenced with `type: script` and `command:`. | Keep AgentV divergence | Use script grader wording only for command-backed or LLM-backed scoring components that need explicit score and assertion-result control. | | Skill assertions | Promptfoo includes `skill-used` for checking whether an agent invoked a named skill. | `type: skill-used` and `type: not-skill-used` with `value: ` or a matcher object. | Align with Promptfoo | AgentV evaluates these against normalized tool-call and skill-use trace data. | diff --git a/packages/core/src/evaluation/graders/llm-grader.ts b/packages/core/src/evaluation/graders/llm-grader.ts index c6e82a7c4..9151f85bd 100644 --- a/packages/core/src/evaluation/graders/llm-grader.ts +++ b/packages/core/src/evaluation/graders/llm-grader.ts @@ -1,4 +1,5 @@ import fs from 'node:fs/promises'; +import { tmpdir } from 'node:os'; import path from 'node:path'; import { z } from 'zod'; @@ -142,6 +143,15 @@ const promptfooGradingResultSchema = z }) .passthrough(); +const agentVerdictFileSchema = z + .object({ + pass: z.boolean(), + score: z.number().finite().min(0).max(1), + reason: z.string(), + checks: z.array(promptfooCheckSchema).optional(), + }) + .passthrough(); + const rubricCheckResultSchema = z.object({ id: z.string().describe('The ID of the rubric item being checked'), satisfied: z.boolean().describe('Whether this rubric requirement is met'), @@ -216,15 +226,32 @@ function buildTemplateVariables(context: EvaluationContext): Record { - return config?.type === 'llm-grader' || config?.type === 'llm-rubric'; +): config is Extract< + GraderConfig, + { readonly type: 'llm-grader' | 'llm-rubric' | 'agent-rubric' } +> { + return ( + config?.type === 'llm-grader' || + config?.type === 'llm-rubric' || + config?.type === 'agent-rubric' + ); +} + +function isAgentRubricConfig(config: GraderConfig | undefined): boolean { + return config?.type === 'agent-rubric'; +} + +function isAgentCapableGraderProvider(provider: Provider | undefined): boolean { + return provider ? isAgentProvider(provider) || provider.kind === 'agentv' : false; } function resolveContentBasePath(context: EvaluationContext): string | undefined { @@ -437,6 +464,14 @@ export class LlmGrader implements Grader { // Delegate mode: grader target provider is an agent provider — send prompt via invoke() if (this.graderTargetProvider) { + if ( + isAgentRubricConfig(preparedContext.evaluator) && + !isAgentCapableGraderProvider(this.graderTargetProvider) + ) { + throw new Error( + `agent-rubric evaluator '${preparedContext.evaluator?.name ?? 'agent-rubric'}' requires an agent-capable grader provider`, + ); + } return this.evaluateWithGraderTarget(preparedContext); } @@ -444,6 +479,14 @@ export class LlmGrader implements Grader { if (!graderProvider) { throw new Error('No grader provider available for LLM grading'); } + if ( + isAgentRubricConfig(preparedContext.evaluator) && + !isAgentCapableGraderProvider(graderProvider) + ) { + throw new Error( + `agent-rubric evaluator '${preparedContext.evaluator?.name ?? 'agent-rubric'}' requires an agent-capable grader provider`, + ); + } // Built-in agent mode: agentv provider → provider.invoke() with filesystem tools if (graderProvider.kind === 'agentv') { @@ -755,19 +798,20 @@ export class LlmGrader implements Grader { ); } - const systemPrompt = this.buildAgentSystemPrompt(context); - const userPrompt = this.buildAgentUserPrompt(context); - const config = context.evaluator; const rubrics = getRubrics(config); const fsTools = createFilesystemTools(workspacePath); + const verdictFile = await createAgentVerdictFile(workspacePath); + const systemPrompt = this.buildAgentSystemPrompt(context, verdictFile.path); + const userPrompt = this.buildAgentUserPrompt(context, verdictFile.path); const graderRawRequest: JsonObject = { mode: 'built-in', systemPrompt, userPrompt, maxSteps: this.maxSteps, + verdict_path: verdictFile.path, }; try { @@ -791,6 +835,19 @@ export class LlmGrader implements Grader { tool_calls: toolCallCount, }; + const verdictScore = await this.parseAgentVerdictFile({ + verdictPath: verdictFile.path, + rubrics, + graderRawRequest, + details, + graderTarget: graderProvider.targetName, + config: context.evaluator, + fallbackText: promptfooFallbackText(context.evaluator, context.evalCase.criteria), + }); + if (verdictScore) { + return verdictScore; + } + return this.parseAgentResult( text, rubrics, @@ -811,6 +868,8 @@ export class LlmGrader implements Grader { graderTarget: graderProvider.targetName, details: { mode: 'built-in', error: message }, }; + } finally { + await verdictFile.cleanup(); } } @@ -848,13 +907,15 @@ export class LlmGrader implements Grader { provider: Provider, modeLabel: string, ): Promise { - const workspacePath = context.workspacePath; - const prompt = this.buildDelegatedPrompt(context); + const workspacePath = context.workspacePath ?? resolveContentBasePath(context) ?? process.cwd(); + const verdictFile = await createAgentVerdictFile(workspacePath); + const prompt = this.buildDelegatedPrompt(context, verdictFile.path); const graderRawRequest: JsonObject = { mode: modeLabel, grader_target: provider.targetName, prompt, + verdict_path: verdictFile.path, }; try { @@ -888,6 +949,19 @@ export class LlmGrader implements Grader { grader_target: provider.targetName, }; + const verdictScore = await this.parseAgentVerdictFile({ + verdictPath: verdictFile.path, + rubrics, + graderRawRequest, + details, + graderTarget: provider.targetName, + config: context.evaluator, + fallbackText: promptfooFallbackText(context.evaluator, context.evalCase.criteria), + }); + if (verdictScore) { + return verdictScore; + } + return this.parseAgentResult( assistantContent, rubrics, @@ -914,6 +988,8 @@ export class LlmGrader implements Grader { error: message, }, }; + } finally { + await verdictFile.cleanup(); } } @@ -925,7 +1001,7 @@ export class LlmGrader implements Grader { * Build system prompt for built-in agent mode. * Includes output format instructions. */ - private buildAgentSystemPrompt(context: EvaluationContext): string { + private buildAgentSystemPrompt(context: EvaluationContext, verdictPath?: string): string { const config = context.evaluator; const rubrics = getRubrics(config); @@ -943,6 +1019,9 @@ export class LlmGrader implements Grader { } else { parts.push(buildOutputSchema()); } + if (verdictPath) { + parts.push('', buildAgentVerdictFileInstructions(verdictPath)); + } return parts.join('\n'); } @@ -951,7 +1030,7 @@ export class LlmGrader implements Grader { * Build user prompt for built-in agent mode. * Uses custom template if provided, otherwise builds default prompt. */ - private buildAgentUserPrompt(context: EvaluationContext): string { + private buildAgentUserPrompt(context: EvaluationContext, verdictPath?: string): string { const formattedQuestion = context.promptInputs.question && context.promptInputs.question.trim().length > 0 ? context.promptInputs.question @@ -1010,6 +1089,10 @@ export class LlmGrader implements Grader { ); } + if (verdictPath) { + parts.push('', buildAgentVerdictFileInstructions(verdictPath)); + } + return parts.join('\n'); } @@ -1017,7 +1100,7 @@ export class LlmGrader implements Grader { * Build the full evaluation prompt for delegate mode (agent providers). * Combines task context, criteria, candidate info, and output format instructions. */ - private buildDelegatedPrompt(context: EvaluationContext): string { + private buildDelegatedPrompt(context: EvaluationContext, verdictPath?: string): string { const formattedQuestion = context.promptInputs.question && context.promptInputs.question.trim().length > 0 ? context.promptInputs.question @@ -1039,7 +1122,9 @@ export class LlmGrader implements Grader { : buildRubricFormatInstructions() : buildOutputSchema(); - return `${customPrompt}\n\n${outputSchema}`; + return `${customPrompt}\n\n${outputSchema}${ + verdictPath ? `\n\n${buildAgentVerdictFileInstructions(verdictPath)}` : '' + }`; } const parts: string[] = [ @@ -1081,6 +1166,9 @@ export class LlmGrader implements Grader { } else { parts.push(buildOutputSchema()); } + if (verdictPath) { + parts.push('', buildAgentVerdictFileInstructions(verdictPath)); + } return parts.join('\n'); } @@ -1190,6 +1278,62 @@ export class LlmGrader implements Grader { } } + private async parseAgentVerdictFile(options: { + readonly verdictPath: string; + readonly rubrics: readonly RubricItem[] | undefined; + readonly graderRawRequest: JsonObject; + readonly details: JsonObject; + readonly graderTarget?: string; + readonly config?: GraderConfig; + readonly fallbackText: string; + }): Promise { + let content: string; + try { + content = await fs.readFile(options.verdictPath, 'utf8'); + } catch (error) { + if (isMissingFileError(error)) { + return undefined; + } + return buildAgentVerdictFailure({ + error: error instanceof Error ? error.message : String(error), + graderRawRequest: options.graderRawRequest, + details: options.details, + graderTarget: options.graderTarget, + }); + } + + try { + const raw = JSON.parse(content); + const parsed = agentVerdictFileSchema.parse(raw); + const normalized = normalizePromptfooGradingResult({ + raw: parsed, + config: options.config, + fallbackText: options.fallbackText, + rubrics: options.rubrics, + }); + return { + score: normalized.score, + verdict: normalized.verdict, + assertions: normalized.assertions, + expectedAspectCount: options.rubrics?.length ?? Math.max(normalized.assertions.length, 1), + graderRawRequest: options.graderRawRequest, + graderTarget: options.graderTarget, + details: { + ...options.details, + ...normalized.details, + verdict_source: 'file', + }, + }; + } catch (error) { + return buildAgentVerdictFailure({ + error: error instanceof Error ? error.message : String(error), + graderRawRequest: options.graderRawRequest, + details: options.details, + graderTarget: options.graderTarget, + }); + } + } + // --------------------------------------------------------------------------- // LLM mode prompt builders // --------------------------------------------------------------------------- @@ -1474,6 +1618,69 @@ export function buildOutputSchema(): string { ].join('\n'); } +function buildAgentVerdictFileInstructions(verdictPath: string): string { + return [ + 'Agent grader verdict file contract:', + `Write exactly one JSON object to the verdict JSON file at: ${verdictPath}`, + 'The JSON object must match this shape: {"pass": boolean, "score": number between 0.0 and 1.0, "reason": string}.', + 'Do not write Markdown, comments, arrays, or multiple JSON objects to the verdict file.', + 'After writing the file, your final assistant message may be brief; AgentV reads the verdict file first.', + ].join('\n'); +} + +async function createAgentVerdictFile( + workspacePath: string | undefined, +): Promise<{ readonly path: string; cleanup(): Promise }> { + const root = workspacePath + ? path.join(workspacePath, '.agentv', 'tmp') + : path.join(tmpdir(), 'agentv-grader'); + await fs.mkdir(root, { recursive: true }); + const directory = await fs.mkdtemp(path.join(root, 'grader-verdict-')); + const verdictPath = path.join(directory, 'verdict.json'); + return { + path: verdictPath, + async cleanup() { + await fs.rm(directory, { recursive: true, force: true }); + }, + }; +} + +function isMissingFileError(error: unknown): boolean { + return ( + typeof error === 'object' && + error !== null && + 'code' in error && + (error as { code?: unknown }).code === 'ENOENT' + ); +} + +function buildAgentVerdictFailure(options: { + readonly error: string; + readonly graderRawRequest: JsonObject; + readonly details: JsonObject; + readonly graderTarget?: string; +}): EvaluationScore { + return { + score: 0, + verdict: 'fail', + assertions: [ + { + text: 'Failed to parse llm-grader agent verdict file as valid evaluation JSON', + passed: false, + evidence: options.error, + }, + ], + expectedAspectCount: 1, + graderRawRequest: options.graderRawRequest, + graderTarget: options.graderTarget, + details: { + ...options.details, + verdict_source: 'file', + verdict_file_error: options.error, + }, + }; +} + export function buildPromptfooRubricOutputSchema(): string { return [ 'You must respond with a single JSON object matching this schema:', diff --git a/packages/core/src/evaluation/loaders/grader-parser.ts b/packages/core/src/evaluation/loaders/grader-parser.ts index 814be2ce4..f29f4f1f9 100644 --- a/packages/core/src/evaluation/loaders/grader-parser.ts +++ b/packages/core/src/evaluation/loaders/grader-parser.ts @@ -67,7 +67,6 @@ function removedGraderReplacement(type: string): string | undefined { } const UNSUPPORTED_PROMPTFOO_ASSERTION_TYPES = new Set([ - 'agent-rubric', 'answer-relevance', 'bleu', 'classifier', @@ -1684,7 +1683,10 @@ async function parseGraderList( // Parse prompt field - can be string (text template) or object (executable script) const rawPrompt = - rawEvaluator.prompt ?? (typeValue === 'llm-rubric' ? defaultRubricPrompt : undefined); + rawEvaluator.prompt ?? + (typeValue === 'llm-rubric' || typeValue === 'agent-rubric' + ? defaultRubricPrompt + : undefined); const parsedPrompt = await parsePromptField(rawPrompt, name, evalId, searchRoots); const { prompt, promptPath, resolvedPromptPath, resolvedPromptScript, promptScriptConfig } = parsedPrompt; @@ -1758,11 +1760,11 @@ async function parseGraderList( const llmTemperature = typeof rawTempLlm === 'number' && rawTempLlm >= 0 && rawTempLlm <= 2 ? rawTempLlm : undefined; - if (typeValue === 'llm-rubric') { + if (typeValue === 'llm-rubric' || typeValue === 'agent-rubric') { for (const removedField of ['criteria', 'rubric_item', 'rubricItem', 'rubrics'] as const) { if (rawEvaluator[removedField] !== undefined) { throw new Error( - `Unsupported llm-rubric field '${removedField}' in '${evalId}' for evaluator '${name}'. Use 'value' instead.`, + `Unsupported ${typeValue} field '${removedField}' in '${evalId}' for evaluator '${name}'. Use 'value' instead.`, ); } } @@ -1783,14 +1785,14 @@ async function parseGraderList( if (!value && (!structuredRubrics || structuredRubrics.length === 0) && !prompt) { logWarning( - `Skipping llm-rubric evaluator '${name}' in '${evalId}': expected value or prompt`, + `Skipping ${typeValue} evaluator '${name}' in '${evalId}': expected value or prompt`, ); continue; } pushEvaluator({ name, - type: 'llm-rubric', + type: typeValue, prompt, promptPath, ...(resolvedPromptPath ? { resolvedPromptPath } : {}), diff --git a/packages/core/src/evaluation/registry/builtin-graders.ts b/packages/core/src/evaluation/registry/builtin-graders.ts index a3df32524..260212bce 100644 --- a/packages/core/src/evaluation/registry/builtin-graders.ts +++ b/packages/core/src/evaluation/registry/builtin-graders.ts @@ -83,6 +83,10 @@ function formatRubricValue(value: unknown): string { return typeof value === 'string' ? value : JSON.stringify(value, null, 2); } +function isAgentCapableGraderProvider(provider: Provider | undefined): boolean { + return provider ? isAgentProvider(provider) || provider.kind === 'agentv' : false; +} + /** * Factory for `llm-grader` evaluators. * Creates a wrapper that resolves custom prompts at evaluation time and @@ -105,14 +109,19 @@ export const llmGraderFactory: GraderFactoryFn = (config, context) => { } if (!graderTargetProvider) { throw new Error( - `llm-grader evaluator '${c.name}': target '${c.target}' not found in targets`, + `llm-grader evaluator '${c.name}': provider '${c.target}' not found in configured providers`, ); } // Only pass graderTargetProvider for agent providers (delegate mode). // LLM providers use the normal resolveGraderProvider path for structured JSON mode. // The agentv provider drives the built-in agent loop directly, so include // it alongside AGENT_PROVIDER_KINDS even though it doesn't spawn a subprocess. - const isAgent = isAgentProvider(graderTargetProvider) || graderTargetProvider.kind === 'agentv'; + const isAgent = isAgentCapableGraderProvider(graderTargetProvider); + if (c.type === 'agent-rubric' && !isAgent) { + throw new Error( + `agent-rubric evaluator '${c.name}': provider '${c.target}' must resolve to an agent-capable grader provider`, + ); + } evaluator = new LlmGrader({ resolveGraderProvider: async (evalContext) => { if (graderTargetProvider) return graderTargetProvider; @@ -161,7 +170,7 @@ export const llmGraderFactory: GraderFactoryFn = (config, context) => { let graderTemplateOverride: string | undefined; let evalCase = evalContext.evalCase; - if (c.type === 'llm-rubric' && c.value !== undefined) { + if ((c.type === 'llm-rubric' || c.type === 'agent-rubric') && c.value !== undefined) { evalCase = { ...evalCase, criteria: formatRubricValue(c.value) }; } if (customPrompt) { @@ -186,6 +195,9 @@ export const llmGraderFactory: GraderFactoryFn = (config, context) => { export const llmRubricFactory: GraderFactoryFn = (config, context) => llmGraderFactory(config as LlmRubricGraderConfig, context); +export const agentRubricFactory: GraderFactoryFn = (config, context) => + llmGraderFactory(config as import('../types.js').AgentRubricGraderConfig, context); + /** Factory for subprocess-backed script evaluators. */ export const scriptFactory: GraderFactoryFn = (config, context) => { const c = config as ScriptGraderConfig; @@ -427,6 +439,7 @@ export function createBuiltinRegistry(): GraderRegistry { registry .register('llm-grader', llmGraderFactory) .register('llm-rubric', llmRubricFactory) + .register('agent-rubric', agentRubricFactory) .register('script', scriptFactory) .register('tool-trajectory', toolTrajectoryFactory) .register('trajectory:tool-used', trajectoryFactory) diff --git a/packages/core/src/evaluation/types.ts b/packages/core/src/evaluation/types.ts index bc8755b78..4c331f9c3 100644 --- a/packages/core/src/evaluation/types.ts +++ b/packages/core/src/evaluation/types.ts @@ -194,6 +194,7 @@ const GRADER_KIND_VALUES = [ 'execution-metrics', 'assert-set', 'llm-rubric', + 'agent-rubric', 'contains', 'contains-any', 'contains-all', @@ -472,7 +473,7 @@ export type LlmGraderConfig = { readonly min_score?: number; /** When true, inverts the grader score (1 - score) and swaps pass/fail verdict */ readonly negate?: boolean; - /** Optional target override for this grader (uses a named LLM target from targets.yaml). */ + /** Internal normalized provider override for this grader (authored as assertion `provider`). */ readonly target?: string; /** Pass-through configuration for custom evaluator prompts (legacy, prefer prompt.config) */ readonly config?: Record; @@ -490,7 +491,14 @@ export type LlmRubricGraderConfig = Omit & { readonly value?: JsonValue; }; -export type LlmBackedGraderConfig = LlmGraderConfig | LlmRubricGraderConfig; +export type AgentRubricGraderConfig = Omit & { + readonly type: 'agent-rubric'; +}; + +export type LlmBackedGraderConfig = + | LlmGraderConfig + | LlmRubricGraderConfig + | AgentRubricGraderConfig; /** * Score range definition for analytic rubric scoring. @@ -955,6 +963,7 @@ export type GraderConfig = ( | CodeGraderConfig | LlmGraderConfig | LlmRubricGraderConfig + | AgentRubricGraderConfig | ToolTrajectoryGraderConfig | SkillUsedGraderConfig | FieldAccuracyGraderConfig diff --git a/packages/core/src/evaluation/validation/eval-file.schema.ts b/packages/core/src/evaluation/validation/eval-file.schema.ts index 0165b4722..ea95d4b9d 100644 --- a/packages/core/src/evaluation/validation/eval-file.schema.ts +++ b/packages/core/src/evaluation/validation/eval-file.schema.ts @@ -309,6 +309,7 @@ const PromptfooAssertionSchema = EvaluatorCommonSchema.extend({ type: z.enum([ 'assert-set', 'llm-rubric', + 'agent-rubric', 'javascript', 'python', 'webhook', diff --git a/packages/core/src/evaluation/validation/eval-validator.ts b/packages/core/src/evaluation/validation/eval-validator.ts index 3acfb958c..673e83396 100644 --- a/packages/core/src/evaluation/validation/eval-validator.ts +++ b/packages/core/src/evaluation/validation/eval-validator.ts @@ -52,6 +52,7 @@ const ASSERTION_TYPES_WITH_ARRAY_VALUE = new Set([ const PROMPTFOO_ASSERTION_TYPES = new Set([ 'assert-set', 'llm-rubric', + 'agent-rubric', 'javascript', 'python', 'webhook', @@ -70,7 +71,6 @@ const REMOVED_ASSERTION_TYPE_REPLACEMENTS = new Map([ ]); const UNSUPPORTED_PROMPTFOO_ASSERTION_TYPES = new Set([ - 'agent-rubric', 'answer-relevance', 'bleu', 'classifier', diff --git a/packages/core/test/evaluation/graders.test.ts b/packages/core/test/evaluation/graders.test.ts index bcf659b9a..ad5c21ac9 100644 --- a/packages/core/test/evaluation/graders.test.ts +++ b/packages/core/test/evaluation/graders.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it, spyOn } from 'bun:test'; -import { mkdtemp, rm } from 'node:fs/promises'; +import { mkdtemp, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { dirname, join } from 'node:path'; import { fileURLToPath } from 'node:url'; @@ -60,6 +60,29 @@ class CapturingProvider implements Provider { } } +class VerdictWritingAgentProvider implements Provider { + readonly id = 'agent'; + readonly kind = 'codex-cli' as const; + readonly targetName = 'agent'; + lastRequest?: ProviderRequest; + lastVerdictPath?: string; + + constructor(private readonly verdict: string | Record) {} + + async invoke(request: ProviderRequest): Promise { + this.lastRequest = request; + const prompt = `${request.systemPrompt ?? ''}\n${request.question}`; + const match = prompt.match(/\/[^\s'"]*verdict\.json/); + if (!match) { + throw new Error('No verdict file path found in grader prompt'); + } + this.lastVerdictPath = match[0]; + const content = typeof this.verdict === 'string' ? this.verdict : JSON.stringify(this.verdict); + await writeFile(match[0], content); + return textResponse('assistant text should not be parsed when verdict file exists'); + } +} + class SequenceCapturingProvider implements Provider { readonly id = 'sequence-capturing'; readonly kind = 'mock' as const; @@ -1381,6 +1404,140 @@ describe('LlmGrader (llm-grader)', () => { // Should NOT contain the default template's structure expect(userPrompt).not.toContain('[[ ## answer ## ]]'); }); + + it('uses an agent grader verdict file before final assistant text', async () => { + const tempDir = await mkdtemp(join(tmpdir(), 'agentv-agent-rubric-')); + const graderProvider = new VerdictWritingAgentProvider({ + pass: true, + score: 0.9, + reason: 'Agent inspected the workspace evidence.', + }); + + try { + const evaluator = new LlmGrader({ + resolveGraderProvider: async () => graderProvider, + }); + + const result = await evaluator.evaluate({ + evalCase: { ...baseTestCase, evaluator: 'agent-rubric' }, + candidate: 'Answer', + target: baseTarget, + provider: graderProvider, + attempt: 0, + promptInputs: { question: '' }, + now: new Date(), + workspacePath: tempDir, + evaluator: { + name: 'agent-rubric', + type: 'agent-rubric', + value: 'Inspect the workspace evidence', + }, + }); + + expect(result.score).toBeCloseTo(0.9); + expect(result.verdict).toBe('pass'); + expect(result.assertions[0]?.evidence).toBe('Agent inspected the workspace evidence.'); + expect(result.details?.verdict_source).toBe('file'); + expect(graderProvider.lastRequest?.question).toContain('verdict.json'); + } finally { + await rm(tempDir, { recursive: true, force: true }); + } + }); + + it('creates delegate agent verdict files under target cwd when workspacePath is absent', async () => { + const tempDir = await mkdtemp(join(tmpdir(), 'agentv-agent-rubric-cwd-')); + const graderProvider = new VerdictWritingAgentProvider({ + pass: true, + score: 1, + reason: 'Verdict file was writable under the eval cwd.', + }); + + try { + const evaluator = new LlmGrader({ + resolveGraderProvider: async () => graderProvider, + }); + + const result = await evaluator.evaluate({ + evalCase: { ...baseTestCase, evaluator: 'agent-rubric' }, + candidate: 'Answer', + target: { ...baseTarget, config: { cwd: tempDir } }, + provider: graderProvider, + attempt: 0, + promptInputs: { question: '' }, + now: new Date(), + evaluator: { + name: 'agent-rubric', + type: 'agent-rubric', + value: 'Inspect writable eval cwd evidence', + }, + }); + + expect(result.verdict).toBe('pass'); + expect(result.details?.verdict_source).toBe('file'); + expect(graderProvider.lastRequest?.cwd).toBe(tempDir); + expect(graderProvider.lastVerdictPath?.startsWith(join(tempDir, '.agentv', 'tmp'))).toBe( + true, + ); + } finally { + await rm(tempDir, { recursive: true, force: true }); + } + }); + + it('fails closed when an agent grader verdict file has invalid JSON', async () => { + const tempDir = await mkdtemp(join(tmpdir(), 'agentv-agent-rubric-invalid-')); + const graderProvider = new VerdictWritingAgentProvider('{not json'); + + try { + const evaluator = new LlmGrader({ + resolveGraderProvider: async () => graderProvider, + }); + + const result = await evaluator.evaluate({ + evalCase: { ...baseTestCase, evaluator: 'agent-rubric' }, + candidate: 'Answer', + target: baseTarget, + provider: graderProvider, + attempt: 0, + promptInputs: { question: '' }, + now: new Date(), + workspacePath: tempDir, + evaluator: { + name: 'agent-rubric', + type: 'agent-rubric', + value: 'Inspect the workspace evidence', + }, + }); + + expect(result.verdict).toBe('fail'); + expect(result.assertions[0]?.text).toContain('verdict file'); + expect(result.details?.verdict_source).toBe('file'); + } finally { + await rm(tempDir, { recursive: true, force: true }); + } + }); + + it('rejects agent-rubric when an explicit grader provider is not agent-capable', () => { + const graderProvider = new CapturingProvider(textResponse('{}'), 'plain-llm'); + + expect(() => + llmGraderFactory( + { + name: 'agent-rubric', + type: 'agent-rubric', + value: 'Inspect the workspace evidence', + target: 'plain-llm', + }, + { + graderProvider, + targetResolver: () => graderProvider, + llmGrader: new LlmGrader({ + resolveGraderProvider: async () => graderProvider, + }), + registry: {} as never, + }, + ), + ).toThrow('agent-rubric evaluator'); + }); }); describe('ScriptGrader', () => { diff --git a/packages/core/test/evaluation/loaders/grader-parser.test.ts b/packages/core/test/evaluation/loaders/grader-parser.test.ts index 0d0918f12..bf2f92967 100644 --- a/packages/core/test/evaluation/loaders/grader-parser.test.ts +++ b/packages/core/test/evaluation/loaders/grader-parser.test.ts @@ -380,6 +380,33 @@ describe('parseGraders - deterministic assertion types', () => { }); }); + it('parses promptfoo-compatible agent-rubric as an agent-backed rubric grader', async () => { + const evaluators = await parseGraders( + { + assert: [ + { + metric: 'agent-check', + type: 'agent-rubric', + value: 'Inspect the workspace and verify the claimed file exists', + provider: 'codex-grader', + max_steps: 4, + }, + ], + }, + undefined, + [tempDir], + 'test-1', + ); + + expect(evaluators?.[0]).toMatchObject({ + name: 'agent-check', + type: 'agent-rubric', + value: 'Inspect the workspace and verify the claimed file exists', + target: 'codex-grader', + max_steps: 4, + }); + }); + it('inherits default rubric prompt for llm-rubric assertions only', async () => { const evaluators = await parseGraders( { diff --git a/packages/core/test/evaluation/validation/eval-file-schema.test.ts b/packages/core/test/evaluation/validation/eval-file-schema.test.ts index 5400eec14..685a068ab 100644 --- a/packages/core/test/evaluation/validation/eval-file-schema.test.ts +++ b/packages/core/test/evaluation/validation/eval-file-schema.test.ts @@ -577,6 +577,25 @@ describe('EvalFileSchema input shorthand', () => { expect(result.success).toBe(true); }); + it('accepts promptfoo agent-rubric assertions in schema validation', () => { + const result = EvalFileSchema.safeParse({ + tests: [ + { + ...baseTest, + assert: [ + { + type: 'agent-rubric', + value: 'Inspect the workspace and verify the evidence', + provider: 'codex-grader', + }, + ], + }, + ], + }); + + expect(result.success).toBe(true); + }); + it('rejects stale skill-trigger assertions with migration guidance', () => { const positive = EvalFileSchema.safeParse({ tests: [ diff --git a/packages/core/test/evaluation/validation/eval-validator.test.ts b/packages/core/test/evaluation/validation/eval-validator.test.ts index 0d788ce52..a754b77a9 100644 --- a/packages/core/test/evaluation/validation/eval-validator.test.ts +++ b/packages/core/test/evaluation/validation/eval-validator.test.ts @@ -2025,6 +2025,27 @@ tests: expect(result.errors.filter((e) => e.severity === 'error')).toHaveLength(0); }); + it('accepts promptfoo agent-rubric assertions', async () => { + const filePath = path.join(tempDir, 'assert-agent-rubric-supported.yaml'); + await writeFile( + filePath, + `prompts: + - "Review the answer" +tests: + - id: test-1 + assert: + - type: agent-rubric + value: "Inspect the workspace and verify the evidence" + provider: codex-grader +`, + ); + + const result = await validateEvalFile(filePath); + + expect(result.valid).toBe(true); + expect(result.errors.filter((e) => e.severity === 'error')).toHaveLength(0); + }); + it('accepts promptfoo-compatible skill-used assertions', async () => { const filePath = path.join(tempDir, 'assert-skill-used.yaml'); await writeFile( diff --git a/packages/sdk/src/assertion.ts b/packages/sdk/src/assertion.ts index 4c74c3e9c..d5075d0c6 100644 --- a/packages/sdk/src/assertion.ts +++ b/packages/sdk/src/assertion.ts @@ -39,6 +39,7 @@ export type AssertionType = // kebab-case (canonical internal form) | 'llm-grader' | 'llm-rubric' + | 'agent-rubric' | 'script' | 'assert-set' /** @deprecated Authored eval YAML rejects this compatibility-only runtime type. */ diff --git a/skills-data/agentv-eval-writer/SKILL.md b/skills-data/agentv-eval-writer/SKILL.md index 6bebb0cef..21a7f4c28 100644 --- a/skills-data/agentv-eval-writer/SKILL.md +++ b/skills-data/agentv-eval-writer/SKILL.md @@ -39,7 +39,7 @@ Use `@agentv/sdk` for TypeScript helper imports. Do not use `@agentv/eval` for n ## Authoring Checklist - Put grading criteria in `assert`, not in test-level `criteria`. Plain assertion strings become an `llm-rubric` grader. -- Prefer plain assertion strings for semantic checks when the default rubric grader can judge them. Use `type: llm-rubric` for structured criteria, custom prompts, custom grader targets, or assertion-level transforms, and `type: script` when grading must execute code. +- Prefer plain assertion strings for semantic checks when the default rubric grader can judge them. Use `type: llm-rubric` for structured criteria, custom prompts, custom grader providers, or assertion-level transforms. Use `type: agent-rubric` when the grader itself must be an agent-capable provider that can inspect the workspace. Use `type: script` when grading must execute code. - Put reference answers in `tests[].vars.expected_output` or `default_test.vars.expected_output`, and consume them with an explicit assertion such as `type: llm-rubric` with `value: "Matches the reference answer: {{ expected_output }}"`. Do not write criteria, scoring instructions, or "the agent should..." rubric prose as the reference answer. - For historical or repo-state evals, materialize the repo through a pinned `environment` setup recipe. Mentioning a SHA only in prompt prose is not enough because the agent needs an actual checkout to inspect. @@ -151,7 +151,7 @@ tests: | `id` | yes | Unique identifier | | `vars` | yes when the prompt needs row data | Prompt-template variables for this row | | `vars.expected_output` | no | Conventional reference-answer var consumed by explicit graders | -| `assert` | yes | Graders: deterministic checks, `llm-rubric` checks, script graders, or plain string rubric criteria | +| `assert` | yes | Graders: deterministic checks, `llm-rubric` / `agent-rubric` checks, script graders, or plain string rubric criteria | | `execution` | no | Per-case grader/default overrides such as `skip_defaults`; target selection belongs in top-level `target` or CLI `--target` | | `environment` | no | Per-case coding-agent testbed config (overrides suite-level) | | `metadata` | no | Arbitrary key-value pairs passed to setup/teardown scripts | @@ -387,7 +387,7 @@ tests: When `assert` is defined, **only the declared graders run**. For semantic checks, add plain rubric strings. If you need a custom LLM prompt or -grader target, declare `llm-rubric` explicitly: +grader provider, declare `llm-rubric` explicitly: ```yaml prompts: @@ -560,7 +560,7 @@ See the Script Graders docs for the full stdin/stdout contract. - name: quality type: llm-rubric prompt: ./prompts/eval.md # markdown template or command config - target: grader_gpt_5_mini # optional: override the grader target for this grader + provider: grader_gpt_5_mini # optional: override the grader provider for this grader model: gpt-5-chat # optional model override config: # passed to prompt templates as context.config strictness: high @@ -568,7 +568,7 @@ See the Script Graders docs for the full stdin/stdout contract. Variables: `{{criteria}}`, `{{input}}`, `{{expected_output}}`, `{{output}}`, `{{metadata}}`, `{{metadata_json}}`, `{{rubrics}}`, `{{rubrics_json}}`, `{{file_changes}}`, `{{tool_calls}}` - Markdown templates: use `{{variable}}` syntax - TypeScript templates: use `definePromptTemplate(fn)` from `@agentv/sdk`, receives context object with all variables + `config` -- Use `target:` to run different `llm-rubric` graders against different named LLM targets in the same eval (useful for grader panels / ensembles) +- Use `provider:` to run different `llm-rubric` graders against different named LLM providers in the same eval (useful for grader panels / ensembles) ### assert-set ```yaml @@ -862,7 +862,7 @@ export default defineScriptGrader(({ output, trace }) => { }); ``` -Use `defineScriptGrader()` when the custom component is a command-backed grader with explicit score control, check arrays, workspace commands, or LLM calls through a grader target. `defineScriptGrader()` scripts are referenced in YAML with `type: script` and `command: [bun, run, grader.ts]`. Plain Vitest workspace verifier files can use `command: [agentv, eval, graders/check.test.ts]`. +Use `defineScriptGrader()` when the custom component is a command-backed grader with explicit score control, check arrays, workspace commands, or LLM calls through a grader provider. `defineScriptGrader()` scripts are referenced in YAML with `type: script` and `command: [bun, run, grader.ts]`. Plain Vitest workspace verifier files can use `command: [agentv, eval, graders/check.test.ts]`. ### Convention-Based Discovery diff --git a/skills-data/agentv-eval-writer/references/rubric-evaluator.md b/skills-data/agentv-eval-writer/references/rubric-evaluator.md index 5a5c9e4c7..b97174614 100644 --- a/skills-data/agentv-eval-writer/references/rubric-evaluator.md +++ b/skills-data/agentv-eval-writer/references/rubric-evaluator.md @@ -1,12 +1,12 @@ # Rubric Graders -Rubrics are defined as `assert` entries with plain strings or `type: llm-rubric`. They support binary checklist grading and score-range analytic grading. +Rubrics are defined as `assert` entries with plain strings, `type: llm-rubric`, or `type: agent-rubric`. They support binary checklist grading and score-range analytic grading. Use `agent-rubric` only when the grader provider is agent-capable and should inspect workspace evidence. ## Field Reference | Field | Type | Default | Description | |-------|------|---------|-------------| -| `type` | string | required | Use `llm-rubric` for a structured rubric; plain strings in `assert` use the same grader path | +| `type` | string | required | Use `llm-rubric` for a structured rubric; use `agent-rubric` for Promptfoo-compatible agent-backed rubric checks; plain strings in `assert` use the same non-agent rubric path | | `value` | array | required | List of criterion strings or objects | | `required` | boolean or number | - | Gate: `true` requires score >= 0.8; a number (0–1) sets a custom threshold | @@ -117,6 +117,23 @@ Array format is also accepted: Ranges must be integers 0-10, non-overlapping, covering all values 0-10. +## Agent Rubric Mode + +`agent-rubric` accepts the same `value`, `provider`, `max_steps`, `required`, and +`min_score` fields as `llm-rubric`, but the resolved grader provider must be +agent-capable: + +```yaml +assert: + - type: agent-rubric + provider: codex-grader + value: Inspect the workspace and verify the claimed files exist. +``` + +AgentV asks the grader agent to write a verdict JSON file shaped like +`{"pass": boolean, "score": number, "reason": string}`. The score must be a +finite 0-1 value; invalid verdict files fail closed. + ## Scoring **Checklist:** `score = sum(satisfied weights) / sum(all weights)`