agent-eval-framework

An offline golden-set eval harness, an LLM-as-judge with built-in bias checks, and a failure taxonomy for AI agent systems — extracted from AI Product Engineer Copilot, where the rigor layer was designed to be reusable against any agent that can produce a scoreable output, not just that project's own PRD/roadmap agent.

What's in here

• src/golden-set.ts — loads and validates a golden dataset; every case must have an input and a rubric.
• src/regression.ts — runRegression flags any case whose current score drops below its recorded baseline; assertReproducible enforces fixed temperature or a fixed seed so eval runs are comparable across time.
• src/judge.ts — Judge wraps a model call and validates its output against a structured, parseable schema (score, rationale, citedEvidence) instead of trusting free text. verifyJudgeSanity checks the judge separates a known-good output from a known-bad one before its verdicts on real cases are trusted.
• src/bias.ts — checkPositionBias runs a pairwise comparison both forward and swapped, and flags a judge whose winner flips based on position or verbosity rather than substance.
• src/taxonomy.ts — tagFailures assigns zero or more of hallucination | planning | tool | context from structured signals (e.g. a claimed metric not present in the tool's ground-truth output is tagged hallucination, not inferred from prose).

Install

Not published to npm yet — install directly from GitHub:

npm install github:vrjgamer/agent-eval-framework

Usage

import { loadGoldenSet, runRegression, Judge, checkPositionBias, tagFailures } from "agent-eval-framework";

Development

npm install
npm test        # vitest
npm run typecheck
npm run build

License

MIT — see LICENSE.