Skip to content

feat(grader): support agent-rubric compatibility - #1717

Merged
christso merged 2 commits into
mainfrom
feat/av-8pjk-agent-rubric
Jul 7, 2026
Merged

feat(grader): support agent-rubric compatibility#1717
christso merged 2 commits into
mainfrom
feat/av-8pjk-agent-rubric

Conversation

@christso

@christso christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • adds Promptfoo-compatible agent-rubric as a supported AgentV assertion mapped onto the existing structured LLM grader path
  • keeps public authoring on provider/default grader selection and does not add top-level graders:
  • enforces structured agent-grader verdict JSON with file-first verdict handling and fallback validation
  • updates docs, eval-writer skill guidance, SDK assertion typing, parser/schema/validator coverage

Validation

  • bun test packages/core/test/evaluation/loaders/grader-parser.test.ts packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts packages/core/test/evaluation/graders.test.ts
  • bun --filter @agentv/core typecheck
  • bun --filter @agentv/sdk typecheck
  • bun --filter @agentv/core build
  • bun --filter @agentv/sdk build
  • bun --filter agentv build
  • bun --filter @agentv/core lint
  • git diff --check
  • env LOCAL_OPENAI_PROXY_BASE_URL=http://127.0.0.1:10531/v1 LOCAL_OPENAI_PROXY_API_KEY=dummy LOCAL_OPENAI_PROXY_MODEL=gpt-5.4-mini bun apps/cli/src/cli.ts validate .agentv/results/av-8pjk-agent-rubric-dogfood/source/evals/agent-rubric.eval.yaml

Live dogfood

  • endpoint: http://127.0.0.1:10531/v1
  • model: gpt-5.4-mini
  • command: env LOCAL_OPENAI_PROXY_BASE_URL=http://127.0.0.1:10531/v1 LOCAL_OPENAI_PROXY_API_KEY=dummy LOCAL_OPENAI_PROXY_MODEL=gpt-5.4-mini bun /home/entity/projects/EntityProcess/agentv__worktrees/av-8pjk-agent-rubric-finish/apps/cli/src/cli.ts eval run evals/agent-rubric.eval.yaml --workers 1
  • final run: 2026-07-07T06-09-22-804Z, 1/1 passed, agent-rubric:agent-rubric score 1
  • grading artifact records delegate mode with grader_target: codex-agent-grader and verdict_source: file
  • private evidence: EntityProcess/agentv-private branch evidence/av-8pjk-agent-rubric commit 97f0463, path dogfood/av-8pjk-agent-rubric/

Notes

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: f13ca00
Status: ✅  Deploy successful!
Preview URL: https://dacaf410.agentv.pages.dev
Branch Preview URL: https://feat-av-8pjk-agent-rubric.agentv.pages.dev

View logs

@christso
christso merged commit 4ed8b31 into main Jul 7, 2026
9 checks passed
@christso
christso deleted the feat/av-8pjk-agent-rubric branch July 7, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant