Skip to content

feat(core): add Promptfoo-compatible transform runtime - #1648

Merged
christso merged 2 commits into
mainfrom
promptfoo-transform-runtime
Jul 4, 2026
Merged

feat(core): add Promptfoo-compatible transform runtime#1648
christso merged 2 commits into
mainfrom
promptfoo-transform-runtime

Conversation

@christso

@christso christso commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Beads are the source of truth for scope and follow-up. Review summary for: av-kfik.34, av-kfik.34.1.

This PR implements Promptfoo-compatible transform runtime support for authored AgentV eval YAML:

  • default_test.options.transform inheritance with test-level tests[].options.transform override
  • tests[].options.transform applied once to provider output before grading/assertions
  • tests[].assert[].transform applied per assertion only, with isolation from sibling assertions
  • file-based transforms such as file://scripts/xlsx-to-markdown.js
  • inline string transforms for simple authored YAML cases
  • hard rejection of deprecated postprocess with actionable transform guidance
  • raw provider output and transformed judge/assertion input traceability in run artifacts

Promptfoo source evidence checked locally at /home/entity/projects/promptfoo/promptfoo, commit 6bfc5a0c7f16f9c4717ac731d276b578e63d0769:

  • src/evaluator.ts transformRunEvalResponse
  • src/assertions/index.ts assertion-level transform
  • src/util/transform.ts
  • src/types/index.ts
  • examples/config-transform-file and related transform examples

Out of scope / coordinated follow-up:

  • Provider-level transform ordering is preserved where AgentV already provides transformed output to the evaluator path; broader provider/target transform parity remains coordinated under av-kfik.34.3.
  • Public preprocessor docs/examples cleanup remains under av-kfik.34.2 / av-kfik.34.4.

Validation

Focused and adjacent tests:

  • bun test packages/core/test/evaluation/transform-runtime.test.ts -> 7 pass
  • bun test packages/core/test/evaluation/loaders/grader-parser.test.ts -> 112 pass
  • bun test packages/core/test/evaluation/orchestrator.test.ts -> 97 pass
  • bun test packages/core/test/evaluation/validation/eval-file-schema.test.ts -> 31 pass

Repo gates:

  • bun run lint -> pass
  • bun run typecheck -> pass
  • bun test -> 3161 pass, 4 fail in examples/features/vitest-workspace-grader/graders/welcome-banner.test.ts; failures are ENOENT for /app/page.tsx because those example verifier tests expect a prepared workspace, not this repo root.

Live dogfood:

  • Used copied primary-checkout .env plus local OpenAI-compatible proxy at http://127.0.0.1:10531/v1.
  • Ran source checkout command with local-openai target and local-openai-grader live LLM grader.
  • Eval used tests[].options.transform: file://scripts/xlsx-to-markdown.js to convert live provider spreadsheet-style JSON into markdown before llm-rubric grading.
  • Result: PASS, 1/1, mean score 100%, threshold 0.6.
  • Private evidence: agentv-private:evidence/av-kfik-34-1-transform-runtime at commit cef5663.

Notes

This PR is intentionally runtime-focused and does not migrate public docs/examples away from preprocessors. Those broader cleanup tasks remain in the Beads called out above.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 4, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4ff7473
Status: ✅  Deploy successful!
Preview URL: https://1eece0fb.agentv.pages.dev
Branch Preview URL: https://promptfoo-transform-runtime.agentv.pages.dev

View logs

@christso
christso marked this pull request as ready for review July 4, 2026 12:02
@christso
christso merged commit 5381143 into main Jul 4, 2026
8 checks passed
@christso
christso deleted the promptfoo-transform-runtime branch July 4, 2026 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant