Skip to content

fix(sdk): align eval config providers surface - #1722

Merged
christso merged 2 commits into
mainfrom
av-icbt-sdk-provider-surface
Jul 7, 2026
Merged

fix(sdk): align eval config providers surface#1722
christso merged 2 commits into
mainfrom
av-icbt-sdk-provider-surface

Conversation

@christso

@christso christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

TypeScript eval authoring now follows the same Promptfoo-shaped provider surface as YAML. SDK users can author providers entries with id/label, select defaults.provider and defaults.grader, and set grader providers through defaultTest.options.provider, tests[].options.provider, or assertion-level provider without falling back to top-level target, targets, or graders.

The TS eval loader no longer rewrites YAML-aligned target into providers; stale target-era eval files now fail with provider migration guidance. options.provider is also carried into LLM-backed assertions as the inherited grader provider, while assertion-level provider still wins.

The remaining target-named SDK API is createTargetClient() and related target-proxy types. Those are documented as runtime-only script-grader proxy APIs, not eval authoring syntax.

Validation

  • bun test packages/sdk/test/eval-authoring.test.ts
  • bun test packages/core/test/evaluation/loaders/ts-eval-loader.test.ts packages/core/test/evaluation/loaders/grader-parser.test.ts
  • bun --filter @agentv/core typecheck
  • bun --filter @agentv/sdk typecheck
  • bunx biome check ...touched files...
  • bun --filter @agentv/sdk test
  • bun --filter @agentv/sdk build

Live dogfood was attempted after copying .env from the primary checkout. It is blocked locally: eval-file inline providers are still ignored by the current CLI provider resolution path covered by #1718, catalog openai starts but fails with pi-ai call failed: Connection error, and catalog local-openai requires missing LOCAL_OPENAI_PROXY_API_KEY.

Related

Related: av-icbt
Related: #1718, #1719


Compound Engineering
GPT-5

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 7, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 1ab307b
Status: ✅  Deploy successful!
Preview URL: https://4730d8a3.agentv.pages.dev
Branch Preview URL: https://av-icbt-sdk-provider-surface.agentv.pages.dev

View logs

@christso
christso force-pushed the av-icbt-sdk-provider-surface branch from 89a2524 to 40eb768 Compare July 7, 2026 08:16
@christso
christso force-pushed the av-icbt-sdk-provider-surface branch from 40eb768 to 1ab307b Compare July 7, 2026 08:18
@christso
christso merged commit 79d156e into main Jul 7, 2026
9 checks passed
@christso
christso deleted the av-icbt-sdk-provider-surface branch July 7, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant