Skip to content

feat(core): normalize public providers schema - #1708

Merged
christso merged 2 commits into
mainfrom
feat/av-ctfu-providers-schema
Jul 7, 2026
Merged

feat(core): normalize public providers schema#1708
christso merged 2 commits into
mainfrom
feat/av-ctfu-providers-schema

Conversation

@christso

@christso christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add public authored providers support and normalize providers[].id/label into the existing internal target model
  • hard-error removed public targets, suite/assertion target fields, defaults.target, and graders with migration guidance
  • support Promptfoo-style colon provider specs, Promptfoo Codex IDs, AgentV-native agentv:codex-cli, and migrate the repo-local provider catalog

Validation

  • bun test packages/core/test/evaluation/loaders/config-loader.test.ts packages/core/test/evaluation/providers/targets-file.test.ts packages/core/test/evaluation/loaders/grader-parser.test.ts packages/core/test/evaluation/eval-inline-experiment.test.ts packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts packages/core/test/evaluation/validation/targets-validator.test.ts packages/core/test/evaluation/providers/targets.test.ts
  • bunx biome check <changed source/test/schema files>
  • bun --filter @agentv/core typecheck
  • bun --filter agentv typecheck
  • bun apps/cli/src/cli.ts validate .agentv/targets.yaml
  • post-catalog focused rerun: bun test packages/core/test/evaluation/providers/targets-file.test.ts packages/core/test/evaluation/validation/targets-validator.test.ts packages/core/test/evaluation/providers/targets.test.ts packages/core/test/evaluation/loaders/config-loader.test.ts
  • git diff --check

Dogfood

Attempted live dogfood after copying .env from the primary checkout. The run reached provider invocation, but OPENAI_API_KEY in the copied env resolves to dummy, so OpenAI returned HTTP 401. This PR remains draft until a live provider + real LLM grader run is completed with real credentials.

Follow-up

  • Promptfoo export/transpile should lower AgentV-only agentv:codex-cli to a generated TS file://...:callApi provider or a complete package provider id such as package:@agentv/promptfoo-providers:CodexCliProvider, not shell/sh exec wrappers.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 7, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: b3130d3
Status: ✅  Deploy successful!
Preview URL: https://81c48f96.agentv.pages.dev
Branch Preview URL: https://feat-av-ctfu-providers-schem.agentv.pages.dev

View logs

@christso

christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

CI follow-up pushed in b3130d34.

Root cause fixed: stale generated eval bundles, rerun override catalogs, TS eval lowering, CLI tests, and example eval YAML still authored old target/targets shapes after the public providers schema break.

Local validation:

  • bun test packages/core/test/evaluation/interpolation-integration.test.ts packages/core/test/evaluation/yaml-parser-metadata.test.ts packages/core/test/evaluation/validation/config-validator.test.ts packages/core/test/evaluation/loaders/ts-eval-loader.test.ts - 63 pass
  • bun test apps/cli/test/eval.integration.test.ts apps/cli/test/commands/grade/grade-prepared.test.ts apps/cli/test/commands/runs/rerun.test.ts apps/cli/test/commands/prepare/prepare.test.ts apps/cli/test/commands/eval/bundle.test.ts apps/cli/test/commands/eval/targets.test.ts apps/cli/test/commands/validate/validate-files.test.ts - 62 pass
  • bun test apps/cli/test/commands/eval/artifact-writer.test.ts apps/cli/test/commands/eval/task-bundle.test.ts - 76 pass
  • bun test apps/cli/test/commands/validate/validate-files.test.ts - 6 pass
  • bun run test - passed across workspace
  • bun run typecheck - passed
  • bun run build - passed
  • bun run lint - passed
  • bun run validate:examples - Total 109, Valid 109, Invalid 0
  • rg -n "^target:\s*|^targets:" examples --glob '*.{yaml,yml}' - no matches
  • git diff --check - clean

Live dogfood:

  • Used local proxy at http://127.0.0.1:10531/v1 with explicit OPENAI_BASE_URL, OPENAI_ENDPOINT, LOCAL_OPENAI_PROXY_BASE_URL, OPENAI_API_KEY=dummy.
  • Temp eval used public providers, defaults.provider, provider catalog labels candidate/grader, and assertion-level provider: grader.
  • Command: bun apps/cli/src/cli.ts eval run tmp-av-ctfu-provider-dogfood.eval.yaml --targets tmp-av-ctfu-providers.yaml --output /tmp/agentv-av-ctfu-provider-dogfood-results --no-results-push --workers 1
  • Result: PASS, 1/1 scored >= 80%, summary at /tmp/agentv-av-ctfu-provider-dogfood-results/summary.json with passed_tests: 1, failed_tests: 0, target candidate.

Fresh CI is running on commit b3130d34.

@christso
christso marked this pull request as ready for review July 7, 2026 01:23
@christso
christso merged commit 7a68de6 into main Jul 7, 2026
8 checks passed
@christso
christso deleted the feat/av-ctfu-providers-schema branch July 7, 2026 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant