Skip to content

feat(provider-surface): migrate SDK CLI docs to providers - #1707

Merged
christso merged 2 commits into
mainfrom
feat/av-uttb-provider-surface-migration-docs
Jul 7, 2026
Merged

feat(provider-surface): migrate SDK CLI docs to providers#1707
christso merged 2 commits into
mainfrom
feat/av-uttb-provider-surface-migration-docs

Conversation

@christso

@christso christso commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Updates the SDK/programmatic eval authoring surface to use providers, provider, and default_test, with hard errors for old target authoring.
  • Adds migration rewrites for target-shaped YAML into provider-shaped YAML, including .agentv/providers.yaml guidance and focused coverage.
  • Updates CLI run/grade flags and docs/examples/templates to the provider public surface while preserving current internal run-artifact target vocabulary.

Validation

  • bun test scripts/migrate-hard-deprecations.test.ts
  • bun test packages/sdk/test/eval-authoring.test.ts
  • bun test apps/cli/test/commands/eval/run-command-options.test.ts
  • bun --filter @agentv/sdk typecheck
  • bun --filter agentv typecheck
  • git diff --check
  • git diff --name-only --diff-filter=ACM -- "*.ts" "*.md" "*.mdx" "*.yaml" | xargs bunx biome format --write

Notes

  • Left core parser/schema normalization files untouched for av-ctfu.
  • Did not run broad example validation because provider-shaped authored YAML depends on the concurrent core provider-surface parser work.
  • Live dogfood is not applicable for this isolated SDK/CLI/docs migration slice until the core provider authoring runtime lands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 6, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: a09cb23
Status: ✅  Deploy successful!
Preview URL: https://30faa097.agentv.pages.dev
Branch Preview URL: https://feat-av-uttb-provider-surfac.agentv.pages.dev

View logs

@christso

christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

CI triage update for this branch:

  • Previous failing run: https://github.com/EntityProcess/agentv/actions/runs/28831912292
  • Test failure was branch-local, not feat(core): normalize public providers schema #1708-blocked: the migration sweep had rewritten three core test fixtures that intentionally exercise removed authored YAML (expected_output and workspace.isolation). Fixed in f388b608 test(provider-surface): restore removed-field fixtures. Local validation now passes with bun --filter @agentv/core test (2304 pass, 0 fail), plus the earlier focused 3-file core test run.
  • Validate Evals is blocked on feat(core): normalize public providers schema #1708 / av-ctfu: the job reported 55 invalid example eval files, all with [providers] Top-level providers is not a runtime alias in AgentV eval YAML..., because this PR intentionally migrates docs/examples to provider-shaped YAML while current main still rejects top-level providers.
  • No parser/schema changes were made on this branch. Planned next step: after feat(core): normalize public providers schema #1708 merges, rebase this branch onto updated main and rerun the relevant validation/CI.

@christso

christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

CI follow-up for the rerun after f388b60:

  • Test exposed one additional branch-local SDK failure in run 28833015259: packages/sdk/test/grader-helpers.test.ts still authored grader overrides as assertion target, and defineEval() correctly rejected eval.tests[0].assert[5].target.
  • Fixed in 1438ab31 fix(sdk): emit grader provider overrides: SDK grader helpers now emit provider for LLM grader routing and script-grader proxy config, and nearby SDK/docs examples no longer show rejected target authoring.
  • Local validation passed: bun test packages/sdk/test/grader-helpers.test.ts; bun --filter @agentv/sdk test (97 pass / 0 fail); bun --filter @agentv/sdk build; git diff --check; targeted scan for rejected docs keys in the touched docs scope.
  • Validate Evals remains classified as blocked on feat(core): normalize public providers schema #1708 / av-ctfu. The provider-shaped authored YAML in this PR still needs the core parser/schema normalization from feat(core): normalize public providers schema #1708 before main CI can validate those examples. No parser/schema changes were made on this branch.

After #1708 merges: rebase this branch onto updated main and rerun validation/CI.

@christso

christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

Manual CI dispatch on the latest head 1438ab31: https://github.com/EntityProcess/agentv/actions/runs/28833666507

Result:

  • Passed: Build, Typecheck, Lint, Check Links, Validate Marketplace, and Cloudflare Pages.
  • Validate Evals failed as expected: 55/66 files still reject top-level providers on the current main parser/schema.
  • Test still fails, but the remaining failures are now classified as feat(core): normalize public providers schema #1708-blocked core parser/schema dependency, not another self-contained SDK/CLI fix. The app/CLI tests use provider-shaped fixtures from this branch; current main core rejects them before the CLI assertions can run.

Representative Test failures:

  • apps/cli/test/eval.integration.test.ts cases fail early with Invalid eval runtime config in eval file: top-level providers is not a runtime alias....
  • apps/cli/test/commands/grade/grade-prepared.test.ts / prepare tests fail with .agentv/targets.yaml validation errors: [providers] Top-level providers is not a runtime alias in targets.yaml and [targets] Missing or invalid targets field.
  • apps/cli/test/commands/eval/bundle.test.ts fails for the same current-core .agentv/targets.yaml parser rejection.

No parser/schema changes were made here. Next step remains: after #1708 lands, rebase this branch onto updated main and rerun CI/focused validation.

@christso
christso force-pushed the feat/av-uttb-provider-surface-migration-docs branch from 1438ab3 to 7b74d65 Compare July 7, 2026 02:16
@christso
christso marked this pull request as ready for review July 7, 2026 02:30
@christso
christso merged commit 1824692 into main Jul 7, 2026
8 checks passed
@christso
christso deleted the feat/av-uttb-provider-surface-migration-docs branch July 7, 2026 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant