Skip to content

fix(providers): emit skill call metadata - #1706

Merged
christso merged 1 commit into
mainfrom
fix/av-xdwn-provider-skill-calls
Jul 6, 2026
Merged

fix(providers): emit skill call metadata#1706
christso merged 1 commit into
mainfrom
fix/av-xdwn-provider-skill-calls

Conversation

@christso

@christso christso commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • derive canonical skill call metadata from normalized provider messages and wire it through pi, copilot, cli, and SDK child providers
  • separate confirmed skillCalls from attemptedSkillCalls for failed skill reads, following Promptfoo's useful metadata pattern
  • hard-deprecate the legacy skill-trigger grader surface and reject authored skill-trigger with skill-used / not-skill-used migration guidance
  • update LLM grader docs to reflect target-only grader config

Verification

  • bun test packages/core/test/evaluation/providers/skill-calls.test.ts packages/core/test/evaluation/providers/pi-runtime.test.ts packages/core/test/evaluation/providers/cli.test.ts packages/core/test/evaluation/providers/copilot-sdk.test.ts packages/core/test/evaluation/providers/copilot-cli.test.ts packages/core/test/evaluation/providers/sdk-child-provider.test.ts
  • bun test packages/core/test/evaluation/graders/promptfoo-assertions.test.ts packages/core/test/evaluation/loaders/eval-yaml-transpiler.test.ts packages/core/test/evaluation/loaders/grader-parser.test.ts packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts
  • bun run lint
  • bun run typecheck
  • live dogfood: Codex SDK target + local-openai-grader + skill-used assertion passed 1/1; bundle: .agentv/results/dogfood-av-xdwn/run-skill-used-codex-key

Notes

  • Promptfoo reference checked locally at /home/entity/projects/promptfoo/promptfoo commit 6bfc5a0c7f16f9c4717ac731d276b578e63d0769.
  • Existing local dogfood caveats observed: inline eval target api_key templates are rendered before target secret validation, so the dogfood used the existing local-openai-grader target with LOCAL_OPENAI_PROXY_* and CODEX_API_KEY env overrides.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 0f1e2fe
Status: ✅  Deploy successful!
Preview URL: https://995a0bfe.agentv.pages.dev
Branch Preview URL: https://fix-av-xdwn-provider-skill-c.agentv.pages.dev

View logs

@christso
christso merged commit 62533e9 into main Jul 6, 2026
8 checks passed
@christso
christso deleted the fix/av-xdwn-provider-skill-calls branch July 6, 2026 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant