Skip to content

docs: add provider-neutral AgentsKit eval cookbook - #105

Open
Emerson Braun (EmersonBraun) wants to merge 2 commits into
braintrustdata:mainfrom
EmersonBraun:codex/agentskit-eval
Open

docs: add provider-neutral AgentsKit eval cookbook#105
Emerson Braun (EmersonBraun) wants to merge 2 commits into
braintrustdata:mainfrom
EmersonBraun:codex/agentskit-eval

Conversation

@EmersonBraun

Copy link
Copy Markdown

Summary

  • add a deterministic TypeScript support agent and three evaluation fixtures
  • run the same baseline locally with @agentskit/eval and through Braintrust
  • add required-term and safe-escalation scorers with credential-free tests
  • register the cookbook and Emerson Braun as the author

Why

This gives Braintrust users a provider-neutral evaluation recipe that can run before
they have a model key or production traces. The async task boundary can later be
replaced by any AgentsKit runtime, hosted service, or model provider without rewriting
the dataset or scorers.

Validation

  • pnpm install --frozen-lockfile
  • pnpm --filter agentskit-braintrust-eval test — 3 passed
  • pnpm --filter agentskit-braintrust-eval check
  • pnpm --filter agentskit-braintrust-eval eval:local — 3/3 passed, accuracy 1
  • braintrust eval agent.eval.ts --list --no-send-logs
  • Prettier and registry/author schema-shape validation passed

@EmersonBraun
Emerson Braun (EmersonBraun) marked this pull request as ready for review July 27, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant