Add Hugging Face agent trace analysis cookbook - #102
Merged
Conversation
A guide to importing real agent-eval traces from Hugging Face into Braintrust, scoring them with an LLM judge, and analyzing them with Topics and SQL. Bundles the reusable hf_bt_cookbook importer scripts (streaming HF -> Braintrust Logs and Datasets). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Rewrite the judge intro to explain the Braintrust gateway inline (OpenAI-compatible endpoint fronting all providers, one key). - Switch the deprecated proxy endpoint/doc to the gateway (https://gateway.braintrust.dev/chat/completions, /docs/deploy/gateway). - Fix malformed blog-post markdown link. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Give the concrete cross-judge example (GPT-4.1 graded by GPT-4o). - Add a paragraph on how Topics independently confirmed a failure mode we found by hand (the AppWorld false-success case). - Add the config x benchmark success heatmap after the slicing query. - Fix a typo in the slicing intro sentence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Drop 'in Braintrust' from the title (STYLE.md forbids appending it). - Add trailing periods to the 'By the end', 'You'll need', and 'Next steps' bullet lists. - Link the bt CLI mention to /reference/cli/quickstart. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Show the OpenTelemetry GenAI parts format vs the normalized OpenAI shape. - Link import_logs.py and import_dataset.py to their files on GitHub. - Remove the Topics Tasks facet image (and unused asset). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Old /guides/* paths don't exist in the current docs; point to /annotate/datasets/index and /evaluate/run-evaluations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a new cookbook, Analyzing agent traces from Hugging Face in Braintrust, under
examples/HuggingFaceAgentTraces/. It's the companion guide to the HF agent trace analysis blog post: how to take 1,781 real agent-eval traces published on Hugging Face (the Exgenticagent-llm-tracesdataset), import them into Braintrust, score them with an LLM judge, and analyze them with Topics and SQL.Contents
HuggingFaceAgentTraces.mdx— the walkthrough: import → score → cluster failures with Topics → slice by benchmark/harness/model → build a targeted eval dataset.hf_bt_cookbook/— the reusable importer scripts (from Izzy Hurley's work):import_logs.py— streams an HF trace dataset into Braintrust Logs as a span tree per session, with deterministic IDs so re-runs upsert (which is also how scores write back).import_dataset.py— turns an HF dataset into a gradable Braintrust Dataset.normalize.py,braintrust_logs.py— message/trace normalization and span-tree helpers.assets/— Logs view and Topics screenshots.requirements.txtand aregistry.yamlentry (author:jesswang).Notes
datasets.load_dataset(..., streaming=True), so there's no need to download the 39 Parquet shards locally.SCORE_COLSat it, and re-run — the deterministic IDs upsert the score onto the existing spans.🤖 Generated with Claude Code