Add two showcases: eval regression demo + de-id/re-id sample results#24
Merged
Conversation
Makes the strongest work legible without running anything, and demonstrates the eval-as-CI-gate pattern end to end. agent-eval worked example (skills/agent-eval/examples/): - results_baseline.jsonl + results_regressed.jsonl: a support-ticket summarizer before/after a "be concise" prompt change that looks like a clear win — cheaper ($0.0049→$0.0030), faster (1312→822ms), better format (80%→100%) — but silently breaks 3 accuracy cases - README.md walks the scenario and shows the real gate output: accuracy 62%, 3 pass→fail regressions caught, GATE FAILED exit 1 - CI self-check step asserts the regression example still trips the gate, so it can't bit-rot (passes when the gate correctly fails) deid-reid-harness showcase (skills/deid-reid-harness/RESULTS.md): - A verified run rendered for non-runners: the privacy-utility frontier (ASCII plot + table), the cross-track headline (49/50 records vulnerable despite a perfect Safe Harbor scrub), Track 1/2/3 breakdowns, every figure with a 95% bootstrap CI. All numbers reproduced from the harness. Both linked from their SKILL.md and the root README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013H1oBwjtpzDLGreGTTjoP7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the strongest work legible to someone skimming the repo — no cloning, no Python — and demonstrates the eval-as-CI-gate pattern end to end.
1. agent-eval — worked regression example (
skills/agent-eval/examples/)A support-ticket summarizer, before/after a "be more concise" prompt change that looks like a clear win:
formatpass rateaccuracypass rateEvery dashboard an engineer usually watches went green — but the change silently dropped an account ID + error code, collapsed reproduction steps, and mangled a timeline. The gate catches all three:
results_baseline.jsonl+results_regressed.jsonl(same 20 case IDs)README.mdwalks the scenario with the real output2. deid-reid-harness — sample results (
skills/deid-reid-harness/RESULTS.md)A verified run rendered for non-runners:
regex-baseline-v0at 0.449/1.00 vsover-redact-v0at 0.900/0.60 — opposite corners, neither dominatesBoth showcases are linked from their
SKILL.mdand the root README.Verification (re-run before merge)
Both showcases were re-verified deterministically, not asserted from memory:
20260101(frontier corners, cross-track 49/34/47/32, inference 0.94, re-id min_k=1 / 6 population singletons, bootstrap CI bounds).test_harness.py(31/31) locks them against drift.score_eval.pyoutput (pass rate 16/20, accuracy 62%, cost $0.0030, latency 822ms, the three named regressions), and the gate exits 1 as designed.Scope note (good faith): the deid figures are empirical — reproduced from real harness runs. The agent-eval example's per-case scores are illustrative synthetic data — a teaching scenario for a hypothetical summarizer, not a measurement of a real system. What's verified there is the tool behavior (scoring math, per-category breakdown, regression detection, gate exit codes), which is exercised by the run and by the 32 unit tests.
🤖 Generated with Claude Code