Skip to content

Add two showcases: eval regression demo + de-id/re-id sample results#24

Merged
Hefrock merged 1 commit into
mainfrom
claude/festive-cori-wd84yr
Jul 13, 2026
Merged

Add two showcases: eval regression demo + de-id/re-id sample results#24
Hefrock merged 1 commit into
mainfrom
claude/festive-cori-wd84yr

Conversation

@Hefrock

@Hefrock Hefrock commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Makes the strongest work legible to someone skimming the repo — no cloning, no Python — and demonstrates the eval-as-CI-gate pattern end to end.

1. agent-eval — worked regression example (skills/agent-eval/examples/)

A support-ticket summarizer, before/after a "be more concise" prompt change that looks like a clear win:

Metric Baseline After change
Mean cost $0.0049 $0.0030
Mean latency 1312ms 822ms
format pass rate 80% 100%
accuracy pass rate 100% 62%

Every dashboard an engineer usually watches went green — but the change silently dropped an account ID + error code, collapsed reproduction steps, and mangled a timeline. The gate catches all three:

⚠ 3 regression(s) vs baseline (passed before, failing now):
  acc_03: 1.00 -> 0.40
  acc_06: 0.90 -> 0.30
  acc_08: 0.80 -> 0.50
GATE FAILED — exit 1
  • results_baseline.jsonl + results_regressed.jsonl (same 20 case IDs)
  • README.md walks the scenario with the real output
  • CI self-check: a step asserts the regression example still trips the gate, so it can't bit-rot (the step passes when the gate correctly fails)

2. deid-reid-harness — sample results (skills/deid-reid-harness/RESULTS.md)

A verified run rendered for non-runners:

  • The privacy-utility frontier (ASCII plot + table): regex-baseline-v0 at 0.449/1.00 vs over-redact-v0 at 0.900/0.60 — opposite corners, neither dominates
  • The cross-track headline: 49/50 records (98%, CI 94–100%) remain vulnerable despite a perfect Safe Harbor scrub — 34 re-identifiable, 47 inferable, 32 on both axes
  • Track 1/2/3 breakdowns, every figure with a 95% bootstrap CI

Both showcases are linked from their SKILL.md and the root README.

Verification (re-run before merge)

Both showcases were re-verified deterministically, not asserted from memory:

  • deid-reid — 15/15 headline numbers reproduce exactly from a fresh pipeline run at seed 20260101 (frontier corners, cross-track 49/34/47/32, inference 0.94, re-id min_k=1 / 6 population singletons, bootstrap CI bounds). test_harness.py (31/31) locks them against drift.
  • agent-eval example — 8/8 documented figures match live score_eval.py output (pass rate 16/20, accuracy 62%, cost $0.0030, latency 822ms, the three named regressions), and the gate exits 1 as designed.
  • Test suites: scorer 32/32, deid harness 31/31, MCP server 14/14.
  • CI self-check step confirms the regression example still trips the gate on every push.

Scope note (good faith): the deid figures are empirical — reproduced from real harness runs. The agent-eval example's per-case scores are illustrative synthetic data — a teaching scenario for a hypothetical summarizer, not a measurement of a real system. What's verified there is the tool behavior (scoring math, per-category breakdown, regression detection, gate exit codes), which is exercised by the run and by the 32 unit tests.

🤖 Generated with Claude Code

Makes the strongest work legible without running anything, and demonstrates
the eval-as-CI-gate pattern end to end.

agent-eval worked example (skills/agent-eval/examples/):
- results_baseline.jsonl + results_regressed.jsonl: a support-ticket
  summarizer before/after a "be concise" prompt change that looks like a
  clear win — cheaper ($0.0049→$0.0030), faster (1312→822ms), better
  format (80%→100%) — but silently breaks 3 accuracy cases
- README.md walks the scenario and shows the real gate output: accuracy
  62%, 3 pass→fail regressions caught, GATE FAILED exit 1
- CI self-check step asserts the regression example still trips the gate,
  so it can't bit-rot (passes when the gate correctly fails)

deid-reid-harness showcase (skills/deid-reid-harness/RESULTS.md):
- A verified run rendered for non-runners: the privacy-utility frontier
  (ASCII plot + table), the cross-track headline (49/50 records vulnerable
  despite a perfect Safe Harbor scrub), Track 1/2/3 breakdowns, every
  figure with a 95% bootstrap CI. All numbers reproduced from the harness.

Both linked from their SKILL.md and the root README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H1oBwjtpzDLGreGTTjoP7
@Hefrock
Hefrock merged commit f5ae6a4 into main Jul 13, 2026
4 checks passed
@Hefrock
Hefrock deleted the claude/festive-cori-wd84yr branch July 13, 2026 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants