Better offline analysis: pass-aware training & segmented metrics

[voorhs]

Summary

• Export per-case success info (passed + raw evaluator scores) into tool-suggest samples so it's available downstream.
• Offline eval can now train on successful cases only (--train-on-passed-only), and per-fold metrics are segmented into all / passed / failed buckets so we can compare predictability of successful vs unsuccessful runs.
• Add balanced accuracy, normalized Shannon entropy of the class distribution, and macro mAP to the offline metrics, plus a new batch-jsonl-table --segment={all,passed,failed,compare} view that stacks the segments side-by-side.

Details

samples.py — carry success info

• SQL forwards attributes->'scores' from each case: … span through the CTE chain.
• Sample.data now includes passed: bool (computed the same way as src/report/file_writing.py:_parse_case_row — all evaluator values ≈ 1.0, with PASSED_EPS = 1e-9) and the raw scores dict.
• Caveat: samples.py's SQL still dedupes to the latest evaluate root per case_name, so passed reflects that latest run.

offline_eval.py — pass-aware training

• New --train-on-passed-only flag on OfflineEvalCommonArgs (inherited by both main and batch-redo-repos).
• _prepare_folds validates every sample carries data['passed'] when the flag is on — hard error pointing to re-exporting via samples.py (older repos are not silently fallback'd).
• After splitting, the train set is filtered to passed=True samples; test set is untouched. Folds whose train split becomes empty are skipped, recorded as a synthetic FoldResult with error="skipped: no passing train samples (...)" so the existing error is None and n_test_samples_scored > 0 filter at downstream summaries handles them naturally.

New metrics

• Balanced accuracy (sklearn macro recall on top-1 predictions; multiclass only — 0.0 in multilabel).
• Normalized Shannon entropy of the y_true distribution (H(p)/log₂(K); 0 → single class, 1 → uniform) to spot fold-level imbalance.
• Macro mAP across classes via sklearn.metrics.average_precision_score(..., average='macro'); score derivation is 1/(1+rank) because full_ranked_tool_ids only exposes ids. Restricted to labels in y_true so each per-class AP is well-defined.

Segmented reporting (all / passed / failed)

• FoldResult gains metrics_passed, metrics_failed, n_test_passed, n_test_failed (all defaulted, so existing call sites still work).
• evaluate_fold partitions scored test samples by data['passed'] and aggregates each bucket via shared _aggregate_for_items helper. Empty buckets → None.
• JSONL repo_suggester_summary now carries mean_over_folds, mean_over_folds_passed, mean_over_folds_failed.
• batch-jsonl-table adds --segment={all,passed,failed,compare}. The pivot column set now also surfaces balanced_accuracy and mAP. compare stacks the three segments under headings so you can read predictability of successful vs unsuccessful runs side-by-side.
• Repos exported before the passed change will show em-dash cells in passed/failed segments — intentional rather than silent fallback.

Test plan

• uv run samples.py --experiment <known-passing-exp> and confirm Sample.data carries passed and scores.
• uv run offline_eval.py --repo <new-repo>.jsonl baseline still produces the same mean_over_folds numbers as before for the all segment.
• uv run offline_eval.py --repo <new-repo>.jsonl --train-on-passed-only runs successfully; verify the warning "Fold N: no passing samples …; skipping fold." appears only when expected.
• uv run offline_eval.py --repo <old-repo>.jsonl --train-on-passed-only raises ValueError mentioning re-export.
• uv run offline_eval.py batch-redo-repos writes mean_over_folds_passed / mean_over_folds_failed keys per repo_suggester_summary.
• uv run offline_eval.py batch-jsonl-table <jsonl> --segment compare prints three stacked sections.
• uv run offline_eval.py batch-jsonl-table <jsonl> --segment passed and --segment failed each print one table.
• Sanity-check that balanced_accuracy, class_entropy_normalized, mean_average_precision are in a reasonable range on a known repo.

🤖 Generated with Claude Code