Capabilities Spike Analysis

Empirical analysis of how various factors vary with task difficulty for AIs within METR's time horizon benchmark tasks. Core question: can we predict how much harder a task is for AI than for humans by looking at properties of the task?

See the blog post.

Status: The core results from this analysis that are used in the blog post (results/raw/metrics_plots/bonferroni_promising.png and results/raw/metrics_plots/bonferroni_interest_factors.png) are the outputs I'm most confident in. The rest of the pipeline (Venn diagrams, combo search, permutation tests, regularization, subset on human hours, etc.) is exploratory and has not been as thoroughly validated — treat those outputs with appropriate skepticism.

An LLM-as-judge agent grades each task on ~25 binary factors (plus 3 scalars). This repo takes those grades (in grades.csv) and runs a statistical pipeline to identify factor combinations that best explain AI performance residuals.

Setup

Devcontainer (recommended)

Open the repo in VS Code or Cursor and use "Reopen in Container" (Cmd+Shift+P → "Dev Containers: Reopen in Container"). This gives you a working environment with Python 3.13, DVC, and all project dependencies pre-installed.

Prerequisites:

• Docker running locally

Local setup

uv venv && uv pip install -e ".[dev]"
source .venv/bin/activate

Usage

Full pipeline via DVC

dvc repro

The pipeline reads tuning knobs from params.yaml. Edit that file and re-run dvc repro to change settings.

Individual scripts

# 1. Add correlation row (one per residual variant)
python add_correlations.py grades.csv -o results/raw/grades_with_corr.csv --residual-variant raw
python add_correlations.py grades.csv -o results/pearson/grades_with_corr.csv --residual-variant pearson

# 2. Heatmaps (one set per residual variant)
python plot_heatmap.py results/raw/grades_with_corr.csv -o results/raw/heatmap --rubric rubric.json --residual-variant raw

# 3. Venn diagrams
python plot_venn.py grades.csv -o results/raw/venn --rubric rubric.json --residual-variant raw

# 4. Confidence histograms
python plot_confidence.py grades.csv -o confidence_histograms.png

# 5. Per-factor metrics
python compute_factor_metrics.py grades.csv -o results/raw/factor_metrics.csv --residual-variant raw

# 6. Combo search (AND/OR/DNF combinations)
python search_combos.py grades.csv -o results/raw/combo_metrics.csv --residual-variant raw

# 7. Permutation null test
python permutation_test.py grades.csv -o results/raw/permutation_nulls.json --residual-variant raw

# 8. Metrics plots
python plot_metrics.py results/raw/factor_metrics.csv results/raw/combo_metrics.csv results/raw/permutation_nulls.json --grades grades.csv -o results/raw/metrics_plots --residual-variant raw

Tests

pytest tests/

Reproducing plots from blog post

After running dvc repro, the plots are located at results/raw/metrics_plots/bonferroni_promising.png and results/raw/metrics_plots/bonferroni_interest_factors.png.

Data

grades.csv contains one row per task with:

• Task identifiers (task, task_id, task_family)
• Residual values (residual, pearson_residual) — positive means harder for AI
• ~25 binary factor scores assigned by the grading model
• Per-factor grader confidence values
• Human-graded factor scores (where available)

rubric.json contains {factor_name: definition} for each graded factor.

Both files are committed directly to this repo. They were generated from an internal pipeline that fetches grading transcripts and combines them with human-graded data.

DVC pipeline

Stages in dvc.yaml (most loop over residual_variants: [raw, pearson]):

Stage	Output
correlations	results/{variant}/grades_with_corr.csv
heatmap	results/{variant}/heatmap_*.png
venn	results/{variant}/venn_*.png
confidence_histograms	confidence_histograms.png
factor_metrics	results/{variant}/factor_metrics.csv
combo_search	results/{variant}/combo_metrics.csv
permutation_test	results/{variant}/permutation_nulls.json
plot_metrics	results/{variant}/metrics_plots/

Key files

File	Purpose
analysis.py	Shared module: data loading, metric functions, combo enumeration, bootstrap
add_correlations.py	Adds correlation row to grades CSV
compute_factor_metrics.py	Per-factor metrics with bootstrap CIs
search_combos.py	AND/OR/DNF combo search over factor pairs/triples
permutation_test.py	Permutation null test for combo search significance
plot_metrics.py	Generates all metrics plots (bar charts, permutation histograms, Bonferroni CIs)
plot_heatmap.py	Factor-grade heatmaps
plot_venn.py	Venn diagrams of factor overlaps
plot_confidence.py	Histograms of grader self-reported confidence per factor
tests/test_analysis.py	Unit tests for analysis module