parameter-golf-checker

Experimental reviewer aid for OpenAI Parameter Golf submissions.

Runs AST-based static analysis on train_gpt.py files to surface patterns that commonly correspond to compliance issues. This tool is a first-pass triage helper for human reviewers, not an authoritative PASS/FAIL judge. It errs heavily on the side of WARN. No GPU required.

⚠️ Every PASS is a weak signal, every WARN means "human should look at this," and every FAIL is a hypothesis, not a verdict. False positives and false negatives are expected. Use it to narrow down which submissions need deeper review — not to approve or reject anything.

What it checks

Based on the rules in Issue #1017 and patterns from closed/flagged submissions:

Check	What it detects
C1 Causality	Missing causal masking in attention modules
C2 Normalization	Logits not passed through a normalized loss (cross_entropy, nll_loss, log_softmax)
C3 Score-before-update	TTT adapting on val_tokens before scoring them under no_grad
C4 Single pass	Multi-pass evaluation, reversed() on val data
Artifact size	Estimated model + script size vs 16 MB limit
N-gram cache	Oracle/ground-truth selection in n-gram backoff
SLOT/LBFGS	Second-order optimization on val data
Network access	urllib/requests/socket imports

Each check returns PASS, WARN, FAIL, or N/A with line numbers and a short explanation. PASS means "no pattern this tool recognizes as a violation was found" — not "this is compliant." FAIL is only emitted for the most unambiguous illegal patterns (e.g. C3: direct val_tokens reference inside a TTT loop with no no_grad scoring block in the same function). Everything in between is WARN.

Usage

# Check a local file
python pgolf_checker.py --file path/to/train_gpt.py

# Check a GitHub PR (requires `gh` CLI)
python pgolf_checker.py --pr 1169

# JSON output for automation
python pgolf_checker.py --file train_gpt.py --format json

Example output

### Compliance Review — train_gpt.py

**Status:** WARN

| Check | Status | Details |
|-------|--------|---------|
| C1 Causality | PASS | Causal masking detected at lines 355, 359, 463 |
| C2 Normalization | PASS | Standard loss function detected at lines 621, 1126 |
| C3 Score-before-update | WARN | TTT on val data detected. no_grad scoring found at line 1183 before adapt at 1279. Appears legal. |
| C4 Single pass | WARN | Multiple loops over val data in eval_val_ttt_lora() |
| Artifact size | PASS | Script: 70 KB. Est. total artifact: 12.3 MB |
| N-gram cache | N/A | No n-gram related code detected |
| SLOT/LBFGS | WARN | Second-order optimization detected at line 1139 |
| Network access | PASS | No network imports detected |

How it works

Pure Python AST analysis. No dependencies beyond the standard library (plus gh CLI if using --pr). Each check in checks/ parses the submission's AST and looks for known violation patterns:

• Score-before-update (the highest-value check) walks For loops containing both loss.backward() and optimizer.step(), detects if they reference val data, then verifies a preceding torch.no_grad() scoring block exists.
• Causality looks for is_causal=True, torch.tril, or flash-attention usage in attention modules.
• N-gram cache searches for Counter/defaultdict usage alongside n-gram naming patterns and flags oracle/ground_truth references.

Limitations

This is static analysis — it catches patterns, not semantics. Things it fundamentally cannot see:

• Aliasing (x = val_tokens; ttt(x)) and helper-indirected scoring
• Decorator-based or context-manager-based no_grad wiring through custom helpers
• Dynamic imports, metaprogramming, exec()
• Whether a cached data structure is populated with future tokens or past tokens
• Whether a flash_attn call was made with is_causal=True

What the tool is actually good for:

• Narrowing a 1,500-submission queue down to a review shortlist
• Catching obviously un-obfuscated illegal TTT (direct val_tokens reference + optimizer.step() with no prior no_grad scoring)
• Flagging oracle/ground_truth identifiers near n-gram-style data structures
• Spotting raw LBFGS / scipy.optimize / linalg.solve near val references

What the tool is not good for:

• Declaring any submission compliant or non-compliant on its own authority
• Any submission that obfuscates via lzma/base85 — the tool will attempt decompression but may miss unusual wrapper shapes
• Replacing a runtime compliance harness hooked into the actual scoring function

Tests

python tests/test_checker.py

Fixtures in tests/fixtures/:

• clean_basic.py — minimal compliant submission
• violation_ttt_before_score.py — illegal TTT pattern
• violation_slot_on_val.py — LBFGS on val data
• violation_ngram_oracle.py — n-gram with oracle selection

Contributing

If you find a submission pattern this tool misses (either a false positive or a missed violation), open an issue with the relevant code snippet.

License

MIT