Evaluation issue.

[elginarda-233]

Held-out eval scores the canonical recipe worse than uniform-random — likely a tokenizer/dataset mismatch in the FineWeb eval

Summary

After the eval switched to FineWeb-Edu, the validator scores every gpt2-trained submission — including the canonical reference recipe — worse than a uniform-random model. A real FineWeb-Edu language model cannot be worse than random on FineWeb-Edu, so this points to the held-out eval (active_tokens.bin) being in a different token space than the repo's training pipeline (tiktoken "gpt2").

Reproduction (no secret data needed)

1. Take the canonical h100_proxy recipe and make a trivial config-only change.
2. Train on FineWeb-Edu via the repo's data.prepare --source fineweb-edu (tiktoken gpt2), submit through the normal proof flow.
3. Validator returns val_bpb ≈ 4.5+, while the same checkpoint measured locally on a held-out FineWeb-Edu slice (identical run_hidden_eval, seq_len = max_seq_len//2) gives ≈ 1.4–1.8.

This reproduces for the bare reference recipe — it is not specific to any particular patch.

Why this is impossible for an honest FineWeb LM (the math)

compute_val_bpb uses DEFAULT_BYTES_PER_TOKEN = 4.0 (the non-sealed run_hidden_eval path passes no bytes_per_token), so:

val_bpb 4.56  →  nats/token = 4.56 × ln(2) × 4.0 ≈ 12.6 nats
uniform-random over vocab 50257 = ln(50257) ≈ 10.8 nats
12.6 > 10.8  →  the model is WORSE than uniform-random

A model with training loss ≈ 3.8 nats on FineWeb-Edu has clearly learned English; it is strictly better than uniform-random on any FineWeb-Edu slice (≈ 1.4–2.0 bpb). To exceed random cross-entropy, the model must be confidently predicting the wrong tokens — which happens precisely when the eval token IDs do not correspond to the same tokenizer the model was trained on.

Root-cause hypothesis

The held-out eval stream is not in the same token space as the training data:

• Training/data pipeline: data/tokenizer.py → tiktoken.get_encoding("gpt2"), EOT_TOKEN = 50256, shards/eval stored as uint16.
• eval/val_bpb.py consumes the raw token IDs from active_tokens.bin directly (no re-tokenization), normalized by DEFAULT_BYTES_PER_TOKEN = 4.0.
• If the FineWeb-Edu held-out eval was produced with a different tokenizer / vocab ordering (or a non-gpt2 BPE that still has 50257 entries), every gpt2-trained model sees scrambled targets → CE above random.

This is consistent with the eval having been swapped to FineWeb-Edu through a pipeline other than the repo's data.prepare.

Impact

• The eval currently cannot rank recipes: every honest gpt2 submission (including the reference) lands at ~random, so the "king" is just whichever model is least badly miscalibrated, not the best recipe.
• Counterintuitively, a near-uniform (under-trained / high-entropy) model scores better (closer to random ≈ 3.9 bpb) than a well-trained one (~4.5+), which inverts the intended incentive.

Suggested verification / fix

1. Confirm the held-out active_tokens.bin is tokenized with tiktoken "gpt2" (EOT 50256), uint16, FineWeb-Edu — ideally produced by the same data.prepare --source fineweb-edu miners use.
2. Sanity check: run the canonical recipe checkpoint against the held-out eval and confirm val_bpb is in the ~1.4–2.0 range (better than random). If it is ~4.5, the eval tokenization is the culprit.
3. Verify bytes_per_token is appropriate for the eval stream (4.0 for gpt2 English).

Happy to provide more detail or a minimal repro script. Thanks!