Encoder vectorization + CLI flags for training on larger corpora

[modulovalue]

Two commits that make it practical to point this codebase at corpora bigger and more diverse than Shakespeare. Tested locally on a 500 MB Wikipedia dump (522M chars, 7003-char vocab); steady at 27 it/s on M-series MPS using the same model.

Changes

1. Vectorize Encoder vocab build and bulk encode with numpy

The original Encoder ran two pure-Python passes: a per-character dict-membership check to build the vocab, and a per-character lookup to encode. Fine on Shakespeare, but on a 500 MB wiki dump those Python loops dominate startup (estimated minutes each). New code:

• set(text) for vocab (one C pass)
• encode_array(text): encodes text to UTF-32-LE, views as uint32 codepoints, gathers through a precomputed lookup table -- returns np.ndarray[int32]
• caches the inverse dict so decode() doesn't rebuild it on every call
• chunked bulk encode (4M chars/block) so peak memory stays bounded

encode, decode, vocab are unchanged. encode_array is additive.

Measured on a 500 MB wiki dump (522M characters):

• vocab build: ~2.6 s (was estimated at minutes)
• encode_array: ~0.8 s (was estimated at minutes)

2. Make dataset path configurable and add training CLI flags

• Transformer(data_path=...) constructor argument; threaded through train.py, sample.py, export_onnx.py via a --data flag.
• --batch-size and --seq-len flags so hyperparameters can be tuned without editing source.
• --resume <checkpoint.pt> flag that loads a saved state_dict before training -- useful for picking up long runs after a crash. Only model weights are restored; optimizer state and step counter are not.
• Use the new encoder.encode_array() and store the corpus as int32 on device. Vocab fits easily in 32 bits, so int64 was wasting 50% of corpus memory. On a 500 MB corpus this saves ~2 GB of device memory.
• Read the corpus with f.read() instead of \"\\n\".join(f.readlines()) -- the old form silently doubled every newline (no vocab change, since the encoder was building from the same join'd text).

Defaults are unchanged, so uv run train --device mps behaves exactly the same as before. Sanity-checked on Shakespeare: same it/s, matching loss curve.

Test plan

• uv run train --device mps (defaults to Shakespeare) -- identical it/s and loss curve
• uv run train --device mps --data <other.txt> -- runs cleanly on multilingual corpora
• Full 500 MB wiki dump trains at ~27 it/s on M-series MPS
• --resume checkpoints/checkpoint.pt loads a previous state_dict and continues training

[Encrux]

Looks good.

One thing: If you want to resume from a previous checkpoint, you also want to make sure you're resuming the optimizer state imo.