Granule

Byte-level BPE tokenizer built from scratch, benchmarked against tiktoken on compression and speed. Zero non-stdlib runtime dependencies for the core tokenizer.

The cost of not writing in English

Tokenizers are trained mostly on English. Everyone else pays for that in tokens — and tokens are context window, latency and money. The table below is the point of this repository: the same idea written in Tamil, Hindi or Arabic costs several times more tokens than in English, under the tokenizers actually used in production.

Read cl100k cost vs English as the multiplier on your bill. Every number here is produced by granule bench --report on the corpora in data/; none of it is written by hand.

CPU: AMD64 Family 23 Model 96 Stepping 1, AuthenticAMD

Compression by language

At the largest vocab size benchmarked. cost vs English is cl100k chars/token for english divided by this corpus's -- the token multiplier for writing the same text in this script.

Corpus	bytes/char	granule c/t	granule b/t	cl100k c/t	o200k c/t	cl100k cost vs English
arabic	1.776	2.4807	4.4065	1.453	3.0237	2.91x
english	1.005	2.6658	2.6781	4.2352	4.283	1.0x
hindi	2.502	1.5605	3.9044	1.04	2.9641	4.07x
jsonlines	1.0	2.8719	2.8719	2.8349	2.8316	1.49x
python	1.0	3.3944	3.3944	4.54	4.51	0.93x
tamil	2.582	1.8308	4.7278	0.7909	2.7595	5.35x

Encode throughput vs tiktoken

Pure-Python encoder against Rust-backed tiktoken; measured at the largest vocab size benchmarked.

Corpus	granule MB/s	cl100k MB/s	o200k MB/s	Slowdown vs cl100k
arabic	1.9088	7.5348	5.6317	3.9x
english	2.1849	7.7008	4.6009	3.5x
hindi	3.3993	5.7366	7.7116	1.7x
jsonlines	3.1387	4.4225	3.1555	1.4x
python	2.1977	2.9489	3.4563	1.3x
tamil	3.6744	5.8713	6.5219	1.6x

Full results

Naive/Fast train times are measured on the same 56 KB prefix of each corpus; everything else is measured on the full corpus.

Corpus	Vocab	c/t	b/t	Fertility	cl100k c/t	o200k c/t	Naive(ms)	Fast(ms)	Speedup	Thru(MB/s)	Overlap(%)
arabic	4096	2.1203	3.7663	2.7738	1.453	3.0237	16876.54	1448.98	11.65x	2.0017	7.16
arabic	8192	2.3057	4.0955	2.5508	1.453	3.0237	20111.36	877.85	22.91x	1.8592	7.16
arabic	16384	2.4807	4.4065	2.3708	1.453	3.0237	20271.69	1056.07	19.2x	1.9088	7.2
english	4096	2.3069	2.3175	2.685	4.2352	4.283	14478.88	186.03	77.83x	2.1482	82.4
english	8192	2.5055	2.517	2.4721	4.2352	4.283	13968.28	197.01	70.9x	2.1469	70.06
english	16384	2.6658	2.6781	2.3235	4.2352	4.283	12654.92	162.6	77.83x	2.1849	53.3
hindi	4096	1.5326	3.8346	3.6132	1.04	2.9641	8115.2	124.82	65.02x	3.5806	29.01
hindi	8192	1.5536	3.8872	3.5643	1.04	2.9641	8812.38	139.56	63.14x	3.6525	26.97
hindi	16384	1.5605	3.9044	3.5486	1.04	2.9641	8634.39	156.48	55.18x	3.3993	26.39
jsonlines	4096	2.8719	2.8719	40.9864	2.8349	2.8316	2186.79	17.79	122.92x	2.793	58.08
jsonlines	8192	2.8719	2.8719	40.9864	2.8349	2.8316	2031.51	16.23	125.17x	3.2851	58.08
jsonlines	16384	2.8719	2.8719	40.9864	2.8349	2.8316	2363.65	16.41	144.04x	3.1387	58.08
python	4096	3.1202	3.1202	3.1303	4.54	4.51	5135.23	71.32	72.0x	2.3953	81.85
python	8192	3.2971	3.2971	2.9623	4.54	4.51	5844.77	67.04	87.18x	2.2579	67.45
python	16384	3.3944	3.3944	2.8774	4.54	4.51	5107.56	86.5	59.05x	2.1977	51.65
tamil	4096	1.8028	4.6556	4.9849	0.7909	2.7595	8537.5	132.09	64.63x	3.8351	29.38
tamil	8192	1.8288	4.7227	4.9141	0.7909	2.7595	7234.57	105.78	68.39x	3.0618	28.89
tamil	16384	1.8308	4.7278	4.9088	0.7909	2.7595	7374.25	135.46	54.44x	3.6744	28.26

Regenerated with granule bench --report. Corpora, licenses and SHA256s are in data/README.md; raw numbers in results/results.json.

Quickstart

# Install from source (not published to PyPI)
git clone https://github.com/Mirdula18/Granule
cd Granule
pip install -e ".[bench]"

# Train a tokenizer from a text file
granule train corpus.txt 5000 --pattern gpt4 --verbose

# Encode/decode text
granule encode model.json "Hello, world!"
granule encode model.json "12,34,56" --decode

# Run benchmarks (requires tiktoken)
granule bench --report

Python API

from granule import Tokenizer

# Train
tok = Tokenizer.train(
    corpus="corpus.txt",
    vocab_size=5000,
    pattern="gpt4",           # "gpt2", "gpt4", or "none"
    special_tokens=["<|endoftext|>"],
    fast=True,                # use delta-update trainer
    verbose=True,
)

# Encode / decode
ids = tok.encode("Hello, world!")
text = tok.decode(ids)

# Persist
tok.save("model.json")
tok = Tokenizer.load("model.json")

See SPEC.md for the full API contract.

Features

• Two trainers: naive (O(N) per merge, the correctness oracle) and fast (delta-update with heap, matches naive exactly).
• Pre-tokenization patterns: GPT-2 and GPT-4 patterns sourced from tiktoken's published source.
• Special tokens: reserve token IDs for sentinel markers during training; control matching with allowed_special on encode.
• Save/load: human-inspectable JSON model files with .vocab sidecar for debugging.
• Deterministic: same corpus + settings → identical tokenizer every time. Ties broken by lowest pair id.

Development

pip install -e ".[dev]"
pytest -q                       # all tests
pytest -q -k "not slow"         # fast loop
ruff check src tests            # lint
ruff format src tests           # format
mypy src                        # type check

Benchmarking requires the optional [bench] extra:

pip install -e ".[bench]"
granule bench --report

Limitations

Each of these is tracked as an issue; the list here is the summary, the issue has the evidence and the acceptance criteria.

• Throughput (#7): the pure-Python encoder is slower than tiktoken's Rust-backed implementation — a deliberate trade-off, and an explicit non-goal in SPEC.md. The measured ratio per corpus is the "Slowdown vs cl100k" column above, generated by granule bench --report and never written by hand. Timed runs start from a cold pre-token cache, so the figure reflects doing the work rather than re-reading it.
• Pre-tokenization (#2): the GPT-2/GPT-4 regexes are Python re translations of upstream's \p{...} classes, and the GPT-4 one diverges from upstream on leading-space attachment — "hello world" splits as ['hello', ' ', 'world'] where cl100k gives ['hello', ' world']. This inflates our pre-token count by ~69% on English and understates the compression numbers above.
• Fertility on jsonlines (#4): that corpus is compact JSON with no spaces, so a whole record counts as one whitespace-delimited "word" and fertility reads in the tens. As SPEC.md notes, fertility is meaningless without whitespace word boundaries; it is reported anyway rather than hidden.
• Naive trainer timing (#3): the naive trainer is O(merges × corpus symbols), so the Naive/Fast columns are measured on a bounded prefix of each corpus. Compression and throughput use the full corpus. Because the fast trainer's advantage grows with corpus size, this understates the speedup.
• No streaming (#5): the entire corpus is loaded into memory during training.
• Corpus scale (#6): ~1.5 MB per language is enough to make vocab-16384 compression meaningful, but far short of the billions of tokens a production tokenizer is trained on.
• No demo GIF yet (#8): a terminal recording showing granule in action is a future improvement.

License

MIT