[Kimi] Convert the tiktoken vocabulary at download time by weifengpy · Pull Request #4262 · pytorch/torchtitan

weifengpy · 2026-08-21T15:44:39Z

kimi_k2_5, kimi_vl_a3b and moonlight_16b_a3ball raise "No supported tokenizer files found"

this PR tries to fix without bring back tiktoken dependency

Every non-debug Kimi config fails while building its tokenizer. Moonshot publishes vocabularies as ``tiktoken.model``, and torchtitan's tokenizer reads ``tokenizer.json`` or a vocab/merges pair, so ``moonlight_16b_a3b``, ``kimi_vl_a3b`` and ``kimi_k2_5`` all raise "No supported tokenizer files found". Only the debug model works, because it points at the bundled test tokenizer. This has been true since #3532 added those configs, 12 months after #1333 removed tiktoken support and standardized on HuggingFace tokenizers. Rebuild the equivalent tokenizer.json during ``download_hf_assets.py`` rather than teaching the runtime a second vocabulary format, which keeps #1333's decision intact. Reading the tiktoken format and recovering BPE merge pairs from its ranks is delegated to ``transformers.convert_slow_tokenizer.TikTokenConverter``, so neither becomes torchtitan's to track as tiktoken evolves. The pre-tokenizer regex is read from the model's own tokenizer module via its ``tokenizer_config.json`` auto_map, not hardcoded here, so it cannot silently drift from upstream; ``--pattern`` overrides it for models that expose none. That leaves one thing upstream cannot supply: special-token ids. ``added_tokens_decoder`` names only some reserved slots -- Moonlight names 9 across ids 163584-163839 -- and the unnamed ones still occupy ids, so they are filled with placeholders. Conversion happens at download rather than at tokenizer load because every rank builds a tokenizer; converting at load would have them race to write the same file, and guarding that needs process-group awareness the tokenizer component should not have. ``transformers`` and ``tiktoken`` are needed to run the script. Both are already used under scripts/ -- numerical_tests_kimi.py imports transformers and documents needing tiktoken -- and trust_remote_code is already used there for this same model family. tiktoken joins requirements-dev.txt so CI exercises the conversion. Nothing reaches the training runtime, which still reads only tokenizer.json. Test plan: - pytest tests/unit_tests/test_convert_tiktoken.py -- 10 passed. Special-token id placement is covered without any optional dependency; the conversion tests skip when transformers/tiktoken are absent. - Cross-checked against reference tiktoken on the real Moonlight vocabulary: 812 strings from repo sources plus CJK, emoji, whitespace and literal special-token text encode to identical ids, 0 mismatches. - The delegated output was also compared against a hand-rolled implementation of the same conversion over 667 strings: 0 mismatches, and the regex read from the model matched the published pat_str exactly. - Converted vocabulary is 163840 tokens, matching moonlight_16b_a3b's decoder_vocab_size; moonlight_16b_a3b().tokenizer.build() now succeeds from its own hf_assets_path with bos=163584 eos=163585. - End to end: `download_hf_assets.py --repo_id moonshotai/Moonlight-16B-A3B --assets tokenizer` fetches tiktoken.model plus the tokenization module and writes tokenizer.json. ghstack-source-id: 77a1c96 Pull-Request: #4262

Kimi's non-debug configs cannot build a tokenizer. Moonshot ships vocabularies as ``tiktoken.model``, and torchtitan reads ``tokenizer.json`` or a vocab/merges pair, so ``moonlight_16b_a3b``, ``kimi_vl_a3b`` and ``kimi_k2_5`` all raise "No supported tokenizer files found". Only the debug model works, because it points at the bundled test tokenizer. That has been true since #3532 added them, a year after #1333 dropped tiktoken and standardized on HuggingFace tokenizers. Convert during ``download_hf_assets.py``. Nothing is added to the runtime, the install, or the dev requirements -- the conversion uses ``tokenizers``, already required, and ``tiktoken`` stays gone. Two things the file does not store have to be reconstructed. A BPE model needs the pair each token was merged from, but tiktoken records only the rank it was learned at; replaying the merge loop over a token's own bytes, stopping below its rank, leaves the two halves. And ``tokenizers`` keys its vocabulary by ``str`` where tiktoken uses bytes, so the usual byte-to-character mapping applies. The pre-tokenizer regex is not in the file either and is model-specific, so Moonshot's is a named constant. Special tokens keep their configured ids: ``added_tokens_decoder`` names only some reserved slots, and the unnamed ones still occupy ids, so they are filled with placeholders. Test plan: - pytest tests/unit_tests/test_convert_tiktoken.py -- 3 passed. - Cross-checked against reference tiktoken on the real Moonlight vocabulary: 812 strings from repo sources plus CJK, emoji, whitespace and literal special-token text encode to identical ids, 0 mismatches. tiktoken was installed locally for that check only. - 163840 tokens, matching moonlight_16b_a3b's decoder_vocab_size; ``moonlight_16b_a3b().tokenizer.build()`` now succeeds from its own hf_assets_path with bos=163584 eos=163585. ghstack-source-id: da16c92 Pull-Request: #4262

Kimi's non-debug configs cannot build a tokenizer. The Kimi family ships its vocabulary as ``tiktoken.model``, and torchtitan reads ``tokenizer.json`` or a vocab/merges pair, so ``kimi_k2_5``, ``kimi_vl_a3b`` and ``moonlight_16b_a3b`` all raise "No supported tokenizer files found". Only the debug model works, because it points at the bundled test tokenizer. That has been true since #3532 added them, a year after #1333 dropped tiktoken and standardized on HuggingFace tokenizers. Convert during ``download_hf_assets.py``. Nothing is added to the runtime, the install, or the dev requirements -- the conversion uses ``tokenizers``, already required, and ``tiktoken`` stays gone. Two things the file does not store have to be reconstructed. A BPE model needs the pair each token was merged from, but tiktoken records only the rank it was learned at; replaying the merge loop over a token's own bytes, stopping below its rank, leaves the two halves. And ``tokenizers`` keys its vocabulary by ``str`` where tiktoken uses bytes, so the usual byte-to-character mapping applies. The pre-tokenizer regex is not in the file either, so Kimi's is a named constant. Special tokens keep their configured ids: ``added_tokens_decoder`` names only some reserved slots, and the unnamed ones still occupy ids, so they are filled with placeholders. Test plan: - pytest tests/unit_tests/test_convert_tiktoken.py -- 3 passed. - Cross-checked against reference tiktoken on the real Kimi vocabulary: 812 strings from repo sources plus CJK, emoji, whitespace and literal special-token text encode to identical ids, 0 mismatches. tiktoken was installed locally for that check only. - 163840 tokens, matching the Kimi decoder_vocab_size; ``moonlight_16b_a3b().tokenizer.build()`` now succeeds from its own hf_assets_path with bos=163584 eos=163585. ghstack-source-id: 9b29e5e Pull-Request: #4262

Kimi's non-debug configs cannot build a tokenizer. The Kimi family ships its vocabulary as ``tiktoken.model``, and torchtitan reads ``tokenizer.json`` or a vocab/merges pair, so ``kimi_k2_5``, ``kimi_vl_a3b`` and ``moonlight_16b_a3b`` all raise "No supported tokenizer files found". Only the debug model works, because it points at the bundled test tokenizer. That has been true since #3532 added them, a year after #1333 dropped tiktoken and standardized on HuggingFace tokenizers. Convert during ``download_hf_assets.py``. Nothing is added to the runtime, the install, or the dev requirements -- the conversion uses ``tokenizers``, already required, and ``tiktoken`` stays gone. Two things the file does not store have to be reconstructed. A BPE model needs the pair each token was merged from, but tiktoken records only the rank it was learned at; replaying the merge loop over a token's own bytes, stopping below its rank, leaves the two halves. And ``tokenizers`` keys its vocabulary by ``str`` where tiktoken uses bytes, so the usual byte-to-character mapping applies. The pre-tokenizer regex is not in the file either, so Kimi's is a named constant. Special tokens keep their configured ids: ``added_tokens_decoder`` names only some reserved slots, and the unnamed ones still occupy ids, so they are filled with placeholders. Test plan: - Cross-checked against reference tiktoken on the real Kimi vocabulary: 812 strings from repo sources plus CJK, emoji, whitespace and literal special-token text encode to identical ids, 0 mismatches. tiktoken was installed locally for that check only. - 163840 tokens, matching the Kimi decoder_vocab_size; ``moonlight_16b_a3b().tokenizer.build()`` now succeeds from its own hf_assets_path with bos=163584 eos=163585. ghstack-source-id: c4b9ef6 Pull-Request: #4262

Kimi's non-debug configs cannot build a tokenizer. The Kimi family ships its vocabulary as ``tiktoken.model``, and torchtitan reads ``tokenizer.json`` or a vocab/merges pair, so ``kimi_k2_5``, ``kimi_vl_a3b`` and ``moonlight_16b_a3b`` all raise "No supported tokenizer files found". Only the debug model works, because it points at the bundled test tokenizer. That has been true since #3532 added them, a year after #1333 dropped tiktoken and standardized on HuggingFace tokenizers. Convert during ``download_hf_assets.py``. Nothing is added to the runtime, the install, or the dev requirements -- the conversion uses ``tokenizers``, already required, and ``tiktoken`` stays gone. Two things the file does not store have to be reconstructed. A BPE model needs the pair each token was merged from, but tiktoken records only the rank it was learned at; replaying the merge loop over a token's own bytes, stopping below its rank, leaves the two halves. And ``tokenizers`` keys its vocabulary by ``str`` where tiktoken uses bytes, so the usual byte-to-character mapping applies. The pre-tokenizer regex is not in the file either, so Kimi's is a named constant. Special tokens keep their configured ids: ``added_tokens_decoder`` names only some reserved slots, and the unnamed ones still occupy ids, so they are filled with placeholders. Test plan: - Cross-checked against reference tiktoken on the real Kimi vocabulary: 812 strings from repo sources plus CJK, emoji, whitespace and literal special-token text encode to identical ids, 0 mismatches. tiktoken was installed locally for that check only. - 163840 tokens, matching the Kimi decoder_vocab_size; ``moonlight_16b_a3b().tokenizer.build()`` now succeeds from its own hf_assets_path with bos=163584 eos=163585. ghstack-source-id: 8e87830 Pull-Request: #4262

tianyu-l · 2026-08-21T18:49:46Z

+        - tiktoken.model - tiktoken vocabulary (Kimi models); converted to
+          tokenizer.json after download, since the runtime cannot read it


I vaguely remember in the past we worked hard to modernize the file, but is this hitting a concrete limit with Kimi? @felipemello1 if you still remember the details.

Did you not hit this issue? @shuhuayu

I don't :/. Joe worked on this more than I did.

this worked for me python scripts/download_hf_assets.py --repo_id moonshotai/Moonlight-16B-A3B --assets tokenizer .

will check if this is due to some library difference in my conda env

meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 21, 2026

weifengpy changed the title ~~[tokenizer] Convert tiktoken vocabularies at download time~~ [Kimi] Convert the tiktoken vocabulary at download time Aug 21, 2026

weifengpy marked this pull request as ready for review August 21, 2026 16:14

weifengpy requested review from fegin, tianyu-l, wconstab and wwwjn as code owners August 21, 2026 16:14

weifengpy mentioned this pull request Aug 21, 2026

[kimi k2_7] add kimi k2_7 #3532

Merged

weifengpy force-pushed the gh/weifengpy/64/head branch from f91395e to a92b4a9 Compare August 21, 2026 16:19

weifengpy requested review from IvanKobzarev, SherlockNoMad, aditvenk, sanketpurandare and xmfan as code owners August 21, 2026 16:19

weifengpy changed the base branch from gh/weifengpy/64/base to main August 21, 2026 16:19

weifengpy marked this pull request as draft August 21, 2026 16:19

weifengpy marked this pull request as ready for review August 21, 2026 16:22

tianyu-l requested a review from felipemello1 August 21, 2026 18:46

tianyu-l reviewed Aug 21, 2026

View reviewed changes

weifengpy requested a review from shuhuayu August 21, 2026 19:05

weifengpy marked this pull request as draft August 22, 2026 01:45

Navigation Menu

Uh oh!

[Kimi] Convert the tiktoken vocabulary at download time - #4262

[Kimi] Convert the tiktoken vocabulary at download time#4262
weifengpy wants to merge 1 commit into
mainfrom
gh/weifengpy/64/head

weifengpy commented Aug 21, 2026 •

edited

Loading

Uh oh!

tianyu-l Aug 21, 2026

Uh oh!

felipemello1 Aug 21, 2026

Uh oh!

shuhuayu Aug 21, 2026

Uh oh!

weifengpy Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		- tiktoken.model - tiktoken vocabulary (Kimi models); converted to
		tokenizer.json after download, since the runtime cannot read it

Uh oh!

Conversation

weifengpy commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tianyu-l Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

felipemello1 Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

shuhuayu Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

weifengpy Aug 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

weifengpy commented Aug 21, 2026 •

edited

Loading