fix: implement add_tokens instead of silently ignoring it

[murphymatt]

Summary

_TokenizerShim.add_tokens and add_special_tokens returned 0 and threw their
argument away, so the patched transformers backend could not grow its
vocabulary. That makes fastokens unusable on any checkpoint whose embedding
matrix is padded above the tokenizer's token count — a common shape. Loaders
reconcile the two by appending placeholder tokens until len(tokenizer) reaches
the embedding count and then assert that it did:

tokenizer.add_tokens([f"<|padding_token_{i}|>" for i in range(num_embeddings - len(tokenizer))])
if len(tokenizer) != num_embeddings:
    raise ...   # always, under fastokens

The failure is fail-closed at load, so nothing is ever mis-tokenized; the
backend is simply unavailable on those models. Models whose tokenizer and
embedding counts already agree never enter the padding branch, which is why this
went unnoticed.

Tracing it surfaced a second stored-but-unused value: encode_special_tokens,
which transformers sets for every split_special_tokens=True encode. The shim
kept the flag and encoded as though it were unset, so control-token strings in
untrusted text still produced control-token ids. Callers pair that flag with
add_special_tokens to promote ordinary added tokens to special first, so both
no-ops had to go for that path to work at all.

What changed

Vocabulary extension is now native, and ids are assigned the way tokenizers
assigns them so the two backends stay interchangeable under the same loader:

• a content already in the added vocabulary keeps its id and only its flags can
  change — this is what promotes a token to special;
• a content the model already carries becomes an added token at that same id,
  leaving the vocabulary size untouched;
• anything else is appended contiguously above the vocabulary, which lands
  padding placeholders exactly on the padded embedding rows.

Tokenizer::add_tokens compiles the new set before committing it, so a rejected
batch cannot leave a half-extended tokenizer, and it drops the opt-in prefix
cache, whose entries predate the new tokens. AddedTokenPolicy threads through
encode, adding the SkipSpecial mode that leaves special tokens as ordinary
text while the rest of the added vocabulary still matches — the exact rule
upstream applies under encode_special_tokens.

On the Python side the shim reads added tokens back from the backend instead of
the constructor JSON, so extensions survive to_str, pickling, deepcopy, and
added_tokens_decoder. get_vocab unions the id scan with the added entries,
which also keeps len(get_vocab()) == get_vocab_size() when declared ids leave a
gap.

One divergence is deliberately left alone: get_vocab_size answers the full
vocabulary for both values of with_added_tokens, where upstream returns the
base vocabulary for False. The backend does not track the split, and
over-reporting is the safer direction for callers that size embedding tables or
logit masks from it, so it is documented in place rather than changed here.

Test plan

• cargo test — 264 pass, 16 new. Includes a parity test that runs the same
  batches through tokenizers and this crate and compares ids and
  vocabulary sizes across all four cases (new contents, a repeat, a string
  the model already has, a promotion to special).
• New python/tests/test_add_tokens.py covers the shim directly: padding up
  to an embedding count, idempotence, promotion, serialization round-trips,
  and special-token splitting. Wired into CI alongside the existing
  transformers job.
• python/tests/test_patch_transformers.py asserts end-to-end equality with
  the unpatched backend for both padding and split_special_tokens.
• Green on transformers 4.57.1 and 5.3.0 (both patch paths).
• cargo fmt --check, cargo clippy -- -D warnings.

Made with Cursor

[AlonKejzman]

Thank you for your contribution!