fix: sanitize invalid ids before fast decode

[froststeam]

Background

fastokens.patch_transformers() replaces the Hugging Face tokenizer backend with the fastokens Python compatibility shim.

The serving bug here is not on the normal decode path. It happens when invalid token IDs reach detokenization input, such as negative sentinels or values larger than u32::MAX. In that case the native fast decode call can raise:

OverflowError: out of range integral type conversion attempted

The important part is that valid decode requests must stay on the fast path. We only need a fallback for malformed input.

Changes

• Keep decode() and decode_batch() on the normal fast path.
• Add a narrow fallback in _TokenizerShim that only runs when the native decode path raises OverflowError.
• On that exceptional path, drop token IDs outside [0, 2^32 - 1] and retry the decode.
• Add a regression test that exercises mixed valid/invalid IDs through the patched tokenizer.

This keeps the fast path unchanged and makes malformed input fail open instead of crashing the detokenizer process.

Performance impact

I re-ran the valid-input benchmark against the current wheel and compared it with the baseline wheel built from the parent commit.

The current implementation stays on the same fast path for valid input, so there is no additional filtering or conversion work in the normal case. In local measurements, the valid-path numbers were effectively unchanged and the differences were within normal run-to-run variance.

Testing

Validated locally:

maturin build --release -i python3.10

Smoke test on the built wheel:

fastokens.patch_transformers()
tok.decode([1, -1, 2, 2**32, 3])
tok.batch_decode([[1, -1, 2], [2**32, 3]])