fix: handle invalid decode ids in native binding

[froststeam]

Background

fastokens.patch_transformers() swaps the Hugging Face tokenizer backend for the native fastokens implementation.

The crash happens only when invalid decode IDs reach detokenization, for example negative sentinels or values larger than u32::MAX. In that case the native conversion path can raise:

OverflowError: out of range integral type conversion attempted

That should not take down the detokenizer process. The common case, where all token IDs are valid, should stay on the same fast path.

What changed

• Kept the Python shim thin: it still forwards directly to _fast.decode() / _fast.decode_batch().
• Moved the fallback into the native PyO3 binding, where the conversion error actually happens.
• First try the normal Vec<u32> extraction.
• Only if that hits OverflowError, retry by extracting signed integers and dropping IDs outside [0, 2^32 - 1].
• Applied the same logic to both single decode and batch decode.

Why this shape

This is the narrowest place to fix the bug:

• it catches the boundary conversion failure at the native entry point,
• it avoids a Python-side pre-scan on every decode,
• it keeps the normal path unchanged for valid inputs,
• and it makes malformed decode input fail open instead of crashing the process.

Performance

The common path still uses the normal Vec<u32> extraction, so there is no extra pre-filtering pass for valid inputs.

I re-ran local decode microbenchmarks against the baseline wheel. The numbers stayed in the same range, and the key property is that the fallback only runs on the exceptional overflow path, not on the hot path.

Testing

Validated locally:

cargo test --manifest-path python/Cargo.toml decode_mixed_valid_and_unknown_ids -- --nocapture
maturin build --release -i python3.10

Smoke test on the built wheel:

fastokens.patch_transformers()
tok.decode([1, -1, 2, 2**32, 3])
tok.batch_decode([[1, -1, 2], [2**32, 3]])

[AlonKejzman]

Thank you for your contribution!