Latest version of transformers breaks Infinity Server

[vrdn-23]

System Info

INFO     2025-05-20 20:03:58,938 datasets INFO: PyTorch version     config.py:54
         2.6.0 available.                                                       
Traceback (most recent call last):
  File "/app/.venv/bin/infinity_emb", line 4, in <module>
    from infinity_emb.cli import cli
  File "/app/.venv/lib/python3.12/site-packages/infinity_emb/__init__.py", line 27, in <module>
    from infinity_emb.engine import AsyncEmbeddingEngine, AsyncEngineArray  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/infinity_emb/engine.py", line 11, in <module>
    from infinity_emb.inference import (
  File "/app/.venv/lib/python3.12/site-packages/infinity_emb/inference/__init__.py", line 4, in <module>
    from infinity_emb.inference.batch_handler import BatchHandler
  File "/app/.venv/lib/python3.12/site-packages/infinity_emb/inference/batch_handler.py", line 39, in <module>
    from infinity_emb.transformer.utils import get_lengths_with_tokenize
  File "/app/.venv/lib/python3.12/site-packages/infinity_emb/transformer/utils.py", line 9, in <module>
    from infinity_emb.transformer.classifier.torch import SentenceClassifier
  File "/app/.venv/lib/python3.12/site-packages/infinity_emb/transformer/classifier/torch.py", line 8, in <module>
    from infinity_emb.transformer.acceleration import (
  File "/app/.venv/lib/python3.12/site-packages/infinity_emb/transformer/acceleration.py", line 11, in <module>
    from optimum.bettertransformer import (  # type: ignore[import-untyped]
  File "/app/.venv/lib/python3.12/site-packages/optimum/bettertransformer/__init__.py", line 20, in <module>
    raise RuntimeError(
RuntimeError: BetterTransformer requires transformers<4.49 but found 4.50.0. `optimum.bettertransformer` is deprecated and will be removed in optimum v2.0.

We might need to pin the transformers version or maybe even deprecate the better transformer version

Information

• Docker + cli
• pip + cli
• pip + usage of Python interface

Tasks

• An officially supported CLI command
• My own modifications

Reproduction

uv run infinity_emb v2 --model-id any-model --port 8000 --url-prefix /v1

[vrdn-23]

cc @michaelfeil

[mtrmarko]

I think the issue lies with optimum dependency. I tried the latest version of optimum with same results. I opened a bug with optimum project. See huggingface/optimum#2277. For the time being I was forced to downgrade transformers, I could have tried removing the runtime error from the code I referenced in the issue, but did not feel like scripting that inside the docker image build. :)

[wirthual]

@michaelfeil What do you think is the preferred way forward with this issue. Should bettertransformers be dropped from infinity or should some version check for transformers>4.49 be introduced for backwards compatibility?

[bannert1337]

A version check would be nice as a short term fix, as it currently is unable to even run with --no-bettertransformer

[wirthual]

This branch should overcome this problem: #600. Happy to receive feedback if its working as intended.