Tenstorrent Inference Server (tt-inference-server) is the repo of available model APIs for deploying on Tenstorrent hardware.
https://github.com/tenstorrent/tt-inference-server
Please follow setup instructions for the model you want to serve, Model Name in tables below link to corresponding implementation.
Note: models with Status [🛠️ Experimental] are under active development. If you encounter setup or stability problems with any model please file an issue and our team will address it.
For automated and pre-configured vLLM inference server using Docker please see the Model Readiness Workflows User Guide. The list below shows the default model implementations supported.
| Model Weights |
Hardware |
Status |
tt-metal commit |
vLLM commit |
Docker Image |
| AFM-4.5B |
n300, WH-QuietBox/WH-LoudBox (T3K) |
🛠️ Experimental |
ae65ee5 |
35f023f |
0.3.0-ae65ee5-35f023f |
| gemma-3-1b-it |
n150 |
🛠️ Experimental |
c254ee3 |
c4f2327 |
0.3.0-c254ee3-c4f2327 |
gemma-3-4b-it
medgemma-4b-it |
n300, n150 |
🛠️ Experimental |
c254ee3 |
c4f2327 |
0.3.0-c254ee3-c4f2327 |
gemma-3-27b-it
medgemma-27b-it |
WH-QuietBox/WH-LoudBox (T3K) |
🛠️ Experimental |
c254ee3 |
c4f2327 |
0.3.0-c254ee3-c4f2327 |
| Qwen2.5-VL-3B-Instruct |
n300, WH-QuietBox/WH-LoudBox (T3K), n150 |
🛠️ Experimental |
5bf679a |
48eba14 |
0.3.0-5bf679a-48eba14 |
| Qwen2.5-VL-7B-Instruct |
n300, WH-QuietBox/WH-LoudBox (T3K), n150 |
🛠️ Experimental |
5bf679a |
48eba14 |
0.3.0-5bf679a-48eba14 |
| Qwen2.5-VL-32B-Instruct |
WH-QuietBox/WH-LoudBox (T3K) |
🛠️ Experimental |
5bf679a |
48eba14 |
0.3.0-5bf679a-48eba14 |
| Qwen2.5-VL-72B-Instruct |
WH-QuietBox/WH-LoudBox (T3K) |
🟡 Functional |
5bf679a |
48eba14 |
0.3.0-5bf679a-48eba14 |
| Qwen3-8B |
n150, n300, WH-QuietBox/WH-LoudBox (T3K), Galaxy |
🟡 Functional |
e95ffa5 |
48eba14 |
0.3.0-e95ffa5-48eba14 |
| Qwen3-32B |
WH-QuietBox/WH-LoudBox (T3K), Galaxy |
🟡 Functional |
e95ffa5 |
48eba14 |
0.3.0-e95ffa5-48eba14 |
| Mistral-7B-Instruct-v0.3 |
n300, WH-QuietBox/WH-LoudBox (T3K), n150 |
🟢 Complete |
9b67e09 |
a91b644 |
0.3.0-9b67e09-a91b644 |
| QwQ-32B |
WH-QuietBox/WH-LoudBox (T3K), Galaxy |
🟡 Functional |
e95ffa5 |
48eba14 |
0.3.0-e95ffa5-48eba14 |
Qwen2.5-72B
Qwen2.5-72B-Instruct |
WH-QuietBox/WH-LoudBox (T3K), Galaxy |
🟡 Functional |
13f44c5 |
0edd242 |
0.3.0-13f44c5-0edd242 |
Qwen2.5-7B
Qwen2.5-7B-Instruct |
n300 |
🛠️ Experimental |
5b5db8a |
e771fff |
0.3.0-5b5db8a-e771fff |
Llama-3.3-70B-Instruct
Llama-3.1-70B
Llama-3.1-70B-Instruct
DeepSeek-R1-Distill-Llama-70B |
Galaxy |
🟢 Complete |
e95ffa5 |
48eba14 |
0.3.0-e95ffa5-48eba14 |
Llama-3.3-70B-Instruct
Llama-3.1-70B
Llama-3.1-70B-Instruct
DeepSeek-R1-Distill-Llama-70B |
WH-QuietBox/WH-LoudBox (T3K) |
🟡 Functional |
9b67e09 |
a91b644 |
0.3.0-9b67e09-a91b644 |
Llama-3.3-70B-Instruct
Llama-3.1-70B
Llama-3.1-70B-Instruct
DeepSeek-R1-Distill-Llama-70B |
BH-QuietBox (P150X4) |
🟡 Functional |
55fd115 |
aa4ae1e |
0.3.0-55fd115-aa4ae1e |
Llama-3.2-11B-Vision
Llama-3.2-11B-Vision-Instruct |
n300, WH-QuietBox/WH-LoudBox (T3K) |
🟡 Functional |
v0.61.1-rc1 |
5cbc982 |
0.3.0-v0.61.1-rc1-5cbc982 |
Llama-3.2-90B-Vision
Llama-3.2-90B-Vision-Instruct |
WH-QuietBox/WH-LoudBox (T3K) |
🟡 Functional |
v0.61.1-rc1 |
5cbc982 |
0.3.0-v0.61.1-rc1-5cbc982 |
Llama-3.2-1B
Llama-3.2-1B-Instruct |
n300, WH-QuietBox/WH-LoudBox (T3K), n150 |
🟡 Functional |
9b67e09 |
a91b644 |
0.3.0-9b67e09-a91b644 |
Llama-3.2-3B
Llama-3.2-3B-Instruct |
n300, WH-QuietBox/WH-LoudBox (T3K), n150 |
🟡 Functional |
20edc39 |
03cb300 |
0.3.0-20edc39-03cb300 |
Llama-3.1-8B
Llama-3.1-8B-Instruct |
n300, WH-QuietBox/WH-LoudBox (T3K), n150 |
🟢 Complete |
9b67e09 |
a91b644 |
0.3.0-9b67e09-a91b644 |
Llama-3.1-8B
Llama-3.1-8B-Instruct |
p100, p150 |
🛠️ Experimental |
55fd115 |
aa4ae1e |
0.3.0-55fd115-aa4ae1e |
Llama-3.1-8B
Llama-3.1-8B-Instruct |
BH-QuietBox (P150X4) |
🟢 Complete |
55fd115 |
aa4ae1e |
0.3.0-55fd115-aa4ae1e |
Llama-3.1-8B
Llama-3.1-8B-Instruct |
Galaxy |
🟡 Functional |
e95ffa5 |
48eba14 |
0.3.0-e95ffa5-48eba14 |
| Qwen2.5-Coder-32B-Instruct |
WH-QuietBox/WH-LoudBox (T3K) |
🛠️ Experimental |
17a5973 |
aa4ae1e |
0.3.0-17a5973-aa4ae1e |
| stable-diffusion-xl-base-1.0 |
Galaxy, n300, WH-QuietBox/WH-LoudBox (T3K), n150 |
🟢 Complete |
13f44c5 |
N/A |
0.4.0-e95ffa59adbe39237525161272141cbbb603c686 |
| stable-diffusion-3.5-large |
WH-QuietBox/WH-LoudBox (T3K), Galaxy |
🟢 Complete |
13f44c5 |
N/A |
0.4.0-e95ffa59adbe39237525161272141cbbb603c686 |
whisper-large-v3
distil-large-v3 |
Galaxy, WH-QuietBox/WH-LoudBox (T3K), n150 |
🟢 Complete |
13f44c5 |
N/A |
0.4.0-e95ffa59adbe39237525161272141cbbb603c686 |
| resnet-50 |
n300, n150 |
🛠️ Experimental |
2496be4 |
N/A |
0.2.0-2496be4518bca0a7a5b497a4cda3cfe7e2f59756 |
| vovnet |
n300, n150 |
🛠️ Experimental |
2496be4 |
N/A |
0.2.0-2496be4518bca0a7a5b497a4cda3cfe7e2f59756 |
| mobilenetv2 |
n300, n150 |
🛠️ Experimental |
2496be4 |
N/A |
0.2.0-2496be4518bca0a7a5b497a4cda3cfe7e2f59756 |