exllamav3

LLM inference server for ExLlamaV3 / EXL3, with OpenAI- and Anthropic-compatible APIs optimized for Agent workloads.

Run a 177B model (Qwen3.8-Flash-Next) on an 8 GB laptop GPU. Measured: 6.6 GiB VRAM, 47.8 GiB RAM, 34-35 tok/s. The n-gram table stays on disk; the experts run on CPU.

Blackwell (sm120) build of Mia'a AI Lab's Qwen3.8-27B EXL3 deployment kit. Measured on one RTX 5090: 193 tok/s decode, full native 262k context in 25.4 GiB.

Serve EXL3 (ExLlamaV3 trellis) quantized models on vLLM fork runtimes — any architecture, mixed per-layer bitrates, composable with source-format non-routed weights

FastAPI LLM pool for local and OpenAI-compatible remote inference, with multimodal input, scheduling, replicas, metrics, and admin APIs.