Rust LLM inference engine built to run the latest large GGUF models on an HPE ProLiant DL580 Gen9: NUMA-aware CPU, optional CUDA, OpenAI-compatible server (GLM5Next, Qwen3.5, Llama)
rust cuda self-hosted avx2 llama numa glm homelab inference-engine mixture-of-experts openai-api cpu-inference llm local-llm llm-inference qwen gguf speculative-decoding hpe-proliant dl580
-
Updated
Sep 6, 2026 - Rust