Skip to content

Add managed TensorRT-LLM serve backend - #1

Open
Bobcat wants to merge 1 commit into
mainfrom
feat/trtllm-serve-backend
Open

Add managed TensorRT-LLM serve backend#1
Bobcat wants to merge 1 commit into
mainfrom
feat/trtllm-serve-backend

Conversation

@Bobcat

@Bobcat Bobcat commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add trtllm_serve as a managed local backend alongside llama.cpp and vLLM serve.
  • Start and stop TensorRT-LLM in its own process group, wait for health, and forward chat-completion requests.
  • Expose TensorRT-LLM runtime settings through model definitions and router administration.
  • Add a disabled Gemma 4 26B A4B NVFP4 example with bounded compilation concurrency.
  • Cover configuration parsing, lifecycle, request translation, response handling, and router dispatch.

Validation

  • 197 passed, 7 skipped, 6 subtests passed
  • Loaded /home/gunnar/models/nvidia/Gemma-4-26B-A4B-NVFP4 with TensorRT-LLM 1.3.0rc25.
  • Completed text generation through the managed llm-pool backend.
  • Verified startup health checks, graceful process-group shutdown, and the OpenAI-compatible response mapping.

Scope

The example model remains disabled by default. The repository does not install TensorRT-LLM or its CUDA/OpenMPI runtime.

MTP speculative decoding is intentionally not enabled. The current upstream shared-KV implementation depends on the trtllm-gen attention backend used on SM100/SM103, while this RTX PRO 6000 SM120 system uses FlashInfer FA2. An eager SM120 proof of concept worked, but CUDA graphs and batch sizes above one are not ready for this backend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant