Vulkan llama-server (AMD), parameterized via .env

A single OpenAI-compatible llama-server running the official Vulkan image directly. Models are loaded from HuggingFace. All runtime flags are driven from .env. No orchestrator.

Layout

• .env — every tunable parameter (model, flags, paths, port, GPU GIDs)
• compose.yaml — the llama service + AMD /dev/dri passthrough + healthcheck
• setup-env.sh — resolves host-specific values (RENDER_GID, VIDEO_GID, HF_CACHE_DIR) and writes them into .env

First-time setup

# Resolves GPU group IDs and your HF cache dir, writes them into .env
bash setup-env.sh

docker compose up -d

setup-env.sh fills in:

• RENDER_GID / VIDEO_GID — read from getent group render/video
• HF_CACHE_DIR — set to $HOME/.cache/llama.cpp, created if missing; mounted into the container so downloads persist across restarts

Choosing a model

Set HF_REPO and HF_FILE in .env:

HF_REPO=unsloth/Qwen3-Coder-30B-A3B-GGUF
HF_FILE=Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf

The file is downloaded on first start and cached in HF_CACHE_DIR — no re-download on restart. For gated/private repos, uncomment HF_TOKEN in .env.

Use it (standard OpenAI API)

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"x","messages":[{"role":"user","content":"hi"}]}'

Changing the model or any flag

Edit .env, then:

docker compose up -d

Compose recreates the container with the new parameters (full stop + reload, weights into VRAM, KV cache starts empty).

Caveats

1. Pin BASE_TAG to a dated build (server-vulkan-bNNNN) for stability; see ghcr.io/ggml-org/llama.cpp tags. server-vulkan-latest floats.
2. The Vulkan image sometimes can't see the GPU until extra GL libs are present (libglvnd0 libgl1...). If it loads on CPU, that's the cause.
3. -fa on on Vulkan can disable integer-dot on some builds (perf quirk, not a correctness issue). Benchmark on vs off.
4. The healthcheck assumes curl exists in the image; the slim server image may not include it. Drop the block or add curl via a tiny Dockerfile if so.
5. --slot-save-path enables /slots/<id>/save|restore so you can persist KV to disk manually; nothing automatic happens on a restart.