Misc. bug: Is this a backend VRAM tracking bug?

[Diablo-D3]

Name and Version

version: 9512 (0dbfa66)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-cli

Command line

llama-vulkan/llama-cli.exe -hf unsloth/Qwen3.6-27B-MTP-GGUF:Q5_K_XL -ngl all --fit on -lv 4 --no-mmap --fa on --cache-type-k x --cache-type-v x --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-type-k x --spec-draft-type-v x 

Problem description & steps to reproduce

When going through iterations of KV cache and draft KV cache quantization, VRAM usage is not consistent across different backends.

HIP:

kv cache  | draft kv  | context
f16/f16   | ntp       |   49408
q8_0/q8_0 | ntp       |   82944
f16/f16   | f16/f16   |   13312
f16/f16   | q8_0/q8_0 |    4608
q8_0/q8_0 | f16/f16   |   22784
q8_0/q8_0 | q8_0/q8_0 |    8192

Vulkan:

kv cache  | draft kv  | context
f16/f16   | ntp       |   39936
q8_0/q8_0 | ntp       |   74496
f16/f16   | f16/f16   |    8960
f16/f16   | q8_0/q8_0 |   16640
q8_0/q8_0 | f16/f16   |   17408
q8_0/q8_0 | q8_0/q8_0 |   31488 

According to #24102 (comment) the HIP numbers are correct, but the Vulkan numbers are not.

First Bad Commit

N/A

Relevant log output

N/A

[regunakyle]

FWIW I also tested different configurations:

fit	VRAM usage (MiB) after initial load	Context size in UI	Spec draft quant
yes	22604	66560	q4
yes	22604	74752	f16
yes (--fit-target 64)	23570	90112	q4
yes (--fit-target 64)	23582	98560	f16
no (-c 45000)	19786	45056	q4
no (-c 45000)	19808	45056	f16
no (-c 5000)	18128	5120	q4
no (-c 5000)	18102	5120	f16

Setup: Fedora Server 44, Nvidia 3090, version: 9528 (2016bf2)
built with GNU 16.1.1 for Linux x86_64

Full command:

/home/eleung/llama.cpp/build/bin/llama-server \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --presence_penalty 0.0 \
    --min-p 0.00 \
    --gpu-layers all \
    -m /home/eleung/huggingface/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-UD-Q4_K_XL.gguf \
    -a llama.cpp \
    --host 0.0.0.0 \
    --cache-type-k q8_0 --cache-type-v q8_0 \
    --chat-template-kwargs '{"preserve_thinking":true}' \
    --flash-attn on \
    --ui-mcp-proxy \
    --jinja --chat-template-file /home/eleung/huggingface/unsloth/Qwen3.6-27B-MTP-GGUF/chat_template.jinja \
    --ubatch-size 2048 \
    --parallel 1 -kvu \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    <-fit on/off> \
    <--spec-draft-type-k q4_0 --spec-draft-type-v q4_0>

[JohannesGaessler]

The reported memory sizes are from the perspective of the ggml library but there is no support for measuring memory that is allocated internally by a ggml backend. So the numbers are only directly comparable if runtime allocations are negligible which they tend not to be for quantized KV cache due to temporary buffers for type conversions. In the CUDA/HIP backends this has recently become measurable in #23907 .