Eval bug: Qwen 3.6 27B forcing full prompt re-processing due to lack of cache data

[tarunkapadia-2266]

Name and Version

PS D:\llm\llama-b8977-bin-win-hip-radeon-x64> .\llama-cli --version
HIP Library Path: C:\WINDOWS\SYSTEM32\amdhip64_7.dll
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16368 MiB):
Device 0: AMD Radeon RX 7800 XT, gfx1101 (0x1101), VMM: no, Wave Size: 32, VRAM: 16368 MiB
load_backend: loaded ROCm backend from D:\llm\llama-b8977-bin-win-hip-radeon-x64\ggml-hip.dll
load_backend: loaded RPC backend from D:\llm\llama-b8977-bin-win-hip-radeon-x64\ggml-rpc.dll
load_backend: loaded CPU backend from D:\llm\llama-b8977-bin-win-hip-radeon-x64\ggml-cpu-zen4.dll
version: 8977 (b1d5f5b)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

GGML backends

HIP

Hardware

PC Spec:
Ryzen 7700x
32GB DDR5 5200 Mhz
16 GB 7800 XT
2TB nvme SSD

Models

Qwen 3.6 27B @ IQ4_XS / Q4_K_M / UD-IQ3_XXS / Q3_K_M

Problem description & steps to reproduce

llama server start up script... also i am using opencode as the harness

#!/bin/bash

QWEN_3_6_27_Q3_K_M='models\\Qwen3.6-27B-Q3_K_M.gguf'
QWEN_3_6_27_UD-IQ3_XXS='models\\Qwen3.6-27B-UD-IQ3_XXS.gguf'
QWEN_3_6_27_Q4_K_M='models\\Qwen3.6-27B-Q4_K_M.gguf'
QWEN_3_6_27_IQ4_XS='models\\Qwen3.6-27B-IQ4_XS.gguf'

# --- CORE PATHS ---
LLAMA_BIN="llama-b8977-bin-win-hip-radeon-x64/llama-server.exe"
MODELS_PATH="models/"
SLOT_PATH="models/cache"
PROMPT_CACHE='models/cache/main_cache.bin'

# --- CORE SETTINGS ---
CTX_SIZE=131072 #65336
GPU_LAYERS=99
THREADS=16
K_TYPE="q8_0"
V_TYPE="q8_0"
PORT=8081
ADDR="0.0.0.0"

# --- AGENTIC TUNING ---
TEMP="0.6"
MIN_P="0.0"
TOP_P="0.95"
TOP_K="20"
PRESENCE_PENALTY="0.0"
REPEAT_PENALTY="1.0"

# Optimized for Branching/Multi-Agent (OpenCode)
BATCH_SIZE=512
UBATCH_SIZE=512
CACHE_RAM_SIZE=18432 # 18GB RAM for snapshots
CTX_CHECKPOINTS=64 # Sufficient for 18GB
CHECKPOINT_INTERVAL=8192 # Smaller interval = Faster switching between agents

# --- CONSTRUCT COMMAND ---

CMD="$LLAMA_BIN \
  --model \"$QWEN_3_6_27_Q3_K_M\" \
  --models-max 1 \
  --ctx-size $CTX_SIZE \
  --n-gpu-layers $GPU_LAYERS \
  --threads $THREADS \
  --flash-attn on \
  --cache-type-k $K_TYPE \
  --cache-type-v $V_TYPE \
  --batch-size $BATCH_SIZE \
  --ubatch-size $UBATCH_SIZE \
  --cache-ram $CACHE_RAM_SIZE \
  --checkpoint-every-n-tokens $CHECKPOINT_INTERVAL \
  --cache-reuse 1 \
  --ctx-checkpoints $CTX_CHECKPOINTS \
  --log-colors on \
  --slot-prompt-similarity 0.1 \
  --metrics \
  --reasoning on \
  --parallel 2 \
  --cache-prompt \
  --no-prefill-assistant \
  --cont-batching \
  --port $PORT \
  --host $ADDR \
  --jinja \
  --no-mmap \
  --embeddings \
  --webui \
  --chat-template-kwargs '{\"enable_thinking\": true, \"preserve_thinking\": true}'"

# --- ADD SAMPLING ---
CMD="$CMD --temp $TEMP --min-p $MIN_P --top-p $TOP_P --top-k $TOP_K"
CMD="$CMD --presence-penalty $PRESENCE_PENALTY --repeat-penalty $REPEAT_PENALTY"

echo "-------------------------------------------------------"
echo "Launching Llama.cpp Server"
echo "-------------------------------------------------------"
eval $CMD

First Bad Commit

No response

Relevant log output

Logs

srv  update_slots: all slots are idle
srv  params_from_: Chat format: peg-native
slot get_availabl: id  1 | task -1 | selected slot by LCP similarity, sim_best = 0.965 (> 0.100 thold), f_keep = 0.967
slot launch_slot_: id  1 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ?top-p -> ?min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id  1 | task 93797 | processing task, is_child = 0
slot update_slots: id  1 | task 93797 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 9287
slot update_slots: id  1 | task 93797 | n_past = 8962, slot.prompt.tokens.size() = 9267, seq_id = 1, pos_min = 9266, n_swa = 0
slot update_slots: id  1 | task 93797 | Checking checkpoint with [8886, 8886] against 8962...
slot update_slots: id  1 | task 93797 | restored context checkpoint (pos_min = 8886, pos_max = 8886, n_tokens = 8887, n_past = 8887, size = 149.626 MiB)
slot update_slots: id  1 | task 93797 | n_tokens = 8887, memory_seq_rm [8887, end)
slot update_slots: id  1 | task 93797 | prompt processing progress, n_tokens = 9283, batch.n_tokens = 396, progress = 0.999569
slot update_slots: id  1 | task 93797 | n_tokens = 9283, memory_seq_rm [9283, end)
reasoning-budget: activated, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
slot init_sampler: id  1 | task 93797 | init sampler, took 0.99 ms, tokens: text = 9287, total = 9287
slot update_slots: id  1 | task 93797 | prompt processing done, n_tokens = 9287, batch.n_tokens = 4
slot create_check: id  1 | task 93797 | created context checkpoint 8 of 64 (pos_min = 9282, pos_max = 9282, n_tokens = 9283, size = 149.626 MiB)
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
reasoning-budget: deactivated (natural end)
slot print_timing: id  1 | task 93797 |
prompt eval time =    2374.07 ms /   400 tokens (    5.94 ms per token,   168.49 tokens per second)
       eval time =   10536.28 ms /   111 tokens (   94.92 ms per token,    10.54 tokens per second)
      total time =   12910.35 ms /   511 tokens
slot      release: id  1 | task 93797 | stop processing: n_tokens = 9397, truncated = 0
srv  update_slots: all slots are idle
srv  params_from_: Chat format: peg-native
slot get_availabl: id  1 | task -1 | selected slot by LCP similarity, sim_best = 0.928 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id  1 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ?top-p -> ?min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id  1 | task 93910 | processing task, is_child = 0
slot update_slots: id  1 | task 93910 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 10121
slot update_slots: id  1 | task 93910 | n_tokens = 9397, memory_seq_rm [9397, end)
slot update_slots: id  1 | task 93910 | prompt processing progress, n_tokens = 9609, batch.n_tokens = 212, progress = 0.949412
slot update_slots: id  1 | task 93910 | n_tokens = 9609, memory_seq_rm [9609, end)
slot update_slots: id  1 | task 93910 | prompt processing progress, n_tokens = 10117, batch.n_tokens = 508, progress = 0.999605
slot create_check: id  1 | task 93910 | created context checkpoint 9 of 64 (pos_min = 9608, pos_max = 9608, n_tokens = 9609, size = 149.626 MiB)
slot update_slots: id  1 | task 93910 | n_tokens = 10117, memory_seq_rm [10117, end)
reasoning-budget: activated, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
slot init_sampler: id  1 | task 93910 | init sampler, took 1.08 ms, tokens: text = 10121, total = 10121
slot update_slots: id  1 | task 93910 | prompt processing done, n_tokens = 10121, batch.n_tokens = 4
slot create_check: id  1 | task 93910 | created context checkpoint 10 of 64 (pos_min = 10116, pos_max = 10116, n_tokens = 10117, size = 149.626 MiB)
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
reasoning-budget: deactivated (natural end)
slot print_timing: id  1 | task 93910 |
prompt eval time =    4096.00 ms /   724 tokens (    5.66 ms per token,   176.76 tokens per second)
       eval time =   24213.29 ms /   250 tokens (   96.85 ms per token,    10.32 tokens per second)
      total time =   28309.29 ms /   974 tokens
slot      release: id  1 | task 93910 | stop processing: n_tokens = 10370, truncated = 0
srv  update_slots: all slots are idle
srv  params_from_: Chat format: peg-native
slot get_availabl: id  1 | task -1 | selected slot by LCP similarity, sim_best = 0.902 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id  1 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ?top-p -> ?min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id  1 | task 94163 | processing task, is_child = 0
slot update_slots: id  1 | task 94163 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 11502
slot update_slots: id  1 | task 94163 | n_tokens = 10370, memory_seq_rm [10370, end)
slot update_slots: id  1 | task 94163 | prompt processing progress, n_tokens = 10882, batch.n_tokens = 512, progress = 0.946096
slot update_slots: id  1 | task 94163 | n_tokens = 10882, memory_seq_rm [10882, end)
slot update_slots: id  1 | task 94163 | prompt processing progress, n_tokens = 10990, batch.n_tokens = 108, progress = 0.955486
slot update_slots: id  1 | task 94163 | n_tokens = 10990, memory_seq_rm [10990, end)
slot update_slots: id  1 | task 94163 | prompt processing progress, n_tokens = 11498, batch.n_tokens = 508, progress = 0.999652
slot create_check: id  1 | task 94163 | created context checkpoint 11 of 64 (pos_min = 10989, pos_max = 10989, n_tokens = 10990, size = 149.626 MiB)
slot update_slots: id  1 | task 94163 | n_tokens = 11498, memory_seq_rm [11498, end)
reasoning-budget: activated, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
slot init_sampler: id  1 | task 94163 | init sampler, took 1.22 ms, tokens: text = 11502, total = 11502
slot update_slots: id  1 | task 94163 | prompt processing done, n_tokens = 11502, batch.n_tokens = 4
slot create_check: id  1 | task 94163 | created context checkpoint 12 of 64 (pos_min = 11497, pos_max = 11497, n_tokens = 11498, size = 149.626 MiB)
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
reasoning-budget: deactivated (natural end)
slot print_timing: id  1 | task 94163 |
prompt eval time =    6487.16 ms /  1132 tokens (    5.73 ms per token,   174.50 tokens per second)
       eval time =   17400.56 ms /   177 tokens (   98.31 ms per token,    10.17 tokens per second)
      total time =   23887.72 ms /  1309 tokens
slot      release: id  1 | task 94163 | stop processing: n_tokens = 11678, truncated = 0
srv  update_slots: all slots are idle
srv  params_from_: Chat format: peg-native
slot get_availabl: id  1 | task -1 | selected slot by LCP similarity, sim_best = 0.998 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id  1 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ?top-p -> ?min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id  1 | task 94344 | processing task, is_child = 0
slot update_slots: id  1 | task 94344 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 11697
slot update_slots: id  1 | task 94344 | n_tokens = 11678, memory_seq_rm [11678, end)
slot update_slots: id  1 | task 94344 | prompt processing progress, n_tokens = 11693, batch.n_tokens = 15, progress = 0.999658
slot create_check: id  1 | task 94344 | created context checkpoint 13 of 64 (pos_min = 11677, pos_max = 11677, n_tokens = 11678, size = 149.626 MiB)
slot update_slots: id  1 | task 94344 | n_tokens = 11693, memory_seq_rm [11693, end)
reasoning-budget: activated, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
slot init_sampler: id  1 | task 94344 | init sampler, took 1.24 ms, tokens: text = 11697, total = 11697
slot update_slots: id  1 | task 94344 | prompt processing done, n_tokens = 11697, batch.n_tokens = 4
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
reasoning-budget: deactivated (natural end)
slot print_timing: id  1 | task 94344 |
prompt eval time =     400.78 ms /    19 tokens (   21.09 ms per token,    47.41 tokens per second)
       eval time =    8571.52 ms /    87 tokens (   98.52 ms per token,    10.15 tokens per second)
      total time =    8972.30 ms /   106 tokens
slot      release: id  1 | task 94344 | stop processing: n_tokens = 11783, truncated = 0
srv  update_slots: all slots are idle
srv  params_from_: Chat format: peg-native
slot get_availabl: id  1 | task -1 | selected slot by LCP similarity, sim_best = 0.986 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id  1 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ?top-p -> ?min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id  1 | task 94433 | processing task, is_child = 0
slot update_slots: id  1 | task 94433 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 11956
slot update_slots: id  1 | task 94433 | n_tokens = 11783, memory_seq_rm [11783, end)
slot update_slots: id  1 | task 94433 | prompt processing progress, n_tokens = 11952, batch.n_tokens = 169, progress = 0.999665
slot create_check: id  1 | task 94433 | created context checkpoint 14 of 64 (pos_min = 11782, pos_max = 11782, n_tokens = 11783, size = 149.626 MiB)
slot update_slots: id  1 | task 94433 | n_tokens = 11952, memory_seq_rm [11952, end)
reasoning-budget: activated, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
reasoning-budget: re-activated on new start tag, budget=2147483647 tokens
slot init_sampler: id  1 | task 94433 | init sampler, took 1.15 ms, tokens: text = 11956, total = 11956
slot update_slots: id  1 | task 94433 | prompt processing done, n_tokens = 11956, batch.n_tokens = 4
slot create_check: id  1 | task 94433 | created context checkpoint 15 of 64 (pos_min = 11951, pos_max = 11951, n_tokens = 11952, size = 149.626 MiB)
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
reasoning-budget: deactivated (natural end)
slot print_timing: id  1 | task 94433 |
prompt eval time =    1289.55 ms /   173 tokens (    7.45 ms per token,   134.16 tokens per second)
       eval time =   17754.35 ms /   179 tokens (   99.19 ms per token,    10.08 tokens per second)
      total time =   19043.89 ms /   352 tokens
slot      release: id  1 | task 94433 | stop processing: n_tokens = 12134, truncated = 0
srv  update_slots: all slots are idle
srv  params_from_: Chat format: peg-native
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = 65026105309
slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ?top-p -> ?min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id  0 | task 94614 | processing task, is_child = 0
slot update_slots: id  0 | task 94614 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 86082
srv    send_error: task id = 94614, error: request (86082 tokens) exceeds the available context size (65536 tokens), try increasing it
slot      release: id  0 | task 94614 | stop processing: n_tokens = 12947, truncated = 0
srv          stop: cancel task, id_task = 94614
srv  update_slots: no tokens to decode
srv  update_slots: all slots are idle
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 400
srv  params_from_: Chat format: peg-native
slot get_availabl: id  1 | task -1 | selected slot by LRU, t_last = 65147626039
slot launch_slot_: id  1 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ?top-p -> ?min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id  1 | task 94617 | processing task, is_child = 0
slot update_slots: id  1 | task 94617 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 53564
slot update_slots: id  1 | task 94617 | n_past = 3, slot.prompt.tokens.size() = 12134, seq_id = 1, pos_min = 12133, n_swa = 0
slot update_slots: id  1 | task 94617 | Checking checkpoint with [11951, 11951] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [11782, 11782] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [11677, 11677] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [11497, 11497] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [10989, 10989] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [10116, 10116] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [9608, 9608] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [9282, 9282] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [8886, 8886] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [8590, 8590] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [8447, 8447] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [8249, 8249] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [8162, 8162] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [7909, 7909] against 3...
slot update_slots: id  1 | task 94617 | Checking checkpoint with [7401, 7401] against 3...
slot update_slots: id  1 | task 94617 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 7401, pos_max = 7401, n_tokens = 7402, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 7909, pos_max = 7909, n_tokens = 7910, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 8162, pos_max = 8162, n_tokens = 8163, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 8249, pos_max = 8249, n_tokens = 8250, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 8447, pos_max = 8447, n_tokens = 8448, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 8590, pos_max = 8590, n_tokens = 8591, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 8886, pos_max = 8886, n_tokens = 8887, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 9282, pos_max = 9282, n_tokens = 9283, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 9608, pos_max = 9608, n_tokens = 9609, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 10116, pos_max = 10116, n_tokens = 10117, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 10989, pos_max = 10989, n_tokens = 10990, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 11497, pos_max = 11497, n_tokens = 11498, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 11677, pos_max = 11677, n_tokens = 11678, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 11782, pos_max = 11782, n_tokens = 11783, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | erased invalidated context checkpoint (pos_min = 11951, pos_max = 11951, n_tokens = 11952, n_swa = 0, pos_next = 0, size = 149.626 MiB)
slot update_slots: id  1 | task 94617 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  1 | task 94617 | prompt processing progress, n_tokens = 512, batch.n_tokens = 512, progress = 0.009559
slot update_slots: id  1 | task 94617 | n_tokens = 512, memory_seq_rm [512, end)
slot update_slots: id  1 | task 94617 | prompt processing progress, n_tokens = 1024, batch.n_tokens = 512, progress = 0.019117
slot update_slots: id  1 | task 94617 | n_tokens = 1024, memory_seq_rm [1024, end)
slot update_slots: id  1 | task 94617 | prompt processing progress, n_tokens = 1536, batch.n_tokens = 512, progress = 0.028676
slot update_slots: id  1 | task 94617 | n_tokens = 1536, memory_seq_rm [1536, end)
slot update_slots: id  1 | task 94617 | prompt processing progress, n_tokens = 2048, batch.n_tokens = 512, progress = 0.03823

[Setibs]

I believe it is a problem with OpenCode, the same thing happens when using Hermes.

[gokussjx]

Generalized issue. Curl commands trigger the same against the API.
I can confirm this over both Qwen3.6 27B and 35B A3B models. MMPROJ off or on.

[aldehir]

Attach a repro.

[ashirviskas]

Possibly related, also on Qwen 35B A3B using Vulkan: #22756

[gokussjx]

@aldehir

⬢ [toolbx test]$ python3 repro.py --url http://localhost:11001 --model Qwen3.6-35B-A3B
========================================================================
Pigeonhole reproducer: 5 agents
  A1: 15 paragraphs (~450 tokens estimated)
  A2: 40 paragraphs (~1200 tokens estimated)
  A3: 80 paragraphs (~2400 tokens estimated)
  A4: 130 paragraphs (~3900 tokens estimated)
  A5: 370 paragraphs (~11100 tokens estimated)
Rounds: 3
Server: http://localhost:11001  Model: Qwen3.6-35B-A3B
========================================================================
Each round runs all agents serially in order: A1, A2, ..., AN
Same prompts every round. Cache hits expected from round 2 onward.
========================================================================

--- Round 1 ---
  A1:   0.87s  prompt=   701  completion= 30
  A2:   1.15s  prompt=  1826  completion= 30
  A3:   1.83s  prompt=  3626  completion= 30
  A4:   2.46s  prompt=  5906  completion= 30
  A5:   6.03s  prompt= 16946  completion= 30

--- Round 2 ---
  A1:   0.62s  prompt=   701  completion= 30  <-- LIKELY COLD (cache miss)
  A2:   1.16s  prompt=  1826  completion= 30  <-- LIKELY COLD (cache miss)
  A3:   1.78s  prompt=  3626  completion= 30  <-- LIKELY COLD (cache miss)
  A4:   2.54s  prompt=  5906  completion= 30  <-- LIKELY COLD (cache miss)
  A5:   6.17s  prompt= 16946  completion= 30  <-- LIKELY COLD (cache miss)

--- Round 3 ---
  A1:   0.86s  prompt=   701  completion= 30  <-- LIKELY COLD (cache miss)
  A2:   1.15s  prompt=  1826  completion= 30  <-- LIKELY COLD (cache miss)
  A3:   1.74s  prompt=  3626  completion= 30  <-- LIKELY COLD (cache miss)
  A4:   2.50s  prompt=  5906  completion= 30  <-- LIKELY COLD (cache miss)
  A5:   6.13s  prompt= 16946  completion= 30  <-- LIKELY COLD (cache miss)

========================================================================
Summary (seconds per agent per round):
========================================================================
agent    rnd1    rnd2    rnd3
A1          0.87     0.62     0.86
A2          1.15     1.16     1.15
A3          1.83     1.78     1.74
A4          2.46     2.54     2.50
A5          6.03     6.17     6.13

Expected (cache working): round 1 slow, rounds 2+ fast for all agents.
Bug pattern: one agent stays slow across rounds (its cache keeps
getting evicted because it's the LRU victim every cycle).
========================================================================

[jacekpoplawski]

please test this solution #22826
I will be working on another one, but in my tests it solves similar issue

[b1skit]

Possibly related: #23030

[robertpeal]

I think I am seeing the same issue, and I have some additional diagnostics that may help.

Summary

I am running Qwen3.6 27B through llama-server behind Hermes. I initially suspected Hermes was mutating the prompt and invalidating the cache, but I placed a local HTTP dump proxy between Hermes and llama-server and captured the exact OpenAI-compatible JSON request bodies sent to llama.cpp.

The captured requests were append-only. Consecutive requests had a 100% common prefix relative to the previous request.

Despite that, llama.cpp sometimes falls back to:

n_past = 5188

instead of reusing the large append-only prefix.

This causes either:

• Full prompt re-processing, when no checkpoint exists before n_past = 5188
• Large partial re-processing, when an earlier checkpoint such as 4096 exists

This appears to be llama.cpp-side prompt-cache / checkpoint restore behavior, not frontend prompt mutation, at least in my captured request sequence.

Environment

Frontend:

• Hermes
• OpenAI-compatible API
• Chat format: peg-native

Backend:

• llama.cpp llama-server
• Qwen3.6 27B GGUF
• Windows / CUDA
• Context: 128k, parallel: 1

The issue occurs with MTP disabled as well, so it does not appear to be caused solely by draft-mtp.

Current mitigation config:

llama-server.exe \
  -m "D:\\llm_models\\YOUR-QWEN3.6-MODEL.gguf" \
  --host 0.0.0.0 \
  --port 8081 \
  -c 128000 \
  -ngl 999 \
  --flash-attn \
  --parallel 1 \
  --cache-prompt \
  --no-context-shift \
  --batch-size 2048 \
  --ubatch-size 512 \
  --ctx-checkpoints 32 \
  --checkpoint-every-n-tokens 1024

Previous config that made the issue worse: --batch-size 8192

Request-body capture

I put a small HTTP proxy between Hermes and llama-server:

Hermes → HTTP dump proxy → llama-server

The proxy dumped the raw JSON request bodies. I then compared consecutive requests.

Result: every captured request was append-only relative to the previous one.

Example pairwise comparisons:

A chars: 73420  |  B chars: 75898  |  Common prefix: 73420 (100.00%) → B is append-only relative to A
A chars: 75898  |  B chars: 78688  |  Common prefix: 75898 (100.00%) → append-only
A chars: 78688  |  B chars: 89875  |  Common prefix: 78688 (100.00%) → append-only
A chars: 89875  |  B chars: 91873  |  Common prefix: 89875 (100.00%) → append-only
A chars: 91873  |  B chars: 94273  |  Common prefix: 91873 (100.00%) → append-only
A chars: 94273  |  B chars: 95879  |  Common prefix: 94273 (100.00%) → append-only
A chars: 95879  |  B chars: 102853 |  Common prefix: 95879 (100.00%) → append-only

So the frontend request body was not rewriting earlier messages. Each new request contained the full previous request as a prefix, then appended new messages.

Full reprocess case with --batch-size 8192

With --batch-size 8192, the first usable checkpoint was created at 8192 tokens. When llama.cpp fell back to n_past = 5188, there was no checkpoint at or before that point:

n_past = 5188, slot.prompt.tokens.size() = 50468, seq_id = 0, pos_min = 50467, n_swa = 0
Checking checkpoint with [8191, 8191] against 5188...
forcing full prompt re-processing due to lack of cache data

Because the first checkpoint was after 5188, llama.cpp could not restore any checkpoint and forced a full prompt reprocess.

Partial mitigation with --batch-size 2048

Lowering batch size changed checkpoint placement:

--batch-size 2048 --ubatch-size 512 --ctx-checkpoints 32 --checkpoint-every-n-tokens 1024

llama.cpp created earlier checkpoints:

1024 tokens since last checkpoint at 0, creating new checkpoint during processing at position 4096
created context checkpoint 1 of 32 (pos_min = 2047, pos_max = 2047, n_tokens = 2048)

1024 tokens since last checkpoint at 2048, creating new checkpoint during processing at position 6144
created context checkpoint 2 of 32 (pos_min = 4095, pos_max = 4095, n_tokens = 4096)

Later, when the same fallback happened:

n_past = 5188, slot.prompt.tokens.size() = 30062, seq_id = 0, pos_min = 30061, n_swa = 0
Checking checkpoint with [6143, 6143] against 5188...
Checking checkpoint with [4095, 4095] against 5188...
restored context checkpoint (pos_min = 4095, pos_max = 4095, n_tokens = 4096, n_past = 4096)

This avoided the explicit full reprocess warning. However, it still caused a very large partial reprocess:

prompt eval time = 10453.18 ms / 27519 tokens

So lowering batch size changes the failure mode from full prompt reprocess from zero to restore checkpoint at 4096, then reprocess most of the prompt. This is a mitigation, not a complete fix.

What seems to be happening

The repeated fallback value appears to be n_past = 5188. When the prompt is tens of thousands of tokens long, llama.cpp should be able to reuse most of the append-only prefix. Instead, it sometimes falls back to around 5k tokens.

Then checkpoint behavior determines what happens next:

• Earliest checkpoint > 5188 → no checkpoint can be restored → full prompt reprocess
• Checkpoint ≤ 5188 exists → checkpoint is restored → large partial reprocess from that checkpoint onward

Why I think this is llama.cpp-side

I captured the raw HTTP request bodies sent to llama-server and verified that consecutive requests were append-only with a 100% common prefix relative to the previous request. So at the OpenAI-compatible JSON request level, Hermes is not mutating earlier messages. The fallback to n_past = 5188 appears to happen inside llama.cpp after receiving an append-only request.

Additional related observation: Gemma4

I also saw a similar but not identical behavior with a Gemma4 model. In that case, n_swa was nonzero (n_swa = 1024) and checkpoint restore succeeded, but it still caused a large partial reprocess:

prompt eval time = 12151.29 ms / 23349 tokens

This may be a related broader prompt-cache/checkpoint behavior, but the Qwen3.6 case is the main issue I am reporting.

Questions

1. Why does llama-server fall back to n_past = 5188 when the raw chat request body is append-only?
2. Is this expected behavior for Qwen3.6 hybrid/recurrent / DeltaNet models?
3. Should checkpoint restore be able to recover from a later checkpoint instead of requiring a checkpoint before the fallback n_past?
4. Is there a way to pin an early checkpoint or avoid losing the useful early checkpoint?
5. Is this related to known Qwen3.6 / hybrid recurrent prompt-cache issues?
6. Is there a way to expose more logging around why the prompt matcher picks n_past = 5188 despite append-only request bodies?

Practical mitigation found

--batch-size 2048 --ubatch-size 512
--ctx-checkpoints 32 --checkpoint-every-n-tokens 1024
--cache-prompt --no-context-shift

This creates early checkpoints at 2048 and 4096, preventing n_past = 5188 from always becoming a full reprocess. However, it still does not solve the main problem: llama.cpp still falls back to n_past = 5188 and reprocesses a large tail instead of reusing the full append-only prefix.

[jacekpoplawski]

I think I am seeing the same issue, and I have some additional diagnostics that may help.

please test #22929 and let me know if it fixes your issues

[patrolbants]

this issue is really disappointing for local use. it makes qwen3.6/3.5 totally unusable for long agentic flows

[mfielding92]

I spent over 36 hours getting this working locally on my proxy and I finally have it restoring states for Qwen 3.5/3.6 in seconds from disk!

I also confirmed that it doesn't interfere with other non-qwen models (tested Gemma4 and Step 3.5 Flash, would love for others to confirm some more).

llama.cpp issue #22746 — hybrid recurrent memory cache fix PATCH 1+2

Two surgical patches to tools/server/server-context.cpp that make
llama.cpp's slot save/restore and LCP cache-reuse work cleanly for
hybrid-memory models (Qwen 3.5 / 3.6 family with Gated DeltaNet +
attention). Validated 2026-05-25 against llama.cpp upstream commit
6a257d446.

Patch: Save as server-context.cpp.patch

--- a/tools/server/server-context.cpp	2026-05-25 19:38:54.525341310 -0400
+++ b/tools/server/server-context.cpp	2026-05-26 06:17:19.760128321 -0400
@@ -2673,7 +2673,8 @@
                                     SLT_WRN(slot, "%s\n", st1.str().c_str());
                                 }
 
-                                if (pos_min >= pos_min_thold) {
+                                if (n_swa > 0 && pos_min >= pos_min_thold) {
                                     SLT_WRN(slot, "n_past = %d, slot.prompt.tokens.size() = %d, seq_id = %d, pos_min = %d, n_swa = %d\n", n_past, (int) slot.prompt.tokens.size(), slot.id, pos_min, n_swa);
 
                                     // search for a context checkpoint
@@ -2759,9 +2760,16 @@
 
                     SLT_TRC(slot, "cached n_tokens = %d, memory_seq_rm [%d, end)\n", slot.prompt.n_tokens(), p0);
 
-                    common_context_seq_rm(ctx_tgt, slot.id, p0, -1);
+                    if (!llama_memory_seq_rm(llama_get_memory(ctx_tgt), slot.id, p0, -1)) {
+                        SLT_WRN(slot, "partial seq_rm at p0=%d failed (recurrent backend cannot roll into cache); clearing slot and re-evaluating from scratch\n", p0);
+                        slot.n_prompt_tokens_cache = 0;
+                        slot.prompt.tokens.keep_first(0);
+                        common_context_seq_rm(ctx_tgt, slot.id, 0, -1);
+                    }
                     if (ctx_dft) {
-                        common_context_seq_rm(ctx_dft.get(), slot.id, p0, -1);
+                        if (!llama_memory_seq_rm(llama_get_memory(ctx_dft.get()), slot.id, p0, -1)) {
+                            common_context_seq_rm(ctx_dft.get(), slot.id, 0, -1);
+                        }
                     }
 
                     // If using an alora, there may be uncached tokens that come

---

How to apply

cd <llama.cpp repo root>
patch -p1 < /path/to/server-context.cpp.patch

Or for a dry-run first:

patch -p1 --dry-run < /path/to/server-context.cpp.patch

Then rebuild:

cmake --build build --target llama-server -j$(nproc)

Both patches are inside a single file (tools/server/server-context.cpp)
so the build is incremental — typically 30-60s.

---

The bug:

llama.cpp's server tracks per-slot KV cache state and uses a check at
server-context.cpp:~2676 to detect when sliding-window attention
eviction has gone too far for the cache to be useful. The check is
if (pos_min >= pos_min_thold) where
pos_min_thold = max(0, pos_next - n_swa - 1).

For hybrid Mamba/Gated-DeltaNet + attention models (Qwen 3.5 122B,
Qwen 3.6 27B / 35B):

1. n_swa is 0 (the model has no sliding window).
2. The hybrid memory backend's seq_pos_min returns
   std::max(mem_attn->seq_pos_min, mem_recr->seq_pos_min). The
   recurrent half's seq_pos_min returns the latest token position
   (it's a rolling state, not a per-token cache).
3. So pos_min ≈ pos_next and pos_min_thold = pos_next − 1, making
   pos_min >= pos_min_thold trivially true for any cached slot.
4. The server then searches slot.prompt.checkpoints for an older
   checkpoint to rewind to. For a slot that was just populated via
   /slots/N?action=restore, that list is empty (restore loads the
   GPU state but not the server-side checkpoint metadata).
5. do_reset = true fires. n_past gets set to 0,
   slot.prompt.tokens.keep_first(0) empties the software-side token
   list, and common_context_seq_rm(slot.id, 0, -1) wipes the
   just-restored cache before re-evaluating the full prompt cold.

Observed symptom: every cold load of a hybrid model after a
/slots/restore logs f_keep = 1.000 (LCP picker was happy)
but then emits forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory) and
spends 28+ seconds re-evaluating the 35k-token system+tools prefix
that had just been loaded from disk.

(For the full upstream tracking-issue list, see the "Related GitHub
issues" section near the bottom of this README.)

A second, downstream bug surfaces once the first patch lets the cache
survive: when a new request matches no profile but the slot has
cached content for the previous request, the LCP picker matches a
small prefix (e.g. 3 chat-template tokens). The server then calls
common_context_seq_rm(slot.id, 3, -1) to truncate the cache.
On hybrid recurrent backends with n_rs_seq=0 (the default), partial
rollback fails (src/llama-memory-recurrent.cpp:189 returns false).
common_context_seq_rm aborts via GGML_ABORT, killing llama-server.

---

Patch 1 — gate the pos_min_thold check on n_swa > 0

Location: tools/server/server-context.cpp:2676 (1 line of code,
5 lines of comment).

-    if (pos_min >= pos_min_thold) {
+    if (n_swa > 0 && pos_min >= pos_min_thold) {

Why it works

The whole block guarded by this if exists to handle SWA cache
eviction. With n_swa == 0 there is no sliding window — nothing to
evict — so the block is irrelevant. Gating on n_swa > 0 makes it a
no-op exactly when the underlying assumption (eviction has happened)
is false.

Safety analysis

• Pure attention non-SWA models (n_swa == 0, no recurrent
  backend): their pos_min would be 0 and the original check would
  evaluate false anyway. New gate changes nothing observable.
• Pure recurrent models (n_swa == 0, recurrent only): their
  seq_pos_min returns latest position, the original check was a
  false positive. New gate skips correctly.
• Hybrid Mamba+attention (n_swa == 0, recurrent + attention):
  same false-positive case as above. New gate skips correctly.
• SWA models (n_swa > 0, attention only): the gate evaluates
  true, original behavior preserved. No change.
• Hybrid + iSWA (n_swa > 0, recurrent + SWA-attention, e.g.
  some Gemma variants): the gate evaluates true. The original
  behavior is still potentially broken in the same way, but this
  patch does not address it — verified only against non-iSWA hybrid
  Qwen models so far.

---

Patch 2 — graceful fallback on partial-seq_rm failure

Location: tools/server/server-context.cpp:~2767 (replaces 4
lines with 15 lines + comments).

-    common_context_seq_rm(ctx_tgt, slot.id, p0, -1);
+    if (!llama_memory_seq_rm(llama_get_memory(ctx_tgt), slot.id, p0, -1)) {
+        SLT_WRN(slot, "partial seq_rm at p0=%d failed (recurrent backend cannot roll into cache); clearing slot and re-evaluating from scratch\n", p0);
+        slot.n_prompt_tokens_cache = 0;
+        slot.prompt.tokens.keep_first(0);
+        common_context_seq_rm(ctx_tgt, slot.id, 0, -1);
+    }
     if (ctx_dft) {
-        common_context_seq_rm(ctx_dft.get(), slot.id, p0, -1);
+        if (!llama_memory_seq_rm(llama_get_memory(ctx_dft.get()), slot.id, p0, -1)) {
+            common_context_seq_rm(ctx_dft.get(), slot.id, 0, -1);
+        }
     }

Why it works for Qwen

common_context_seq_rm is a thin wrapper over llama_memory_seq_rm
that aborts on false. Calling the underlying API directly lets the
server detect "partial removal not supported" and recover instead of
crashing.

The recurrent-backend path in
src/llama-memory-recurrent.cpp:182-189 returns false only when
asked to roll back INTO the cached range (i.e. p0 <= cell.pos AND
the rollback distance exceeds n_rs_seq). It returns true for the
common "remove the tail after the cache" case where p0 > cell.pos
(nothing to remove → trivial success). So this patch's fallback
fires exclusively in the failure case worth recovering from.

When the fallback fires, it:

1. Clears slot.n_prompt_tokens_cache (the server's book-keeping).
2. Calls slot.prompt.tokens.keep_first(0) (truncates the
   software-side token list).
3. Calls common_context_seq_rm(slot.id, 0, -1) (full clear — always
   succeeds because recurrent's seq_rm accepts p0 == 0 as the
   special "wipe everything" case).

Downstream code in update_slots() then iterates the unprocessed
tokens of the new request and decodes them from scratch. Slow but
correct, and the process stays alive.

Why ONLY this call site

Several other common_context_seq_rm call sites also do partial
removals (context shift at L2276, speculative-decoding cleanup at
L2395, L3268, context-reuse at L2594). None of those want the
fallback behavior — for context-shift, falling back to "wipe and
re-eval" would silently lose work tokens. The patch is intentionally
scoped to the one site (the post-LCP cache truncation at L2767)
where "give up and start over" is the right semantic.

---

How to verify

After rebuilding, run a multi-turn workload on a hybrid model and
watch the logs for:

• restored profile_N (score=1.0000) followed by
  selected slot by LCP similarity, sim_best = 1.000 (...) f_keep = 1.000
• NO forcing full prompt re-processing due to lack of cache data
  warnings (was the symptom of bug 1)
• If a request lands that matches no profile while the slot is
  populated, you'll see Patch 2's
  partial seq_rm at p0=X failed (recurrent backend cannot roll into cache); clearing slot and re-evaluating from scratch
  warning instead of the process aborting with ggml_abort (was
  the symptom of bug 2)

Empirical results (llama.cpp 6a257d446)

A 4-step test against each model (cold-load with 30k tokens,
drift-swap to 10k different tokens, not match small prompt, drift back to the original 30k tokens. With both patches applied:

Model	Architecture	Step 1 cold	Step 2 drift	Step 3 no-match	Step 4 drift back
Qwen3.6 27B 128k q8_0 Cache	Hybrid GDN+attn	4.37 s	1.48 s	1.46 s	1.31 s
Qwen3.6 27B 201k q4_0 Cache	Hybrid GDN+attn	4.49 s	1.46 s	1.54 s	1.07 s
Qwen3.5 122B MoE 201k	Hybrid GDN+attn (CPU MoE)	11.50 s	4.45 s	4.45 s	4.28 s
Gemma4 26B MoE 201k	SWA (regression)	4.20 s	1.22 s	1.02 s	1.16 s
Step3.5 Flash 190B MoE 196k	SWA (regression)	39.85 s	10.15 s	5.00 s	7.98 s

Without the patches, the Qwen 3.5 row would either spend 8+ minutes
on Step 1 cold prompt evaluation (Qwen 3.5 122B CPU-offload) or
abort llama-server outright on Step 3. The SWA rows are unchanged
by the patches (they don't fire — n_swa > 0 and seq_rm already
handles partial removals).

Related GitHub issues

These are the upstream ggml-org/llama.cpp issues that describe the
same or related cache-invalidation behavior. They're useful as
read-only context — they document the symptom, the model class, and
in #20225 walk through the exact pos_min_thold math the first
patch addresses. None of them have a merged fix as of llama.cpp
commit 6a257d446 (2026-05-21).

Issue	Model / scope	What it documents
ggml-org/llama.cpp#22746	Qwen3.6-27B (IQ4_XS, Q4_K_M, UD-IQ3_XXS, Q3_K_M)	"Forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory)" on multi-turn use. Slot creates checkpoints, walks the chain, fails to match against pos_min, then erases all 14+ checkpoints in one go before reprocessing the full prompt.
ggml-org/llama.cpp#20225	Qwen3.5-122B-A10B	Explicit diagnosis with the pos_min_thold math broken out line-by-line. Notes --swa-full has no effect (confirmed). This is the issue whose analysis matches the Patch 1 fix most directly.
ggml-org/llama.cpp#19977	Original report	"Loses part of the context cache with erased invalidated context checkpoint." The earliest write-up of the same failure mode.
ggml-org/llama.cpp#19794	Qwen3-Coder Next	Same cache-invalidation pattern on a different Qwen3-family release.
ggml-org/llama.cpp#22450	Qwen3.6-35B-A3B (MoE variant)	A worse cousin where slots actually hang in token generation after hitting the same cache-invalidation path (the dense 27B just reprocesses; the MoE variant locks up).
ggml-org/llama.cpp#23371	Qwen3.6-27B UD-Q5_K_XL + MTP + vision	Long-context checkpoint retention growing VRAM until OOM during mmproj restore. Different symptom, related architecture. Not addressed by these patches — only relevant if you use the vision/mmproj path.

[zac9]

I applied reedmayhew18's patch. solved my issue.

AMD 395
Prefill still times out at the 300s mark, but the cache is not invalidated so it immediately reloads and continues. like it did with my previous build.

[mfielding92]

@zac9 Glad to hear Patches 1+2 worked! After applying those, I hit a follow-on
issue: multi-turn requests with any rendering drift (tool-call JSON
re-serialization, missing reasoning_content, etc.) trip the new Patch 2
fallback every turn — the slot gets wiped and the full prompt re-evaluated
at O(N) per turn. So I wrote two more patches that bring that case down
to O(delta) by using llama.cpp's existing context-checkpoint infrastructure
(the same store the SWA path already uses) to rescue the state instead
of wiping.

Patches 3 + 5 — multi-turn drift recovery via LCP boundary backstep + checkpoint rescue

There was a patch 4, then it got reverted because it didn't work. Then patch 5 part A and B worked. Sorry about the versioning.

Three additional changes in tools/server/server-context.cpp, building on
top of Patches 1+2:

• Patch 3 — pre-tokenize chat-template turn-end strings at startup;
  in the LCP cut path, walk n_past back to the nearest such boundary
  in the cached token stream before calling seq_rm.
• Patch 5 Part A — in Patch 2's seq_rm fail handler, before the
  slot wipe, search slot.prompt.checkpoints for an existing checkpoint
  at or before p0-1 and call it->load_tgt() / it->load_dft() to
  restore the saved state.
• Patch 5 Part B — in the do_checkpoint gating logic, force
  do_checkpoint = true when the just-processed batch contains a
  registered turn-boundary token. Guarantees a usable rescue point
  exists at every turn end.

Patch: Save as patches-3-and-5.patch

--- a/tools/server/server-context.cpp
+++ b/tools/server/server-context.cpp
@@ -618,6 +618,33 @@
 // server_context_impl (private implementation)
 //
 
+static llama_pos find_safe_boundary_backstep(
+        const server_tokens & cached,
+        llama_pos             n_past,
+        const std::vector<std::vector<llama_token>> & boundaries) {
+    if (boundaries.empty() || n_past <= 0) {
+        return 0;
+    }
+    for (llama_pos i = n_past; i > 0; --i) {
+        for (const auto & bt : boundaries) {
+            const llama_pos blen = (llama_pos) bt.size();
+            if (blen == 0 || blen > i) {
+                continue;
+            }
+            bool match = true;
+            for (llama_pos k = 0; k < blen; ++k) {
+                if (cached[i - blen + k] != bt[k]) {
+                    match = false;
+                    break;
+                }
+            }
+            if (match) {
+                return i;
+            }
+        }
+    }
+    return 0;
+}
+
 struct server_context_impl {
     friend struct server_context;
 
@@ -667,6 +694,9 @@
     common_context_seq_rm_type ctx_tgt_seq_rm_type = COMMON_CONTEXT_SEQ_RM_TYPE_NO;
     common_context_seq_rm_type ctx_dft_seq_rm_type = COMMON_CONTEXT_SEQ_RM_TYPE_NO;
 
+    std::vector<std::string>              turn_boundary_strings;
+    std::vector<std::vector<llama_token>> turn_boundary_tokens;
+
     common_speculative_ptr spec;
 
     bool add_bos_token = true;
@@ -939,6 +969,22 @@
             SRV_WRN("%s", "speculative decoding will use checkpoints\n");
         }
 
+        if (ctx_tgt_seq_rm_type == COMMON_CONTEXT_SEQ_RM_TYPE_FULL) {
+            turn_boundary_strings = { "<|im_end|>\n", "<|im_end|>" };
+            turn_boundary_tokens.reserve(turn_boundary_strings.size());
+            for (const auto & s : turn_boundary_strings) {
+                auto toks = common_tokenize(vocab, s, false, true);
+                if (toks.empty()) {
+                    SRV_WRN("turn-boundary string '%s' did not tokenize; skipping\n", s.c_str());
+                    continue;
+                }
+                turn_boundary_tokens.push_back(std::move(toks));
+            }
+            if (!turn_boundary_tokens.empty()) {
+                SRV_INF("turn-boundary backstep enabled (%zu boundary string(s))\n", turn_boundary_tokens.size());
+            }
+        }
+
         // initialize slots
         for (int i = 0; i < params_base.n_parallel; i++) {
             slots.emplace_back();
@@ -2736,6 +2782,23 @@
                             SLT_WRN(slot, "n_past was set to %d\n", n_past);
                         }
 
+                        if (n_past > 0 &&
+                            (size_t) n_past < slot.prompt.tokens.size() &&
+                            !slot.prompt.tokens.has_mtmd &&
+                            !turn_boundary_tokens.empty() &&
+                            ctx_tgt_seq_rm_type == COMMON_CONTEXT_SEQ_RM_TYPE_FULL) {
+                            const llama_pos n_past_safe = find_safe_boundary_backstep(
+                                slot.prompt.tokens, (llama_pos) n_past, turn_boundary_tokens);
+                            if (n_past_safe > 0 && n_past_safe < (llama_pos) n_past) {
+                                SLT_INF(slot, "LCP backstep: n_past %d -> %d (turn-boundary aligned)\n",
+                                        (int) n_past, (int) n_past_safe);
+                                n_past = (size_t) n_past_safe;
+                            } else if (n_past_safe == 0) {
+                                SLT_INF(slot, "%s", "LCP backstep: no turn boundary in cached prefix; existing fallback will handle if seq_rm fails\n");
+                            }
+                        }
+
                         slot.n_prompt_tokens_cache = n_past;
                         slot.n_prompt_tokens_processed = 0;
 
@@ -2773,10 +2836,29 @@
                     // still succeed silently as before.
                     if (!llama_memory_seq_rm(llama_get_memory(ctx_tgt), slot.id, p0, -1)) {
-                        SLT_WRN(slot, "partial seq_rm at p0=%d failed (recurrent backend cannot roll into cache); clearing slot and re-evaluating from scratch\n", p0);
-                        slot.n_prompt_tokens_cache = 0;
-                        slot.prompt.tokens.keep_first(0);
-                        common_context_seq_rm(ctx_tgt, slot.id, 0, -1);
+                        auto it = std::find_if(
+                            slot.prompt.checkpoints.rbegin(),
+                            slot.prompt.checkpoints.rend(),
+                            [&](const auto & cur) { return cur.pos_max <= p0 - 1; }
+                        );
+                        if (it != slot.prompt.checkpoints.rend()) {
+                            it->load_tgt(ctx_tgt, slot.id, LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY);
+                            if (ctx_dft) {
+                                it->load_dft(ctx_dft.get(), slot.id, LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY);
+                            }
+                            const llama_pos pos_restored = std::max(it->pos_min + 1, it->pos_max);
+                            const size_t    n_past_new   = std::min(slot.prompt.tokens.size_up_to_pos(pos_restored), (size_t) it->n_tokens);
+                            slot.n_prompt_tokens_cache = n_past_new;
+                            slot.prompt.tokens.keep_first(n_past_new);
+                            SLT_INF(slot, "rescue: seq_rm at p0=%d failed; restored checkpoint at pos=%d (n_past %d -> %zu, size = %.3f MiB)\n",
+                                    (int) p0, (int) pos_restored, (int) p0, n_past_new, (float) it->size() / 1024 / 1024);
+                        } else {
+                            SLT_WRN(slot, "partial seq_rm at p0=%d failed (recurrent backend cannot roll into cache); no checkpoint <= p0-1, clearing slot and re-evaluating from scratch\n", p0);
+                            slot.n_prompt_tokens_cache = 0;
+                            slot.prompt.tokens.keep_first(0);
+                            common_context_seq_rm(ctx_tgt, slot.id, 0, -1);
+                        }
                     }
                     if (ctx_dft) {
                         if (!llama_memory_seq_rm(llama_get_memory(ctx_dft.get()), slot.id, p0, -1)) {
@@ -2915,9 +2997,28 @@
 
                         slot.init_sampler();
                     } else {
+                        bool boundary_just_processed = false;
+                        if (!turn_boundary_tokens.empty() && n_tokens_cur > 0) {
+                            const size_t total      = slot.prompt.tokens.size();
+                            const size_t scan_start = total > (size_t) n_tokens_cur ? total - (size_t) n_tokens_cur : 0;
+                            for (size_t i = scan_start; i < total && !boundary_just_processed; ++i) {
+                                for (const auto & bt : turn_boundary_tokens) {
+                                    if (!bt.empty() && slot.prompt.tokens[i] == bt.back()) {
+                                        boundary_just_processed = true;
+                                        break;
+                                    }
+                                }
+                            }
+                        }
+
                         if (slot.task->n_tokens() < slot.prompt.n_tokens() + n_ubatch) {
                             // near the end of the prompt
                             do_checkpoint = do_checkpoint && true;
+                        } else if (boundary_just_processed) {
+                            do_checkpoint = do_checkpoint && true;
+                            SLT_DBG(slot, "%s", "boundary token in batch, forcing checkpoint candidate\n");
                         } else {
                             // only do non-end checkpoints if the "checkpoint every n tokens" option is set
                             do_checkpoint = do_checkpoint && params_base.checkpoint_every_nt > 0;

---

How to apply

cd <llama.cpp repo root>
patch -p1 < /path/to/patches-3-and-5.patch
cmake --build build --target llama-server -j$(nproc)

Patches 1+2 must already be in place — these build on top of them.

---

The bug left behind by Patches 1+2

With Patches 1+2 applied, the cache survives /slots/restore and
llama-server no longer aborts on partial-seq_rm failure. But every
multi-turn request where the new prompt's rendering drifts from the
cached tokens by even one byte (tool-call JSON re-serialization,
stripped reasoning_content, etc.) trips Patch 2's keep_first(0)
fallback. The result is O(N) full re-eval of the conversation every
turn — typically 30+ seconds on a 36k-token kai-style prompt.

The proxy-side workaround for this was to restore a known-good
/slots/restore snapshot of [sys + tools] on every turn, paying
O(turns × history) per conversation. The patches below remove that
workaround entirely by making the cache survive drift.

---

Patch 3 — LCP boundary backstep

When the LCP picker returns an n_past that lands mid-turn (inside a
<think> block, mid-JSON, etc.), walk back through the cached token
stream to the nearest chat-template turn boundary. The resulting n_past
aligns with a "clean cut" position where the existing checkpoint system
already captures state.

Boundaries are registered at startup as a priority-ordered list, tried
longer-to-shorter at each candidate position:

turn_boundary_strings = { "<|im_end|>\n", "<|im_end|>" };

The dual-list handles both possible cache endings:

Cache ends with…	Matches	Returns
[..., im_end, \n] (typical ChatML profile cache)	<|im_end|>\n	cache.size — aligns with checkpoint.pos_max + 1
[..., im_end] (bare im_end — tool-call boundary, custom template)	<|im_end|>	cache.size — also aligns
mid-content	walks back to find earlier <|im_end|>(\n?)	position immediately after the earlier boundary

The gate condition n_past < slot.prompt.tokens.size() is important:
on cold-load where LCP matches the full cache (n_past == size), the
upcoming seq_rm is a tail no-op that already succeeds. Backstepping
in that case would convert a guaranteed success into a guaranteed
failure (the rollback path would fire on the now-shorter p0). The
gate skips backstep in the happy-path case.

---

Patch 5 Part A — checkpoint rescue in seq_rm fail path

Inside Patch 2's if (!llama_memory_seq_rm(...)) block, before the
existing keep_first(0) wipe, search slot.prompt.checkpoints for a
checkpoint whose pos_max <= p0 - 1. If found, restore via
it->load_tgt() (and it->load_dft() if speculation is on),
truncate slot.prompt.tokens to match, and return — the subsequent
prompt-processing loop re-evaluates from the restored position
forward.

This uses the existing checkpoint store — the same one the SWA
path at ~L2790 already uses via the same load_tgt mechanism.
No new data structures.

The predicate pos_max <= p0 - 1 (rather than <= p0) preserves the
seq_rm contract "remove positions [p0, p1)": after restore,
positions [0, pos_max] remain and [pos_max+1, ...) is gone.
Since Patch 3 backsteps p0 to checkpoint.pos_max + 1, the
predicate becomes a strict equality match in practice.

The existing wipe behavior is preserved as the inner-else branch
— if no checkpoint exists at or before p0 - 1 (e.g. small-prompt
case that doesn't match any saved snapshot), we still wipe + re-eval.
Strictly additive: never worse than current behavior.

---

Patch 5 Part B — force a checkpoint when a turn boundary just got processed

Inside the do_checkpoint gating logic, scan the tokens of the
just-processed batch for any registered turn-boundary. If found, set
do_checkpoint = do_checkpoint && true — i.e. allow the create to
proceed regardless of params_base.checkpoint_every_nt. The existing
spacing / pos_min >= 0 / n_tokens >= 64 / !has_mtmd constraints
still apply below.

Without Part B, the existing system creates checkpoints at "end of
prompt" plus one every checkpoint_every_nt tokens (default 8192).
Neither is guaranteed to land at an <|im_end|>. Part B guarantees
one exists at every turn end, which is exactly the rescue target
Patch 3's backstep produces and Part A consumes.

Memory cost scales with conversation turns (capped at
params_base.n_ctx_checkpoints, default 32). For a typical 5-10
turn conversation: ~150 MiB per checkpoint on 27B GPU, ~1-1.5 GiB
total. Older checkpoints get evicted at the cap.

---

How to verify

After rebuild, watch the journal during a multi-turn workload:

• At startup, you should see one line per gated model:

  turn-boundary backstep enabled (2 boundary string(s))
  

• On a turn that drifts mid-cache, you should see:

  LCP backstep: n_past 36536 -> 36533 (turn-boundary aligned)
  rescue: seq_rm at p0=36533 failed; restored checkpoint at pos=36533 (n_past 36533 -> 36533, size = 149.626 MiB)
  

• And the subsequent prompt eval time should be small (tens of
  tokens, ~100ms), not the full prompt.
• The absence of:

  partial seq_rm at p0=X failed (...); no checkpoint <= p0-1, clearing slot and re-evaluating from scratch
  

  (this only fires when Part A's predicate finds nothing — small-prompt
  drift cases, etc., still hit the existing wipe).

---

Empirical results (Qwen 3.6 27B 128k GPU, q8_0 cache, llama.cpp 6a257d446 + Patches 1-3+5)

Same 4-step swap test as the Patches 1+2 reply — regression-free:

Step	Description	Time
1	kai cold load + profile_0 restore	5.14 s
2	open-webui drift swap to profile_1 (slot restore, no model reload)	4.55 s
3	minimal "hi" prompt — no profile match, falls through to wipe (correctly)	1.55 s
4	kai drift swap back to profile_0	3.54 s

3-turn multi-turn drift test (math conversation with <think> content,
each turn references the previous answer):

Turn	Time	Behavior
1	0.70 s	Drift case — rescue fired (restored checkpoint at pos=36533), 26 tokens re-evaluated at 123 ms
2	0.62 s	f_keep=1.000, 19 tokens
3	0.70 s	f_keep=1.000, 20 tokens

Same test before Patch 5 (with Patches 1+2+3 only): turn 1 was
31.51 s (full re-eval, 36559 tokens). ~45x speedup on the
drift case. The math answers stayed correct (40 → 80 → 70) across
all turns, confirming the restored state produces semantically
equivalent outputs.