Eval bug: [SYCL] empty/gibberish output on hybrid models + ggml_sycl_op_mul_mat crash (qwen3next/qwen35 arch) on Intel Arc Pro B60 — regression between *pinpointed b9128- b9159* (2026-03-23) and server-intel latest (2026-06-03)

[LVLUP-Studio]

Name and Version

Working: build 8477 (ec2b787) with IntelLLVM 2025.2.1 — image built 2026-03-23
Broken: server-intel latest — image built 2026-06-03 (~b9479)

Operating systems

Linux

GGML backends

SYCL

Hardware

2× Intel Arc Pro B60 (24GB each, 46GB combined, PCIe discrete, no XeLink)
AMD Ryzen 7 7700 (host CPU)
64GB DDR5
Level Zero driver 1.15.38308+1
Both GPUs passed via /dev/dri to Docker container

Models

Qwen3-Coder-Next UD-Q3_K_XL (33.79 GiB, arch=qwen3next) — CRASH on latest, FIXED on b8477
https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF

Qwen3-Coder-Next Q4_K_M (48.18 GiB, arch=qwen3next) — CRASH on latest, FIXED on b8477
https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF

Qwen3.6-27B Q4_K_M (15.65 GiB, arch=qwen35) — empty/gibberish content on latest, FIXED on b8477
https://huggingface.co/unsloth/Qwen3.6-27B-GGUF

Qwen3.6-35B-A3B UD-Q4_K_M (arch=qwen35) — empty/gibberish content on latest, FIXED on b8477
https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF

Qwen3-Coder-30B-A3B Q4_K_M (arch=qwen3moe, pure attention) — UNAFFECTED on both builds

Problem description & steps to reproduce

Newer qwen models 3.5+ hybrid models were outputing mostly gibberish and crashing. I've tested most of these models previously and they were working. Honestly thought I did something at first but then realized it was just an update I'm guessing.

Two related symptoms affect all hybrid architecture models (identifiable by
ssm_d_conv / ssm_d_state / ssm_dt_rank fields in print_info) on Intel Arc Pro B60
using server-intel latest (built 2026-06-03). Both are fixed by pinning to b8477.

SYMPTOM 1 - CRASH (qwen3next arch, e.g. Qwen3-Coder-Next):
Model loads fully into VRAM then crashes during warmup mul_mat on SSM/GDN weights.
crash at ggml_sycl_op_mul_mat line:3093.

SYMPTOM 2 - EMPTY/GIBBERISH OUTPUT (qwen35 arch, e.g. Qwen3.6-27B/35B):
Model loads and generates tokens but "content" field in API response is empty or
contains number spam (e.g. 03129816571033482510583...). All output goes to
"reasoning_content" as corrupted tokens. On b8477 the same models produce clean
coherent output in "content".

IMPORTANT CONTEXT - why this is hard to diagnose:
server-intel is a rolling latest tag. Not sure when build/image broke but when I tried most recent build or 2026-06-03 get the broken build. Same
image tag, completely different behavior depending on pull date. At one point I randomly tested Coder-Next about a week ago and although slow it worked. Now I randomly couldn't get any model 3.5 or above to work without gibberish outputs .... Coder-Next had been working in earlier sessions, then
silently broke when the image updated i'm guessing, I just can't pinpoint when, unless this is known already.

Pure attention MoE models (qwen3moe arch, no SSM layers) are completely unaffected
on both builds and work correctly throughout.

Steps to reproduce (using Docker):

1. Pull latest server-intel:
   docker pull ghcr.io/ggml-org/llama.cpp:server-intel

2. Run with dual Arc Pro B60:

   docker run --rm
   --device /dev/dri:/dev/dri
   --group-add 993 --group-add 44
   -e ONEAPI_DEVICE_SELECTOR=level_zero:0;level_zero:1
   -e ZES_ENABLE_SYSMAN=1
   -e UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1
   -e GGML_SYCL_ENABLE_LEVEL_ZERO=1
   -e GGML_SYCL_ENABLE_VMM=1
   -e GGML_SYCL_DISABLE_GRAPH=1
   -p 8010:8010
   ghcr.io/ggml-org/llama.cpp:server-intel
   --model /models/Qwen3-Coder-Next-UD-Q3_K_XL.gguf
   --ctx-size 8192
   --host 0.0.0.0
   --port 8010
   -ngl 999
   --split-mode layer
   --tensor-split 1,1
   --flash-attn off
   --threads 1

3. For qwen3next: crash during warmup at ggml_sycl_op_mul_mat line:3093
   For qwen35: server starts, responds, but content field is empty or number spam

4. Replace image with server-intel-b8477 — identical command — both symptoms fixed

---

Side note for Ollama SYCL users:

This same regression affects users of https://github.com/eleiton/ollama-intel-arc
— an excellent community-maintained project providing native SYCL Ollama builds
for Intel Arc GPUs. It is not a bug in that project. Ollama 0.30.2 pins
LLAMA_CPP_VERSION=b9479 (confirmed via
https://raw.githubusercontent.com/ollama/ollama/v0.30.2/LLAMA_CPP_VERSION) which
pulls the broken llama.cpp commit via FetchContent at build time.

Workaround: add this to ollama-sycl/Dockerfile after WORKDIR /ollama, then rebuild:

RUN echo "ec2b787eb" > LLAMA_CPP_VERSION

Once this regression is fixed in llama.cpp master, Ollama SYCL builds will
automatically pick up the fix.

First Bad Commit

Working: server-intel-b8477 (ec2b787), image built 2026-03-23
Broken: server-intel latest (~b9479), image built 2026-06-03

Regression window: somewhere between 2026-03-23 and 2026-06-03.
llama.cpp ships multiple builds per day so this window contains
hundreds of commits. Have not bisected further.

A git bisect focusing on changes to ggml/src/ggml-sycl/ggml-sycl.cpp
around the ggml_sycl_op_mul_mat function (line ~3093 in broken build)
would likely narrow it down quickly.

Related: #21474 (same crash class on iGPU/Meteor Lake — that reporter
also identified b8477 as working) I just used this image to see if it would work, and it did!

Relevant log output

Logs

--- CRASH on latest (qwen3next arch) ---

load_tensors: offloaded 49/49 layers to GPU
load_tensors: SYCL0 model buffer size = 17595.44 MiB
load_tensors: SYCL1 model buffer size = 16685.41 MiB
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve: graph nodes = 5108
common_init_from_params: warming up the model with an empty run...
Exception caught at ggml/src/ggml-sycl/ggml-sycl.cpp, line:3093
SYCL error: CHECK_TRY_ERROR(op(ctx, src0, src1, dst, ...stream))
in function ggml_sycl_op_mul_mat
signal: aborted (core dumped)

--- EMPTY CONTENT on latest (qwen35 arch) ---

{"choices":[{"finish_reason":"length","index":0,
"message":{"role":"assistant","content":"",
"reasoning_content":"1. = 4. = 3. = 4. = 3. = 4..."}}]}

content is empty, reasoning_content contains number spam

--- BOTH FIXED on server-intel-b8477 (identical commands) ---

qwen3next: server is listening, no crash, ~10 t/s clean output
qwen35: server is listening, clean text in content field,
full HTML page generated successfully

[NeoZhangJianyu]

@LVLUP-Studio

Got it! Let's check it.

[LVLUP-Studio]

Quick update. and sorry for the info dump. I was able to narrow this down a lot more by testing the different images.

I tested the same stack, same hardware, same runtime flags, and the same model across each build to see where it worked and where it started producing bad output. The only thing I changed between tests was the image tag and container name.

The model used for every test was:

Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

The model reports as:

general.architecture = qwen35moe
general.name = Qwen3.6-35B-A3B

This does not look like the container itself is crashing. The container starts, the SYCL backend loads, the model loads, warmup finishes, and the server accepts requests. The actual issue is that it starts generating corrupted / gibberish tokens. Sometimes that then causes the server to return a 500 because the bad generated text breaks parsing around <think>.

This was the compose stack used for every build, with only the image tag and container_name changed for each build:

version: "3.8"
services:
  llama-qwen36-dnn-test:
    image: ghcr.io/ggml-org/llama.cpp:server-intel-bXXXX
    container_name: llama-qwen36-dnn-test-bXXXX
    restart: "no"
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri/renderD129:/dev/dri/renderD129
      - /dev/dri/card1:/dev/dri/card1
      - /dev/dri/card2:/dev/dri/card2
    group_add:
      - "993"
      - "44"
    volumes:
      - /home/llm/models:/models:ro
    ports:
      - "8012:8012"
    environment:
      ZES_ENABLE_SYSMAN: "1"
      ZE_ENABLE_PCI_ID_DEVICE_ORDER: "1"
      ONEAPI_DEVICE_SELECTOR: "level_zero:0;level_zero:1"
      UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS: "1"
      GGML_SYCL_ENABLE_LEVEL_ZERO: "1"
      GGML_SYCL_ENABLE_VMM: "1"
      GGML_SYCL_DISABLE_GRAPH: "1"
      GGML_SYCL_DISABLE_DNN: "1"
    command: >
      --model /models/Qwen3.6-35B-Q4/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
      --ctx-size 8192
      --host 0.0.0.0
      --port 8012
      --jinja
      --parallel 1
      -np 1
      -ngl 999
      --split-mode layer
      --tensor-split 1,1
      --flash-attn off
      --temp 1.0
      --top-p 0.95
      --top-k 20
      --threads 1
    shm_size: 4g

The model has SSM / recurrent fields:

qwen35moe.ssm.conv_kernel = 4
qwen35moe.ssm.state_size = 128
qwen35moe.ssm.group_count = 16
qwen35moe.ssm.time_step_rank = 32
qwen35moe.ssm.inner_size = 4096

The runtime also reports:

fused Gated Delta Net (autoregressive) enabled
fused Gated Delta Net (chunked) enabled

Here is what I tested so far, all with the same model Qwen3.6-35B-A3B-UD-Q4_K_M.gguf and the same compose stack above:

server-intel-b9049: GOOD / coherent
model: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

server-intel-b9070: GOOD / coherent
model: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

server-intel-b9128: GOOD / coherent
model: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

server-intel-b9159: BAD / corrupted output
model: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

server-intel-b9191: BAD / corrupted output
model: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

server-intel-b9209: BAD / corrupted output
model: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

server-intel-b9222: BAD / corrupted output
model: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

server-intel-b9479: BAD / corrupted output
model: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

So the current narrowed window is:

GOOD: b9128 / 856c3adac
BAD:  b9159 / 5c0e9468

Same exact test prompt was used:

curl http://192.168.50.45:8012/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-35B-A3B-UD-Q4_K_M.gguf","messages":[{"role":"user","content":"say hello"}],"max_tokens":50,"stream":false}'

On b9128, the output is coherent. It starts like this:

Here's a thinking process:

1. **Analyze User Input:** The user said "say hello". This is a simple, direct request.
2. **Identify Intent:** The user wants me to greet them.

On b9159, the same prompt produces corrupted output and then the server throws a parse error:

Failed to parse input at pos 22: <think>
### 人喜欢velt6:  ### { each   约不会三高大河 typeas was use way
...

So the 500 looks like a downstream symptom. The actual problem seems to be that the generated reasoning text is already corrupted.

I also checked the commits between b9128 and b9159. These are the ones that stood out:

9ed6e19b 2026-05-14 SYCL: fix multi-GPU system RAM exhaustion by using Level Zero allocations (#21597)
0f45f1a3 2026-05-14 docker : revert stable version of intel compute-runtime (#22968)
42532aff 2026-05-14 unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regression tests (#22110)
5d44db60 2026-05-13 server, webui: support continue generation on reasoning models (#22727)
95d469a9 2026-05-13 server, webui: accept continue_final_message flag for vLLM API compat (#23012)
67b2b7f2 2026-05-14 logs : reduce (#23021)

The strongest suspect to me right now is:

9ed6e19b / #21597
SYCL: fix multi-GPU system RAM exhaustion by using Level Zero allocations

Main reason is that this repro is on 2x Arc Pro B60 GPUs using SYCL Level Zero, VMM, tensor split, and layer split. The issue shows up as generation corruption after the model loads successfully.

The second thing that seems worth checking is:

0f45f1a3 / #22968
docker: revert stable version of intel compute-runtime

That could explain a runtime behavior change inside the server-intel image even if the model graph code itself did not change much.

Earlier I was looking at #22149 because it added SYCL SSM / Gated Delta Net support, but that looks less likely to be the direct trigger now. b9070 contains ad092246 / #22149 and still works, and b9128 also works. The first bad image I have right now is b9159.

Current summary:

Same stack and same model tested across each build.
Model used on every build: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
Last confirmed good: server-intel-b9128 / 856c3adac
First confirmed bad: server-intel-b9159 / 5c0e9468
Likely area: b9128..b9159
Most suspicious change: #21597 / 9ed6e19b, SYCL multi-GPU Level Zero allocation changes
Also worth checking: #22968 / 0f45f1a3, Intel compute-runtime Docker change

I can attach the good b9128 log and bad b9159 log if that helps.