[OpenVINO] Tensor element type mismatch (f32 != f16) when running Qwen2.5-Coder on NPU

[ZijinWukll]

Environment

• llama.cpp build: local build from source (no git tag)
• Hardware: Intel Core Ultra 9 285H (Arrow Lake), no dGPU, Intel NPU
• OS: Windows 11
• OpenVINO: 2025.4
• Model: Qwen2.5-Coder-7B-Instruct (GGUF, both Q4_K_M and Q4_0 tested)

Build Configuration

• GGML_OPENVINO=ON
• GGML_OPENVINO_NO_OPENCL=ON (no OpenCL SDK available on the system)
• Windows SDK + MSVC 19.50

Bug Description

Model loads successfully but fails during inference with:
GGML OpenVINO backend ov::Exception: Exception from infer_request.cpp:224:
Check 'dst->get_element_type() == get_element_type()' failed at itensor.cpp:79:
Tensor element types are not equal. (src: f32 != dst: f16)

This occurs with GGML_OPENVINO_DEVICE set to either NPU or CPU.

Steps to Reproduce

1. Build llama.cpp with GGML_OPENVINO=ON on Windows
2. Download Qwen2.5-Coder-7B-Instruct Q4_K_M or Q4_0 GGUF
3. Run: llama-cli -m <model> --conversation
4. Send any prompt → Compute error with above exception

[twoplan]

Looks like we are in the same boat.
I'm getting an similar error on linux with Llama-3.2-1B-Instruct-Q4_0.gguf

Just compiled llama.cpp build b9525-ad1b88ca0 on Ubuntu 26.04 with OpenVino 2026.2.0
Fails to run on NPU, but works fine with GPU and CPU here.

Loading model... \0.23.206.999 E GGML OpenVINO backend ov::Exception: Exception from src/inference/src/cpp/infer_request.cpp:224:
Check 'dst->get_element_type() == get_element_type()' failed at src/core/src/runtime/itensor.cpp:75:
Tensor element types are not equal. (src: f32 != dst: f16)

./llama-bench -m ./Llama-3.2-1B-Instruct-Q4_0.gguf
OpenVINO: using device GPU

model	size	params	backend	ngl	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	OPENVINO	-1	pp512	8628.66 ± 83.56
llama 1B Q4_0	729.75 MiB	1.24 B	OPENVINO	-1	tg128	90.18 ± 0.45
build: ad1b88c (9525)

[ZijinWukll]

Looks like we are in the same boat. I'm getting an similar error on linux with Llama-3.2-1B-Instruct-Q4_0.gguf

Just compiled llama.cpp build b9525-ad1b88ca0 on Ubuntu 26.04 with OpenVino 2026.2.0 Fails to run on NPU, but works fine with GPU and CPU here.

Loading model... \0.23.206.999 E GGML OpenVINO backend ov::Exception: Exception from src/inference/src/cpp/infer_request.cpp:224:
Check 'dst->get_element_type() == get_element_type()' failed at src/core/src/runtime/itensor.cpp:75:
Tensor element types are not equal. (src: f32 != dst: f16)

./llama-bench -m ./Llama-3.2-1B-Instruct-Q4_0.gguf OpenVINO: using device GPU

model size params backend ngl test t/s
llama 1B Q4_0 729.75 MiB 1.24 B OPENVINO -1 pp512 8628.66 ± 83.56
llama 1B Q4_0 729.75 MiB 1.24 B OPENVINO -1 tg128 90.18 ± 0.45
build: ad1b88c (9525)

---

Thanks for confirming. Same error here on Windows (Core Ultra 9 285H, OpenVINO 2025.4, Qwen2.5-Coder-7B).

The fact that GPU works but NPU doesn't is a useful data point — it suggests the ggml-openvino backend always passes f32 tensors, which the GPU plugin
accepts, but the NPU plugin strictly requires f16. The mismatch likely needs to be handled either by converting tensor types in the ggml-openvino backend
based on the target device, or by explicitly setting input precision during ov::Core::compile_model.

My current workaround is using the CPU backend (GGML_OPENVINO_DEVICE=CPU or just disabling OpenVINO), but that obviously defeats the purpose of using the NPU. Hope this gets fixed in a future release.

[twoplan]

I went back to build 9400, which was confirmed working here.

My bench results on an Intel ultra 7 258V laptop:

export GGML_OPENVINO_DEVICE=NPU
LD_LIBRARY_PATH=/opt/intel/openvino_2026.0.0/runtime/lib/intel64 ./llama-bench -m ./Llama-3.2-1B-Instruct-Q4_0.gguf -fa 1

OpenVINO: using device NPU

model	size	params	backend	ngl	fa	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	OPENVINO	99	1	pp512	1580.45 ± 11.61
llama 1B Q4_0	729.75 MiB	1.24 B	OPENVINO	99	1	tg128	45.88 ± 0.48
build: c8914ad (9400)

export GGML_OPENVINO_DEVICE=GPU
LD_LIBRARY_PATH=/opt/intel/openvino_2026.0.0/runtime/lib/intel64 ./llama-bench -m ./Llama-3.2-1B-Instruct-Q4_0.gguf -fa 1

OpenVINO: using device GPU

model	size	params	backend	ngl	fa	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	OPENVINO	99	1	pp512	8225.19 ± 95.69
llama 1B Q4_0	729.75 MiB	1.24 B	OPENVINO	99	1	tg128	89.58 ± 0.63
build: c8914ad (9400)

export GGML_OPENVINO_DEVICE=CPU
LD_LIBRARY_PATH=/opt/intel/openvino_2026.0.0/runtime/lib/intel64 ./llama-bench -m ./Llama-3.2-1B-Instruct-Q4_0.gguf -fa 1

OpenVINO: using device CPU

model	size	params	backend	ngl	fa	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	OPENVINO	99	1	pp512	434.13 ± 2.64
llama 1B Q4_0	729.75 MiB	1.24 B	OPENVINO	99	1	tg128	64.94 ± 1.38
build: c8914ad (9400)

[ZijinWukll]

Thank you so much for your detailed bisect and benchmark data! This was incredibly helpful.

I followed your approach and can confirm your findings on my side:

Windows setup:

• Intel Core Ultra 9 285H (Arrow Lake)
• OpenVINO 2025.4
• Model: Qwen2.5-Coder-7B-Instruct-Q4_0 (7B, ~4.12 GiB)

Confirmed working commits:

• c8914ad4f (build 9400) — NPU works:

  | model     | size    | params | backend  | ngl | fa | test     | t/s              |
  | qwen 7B   | 4.12 GiB| 7.62 B | OPENVINO |  99 |  1 | pp512    | 99.56 ± 0.99     |
  | qwen 7B   | 4.12 GiB| 7.62 B | OPENVINO |  99 |  1 | tg128    | 4.66 ± 0.07      |
  

• 751ebd17a (build 9401) — still works
• ea02bc37f (ggml 0.13.1, build 9408) — still works:

  | model     | size    | params | backend  | ngl | fa | test     | t/s              |
  | qwen 7B   | 4.12 GiB| 7.62 B | OPENVINO |  99 |  1 | pp512    | 102.76 ± 0.63    |
  | qwen 7B   | 4.12 GiB| 7.62 B | OPENVINO |  99 |  1 | tg128    | 4.81 ± 0.14      |
  

What breaks:
Starting somewhere after ea02bc37f, running with GGML_OPENVINO_DEVICE=NPU fails with:

test_prompt: failed to decode prompt batch, res = -3

I did a rough bisect and narrowed it down to the range ea02bc37f..cb47092b0 (4 commits). Two of them seem suspicious:

1. fe12e422a — "sync: ggml" — a ggml core sync that could have changed tensor type handling
2. 031ddb2e0 — "llama: use f16 mask for FA to save VRAM" — directly related to f16 tensor usage, which aligns with the f32/f16 NPU tensor type mismatch theory

I wasn't able to pinpoint the exact commit (rebuilding 7B models on Windows is time-consuming), but hopefully this narrows it down a bit for anyone who wants to investigate further.

For now, I'm using build ea02bc37f as my daily driver with NPU and it works great. The NPU performance is about 100 t/s prefill and 4.8 t/s generation for the 7B Q4_0 model — not blazing fast, but usable for code assistance, and it keeps the CPU/GPU free for other work.

Thanks again for sharing your bisect method — your approach of using llama-bench with a small Q4_0 model to quickly test each build was exactly the right way to do this. Really appreciate it!