Vulkan MoE GEMV backend for integrated/AMD GPUs (draft, complements #418) - #729
Draft
minne100 wants to merge 1 commit into
Draft
Vulkan MoE GEMV backend for integrated/AMD GPUs (draft, complements #418)#729minne100 wants to merge 1 commit into
minne100 wants to merge 1 commit into
Conversation
…JustVugg#418) Self-contained Vulkan compute backend for the routed-expert GEMVs used by the qwen36 engine. int4 weights are unpacked in-shader with a plain float GEMV so it runs on drivers whose int8/dot-product compiler path is broken (AMD Radeon 780M); an int8 float path and an optional OpSDotKHR IDP shader (blacklisted on known-broken drivers) are also provided. Targets integrated/AMD GPUs (shared system memory, no CUDA) and is complementary to JustVugg#418 (steve-m, CUDA/NVIDIA-oriented), not a competing second backend. Status: DRAFT. Depends on JustVugg#712 (qwen36 engine) as the caller; not yet wired into dev's build. Integration point is vg_init()/vg_expert_ensure(). Fix: slot byte offsets were uint32_t, so a weights pool >= 4GB would wrap/alias (layers 32-39 -> 0-7 at cache>=256). The backend now auto-disables and falls back to the CPU MoE path instead of silently corrupting output. 64-bit shader offsets (buffer_device_address) are a follow-up.
|
multiple files are autogenerated from scripts that are not part of the pr, is that intentional?
|
This was referenced Aug 4, 2026
mgua
pushed a commit
to mgua/colibri
that referenced
this pull request
Aug 9, 2026
grep -rn "VK=1\|glslc\|vulkan" .github/workflows/
(nothing)
The Vulkan backend has never been compiled by any job in this repository.
Not built, not linked, not run. backend_vulkan.c and the four GLSL shaders
could be broken on dev right now and every check would stay green.
Three Vulkan PRs are open as this lands -- JustVugg#891, JustVugg#729, JustVugg#789 -- all written
by people without the hardware, and reviewed by reading the diff. JustVugg#892 and
JustVugg#887 are Vulkan bug reports from users whose cards nobody here has.
WHAT LAVAPIPE BUYS, AND WHAT IT DOES NOT
Mesa's software Vulkan runs on any x86 runner. Locally it produces:
[VK] VRAM pressure-proofing: memory_priority on, memory_budget on
[VK] ready: llvmpipe (LLVM 20.1.2, 256 bits), compute qfam 0, memtype 0,
fused gate+up, absorb attention
[VK] expert tier active: routed quantized experts on the GPU (budget 320)
So it proves: the .comp shaders compile to SPIR-V, the loader finds a
device, queue-family and memory-type selection works, the extension
negotiation works, and the expert tier initialises.
It proves NOTHING about performance -- it is a CPU rasteriser and will be
slower than the ordinary CPU path, so any tok/s measured under it is
meaningless. It also does not reproduce driver-specific behaviour: the
VK_EXT_memory_budget under-reporting on RADV RX 6000 in JustVugg#891 will not
appear here. Those still need @Limalski, @BranBushes, @MasterCATZ and
@krusherpt on real cards.
THE JOB
Two steps, both of which can fail independently:
1. `make colibri VK=1` plus an assertion that all four .spv exist and that
the binary actually links libvulkan. glslc can reject a .comp while
backend_vulkan.c compiles perfectly, so shader compilation is its own
failure mode and gets its own check.
2. Run against Lavapipe with a fabricated config.json -- enough to reach
coli_vk_init(), which runs before any weight is read. The run then fails
on the missing model, which is expected and ignored; what is asserted is
the [VK] banner.
VERIFIED IN BOTH DIRECTIONS
Passing, with the exact commands the job runs:
27532 shaders/attention_absorb.spv linkato a libvulkan OK
19120 shaders/qmatmul.spv PASS: [VK] ready
16924 shaders/qmatmul_gate_up.spv PASS: expert tier active
4180 shaders/rmsnorm.spv
Negative control, one shader removed:
[VK] cannot open shaders/qmatmul.spv
[VK] Vulkan backend unavailable (tried shaders/qmatmul.spv; ...)
PASS: the gate FAILS, as it must
A job that cannot fail is worse than no job, and this repo has shipped one
of those recently: JustVugg#868's release check was named "coli would not resolve
these next to itself" and asserted file existence, so it passed while the
launcher could not select the engine (JustVugg#879).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
|
@minne100 status check: this draft predates several rounds of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Vulkan MoE GEMV backend for integrated / AMD GPUs (DRAFT)
Isolates the Vulkan compute path for the qwen36 engine routed-expert GEMVs so it can be reviewed independently of the engine PR (#712).
What
Relation to #418
#418 (steve-m) is the project Vulkan backend, CUDA/NVIDIA-oriented (expert tier + dense + MLA). This one targets integrated/AMD GPUs (shared memory, no CUDA) — complementary, not competing. Happy to coordinate with steve-m / maintainers toward a single backend if preferred.
Status: DRAFT
Fix included here
This supersedes the Vulkan portion of the now-closed #602 (engine taken over by #712).