Add DeepSeek V4 target-only CPU inference - #165
Conversation
DrewZt
changed the title
rajpratham1
left a comment
rajpratham1
left a comment
There was a problem hiding this comment.
This is a very impressive contribution and clearly represents a significant amount of engineering work. The implementation covers a complete DeepSeek V4 CPU inference pipeline including runtime, expert streaming, quantization, safetensors loading, CLI tooling, documentation, and an extensive unit test suite.
Because this PR introduces an entirely new inference stack across many core components, I'd prefer additional review before approval.
Some areas that would benefit from closer review include:
- Long-term API stability for the new DeepSeek V4 interfaces.
- Memory ownership and lifetime throughout the expert streaming/runtime pipeline.
- Performance characteristics of the NVMe streaming implementation under sustained inference.
- Cross-platform compatibility (Windows/Linux/macOS) for filesystem and I/O paths.
- Validation against larger real-world models beyond the included unit tests.
Overall the direction looks very promising, but given the size and architectural impact of this change, I think it should receive another maintainer review before merging.
|
Thanks for the thoughtful review. I agree that another maintainer review is appropriate given the size of the change. A few clarifications on the areas you mentioned: API stability: the new interfaces are currently scoped to the DeepSeek V4 engine and should be considered experimental. They are not intended to establish a stable generic model API at this stage. I’m happy to address targeted follow-up findings, add more validation, or split parts of the implementation into staged PRs if the maintainers feel that would make review and long-term maintenance easier. |
|
I pushed a follow-up series through The main changes are:
The latest changes pass: The remaining validation work is focused on sustained NVMe behavior and broader full-model comparison against the official Transformers implementation. Those limitations remain documented in the PR and do not represent unresolved API or resource-lifetime issues. I’m marking the PR ready for review and would appreciate another maintainer look, particularly at the revised API boundary and ownership model. |
|
First full-model validation on Linux/x86-64 — plus a build fix and an AVX2 kernel series, branch ready to pull: steve-m/colibri@v4-avx2-kernels (5 commits on top of this PR's Hardware: Ryzen 9 3900X (Zen 2, 12C/24T, AVX2 no AVX-512), 62 GiB RAM, NVMe ~6 GB/s, Manjaro, gcc 15. Model: Build fix you'll want regardless: two amalgam units ( Kernel series (all behind a
Measured, 48-token free-form decode, OMP_NUM_THREADS=12: 0.53 tok/s (PR defaults) → 0.62 (threads = physical cores; SMT only adds barrier traffic) → 0.78 (FP4 int8) → 0.99 (head) → 1.12 tok/s (FP8). Verify phase −27%, TTFT 20 → 16.7 s. Validation at every step: your oracle tool's Also on the branch, off by default: a dual-SSD mirror ( Happy to split any of this into separate PRs against your |
This is extremely helpful — thank you for doing the first independent full-model Linux/x86-64 validation and for documenting the performance progression in such detail. The Ryzen 3900X result directly addresses one of the main remaining validation gaps for this PR, and the 0.53 → 1.12 tok/s breakdown makes it much easier to see where the current bottlenecks are. I checked the branch history and it looks like Those later commits include the revised public/internal API separation, engine/session lifetime accounting, failure-path cleanup, isolated ownership-test objects, and V4 session tokenizer cleanup. The optimization work is still very valuable, but it should be rebased onto One clarification regarding the pthread build issue: I had already addressed the missing declaration problem in Your source-level includes may still be a cleaner and more localized solution, but the issue itself no longer needs a separate build-fix PR. When rebasing, please either drop the overlapping build-fix commit or call out why replacing the current compiler-level include with explicit includes in the two amalgam units would be preferable. For the remaining work, I suggest splitting it into two focused follow-ups:
I also noticed that the branch diff appears to include generated test binaries such as: Please drop those artifacts when preparing the follow-up PRs. After rebasing, please rerun the full-model correctness and performance validation because the runtime and ownership code has changed since The current benchmark result is already valuable as independent validation of the earlier implementation. Once the rebased kernel series is reviewed, I’d be happy to add the reproducible Linux/x86-64 measurements to the documentation with credit to you and the exact hardware and commands. Thanks again — this is a substantial and very useful contribution. Rebasing and splitting it should let us preserve the recent API and ownership work while giving the kernels and storage experiment the focused review they deserve. |
|
Follow-up for @JustVugg: head af86de1 closes the remaining validation findings. The safetensors index now requires payload bytes to equal dtype width times shape numel; V4 config integers, floats, and compress ratios now have finite/integer/range checks. I also removed the dense-cache borrowed config pointer in favor of the engine-owned canonical config and made the DSpark oracle require exact output lengths on both paths. Fresh validation passed make check (all C tests plus 71 Python tests), x86-64-v3/native builds, ASan+UBSan+LSan, and the 48-shard MEMORY_GB=32 oracle (19/19 teacher forcing, 8/8 greedy, 8/8 continuation self-check, exact DSpark on/off identity). This supersedes my earlier blocker summary; AVX2 and dual-SSD work remain out of scope. Could you please review when convenient? |
|
Pushed the rebased AVX2 kernel series to Two changes from the old
The three commits, all behind the
Validation on the rebased branch — Ryzen 9 3900X (Zen 2, AVX2 no AVX-512), 62 GiB, gcc 16.1.1,
Perf — AVX2 on vs the Kernel-only isolation, DSpark disabled (
For reference, DSpark on (same prompt): decode 0.62 → 0.85 tok/s (+36%), TTFT 43.6 → 30.3 s (−30%) — but that pass had asymmetric acceptance (on 3/10 vs off 1/10 speculative tokens), so the isolated The float-path baseline matches the known physical-cores number, so the kill-switch cleanly isolates the kernel gain. TTFT is stable across the DSpark condition (~31 s on / ~48 s off), as expected — the first token is prefill + first decode, before any speculation. (Absolute tok/s is workload-dependent — this was a heavy ~80 GB-streaming pass; a lighter/warmer pass on this box peaks around 1.12 tok/s. The on/off ratio is the stable claim.) |
This looks excellent — thank you for rebasing the series carefully and for preserving the API and ownership work underneath it. The updated scope is exactly what I was hoping for:
I also appreciate the distinction between the DSpark-enabled result and the The zero-copy resident-head path being exercised by the full-model oracle is also useful confirmation that the new path is covered rather than only compiled. Since the kernel branch is stacked on top of #165, I think the cleanest next step is to keep the branch as-is for now and open a dedicated follow-up PR after #165 is merged. At that point it can be rebased onto the resulting upstream Please preserve the current commit separation and validation details in that PR. In particular, the Thanks again — this is a strong follow-up series, and the careful validation and honest performance attribution make it much easier to review. |
|
Pushed follow-up stability fix
Validation:
|
|
I'm interested in this — DeepSeek V4 Flash on CPU with NVMe expert streaming is squarely what colibrì is for, and I'd like it in. The one condition is that I need to run it on my own machine first. Not as a gate to be difficult: it's the rule I've had to learn the hard way this week. So: as soon as I can get a checkpoint on this box, I'll test it and we'll work on it together. That's not a "no" parked forever — it's the next thing I want to do on this front. Two things that would make it land sooner, and I'd rather ask than have you guess:
I saw the stability fix you pushed ( Keeping this open. Let's keep going. |
Thanks — this gives me a clear integration path. I’ll make the tiny DeepSeek V4 fixture the immediate priority. The goal will be a deterministic, independently generated checkpoint and reference that The initial test contract will cover:
The generator will remain available so the fixture is reproducible rather than an opaque committed artifact, while CI itself will use the checked-in tiny checkpoint and reference. I’ll also continue removing local duplication where that does not change runtime or kernel boundaries. For the deeper shared-I/O work—particularly consolidating the streaming read path with the existing DIRECT/URING infrastructure—I would prefer to do that as a focused follow-up after this base engine and the already prepared AVX2 series land. That keeps the current correctness and performance baselines stable, avoids repeatedly invalidating the AVX2 branch, and lets the tiny token-exact oracle protect the later refactor across both scalar and optimized paths. I’ll keep AVX2 and the dual-SSD experiment out of this PR, and I’ll document the shortest full-checkpoint smoke-test path for your machine alongside the tiny fixture. |
|
Implemented the requested deterministic tiny DeepSeek V4 + DSpark oracle at head
Manual local validation (from the repository root): # Dedicated tiny target + DSpark token-exact oracle
make -C c deepseek-v4-tiny-check ARCH=x86-64-v3
# Or run the complete dependency-free check suite
make -C c checkTo exercise the normal runtime explicitly with drafting disabled: cd c
make deepseek-v4 ARCH=x86-64-v3
./deepseek_v4 deepseek_v4_tiny '<t005><t007><t009>' \
--raw-prompt --draft-model deepseek_v4_tiny/dspark --no-dsparkThe generator and regeneration package versions are documented in |
sorry what?! That's impressive for NVMe. |
The prompt 'What is the capital of France' have some uniqueness, the answer token got 100% dspark acceptance rate, that is the fastest speed boost. |
|
On your own Pc, Vincent, how does it compare to GLM in speed if you don't mind me asking (And what are the specs of yours , and what OS) 🤔From: DrewZt ***@***.***>Sent: Saturday, July 18, 2026 12:07:24 pmTo: JustVugg/colibri ***@***.***>Cc: Maikel Frias Mosquea ***@***.***>; Comment ***@***.***>Subject: Re: [JustVugg/colibri] Add DeepSeek V4 Flash CPU inference with NVMe expert streaming (PR #165)DrewZt left a comment (JustVugg/colibri#165)1.12 tok/s (FP8).sorry what?! That's impressive for NVMe.The prompt 'What is the capital of France' have some uniqueness, the answer token got 100% dspark acceptance rate, that is the fastest speed boost.
For another 0% dspark acceptance rate special case 'hello', decode speed will drop to 0.59tok/s, which is slower than no-dspark situation—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.
|
My pc is ai max 395+128gb ram+6gb/s ssd, tested glm with config think=0 and mtp=1, got around 0.7 tok/s |
|
Status check: |
Yes, I’m still actively working on this. Please keep the PR open. I reviewed the impact of the current I’m going to treat this as a minimal forward port onto current Work I will complete before merging #165
I had already done additional local work around redundant expert reads and the streamed I/O path, but I intentionally did not push it onto #165. At that point the branch had a stable correctness baseline and Steve’s AVX2 series was stacked on top of it; changing the I/O and cache boundaries again would have invalidated both his branch and the existing measurements. Work planned after #165 merges
So the immediate objective is to restore #165 on current |
|
The forward port is now pushed. PR #165 now points to The main Local The new PR workflows are currently waiting for maintainer approval. Could you approve them so the checks can run? |
One conflict, in c/coli's cmd_run. dev added the DeepSeek V4 branch (JustVugg#165) directly after a plain banner("run"); this branch moved that banner below env_for so it can report the CUDA backend -- banner("run", COLI_CUDA=="1"). Taking dev's hunk verbatim would have reintroduced the plain banner and printed two. Taking this branch's side would have dropped the V4 path. Resolved by keeping the V4 branch and giving it its own banner("run"), since that path sys.exit()s before reaching the CUDA-aware call. Both intents preserved; neither side loses a line. `python -m ast` parses the result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same conflict as JustVugg#825 and the same resolution, in c/coli's cmd_run. dev added the DeepSeek V4 branch (JustVugg#165) right after a plain banner("run"); this branch moved that banner below env_for so it can report the backend -- banner("run", COLI_CUDA=="1"). Kept the V4 branch and gave it its own banner("run"), since that path sys.exit()s before reaching the CUDA-aware call. Taking either side whole would have printed two banners or dropped the V4 path. `python -m ast` parses the result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DeepSeek V4 (JustVugg#165) landed on dev. Two hunks, both "keep both": - c/Makefile: dev reflowed .PHONY across lines and added the deepseek-v4 targets plus a separate bench-omp-grain line; this branch had added hip-dll to the same list. Took dev's form and re-inserted hip-dll after cuda-dll. - c/tools/clean.py: both sides appended to the artifact list. Kept both blocks. Merged rather than rebased so the commit SHAs, the CI results just unblocked for this PR, and the review comments anchored to them all survive.
Rebasing on dev turned check-env red, which is the point of it: - read by the code but missing from coli_env.h: COLI_MTP_GUARD_PCT - read by the code but missing from coli_env.h: COLI_MTP_GUARD_WINDOW - read by the code but missing from coli_env.h: COLI_V4_EXPERT_PREFETCH - read by the code but missing from coli_env.h: CUDA_EXPERT_LOAD_BALANCE - read by the code but missing from coli_env.h: INK_METAL_SHARED - read by the code but missing from coli_env.h: V4_PREFIX_LOG Types come from the call sites, not from the names: the three read through atoi() are CE_INT, INK_METAL_SHARED and COLI_V4_EXPERT_PREFETCH are tested as switches so CE_BOOL, and V4_PREFIX_LOG is presence-checked like the INK_PREFIX_LOG row already in the table, so CE_STR. Two of them belong to deepseek_v4.c, which JustVugg#165 added and which has no engine bit here, so add CE_DSV4. It is deliberately NOT folded into CE_ALL: that flag means the four engines sharing route_trace.h, rans.h and omp_tune.h, and deepseek_v4.c includes none of the three -- putting it in CE_ALL would claim every shared-header knob is read by V4 too. Also refresh the header comment, which had gone stale in the same way the table would have: 212 -> 220. The "187 scattered getenv() call sites" figure is removed rather than corrected -- nothing verifies it, so it can only rot again. `make check-env` prints the live count and is the one number in that comment that cannot go stale. check-env: 220 variables, registry matches the sources.
## The roster said four families; there are five DeepSeek V4 Flash landed in JustVugg#165 and was tuned in JustVugg#839, but the README still opened with "Four families run today" and its table stopped at OLMoE. Someone scanning the front page had no way to learn the engine exists. It is now in the opening line, in the roster table, and in the hardware table above it -- ~167 GB on disk, 16 GB of RAM minimum and 22 comfortable, measured on the reference box rather than estimated. ## The DeepSeek section described a version that no longer exists It called the path "experimental" and said "DSpark is intentionally kept for a separate stacked follow-up". DSpark is in, and the honest state is more interesting than either claim: - the checkpoint streams with no conversion -- routed experts stay native fp4, dense stays fp8-e4m3 with UE8M0 block scales - greedy, one KV slot, no tools or grammar yet: said plainly, because finding that out from a rejected request is worse - --ram is the knob that matters. 43 x 256 routed experts are ~137 GiB and a token touches 301 of them, so the cache hit rate sets tok/s. It changes speed only, never output. - speculative drafting is implemented, verified, and OFF, with the numbers that made that call: 1 accepted in 15 for the markov drafter, 10 in 24 for full MTP, and a 14-token answer that took 495 seconds to replay its rejected suffixes That last one is the point of documenting it at all. The code stays, the measurement stays beside it, and whoever retries this on faster storage starts from evidence instead of from scratch. ## Repo layout described a tree that has not existed since July It listed `glm.c`, renamed to `colibri.c` in JustVugg#391 three weeks ago, and no other engine -- so the file that runs GLM was wrong and the four files that run everything else were missing. Also absent: quant.h, compat.h, expert_store.h, route_trace.h, kv_prefix.h, the Metal and Vulkan backends, resource_plan.py, and docker/. Every path and every make target in the new listing was checked to exist on this branch before it was written down. The rule behind the layout is now stated, because it is the one that keeps being violated: one .c per model family, over shared single headers. An engine owns its architecture and nothing else. The recurring defects in this tree -- the OpenMP thread count, the KV prefix reuse, the NaN router guard -- are all the same shape: a mechanism that landed in one engine and never reached its siblings. ## Also `#### Other supported models` now sits where the roster table is, so `[Full roster ↓](#other-supported-models)` in the opening paragraph resolves to the table instead of to prose four sections earlier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This is JustVugg#772 rescoped. It was an engine plus its kernels and it would not start below 145.3 GiB of VRAM, which makes it a threshold -- the one thing this project does not ship. @jazir555 and @rafpigna said so, @JustVugg agreed, and the request was not negotiable: a GPU backend here holds the hot part and falls back for the rest. The split @JustVugg proposed is the right one. JustVugg#165 landed the DeepSeek V4 engine with the streaming machinery and no CUDA at all; this was the CUDA with no tier. Neither runs the model on hardware anyone owns. So the engine file belongs to JustVugg#165 and this keeps only the kernels: backend_cuda_dsv4*.{cu,h} dense matmul, batched attention, routed MoE dsv4_mhc.h, dsv4_quant.h the CPU-side formats the kernels consume tests/ three GPU tests, two pure-CPU unit tests tools/ oracles and probes the kernels are checked against Dropped from the old branch: c/deepseek_v4.c (JustVugg#165 owns it), and the c/coli, c/openai_server.py, c/tok.h and docs/api.md edits, which belong to whichever engine lands rather than to a kernel PR. The kernels are self-contained by construction -- backend_cuda_dsv4.h includes only <stdint.h>, and the .cu only its own header and the CUDA runtime -- so they build and their CPU tests run with no engine present. That is what makes them usable as a tier: JustVugg#165 calls in, nothing calls out. Makefile: the kernel rules only. deepseek_v4$(EXE) is gone; the .o rules, dsv4-cuda-test and the two header unit tests remain. DSV4_CUDA_OBJ is empty unless CUDA=1. VLLM_MHC/DEEPGEMM/FLASHINFER are marked UNSUPPORTED in place -- eight configurations, none compiled by CI, each needing an external checkout -- and DEEPGEMM's flag line appends instead of replacing, so it no longer discards $(CUDA_GENCODE) and -ccbin. Verified: colibri, deepseek-v4, cuda-test and bench-omp-grain all still resolve; dsv4-cuda-test resolves; test_dsv4_mhc and test_dsv4_quant build and pass on CPU. The GPU tests need a device and are compile-checked only.
Rebasing on dev turned check-env red, which is the point of it: - read by the code but missing from coli_env.h: COLI_MTP_GUARD_PCT - read by the code but missing from coli_env.h: COLI_MTP_GUARD_WINDOW - read by the code but missing from coli_env.h: COLI_V4_EXPERT_PREFETCH - read by the code but missing from coli_env.h: CUDA_EXPERT_LOAD_BALANCE - read by the code but missing from coli_env.h: INK_METAL_SHARED - read by the code but missing from coli_env.h: V4_PREFIX_LOG Types come from the call sites, not from the names: the three read through atoi() are CE_INT, INK_METAL_SHARED and COLI_V4_EXPERT_PREFETCH are tested as switches so CE_BOOL, and V4_PREFIX_LOG is presence-checked like the INK_PREFIX_LOG row already in the table, so CE_STR. Two of them belong to deepseek_v4.c, which JustVugg#165 added and which has no engine bit here, so add CE_DSV4. It is deliberately NOT folded into CE_ALL: that flag means the four engines sharing route_trace.h, rans.h and omp_tune.h, and deepseek_v4.c includes none of the three -- putting it in CE_ALL would claim every shared-header knob is read by V4 too. Also refresh the header comment, which had gone stale in the same way the table would have: 212 -> 220. The "187 scattered getenv() call sites" figure is removed rather than corrected -- nothing verifies it, so it can only rot again. `make check-env` prints the live count and is the one number in that comment that cannot go stale. check-env: 220 variables, registry matches the sources.
This is JustVugg#772 rescoped. It was an engine plus its kernels and it would not start below 145.3 GiB of VRAM, which makes it a threshold -- the one thing this project does not ship. @jazir555 and @rafpigna said so, @JustVugg agreed, and the request was not negotiable: a GPU backend here holds the hot part and falls back for the rest. The split @JustVugg proposed is the right one. JustVugg#165 landed the DeepSeek V4 engine with the streaming machinery and no CUDA at all; this was the CUDA with no tier. Neither runs the model on hardware anyone owns. So the engine file belongs to JustVugg#165 and this keeps only the kernels: backend_cuda_dsv4*.{cu,h} dense matmul, batched attention, routed MoE dsv4_mhc.h, dsv4_quant.h the CPU-side formats the kernels consume tests/ three GPU tests, two pure-CPU unit tests tools/ oracles and probes the kernels are checked against Dropped from the old branch: c/deepseek_v4.c (JustVugg#165 owns it), and the c/coli, c/openai_server.py, c/tok.h and docs/api.md edits, which belong to whichever engine lands rather than to a kernel PR. The kernels are self-contained by construction -- backend_cuda_dsv4.h includes only <stdint.h>, and the .cu only its own header and the CUDA runtime -- so they build and their CPU tests run with no engine present. That is what makes them usable as a tier: JustVugg#165 calls in, nothing calls out. Makefile: the kernel rules only. deepseek_v4$(EXE) is gone; the .o rules, dsv4-cuda-test and the two header unit tests remain. DSV4_CUDA_OBJ is empty unless CUDA=1. VLLM_MHC/DEEPGEMM/FLASHINFER are marked UNSUPPORTED in place -- eight configurations, none compiled by CI, each needing an external checkout -- and DEEPGEMM's flag line appends instead of replacing, so it no longer discards $(CUDA_GENCODE) and -ccbin. Verified: colibri, deepseek-v4, cuda-test and bench-omp-grain all still resolve; dsv4-cuda-test resolves; test_dsv4_mhc and test_dsv4_quant build and pass on CPU. The GPU tests need a device and are compile-checked only.
…wrong
The site said colibri runs GLM-5.2. It has run four families since v1.3.0, and
two of the model cards contradicted the project outright:
Inkling 975B MoE - Planned -> Live (docs/inkling.md ships; runs on 25 GB)
Kimi K2 1T MoE - Planned -> Kimi K3, 2.8T MoE, Live
Telling visitors that Inkling and Kimi are on the roadmap, while the README
front page says both run today, is the kind of contradiction someone finds in
thirty seconds.
Hero rewritten rather than merely widened. The old line worked because it put
two incompatible things next to each other -- an enormous model, your machine.
Replacing that with a range ('744B to 2.8T') informs and stops landing; the
contradiction was the message. It now reads:
These models do not fit in your machine. They run in it anyway.
Same rhetorical shape as the copy further down the page ('Weights are not state
to hold. They are data to stage.'), so the page speaks with one voice. The
subtitle now also explains WHY it is possible -- a MoE token touches a small
fraction of the weights -- which was missing entirely and is what turns an
unbelievable claim into an understandable one. The numbers move there, where
they serve the reader who wants detail instead of the one who is skimming.
DeepSeek and Qwen3 deliberately stay 'Planned': JustVugg#165 and JustVugg#712 are not merged,
and the site should not promise what the code does not do.
Text only. No CSS, structure or script changes -- the sole markup edits are the
two cards' buttons becoming real links now that both models are runnable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Native fp8 checkpoints write the fmt=8 block scale as one UE8M0 byte per block rather than one f32. The geometry is identical -- same shape, same meaning, same multiply -- so the load path now reads the sidecar with st_read_scale_f32, which accepts either encoding and always yields f32. matmul_fp8 is untouched and stays a single implementation with no branch in the hot loop, which is the point of expanding at load rather than decoding per block. Every other format still goes through st_read_f32_cap exactly as before: fmt 0/1/2/4/5/6 are byte-for-byte unchanged. An f32-scaled fmt=8 container behaves identically too, since st_read_scale_f32 dispatches to st_read_f32 for an F32 sidecar -- the same call that ran before. Requested by DrewZt on JustVugg#165, where it was the one shared-infrastructure blocker for moving the DeepSeek V4 engine onto the common quant path. Verified against the real DeepSeek-V4-Flash-0731 checkpoint: the attention sidecars read back as exactly 2^-12 and 2^-11, and the dequantised weights land at |max| 0.094 / |mean| 0.018. All four engines build; make check 288 tests OK. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add DeepSeek V4 Flash target-only CPU inference (@DrewZt) The engine loads official sharded DeepSeek V4 safetensors, implements target prefill and greedy decode, compressed attention, mHC, routed and shared experts, and RAM-tiered ExpertStore caching. It is wired into `coli run`, `coli chat`, `coli serve` and `coli web`, and reuses the shared `st.h` indexing and `quant.h` fmt7 MXFP4 matmul. Merged with one fixup folded into this commit rather than landing a broken tree: `v4_serve_rss_gb` calls getrusage(RUSAGE_SELF) but the GENERATE_STATS unit never included <sys/resource.h>, so COLI_V4_UNIT_GENERATE_STATS.o failed to compile on Linux and took the tiny-oracle CI job with it. Windows was unaffected because compat.h already supplies a getrusage shim, which is why only one of the fourteen checks was red. The include follows the guard the other engines use (colibri.c:35, inkling.c:31, kimi_k3.c:71). That CI had never run at all: the workflow sat in `action_required` for eighteen days, so the first execution on this branch is also the first signal the author ever received. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DeepSeek V4 (JustVugg#165) landed on dev. Two hunks, both "keep both": - c/Makefile: dev reflowed .PHONY across lines and added the deepseek-v4 targets plus a separate bench-omp-grain line; this branch had added hip-dll to the same list. Took dev's form and re-inserted hip-dll after cuda-dll. - c/tools/clean.py: both sides appended to the artifact list. Kept both blocks. Merged rather than rebased so the commit SHAs, the CI results just unblocked for this PR, and the review comments anchored to them all survive.
## The roster said four families; there are five DeepSeek V4 Flash landed in JustVugg#165 and was tuned in JustVugg#839, but the README still opened with "Four families run today" and its table stopped at OLMoE. Someone scanning the front page had no way to learn the engine exists. It is now in the opening line, in the roster table, and in the hardware table above it -- ~167 GB on disk, 16 GB of RAM minimum and 22 comfortable, measured on the reference box rather than estimated. ## The DeepSeek section described a version that no longer exists It called the path "experimental" and said "DSpark is intentionally kept for a separate stacked follow-up". DSpark is in, and the honest state is more interesting than either claim: - the checkpoint streams with no conversion -- routed experts stay native fp4, dense stays fp8-e4m3 with UE8M0 block scales - greedy, one KV slot, no tools or grammar yet: said plainly, because finding that out from a rejected request is worse - --ram is the knob that matters. 43 x 256 routed experts are ~137 GiB and a token touches 301 of them, so the cache hit rate sets tok/s. It changes speed only, never output. - speculative drafting is implemented, verified, and OFF, with the numbers that made that call: 1 accepted in 15 for the markov drafter, 10 in 24 for full MTP, and a 14-token answer that took 495 seconds to replay its rejected suffixes That last one is the point of documenting it at all. The code stays, the measurement stays beside it, and whoever retries this on faster storage starts from evidence instead of from scratch. ## Repo layout described a tree that has not existed since July It listed `glm.c`, renamed to `colibri.c` in JustVugg#391 three weeks ago, and no other engine -- so the file that runs GLM was wrong and the four files that run everything else were missing. Also absent: quant.h, compat.h, expert_store.h, route_trace.h, kv_prefix.h, the Metal and Vulkan backends, resource_plan.py, and docker/. Every path and every make target in the new listing was checked to exist on this branch before it was written down. The rule behind the layout is now stated, because it is the one that keeps being violated: one .c per model family, over shared single headers. An engine owns its architecture and nothing else. The recurring defects in this tree -- the OpenMP thread count, the KV prefix reuse, the NaN router guard -- are all the same shape: a mechanism that landed in one engine and never reached its siblings. ## Also `#### Other supported models` now sits where the roster table is, so `[Full roster ↓](#other-supported-models)` in the opening paragraph resolves to the table instead of to prose four sections earlier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
9 participants
Summary
This PR now contains the DeepSeek V4 target-only CPU engine requested in the latest review. DSpark speculative decoding has been removed from #165 and preserved on
DrewZt:pr165-full-dspark-backupfor a separate stacked follow-up.The target engine:
st.hfor safetensors indexing/range I/O;quant.hfor canonical fmt7 MXFP4 matmul;coli run,coli chat,coli serve, andcoli web;--no-dsparkas a compatibility no-op.Production is consolidated in
c/deepseek_v4.c. The standalonec/v4launcher, DSpark runtime/build units, DSpark fixtures/tests, and committed.safetensorsfixtures have been removed.Shared infrastructure status
st.hquant.hTODO(upstream-fmt7-rows16): migrate when shared quant exposes a resident rows16 APITODO(upstream-fmt8-ue8m0): replace when shared fmt8 UE8M0 decode existsThe last two paths remain only to keep the target engine usable and are explicitly marked for migration.
Unified serving
openai_server.pydetectsdeepseek_v4, renders native multi-turn V4 markers, and launches the persistentSUBMIT/DATA/DONEprotocol. Serving is target-only, greedy, one KV slot, and rejects tools/grammar. Requests re-prefill context while the engine, dense tensors, head, and expert cache stay warm.Fixture and CI
The target-only fixture is generated from pinned PyTorch
2.13.0+cpu, Transformers5.14.1, and safetensors0.8.0. Its reference comes from officialDeepseekV4ForCausalLM; there is no C-engine oracle fallback. Generated safetensors are ignored and not committed.The V4 CI job checks teacher-forcing and greedy token identity, compressed/long prompts beyond the 64-token prefill boundary, repeated engine/session lifetime,
--no-dsparkcompatibility, and two requests through one persistent server process.Validation
make -C c checkon Windows UCRT64: all C tests and 283 Python tests passed (21 platform skips)make -C c deepseek-v4 ARCH=x86-64-v3st.hpread/mirror and fmt8 loader tests passedThe capital of France is Paris.target_only=1The real-checkpoint output was also reproduced through
c/coli run.Commit structure
feat(st): support V4 checkpoint metadatarefactor(v4): preserve target-only runtimefeat(cli): route DeepSeek V4 through coli servingtest(v4): generate target-only oracle in CIdocs(v4): describe target-only engine splitFollow-up
The stacked DSpark PR will restore the saved speculative runtime on this target baseline, make
--no-dsparka real switch again, restore deterministic DSpark identity tests, and provide on/off performance and acceptance data across high-, medium-, and low-acceptance prompts.