deepseek_v4: size the OpenMP team around the expert loaders

[Blakeolson21]

Summary

The DeepSeek V4 engine never sizes its OpenMP team — unlike every other engine in the project (colibri, inkling, kimi_k3, olmoe size via omp_tune.h; GLM has its own block). The OpenMP runtime default is every logical CPU, which schedules compute threads onto the CPUs that the block pipeline's persistent expert-loader workers need. On a disk-bound decode those loaders are doing the rate-limiting work, so the collision costs measurable throughput: about 9–10% decode on the machine below. This is the same rationale omp_tune.h records for the spin-wait half of the GLM tuning: a busy team steals cores from the I/O pool.

The change: when the user has not chosen a team, set threads = logical CPUs − COLI_V4_EXPERT_LOADER_COUNT (the loader pool's worker count, default 3). OMP_NUM_THREADS and COLI_NO_OMP_TUNE=1 win, exactly like the other engines' tuning. On machines where the reservation would leave fewer than 2 compute threads, the OpenMP default is left alone. The loader-count default moves to deepseek_v4_internal.h so the CLI and the block pipeline agree on one number.

I considered reusing coli_omp_tune_threads() (physical-core sizing) instead. On this box it recovers the same decode (+9.9%), but it pays for it in time-to-first-token: prefill is compute-heavy, and 8 threads prefill ~9% slower than even the untuned default (78.6 s vs 72.1 s median). The loader reservation improves both metrics at once, and it degrades gracefully on bigger machines (reserving 3 of 32 logical CPUs is a smaller bite than halving the team).

Measurements

Hardware: AMD Ryzen 7 6800H (8 cores / 16 threads, Zen 3+, AVX2), 32 GB DDR5-4800, weights on a WD Green SN350 1 TB NVMe (Gen3 x4, ext4), Ubuntu 26.04 LTS, gcc 15.2.0.
Model: DeepSeek-V4-Flash-0731 (release FP4/FP8 weights), memory=auto (~27.4 GiB), CTX=4096, target_only=1.
Binary: release v1.5.0 (8f512fc), unmodified, built with make -C c deepseek-v4.

Method: same 45-token prompt for every run, 160 generated tokens, --no-dspark. Decode tok/s = generated ÷ after_first from the engine's own timing line. To make runs independent, every run restores an identical .coli_usage warm-pin history and sets COLI_V4_SAVE_USAGE=0 (expert-cache hit rate came out 65.29–65.33% on all 12 runs, so the arms saw the same cache behavior). Warm-up policy: one discarded 16-token run first. Run count: 3 per configuration, interleaved with alternating order to cancel drift. Medians reported.

OMP_NUM_THREADS	decode tok/s (median of 3)	vs default	TTFT s (median)
unset (OpenMP default = 16 logical)	0.778	—	72.1
14 (= logical − 2)	0.857	+10.2%	66.7
13 (= logical − 3, what this patch picks)	0.849	+9.1%	69.2
8 (= physical cores, what omp_tune.h picks)	0.855	+9.9%	78.6

Per-run decode values: unset 0.7778/0.7698/0.7790 · 14: 0.8571/0.8601/0.8519 · 13: 0.8486/0.8409/0.8517 · 8: 0.8590/0.8546/0.8329.

14 threads (reserve 2) measured marginally better than 13 on this box; the patch still ties the reservation to COLI_V4_EXPERT_LOADER_COUNT because that is the number the code can justify — the 13-vs-14 gap (~1%) is close to run-to-run spread, and a hardcoded "reserve 2" would silently drift if the loader pool ever changes size. If you'd rather chase the last percent I'm happy to change it.

Behavior of the patched build (base dev), verified on the real model:

• OMP_NUM_THREADS unset → a full 160-token generation completes and logs
  [OMP] deepseek-v4: 13 compute threads (16 logical CPUs minus 3 expert-loader workers); OMP_NUM_THREADS=<n> overrides, COLI_NO_OMP_TUNE=1 disables
• OMP_NUM_THREADS=14 → no [OMP] line, the user's value is used untouched
• COLI_NO_OMP_TUNE=1 → no [OMP] line, OpenMP default preserved

(The performance table above is measured on the unmodified v1.5.0 release binary with manual OMP_NUM_THREADS, which exercises the exact team sizes the patch selects; the patched build sits on dev, whose memory planning differs enough that cross-base decode numbers would not be apples-to-apples.)

This is one machine; I'd welcome numbers from other CPU/storage combinations, in particular boxes with more cores or faster storage.

Validation

• make check on the patched tree (Linux x86-64): 373 tests, OK (52 skipped)
• CUDA changes were tested with make -C c cuda-test (not applicable — CPU-only change)
• Performance claims include hardware, commands, and repeatable measurements (above; exact commands in the collapsed section)

Exact benchmark commands

# from c/, weights in $M, same prompt every run
P="Explain, step by step, how NVMe expert streaming lets a 284B-parameter \
mixture-of-experts model run on a machine with only 32 GB of RAM. Then list \
three practical bottlenecks."
M=/path/to/DeepSeek-V4-Flash-0731

# before every run: restore an identical warm-pin history, freeze it
cp -f coli_usage.pristine "$M/.coli_usage"

# per-arm run (drop OMP_NUM_THREADS for the default arm)
OMP_NUM_THREADS=14 COLI_V4_SAVE_USAGE=0 CTX=4096 \
  ./deepseek_v4 "$M" "$P" --max-tokens 160 --no-dspark

Compatibility

• The default CPU build remains dependency-free (omp.h include is guarded by _OPENMP, same as colibri.c)
• No model files, generated binaries, or benchmark artifacts are included