perf(ds4): reuse F16 KV in fused verification

[pepuscz]

Summary

Add an opt-in --ds4-fused-verify-f16-kv path for single-device HIP DeepSeek V4 verification.

The existing explicit batched verifier converts the complete persistent F16 MLA cache to F32 on every speculative step. This path feeds the same raw and compressed rows directly from the F16 cache into the established explicit attention topology. It keeps F32 key-side accumulation through 512 attention rows to protect the short-context baseline, then avoids the full-cache conversion at longer contexts.

The attended row set, masks, and explicit attention operation order are unchanged. The option remains off by default and is rejected by the feature gate outside a monolithic HIP DeepSeek V4 backend. It can still change generated tokens because the verifier inputs are F16.

Published qualification patches:

• lucebox-ds4-fused-explicit-f16-kv.patch
• lucebox-ds4-explicit-f16-f32-attention.patch

Matched benchmark

Ryzen AI Max+ 395 / Radeon 8060S, ROCm 7.1.1, 100/100/100 W package limits, fixed-high GPU policy, ROCmFPX target, Q4RMFP4 DSpark draft, all six experts, q=4 fused verification, Q4_0 K/V. Each comparison used the same prompt and power policy.

Input	Path	Prefill	Decode	Retrieval	Acceptance
4,096	F32-cache conversion control	137.79 tok/s	26.7 tok/s	5/5	1.00
4,096	F16 cache reuse	138.71 tok/s	27.5 tok/s	5/5	1.00
59,933	F32-cache conversion control	123.24 tok/s	16.4 tok/s	5/5	1.00
59,933	F16 cache reuse	123.20 tok/s	22.5 tok/s	5/5	1.00

The long-context decode gain is 37.2%; prefill is unchanged within run variance. A production composition including this path and the same 512-row F32 cutoff later passed 122,879-token 5/5 retrieval, 1.00 acceptance, and the 30/30 quality gate. The deployment-specific adaptive-prefill and graph-cache limits from that composition are intentionally not part of this PR.

Reproduction setup and benchmark definitions: https://github.com/pepuscz/strix-halo-deepseek-v4-flash

Validation

• Added placement feature-gate coverage; the pure feature-gate binary passes locally
• git diff --check
• The qualified ROCm 7.1.1 source composition built successfully and passed 353/354 server tests; the sole failure was the pre-existing MoeHybridStorageFixture::fractional_route_quota_rounds_over_the_batch expectation

[cubic-dev-ai]

3 issues found across 9 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/deepseek4/deepseek4_backend.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_backend.cpp:761">
P3: When `moe_hybrid_` is set, `--ds4-fused-verify-f16-kv` is silently dropped without any message. The sibling `fused_decode` flag right above logs a clear warning when hybrid expert placement disables it. A single-device HIP DS4 with hybrid expert residency (feature gate allows it) will silently ignore the opt-in flag, so the user gets none of the promised perf with no indication why. Add a matching diagnostic for the f16-KV path.</violation>
</file>

<file name="server/src/deepseek4/deepseek4_internal.h">

<violation number="1" location="server/src/deepseek4/deepseek4_internal.h:317">
P1: When `--ds4-fused-verify-f16-kv` is used without `--ds4-fused-decode`, HIP loading selects hybrid mode and clears this flag, so the opt-in silently has no effect. Include `cfg_.fused_verify_f16_kv` in the monolithic-load predicate so this path is actually enabled.</violation>
</file>

<file name="server/src/deepseek4/deepseek4_graph.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_graph.cpp:2289">
P2: The 512-row 'protect the short-context baseline' F32 path only sets F32 precision on the key-side score GEMM. The fused path keeps `kv_attn` F16, so the value-side projection `context = ggml_mul_mat(ctx, kv_t, probs)` still accumulates in default F16 precision even inside the protected window, meaning short-context attention outputs still diverge from the un-fused F32 topology that the cutoff is meant to preserve. If short-context fidelity is the goal, the context GEMM needs F32 precision (or F32 value rows) as well.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: When --ds4-fused-verify-f16-kv is used without --ds4-fused-decode, HIP loading selects hybrid mode and clears this flag, so the opt-in silently has no effect. Include cfg_.fused_verify_f16_kv in the monolithic-load predicate so this path is actually enabled.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_internal.h, line 317:

<comment>When `--ds4-fused-verify-f16-kv` is used without `--ds4-fused-decode`, HIP loading selects hybrid mode and clears this flag, so the opt-in silently has no effect. Include `cfg_.fused_verify_f16_kv` in the monolithic-load predicate so this path is actually enabled.</comment>

<file context>
@@ -313,6 +314,7 @@ struct DeepSeek4BackendConfig {
     int          max_ctx      = 0;     // 0 = auto from SWA + compression capacity
     int          expert_top_k = 0;     // 0 = use all model-routed experts
     bool         fused_decode = false; // single-graph GPU decode
+    bool         fused_verify_f16_kv = false; // F16 KV in batched verifier attention
 };
 
</file context>

[cubic-dev-ai]

P2: The 512-row 'protect the short-context baseline' F32 path only sets F32 precision on the key-side score GEMM. The fused path keeps kv_attn F16, so the value-side projection context = ggml_mul_mat(ctx, kv_t, probs) still accumulates in default F16 precision even inside the protected window, meaning short-context attention outputs still diverge from the un-fused F32 topology that the cutoff is meant to preserve. If short-context fidelity is the goal, the context GEMM needs F32 precision (or F32 value rows) as well.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_graph.cpp, line 2289:

<comment>The 512-row 'protect the short-context baseline' F32 path only sets F32 precision on the key-side score GEMM. The fused path keeps `kv_attn` F16, so the value-side projection `context = ggml_mul_mat(ctx, kv_t, probs)` still accumulates in default F16 precision even inside the protected window, meaning short-context attention outputs still diverge from the un-fused F32 topology that the cutoff is meant to preserve. If short-context fidelity is the goal, the context GEMM needs F32 precision (or F32 value rows) as well.</comment>

<file context>
@@ -2235,6 +2280,22 @@ static ggml_tensor * build_mla_attention(
+            // Keep Q and accumulation in F32 while retaining the persistent
+            // cache in F16. This isolates the key-side accuracy cost of the
+            // default HIP F16 GEMM path without changing cache topology.
+            ggml_mul_mat_set_prec(scores, GGML_PREC_F32);
+            static std::atomic<bool> f32_k_short_logged{false};
+            if (!f32_k_short_logged.exchange(true)) {
</file context>

[cubic-dev-ai]

P3: When moe_hybrid_ is set, --ds4-fused-verify-f16-kv is silently dropped without any message. The sibling fused_decode flag right above logs a clear warning when hybrid expert placement disables it. A single-device HIP DS4 with hybrid expert residency (feature gate allows it) will silently ignore the opt-in flag, so the user gets none of the promised perf with no indication why. Add a matching diagnostic for the f16-KV path.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_backend.cpp, line 761:

<comment>When `moe_hybrid_` is set, `--ds4-fused-verify-f16-kv` is silently dropped without any message. The sibling `fused_decode` flag right above logs a clear warning when hybrid expert placement disables it. A single-device HIP DS4 with hybrid expert residency (feature gate allows it) will silently ignore the opt-in flag, so the user gets none of the promised perf with no indication why. Add a matching diagnostic for the f16-KV path.</comment>

<file context>
@@ -758,6 +758,7 @@ bool DeepSeek4Backend::load_model() {
     }
     w_.routed_expert_top_k = cfg_.expert_top_k;
     w_.fused_decode = cfg_.fused_decode && !moe_hybrid_;
+    w_.fused_verify_f16_kv = cfg_.fused_verify_f16_kv && !moe_hybrid_;
     if (cfg_.fused_decode && moe_hybrid_) {
         std::fprintf(stderr,
</file context>

Suggested change

	w_.fused_verify_f16_kv = cfg_.fused_verify_f16_kv && !moe_hybrid_;
	w_.fused_verify_f16_kv = cfg_.fused_verify_f16_kv && !moe_hybrid_;
	if (cfg_.fused_verify_f16_kv && moe_hybrid_) {
	std::fprintf(stderr,
	"[deepseek4] fused verify F16 KV unavailable with hybrid expert placement; "
	"using standard verifier\n");
	}