refactor(server): shared layer-split backend + GGUF inspection + c2-gate plumbing

[easel]

Summary

Cross-backend C++ refactor that extracts a shared layer_split_backend, a GGUF inspection helper (gguf_inspect), the c2_spec_decode_permitted predicate (c2_gate.h), dim-aware draft GGUF loading, and the PFLASH_COMPRESS_* rename. Touches all three backends (qwen3, qwen35, gemma4), CMake, and the integration test surface. Also removes seven legacy server/scripts/bench_*.py scripts that have been superseded upstream.

Files

• server/src/common/: new gguf_inspect.{h,cpp}, updates to layer_split_backend.{h,cpp} and model_backend.h
• server/src/qwen35/: new c2_gate.h; refactors across layer_split_forward.{h,cpp}, qwen35_layer_split_adapter.{h,cpp}, gguf_target_loader.cpp, qwen35_dflash_target.h
• server/src/qwen3/: backend, graph, and loader updates (qwen3_backend.cpp, qwen3_graph.cpp, qwen3_loader.cpp)
• server/src/gemma4/: graph + layer-split adapter updates
• server/src/draft/: dim-aware draft GGUF loader
• server/CMakeLists.txt, server/README.md
• server/test/test_server_unit.cpp, server/scripts/test_server_integration.py
• server/scripts/fixtures/agent_cases/cases.json (new)
• Deletes: server/scripts/bench_agent.py, bench_agent_loop.py, bench_daemon.py, bench_he.py, bench_he_http.py, bench_llm.py, bench_server.py

Net: 30 files changed, ~1010 insertions / ~2580 deletions.

Dependencies

None — this PR is independent and forms the foundation for sibling server PRs.

Reverse dependencies (PRs that depend on this one):

• feat(lucebox): hub CLI + autotune/sweep/profile + harness adapters + shell wrapper #335 (lucebox-cli) — consumes server build artifacts
• feat(server): soft-close thinking termination (qwen35 + gemma4) #339 (server-soft-close) — depends on c2_gate.h introduced here
• feat(server): card-driven thinking control + reasoning_content channel + /props schema-4 #341 (server-thinking-control) — depends on c2_gate.h (if still referenced after Phase 1 cleanup)

Test plan

• server/test/test_server_unit.cpp passes on rolled-up branch
• server/scripts/test_server_integration.py passes on rolled-up branch
• qwen3, qwen35, and gemma4 backends all build and produce identical decode output vs. main
• Draft GGUF loader handles dim-mismatched draft models correctly
• PFLASH_COMPRESS_* env-var rename is consistent across server + bench callers

Note: this split PR is not independently buildable. server/CMakeLists.txt also references sources (e.g. server/src/qwen3/anchor_scan.cpp) that land in sibling split PRs, so standalone CMake configuration will fail on missing sources. Validation has to happen on the rolled-up branch (easel/feat/lucebox-docker), which is green.

Part of the PR #285 split

This PR is one of several tightly-scoped PRs carved out of the omnibus PR #285. See that PR for the full context and the other split-out PRs.

Generated with Claude Code (https://claude.com/claude-code)

[cubic-dev-ai]

10 issues found across 31 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/qwen35/c2_gate.h">

<violation number="1" location="server/src/qwen35/c2_gate.h:28">
P2: The gate threshold uses `int` multiplication (`2 * fa_window_cfg`), which can overflow and miscompute spec-decode eligibility for large configs.</violation>
</file>

<file name="server/src/gemma4/gemma4_layer_split_adapter.cpp">

<violation number="1" location="server/src/gemma4/gemma4_layer_split_adapter.cpp:149">
P2: Sampler penalty settings are silently ignored in Gemma4 layer-split after sampler state was removed, causing accepted requests to run as plain greedy decode.</violation>
</file>

<file name="server/src/qwen3/qwen3_loader.cpp">

<violation number="1" location="server/src/qwen3/qwen3_loader.cpp:136">
P2: Weight type detection for Q8_0 is logged but never plumbed into tensor loading. The comment claims Q8_0 support, but `copy_tensor_from_file` (same file) only handles same-type, BF16→F16, BF16→F32, and F16→F32 — no Q8_0 conversion path exists. Loading a Q8_0 GGUF would print "detected weight type: Q8_0" then fail every tensor copy with "unsupported tensor conversion". Additionally, the ternary `wtype == GGML_TYPE_Q8_0 ? "Q8_0" : "BF16"` silently mislabels non-Q8_0 types (F16, Q4_0, etc.) as BF16; `ggml_type_name(wtype)` is already used for error logging in the same function and should be used here instead.</violation>
</file>

<file name="server/src/common/gguf_inspect.cpp">

<violation number="1" location="server/src/common/gguf_inspect.cpp:178">
P2: The hashing loop does not distinguish EOF from read errors, so it can return/cache a SHA-256 for truncated data.</violation>
</file>

<file name="server/scripts/test_server_integration.py">

<violation number="1" location="server/scripts/test_server_integration.py:1108">
P2: Natural-close test encodes an incorrect token-accounting invariant (`content_tokens == 0`), conflicting with the documented `finish_details` contract.</violation>
</file>

<file name="server/src/qwen35/qwen35_dflash_target.h">

<violation number="1" location="server/src/qwen35/qwen35_dflash_target.h:59">
P2: Sticky override state with no reset — `set_fa_window` permanently mutates `fa_window_` but the constructor's original value is lost after any call, so subsequent verify cycles silently use a stale override when the caller omits a set call before a specific verify step.</violation>
</file>

<file name="server/src/qwen35/qwen35_layer_split_adapter.cpp">

<violation number="1" location="server/src/qwen35/qwen35_layer_split_adapter.cpp:403">
P1: AR decode no longer applies sampler logit processing, so repetition/frequency/presence penalties are ignored.</violation>

<violation number="2" location="server/src/qwen35/qwen35_layer_split_adapter.cpp:418">
P2: DFlash gating is too permissive: it allows spec decode when non-temperature logit penalties are active.</violation>
</file>

<file name="server/src/draft/draft_gguf_loader.cpp">

<violation number="1" location="server/src/draft/draft_gguf_loader.cpp:371">
P2: New post-allocation validation returns can leak draft GPU/context resources on load failure.</violation>
</file>

<file name="server/src/common/layer_split_backend.cpp">

<violation number="1" location="server/src/common/layer_split_backend.cpp:60">
P1: The condition `req.sampler.temp > 0.0f` is an incomplete proxy for logit-processing needs, allowing penalty-only requests to silently bypass the sampling-unsupported guard.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: AR decode no longer applies sampler logit processing, so repetition/frequency/presence penalties are ignored.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen35/qwen35_layer_split_adapter.cpp, line 403:

<comment>AR decode no longer applies sampler logit processing, so repetition/frequency/presence penalties are ignored.</comment>

<file context>
@@ -419,24 +397,16 @@ bool Qwen35LayerSplitAdapter::decode_ar(
-                cfg_.run_dflash ? &feature_ring_ : nullptr,
-                /*argmax_out=*/nullptr,
-                sampler_.needs_logit_processing() ? &logits_buf : nullptr)) {
+                cfg_.run_dflash ? &feature_ring_ : nullptr)) {
             return false;
         }
</file context>

[cubic-dev-ai]

P1: The condition req.sampler.temp > 0.0f is an incomplete proxy for logit-processing needs, allowing penalty-only requests to silently bypass the sampling-unsupported guard.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/layer_split_backend.cpp, line 60:

<comment>The condition `req.sampler.temp > 0.0f` is an incomplete proxy for logit-processing needs, allowing penalty-only requests to silently bypass the sampling-unsupported guard.</comment>

<file context>
@@ -57,8 +57,7 @@ GenerateResult LayerSplitBackend::run_from_state(const GenerateRequest & req,
     }
-    if (req.do_sample && req.sampler.needs_logit_processing() &&
-        !adapter_->supports_cpu_sampling()) {
+    if (req.do_sample && req.sampler.temp > 0.0f) {
         result.error = "sampling_unsupported";
         return result;
</file context>

Suggested change

	if (req.do_sample && req.sampler.temp > 0.0f) {
	if (req.do_sample) {

[cubic-dev-ai]

P2: The gate threshold uses int multiplication (2 * fa_window_cfg), which can overflow and miscompute spec-decode eligibility for large configs.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen35/c2_gate.h, line 28:

<comment>The gate threshold uses `int` multiplication (`2 * fa_window_cfg`), which can overflow and miscompute spec-decode eligibility for large configs.</comment>

<file context>
@@ -0,0 +1,31 @@
+                                     int kv_committed) {
+    (void)kv_committed;
+    return (fa_window_override == 0)
+        || (fa_window_override <= 2 * fa_window_cfg);
+}
+
</file context>

[cubic-dev-ai]

P2: Sampler penalty settings are silently ignored in Gemma4 layer-split after sampler state was removed, causing accepted requests to run as plain greedy decode.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/gemma4/gemma4_layer_split_adapter.cpp, line 149:

<comment>Sampler penalty settings are silently ignored in Gemma4 layer-split after sampler state was removed, causing accepted requests to run as plain greedy decode.</comment>

<file context>
@@ -146,25 +146,20 @@ bool Gemma4LayerSplitAdapter::init() {
-    if (req.do_sample && sampler_.seed != 0) {
-        sampler_rng_.seed(sampler_.seed);
-    }
+    (void)req;
 }
 
</file context>

[cubic-dev-ai]

P2: Weight type detection for Q8_0 is logged but never plumbed into tensor loading. The comment claims Q8_0 support, but copy_tensor_from_file (same file) only handles same-type, BF16→F16, BF16→F32, and F16→F32 — no Q8_0 conversion path exists. Loading a Q8_0 GGUF would print "detected weight type: Q8_0" then fail every tensor copy with "unsupported tensor conversion". Additionally, the ternary wtype == GGML_TYPE_Q8_0 ? "Q8_0" : "BF16" silently mislabels non-Q8_0 types (F16, Q4_0, etc.) as BF16; ggml_type_name(wtype) is already used for error logging in the same function and should be used here instead.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen3/qwen3_loader.cpp, line 136:

<comment>Weight type detection for Q8_0 is logged but never plumbed into tensor loading. The comment claims Q8_0 support, but `copy_tensor_from_file` (same file) only handles same-type, BF16→F16, BF16→F32, and F16→F32 — no Q8_0 conversion path exists. Loading a Q8_0 GGUF would print "detected weight type: Q8_0" then fail every tensor copy with "unsupported tensor conversion". Additionally, the ternary `wtype == GGML_TYPE_Q8_0 ? "Q8_0" : "BF16"` silently mislabels non-Q8_0 types (F16, Q4_0, etc.) as BF16; `ggml_type_name(wtype)` is already used for error logging in the same function and should be used here instead.</comment>

<file context>
@@ -133,6 +133,18 @@ bool load_qwen3_drafter_model(const std::string & path,
     out.head_dim   = (int)get_u32(gctx, "qwen3.attention.key_length", 128);
     out.rope_theta = get_f32(gctx, "qwen3.rope.freq_base", 1000000.0f);
 
+    // Detect weight quant type from blk.0.attn_q.weight; support BF16 and Q8_0.
+    ggml_type wtype = GGML_TYPE_BF16;
+    {
</file context>

[cubic-dev-ai]

P2: The hashing loop does not distinguish EOF from read errors, so it can return/cache a SHA-256 for truncated data.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/gguf_inspect.cpp, line 178:

<comment>The hashing loop does not distinguish EOF from read errors, so it can return/cache a SHA-256 for truncated data.</comment>

<file context>
@@ -36,4 +41,292 @@ GgufModelInfo inspect_gguf_model_info(const char * path) {
+    // blow the C++ thread stack.
+    constexpr size_t BUF = 4 * 1024 * 1024;
+    std::vector<uint8_t> buf(BUF);
+    while (f) {
+        f.read(reinterpret_cast<char*>(buf.data()), BUF);
+        std::streamsize got = f.gcount();
</file context>

Suggested change

	while (f) {
	f.read(reinterpret_cast<char*>(buf.data()), BUF);
	std::streamsize got = f.gcount();
	if (got > 0) sha256_update(c, buf.data(), size_t(got));
	}
	return sha256_final(c);
	while (true) {
	f.read(reinterpret_cast<char*>(buf.data()), BUF);
	const std::streamsize got = f.gcount();
	if (got > 0) sha256_update(c, buf.data(), size_t(got));
	if (f.eof()) break;
	if (!f) return {};
	}
	return sha256_final(c);

[cubic-dev-ai]

P2: Natural-close test encodes an incorrect token-accounting invariant (content_tokens == 0), conflicting with the documented finish_details contract.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/scripts/test_server_integration.py, line 1108:

<comment>Natural-close test encodes an incorrect token-accounting invariant (`content_tokens == 0`), conflicting with the documented `finish_details` contract.</comment>

<file context>
@@ -871,3 +897,243 @@ def test_stop_no_match(self):
+        # Phase-2 did not fire — content_tokens stays 0 when the model
+        # self-closes (all generated tokens are reasoning + content interleaved
+        # via the emitter on the phase-1 stream).
+        assert fd["content_tokens"] == 0
+        assert fd["thinking_tokens"] == fd["total_tokens"]
+
</file context>

Suggested change

	assert fd["content_tokens"] == 0
	assert fd["thinking_tokens"] == fd["total_tokens"]
	assert fd["content_tokens"] > 0
	assert fd["thinking_tokens"] + fd["content_tokens"] == fd["total_tokens"]

[cubic-dev-ai]

P2: Sticky override state with no reset — set_fa_window permanently mutates fa_window_ but the constructor's original value is lost after any call, so subsequent verify cycles silently use a stale override when the caller omits a set call before a specific verify step.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen35/qwen35_dflash_target.h, line 59:

<comment>Sticky override state with no reset — `set_fa_window` permanently mutates `fa_window_` but the constructor's original value is lost after any call, so subsequent verify cycles silently use a stale override when the caller omits a set call before a specific verify step.</comment>

<file context>
@@ -53,6 +53,11 @@ class Qwen35DFlashTarget : public DFlashTarget {
+    // Per-call override for the verify-time flash-attention window. Used by
+    // do_spec_decode to widen the window when pflash compression has shrunk
+    // the prompt — see GenerateRequest.fa_window_override.
+    void set_fa_window(int fa) { fa_window_ = fa; }
+
 private:
</file context>

[cubic-dev-ai]

P2: DFlash gating is too permissive: it allows spec decode when non-temperature logit penalties are active.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen35/qwen35_layer_split_adapter.cpp, line 418:

<comment>DFlash gating is too permissive: it allows spec decode when non-temperature logit penalties are active.</comment>

<file context>
@@ -445,7 +415,7 @@ bool Qwen35LayerSplitAdapter::decode_ar(
 
 bool Qwen35LayerSplitAdapter::can_dflash_decode() const {
-    return cfg_.run_dflash && cfg_.draft_path && !sampler_.needs_logit_processing();
+    return cfg_.run_dflash && cfg_.draft_path && sampler_.temp == 0.0f;
 }
 
</file context>

Suggested change

	return cfg_.run_dflash && cfg_.draft_path && sampler_.temp == 0.0f;
	return cfg_.run_dflash && cfg_.draft_path && !sampler_.needs_logit_processing();

[cubic-dev-ai]

P2: New post-allocation validation returns can leak draft GPU/context resources on load failure.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/draft/draft_gguf_loader.cpp, line 371:

<comment>New post-allocation validation returns can leak draft GPU/context resources on load failure.</comment>

<file context>
@@ -349,6 +349,63 @@ bool load_draft_gguf(const std::string & path,
+                (long long)derived_q_dim,
+                out.n_head, out.head_dim, (long long)expected_q_dim);
+            set_last_error(buf);
+            return false;
+        }
+        if (derived_kv_dim != expected_kv_dim) {
</file context>

[easel]

Closing this PR — provenance audit revealed two distinct problems that make a clean rebase impossible:

1. Byte-identical theft from dusterbloom's PR #274 (7 files):

• server/src/common/c2_gate.h
• server/src/inference/drafters/qwen3_drafter.{cpp,h}
• server/src/inference/drafters/qwen35_dflash_target.h
• server/src/inference/loaders/gguf_target_loader.cpp
• server/src/inference/loaders/draft_gguf_loader.cpp
• the c2-gate test cases in server/tests/unit/test_server_unit.cpp

These are dusterbloom's structural-defense / drafter work and should land via #274 (or a successor PR opened by their owner), not under this branch's authorship.

2. Silent reverts of weicj's already-merged work from #273 / #295 / #297 (6 files):

This branch was cut before those PRs merged into main, so the merge here would inadvertently roll back supports_cpu_sampling, prefill_last_logits, and related plumbing. Those reverts must not land.

What survives the audit (Erik's own work):

• server/src/common/gguf_inspect.cpp (~292 LOC additive)
• server/src/common/gguf_inspect.h (~40 LOC additive)

That's a SHA-256 mini-impl + GgufMetadata reader + sidecar caching + llama_ftype_name decode, layered on top of Howard Su's prior gguf_inspect from PR #305. It's being re-opened as a clean, scoped PR: feat/server-gguf-inspect (link in follow-up comment once pushed).

No code from this PR is being silently merged; everything legitimate is being re-introduced via its proper canonical PR.

[easel]

Follow-up: the clean, scoped replacement PR for the surviving gguf_inspect work is #344 — #344 (333 insertions, 0 deletions, on top of #305).