feat(speculation): add generic adaptive activation

harness/benchmarks/QWEN38_DSPARK_ADAPTIVE_SELECTION.md

@@ -0,0 +1,118 @@
+# Qwen3.8 adaptive-selection prompts (historical DSpark baseline)
+
+This six-request workload is a prompt-selection fixture for the concurrent
+adaptive speculation gate. It is deliberately balanced rather than
+representative:
+
+- two prompts where dense DSpark is a strong win;
+- two prompts where dense DSpark is only a marginal win;
+- two prompts where dense autoregressive decode wins.
+
+The machine-readable source is
+`prompts/qwen38_dspark_adaptive_selection.jsonl`. Every row records its
+selection class, expected dense oracle, and the R9700 screening measurement
+that justified the label.
+The exact completion lengths and ordered output hashes are retained locally in
+`QWEN38_DSPARK_ADAPTIVE_SELECTION_R9700.json`.
+
+The dense labels below were collected with DSpark and remain useful priors for
+request selection. They do not prescribe the current proposal mechanism.
+Concurrent runs use whichever single `Speculator` adapter the server registers;
+currently that is DFlash2. The activation engine owns scoring, ranking, cost
+comparison, sticky AR fallback, and telemetry.
+
+## Dense screening baseline
+
+Measured 2026-08-19 on the Radeon AI PRO R9700 (`gfx1201`, ROCm 7.2) with:
+
+- target: Qwen3.8-27B PR #625 requantization, IQ4_XS body, Q5_K output,
+  Q6_K `attn_v` and `ssm_out`;
+- drafter: RadixArk Qwen3.8-27B DSpark, no YaRN, Q8_0, block width 8;
+- build: Release, `gfx1201`, HIP graphs;
+- dense cache/attention: Q8_0 K/V, `--fa-window 2048`;
+- greedy chat requests, up to 256 output tokens;
+- fresh AR process without a drafter;
+- forced DSpark with `DFLASH_QWEN35_SPEC_STEP_RATIO=0`, so the dense
+  decoder could not hide weak speculation behind its own AR bursts.
+
+The throughput columns are model-side decode rates. This was a one-repeat
+selection screen, not a publication measurement.
+
+| Prompt | Class | Dense oracle | AR tok/s | Forced spec tok/s | Spec/AR | Accept | Commit/step |
+| :--- | :--- | :--- | ---: | ---: | ---: | ---: | ---: |
+| `he_09 sum_product` | strong win | speculation | 34.89 | 46.35 | 1.328 | 35.8% | 2.51 |
+| `he_10 rolling_max` | strong win | speculation | 34.89 | 50.30 | 1.442 | 38.9% | 2.72 |
+| `he_02 separate_paren_groups` | marginal win | speculation | 34.95 | 37.69 | 1.078 | 29.0% | 2.03 |
+| `he_03 truncate_number` | marginal win | speculation | 35.15 | 37.44 | 1.065 | 28.9% | 2.02 |
+| `he_08 filter_by_substring` | loss | AR | 35.04 | 33.95 | 0.969 | 26.2% | 1.84 |
+| `prose-01 reproducibility` | loss | AR | 34.90 | 27.10 | 0.777 | 20.8% | 1.45 |
+
+All six selected rows produced identical ordered content hashes under dense AR
+and forced dense speculation. Prompts with mismatched hashes were rejected
+from the performance fixture: HumanEval 01, 05, 06, and 07, plus prose 02,
+03, and 04. Keep those rejected prompts as correctness diagnostics; do not
+interpret their throughput as a valid speculative win or loss.
+
+## Concurrent activation benchmark
+
+The smallest adversarial cohort is fixed at C=3: the two dense strong-win
+prompts (`sum_product` and `rolling_max`) plus the strongest dense AR win
+(`reproducibility`). The wider boundary uses all six prompts at C=6.
+
+The removed DSpark-only matrix runner must not be used for new measurements.
+Launch paired fresh AR, forced-speculation, and adaptive server processes with
+the active DFlash2 drafter, and drive each process with
+`harness/benchmarks/concurrency/concurrent_benchmark.py`. The checked-in JSONL
+is already suitable as `--prompt-file`; select the C=3 IDs explicitly when
+running the smaller cohort. `run_qwen38_dflash2_subsets.sh` remains the core
+forced-mode/refill control.
+
+The dense oracle labels are priors, not hard assertions about the concurrent
+executor. Adaptive mode evaluates each request once at its first target-decode
+boundary. The active adapter returns proposal tokens, an activation score,
+expected yield, and optional conditional hazards. The generic engine ranks
+those scores against the profiled cost table and commits each request to AR or
+speculation through retirement. Evaluation failure emits
+`activation_evaluation_failed` and commits sticky AR. Accepted-token history
+and user identity are not scoring inputs.
+
+The prefill logits produce the first sampled output token before a target-decode
+step exists. A speculative request reuses its activation proposal immediately;
+a request that retires directly from prefill has no target-decode mode to
+activate. The useful behavior is still to preserve ordered greedy output hashes,
+keep known losses in AR, and let measured concurrent costs decide the marginal
+pair.
+
+## Profiling outputs
+
+`DFLASH_STEP_TIMING=1` is enabled by default. Every case retains:
+
+- `server.log`: startup, warmup, and measured request evidence;
+- `benchmark-server.log`: only the measured window;
+- `bench.json`: request and aggregate throughput;
+- `feature-proof.json`: request-correlated execution/correctness proof.
+
+The matrix root adds:
+
+- `profiling.json`: complete machine-readable gate, request, shape, and phase
+  distributions;
+- `profiling.md`: concise activation-regret and bottleneck tables;
+- `summary.md`: oracle-gated aggregate summary, written only when output
+  stability and adaptive criteria pass.
+
+Read the report in this order:
+
+1. Require identical outputs between AR, forced speculation, and adaptive.
+   A mismatch is a correctness failure, not a throughput result.
+2. Use paired concurrent AR/speculation measurements as the empirical oracle;
+   never use the dense prompt label as the final activation answer.
+3. Inspect `Activation outcome against matched pure AR`. It compares each
+   `(live, k, path)` shape to pure AR at the same live concurrency.
+4. Inspect `Initial prediction accuracy` for predicted-versus-realized
+   goodput and realized-versus-AR regret.
+5. Use phase attribution to choose the next optimization: unexpected draft
+   work on k=0, verify, or the structural replay forward.
+
+The profiler's `total_us` uses one common decode-round origin for pure AR,
+adaptive k=0, and speculative rounds. `draft_us` is a subset of that wall
+time; do not add it a second time.

harness/benchmarks/QWEN38_PR625_BASELINE.md

@@ -0,0 +1,83 @@
+# Qwen3.8-27B PR #625 baseline
+
+Establish this dense, single-request baseline before measuring PR #626's
+concurrent paged path. The two paths intentionally do not share cache or
+attention settings.
+
+## Models
+
+Sources:
+
+- target: `bartowski/Qwen3.8-27B-GGUF`, `Qwen3.8-27B-IQ4_XS.gguf`
+- drafter: `RadixArk/Qwen3.8-27B-DSpark`, `model.safetensors`
+
+Prepare the permanent local pair with:
+
+```bash
+TARGET_SOURCE=/path/Qwen3.8-27B-IQ4_XS.gguf \
+DRAFT_SOURCE=/path/RadixArk-Qwen3.8-27B-DSpark/model.safetensors \
+LLAMA_QUANTIZE=/path/llama-quantize \
+server/scripts/prepare_qwen38_pr625_models.sh
+```
+
+This produces and validates:
+
+- target: pure IQ4_XS body, Q5_K `output.weight`, Q6_K `attn_v` and
+  `ssm_out`
+- drafter: no YaRN, Q8_0, capture layers `4,16,28,40,52`, mask token
+  `248077`. PR #625 does not spell out the DSpark quantization command, but
+  Q8_0 reproduces its reported compute regime: this setup measured a 1.78x
+  speculative/plain step-time ratio on the R9700, versus the PR's ~1.8x.
+  The unquantized F16 drafter measured 4.20x and is therefore not the
+  benchmark artifact. Set `DRAFT_SCHEME=f16` or `DRAFT_SCHEME=q4-mix` only
+  for an explicit ablation.
+
+## Build on Radeon AI PRO R9700
+
+Use ROCm 7.2, `gfx1201`, Release, and HIP graphs:
+
+```bash
+cmake -S server -B server/build-pr625-r9700 -G Ninja \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DCMAKE_HIP_COMPILER=/opt/rocm/llvm/bin/clang++ \
+  -DDFLASH27B_GPU_BACKEND=hip \
+  -DDFLASH27B_HIP_ARCHITECTURES=gfx1201 \
+  -DDFLASH27B_HIP_SM80_EQUIV=ON \
+  -DGGML_HIP_GRAPHS=ON \
+  -DDFLASH27B_FA_ALL_QUANTS=ON \
+  -DDFLASH27B_SERVER=ON -DDFLASH27B_TESTS=OFF
+cmake --build server/build-pr625-r9700 -j
+```
+
+## Dense single-request launch
+
+Use `HIP_VISIBLE_DEVICES=0` when the R9700 is the first physical GPU. Inside
+the process it remains `hip:0`.
+
+```bash
+HIP_VISIBLE_DEVICES=0 \
+DFLASH_SINGLE_CHAIN_CHECKPOINT_F32=1 \
+DFLASH_FAST_ROLLBACK_THRESHOLD=1 \
+LUCE_Q8_MEMO=1 \
+DFLASH_KV_ROTATE=0 \
+server/build-pr625-r9700/dflash_server \
+  models/.lucebox/qwen38-pr625/Qwen3.8-27B-PR625-IQ4_XS.gguf \
+  --draft models/.lucebox/qwen38-pr625/Qwen3.8-27B-DSpark-RadixArk-no-yarn-q8_0.gguf \
+  --target-device hip:0 --draft-device hip:0 \
+  --fa-window 2048 --cache-type-k q8_0 --cache-type-v q8_0 \
+  --max-ctx 8192 --prefix-cache-slots 0 --prefill-cache-slots 0 \
+  --decode-mode speculation --host 127.0.0.1 --port 18140
+```
+
+Do not pass `--paged-attention` or `--max-concurrency` for this baseline. For
+the AR control, start a fresh process **without `--draft`** and use
+`--decode-mode ar`; the dense backend otherwise has a loaded drafter and can
+enter its original speculative loop. All target, cache, and attention settings
+stay identical. Use greedy 300-token generations with
+`harness/benchmarks/prompts/qwen38_pr625.jsonl`, and reject a measurement that
+ends before the 300-token cap.
+
+PR #625 reported R9700 decode throughput of 34.3/34.4 tok/s for AR and
+45.6/32.4 tok/s for DSpark on its code/prose prompts. Exact numeric parity
+requires the original unpublished prompts; the structural check is that code
+benefits while prose can remain below AR.

harness/benchmarks/concurrency/FEATURE_MATRIX.md

@@ -0,0 +1,147 @@
+# Qwen3.6 concurrent feature matrix
+
+The bounded Strix Halo measurements collected for the draft implementation are
+recorded in [`STRIX_HALO_RESULTS.md`](STRIX_HALO_RESULTS.md).
+
+## Qwen3.8 adaptive speculation
+
+The DSpark-only matrix runner was removed with the concurrent DSpark execution
+path. The checked-in Qwen3.8 selection prompts remain useful workload fixtures,
+but proposal generation and activation scoring now come from the one active
+`Speculator` adapter (currently DFlash2).
+
+Use `concurrent_benchmark.py` as the common request client for AR, forced
+speculation, and adaptive server processes. Use
+`run_qwen38_dflash2_subsets.sh` for the forced AR/speculation and refill
+controls. Adaptive runs must retain the fail-closed `[spec-activation]` proof:
+one scored or failed decision per measured request, opaque `score_kind`,
+optional hazards, and `activation_evaluation_failed` for sticky-AR fallback.
+
+## Qwen3.6 DDTree/PFlash/KVFlash matrix
+
+`run_qwen36_feature_matrix.sh` extends the PR #596 protocol with feature
+ablations for the complete Strix Halo configuration:
+
+- `ar`: concurrent paged autoregressive control.
+- `ddtree`: adds the decode draft, DDTree, and the recorded budget.
+- `pflash`: adds auto prefill compression, its drafter, and persistent draft
+  residency.
+- `kvflash`: adds bounded KV residency in auto mode and explicitly supplies
+  the hashed prefill drafter for relevance-scored page selection; prefill
+  compression remains off in this ablation.
+- `full`: enables DDTree, PFlash, and KVFlash together with both devices on
+  `hip:0`.
+- `llama`: optional external comparison; its binary is required only when this
+  variant is explicitly requested.
+
+Run the default bounded C4 screening repeat (seven applicable fresh-server cases):
+
+```bash
+MODEL=/opt/models/Qwen3.6-27B-Q4_K_M.gguf \
+DRAFT_MODEL=/opt/models/draft/dflash-draft-3.6-q8_0.gguf \
+PREFILL_DRAFTER=/opt/models/Qwen3-0.6B-BF16.gguf \
+REPEATS=1 \
+harness/benchmarks/concurrency/run_qwen36_feature_matrix.sh
+```
+
+For the canonical AMD Strix Halo recipe, use the published Q8_0 3.6 drafter and make the tuning explicit:
+
+```bash
+MODEL=/opt/models/Qwen3.6-27B-Q4_K_M.gguf \
+DRAFT_MODEL=/opt/models/draft/dflash-draft-3.6-q8_0.gguf \
+PREFILL_DRAFTER=/opt/models/Qwen3-0.6B-BF16.gguf \
+DRAFT_SWA=2048 PREFILL_UBATCH=512 DDTREE_ADAPTIVE=0 \
+VARIANTS=ddtree,pflash,kvflash,full CLIENTS=1,4,8,16 REPEATS=5 \
+harness/benchmarks/concurrency/run_qwen36_feature_matrix.sh
+```
+
+`DDTREE_ADAPTIVE=0` matches the blog's continuous DDTree probe policy; leave it at the default `1` when measuring the concurrent engine's adaptive fallback policy. The concurrent path now records a startup `[parallel-ddtree]` marker and per-request `ddtree_steps`; these are the proof that DDTree actually ran.
+
+On a 128 GiB Strix Halo host, budget roughly 45–90 minutes for this smoke run;
+the long-context AR controls dominate and actual time depends on the build.
+Every row remains independently selectable through `VARIANTS`. For example:
+
+```bash
+WORKLOADS=short CLIENTS=4 VARIANTS=ar,ddtree MAX_TOKENS=256 REPEATS=1 \
+harness/benchmarks/concurrency/run_qwen36_feature_matrix.sh
+```
+
+Use five fresh-process, paired repeats for published measurements:
+
+```bash
+WORKLOADS=short,compression CLIENTS=1,4,8,16 \
+VARIANTS=ar,ddtree,pflash,kvflash,full MAX_TOKENS=256 REPEATS=5 \
+harness/benchmarks/concurrency/run_qwen36_feature_matrix.sh
+```
+
+That full matrix can take roughly 15–30 hours on Strix Halo; keep the generated
+case directory so interrupted or suspect rows can be diagnosed rather than
+quoted.
+
+## Activation workloads
+
+Auto features cannot be validated with the original 400–4,000 word prompts.
+The extension adds two deterministic, disjoint 29-prompt cohorts:
+
+- `compression`: 34K–40K words, chosen after observing 38,130–44,856
+  tokens with the development Qwen GGUF tokenizer.
+- `kv-pressure`: 12K–18K words, which produced 13,463–20,190 tokens with
+  that tokenizer against the runner's explicitly recorded 8K pool cap.
+
+The observed counts are a fixture sanity check, not a claim derived from word
+count and not publication evidence. Each row records the model hash; the proof
+cross-checks logged raw PFlash input against `usage.prompt_tokens`, requires
+it to meet the recorded auto threshold, and uses server-reported effective
+tokens plus actual page traffic for KVFlash.
+
+The bounded default uses `short,compression` at C4. It runs PFlash and the
+full configuration only on `compression`; KVFlash can also be selected on
+`kv-pressure`. Inapplicable pairs are printed as skips and never appear as
+successful rows. AR controls use the same prompts, so feature deltas remain
+paired.
+
+## Fail-closed feature proof
+
+The server must write one JSON object per completed request with this prefix:
+
+```text
+[concurrency-metrics] {"request_id":"...", ...}
+```
+
+Required fields are `effective_prompt_tokens`, `ddtree_steps`,
+`ddtree_suspensions`, `ddtree_accepted_tokens`, `target_forwards`,
+`kvflash_page_ins`, `kvflash_page_outs`, `kvflash_resident_blocks`,
+`kvflash_reselects`,
+`pflash_applied`, `pflash_input_tokens`, and `pflash_output_tokens`.
+
+The proof tool correlates log objects with measured SSE request IDs and also
+checks the log's effective token count against
+`usage.timings.effective_prompt_tokens`. A requested feature fails the case
+unless:
+
+- DDTree has positive step and target-forward counts. Acceptance may be zero.
+  The required per-request suspension counter must be either zero or one. It
+  records adaptive fallback activation, but does not prove whether AR work ran
+  before or after the suspension; temporal claims require direct ordered-log
+  evidence.
+- PFlash reports `pflash_applied=true`, a smaller output prompt, and (in auto
+  mode) an input token count at or above the recorded activation threshold.
+- KVFlash always records an explicit hashed scorer drafter, reports its
+  startup-observed physical pool and enabled metadata, and has a positive
+  resident-block count. `kvflash`-only and `kv-pressure` rows must also show page-in or
+  page-out traffic. For a `full` row, traffic is required only when a
+  server-reported `effective_prompt_tokens` value exceeds that observed pool
+  token limit; zero traffic is valid when PFlash compression fits in the pool.
+
+After health succeeds, the runner fail-closed parses the backend's
+`[parallel-kvflash] physical resident pool ...` and `[paged-attention] ...`
+startup markers into `runtime_observed`. This distinguishes the actual resident
+pool from both the requested auto cap and `--kv-pool-tokens`, which concurrent
+KVFlash intentionally does not use to expand VRAM.
+
+Each case retains the exact shell-escaped command, controlled launch
+environment, literal client process arguments and client-script hash,
+binary/shared-library/target/draft/PFlash-and-KV-scorer hashes, the ordered
+`literal_screenshot_flags` array, all feature values, raw request report,
+server log, and `feature-proof.json`. The summary refuses to
+include a Lucebox row whose proof is missing or invalid.