From 979001d0d35f1bb651af704bfd8c73a8c941d97c Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 11:52:31 -0700
Subject: [PATCH 01/70] fix: flag fallback byte widths in roofline graphs

---
 AUDIT.md                  | 104 ++++++++++++++++++++++++++++++++++++++
 gitm/planner/graph.py     |   9 ++++
 gitm/planner/moe_graph.py |  76 ++++++++++++++++++++++++----
 gitm/planner/roofline.py  |  12 +++++
 tests/test_moe_graph.py   |  34 +++++++++++++
 5 files changed, 224 insertions(+), 11 deletions(-)
 create mode 100644 AUDIT.md

diff --git a/AUDIT.md b/AUDIT.md
new file mode 100644
index 0000000..b1e16f8
--- /dev/null
+++ b/AUDIT.md
@@ -0,0 +1,104 @@
+# Graceful Fallback and Runtime Wiring Audit
+
+## Executive summary
+
+Status: **in progress**. This ledger is the primary deliverable for the audit of
+`gitm/` and `scripts/`. Findings are ranked by the likelihood that a fallback can
+turn missing knowledge into a confident wrong result, with answer-deciding byte
+traffic and dominant expert terms ranked above non-binding estimates.
+
+Highest-severity masks closed: **0 so far**. Wiring gaps confirmed: **3 so far**.
+Deferred findings: **none so far**.
+
+The worktree already contained uncommitted scheduler/serve changes and two new
+expert-signal files before this audit branch was created. They are preserved and
+treated as pre-existing work until their ownership and relevance can be separated;
+they will not be silently absorbed into an audit commit.
+
+## Finding ledger
+
+| Rank | Status | Severity | Location | Distorted term / contract | Surfacing state | Failure scenario | Disposition |
+|---:|---|---|---|---|---|---|---|
+| 1 | open | critical | `gitm/scheduler/loop.py:165-204, 576-586` | Entire sparse-MoE execution graph; especially dominant expert weight bytes and hybrid-attention/KV terms | silent | A live V4 engine is converted into the legacy `ModelSpec` and sent to `predict_graph`; the production loop never calls `spec_from_hf_config`/`predict_moe_graph`, so it can issue optimization claims against the wrong architecture. | Add one execution-graph dispatcher shared in intent with attach: recognize/normalize/validate the final config, build the sparse graph, and refuse graph-based claims with named diagnostics when it cannot be priced. |
+| 2 | open | critical | `gitm/scheduler/loop.py:165-204` | Model identity and every residual | silent | Any parse bug or version-drift attribute error is caught by `except Exception`, returns `None`, and becomes the plausible Llama-2-7B default graph. | Narrow expected absence handling; return a typed resolution result/diagnostic and degrade to measurement-only rather than a default model. |
+| 3 | open | high | `gitm/planner/context.py:156-169`; `gitm/scheduler/loop.py:576-590`; `gitm/serve/attach.py:477-487` | Compute and HBM denominators for every node | silent | Unknown/no GPU SKU makes `hardware_spec_for(None)` return an A100 spec; artifacts then write `hardware: A100-SXM4-80GB` as though detected on a different or absent GPU. | Preserve fail-open only with explicit hardware-fallback provenance and warnings; graph-based claims should not present the fallback SKU as observed hardware. |
+| 4 | open (planner flag fixed; boundary surfacing pending) | high | `gitm/planner/roofline.py:42-54`; sparse graph byte builders | Weight/KV/activation bytes; dominant decode-binding term | planner FLAG added; production consumers pending | Unknown dtype returns 2-byte bf16. Graph peak fallback may incidentally flag an unknown compute dtype, but scalar sizing calls and mixed-byte nodes cannot identify that their byte width was substituted. | Added `weight_bytes_is_fallback`, per-node `bytes_are_fallback`, and `Graph.has_fallback_bytes` with known/unknown/mixed regression tests. Still must share final-spec validation across attach/loop and surface flags in artifacts/reports before closing. |
+| 5 | open | high | `gitm/scheduler/loop.py:322-354`; `gitm/optimizer/report.py:20-29`; report template | Residual magnitude shipped on every Claim | silent clamp | A 10x/18x model error and a 2x error both render `+100%`, hiding that the model is broken and repeating one aggregate as if claim-specific. | Advisor-approved contract: preserve raw residual, derive capped display + saturation, render capped and raw values; label run-level versus target-op scope; reject/surface non-finite values. |
+| 6 | open | high | `gitm/optimizer/monitor.py:83-184` | Residual population / coverage | silent drop | Unclassified kernels and classified ops absent from the graph are skipped, so the loop can report clean residuals over a small, biased fraction without matched/total coverage. | Add total/classified/matched kernel and duration coverage to `Residuals`; serialize and print/report warnings when incomplete. Test unknown and good paths. |
+| 7 | open | high | `gitm/scheduler/loop.py:309-319` | Dense-path MoE expert weight bytes | silent | An unknown quantization method is ignored, leaving `weight_dtype_bytes` at the bf16 default; dominant expert traffic can be overstated while claims look fully priced. | Replace numeric-only extraction with named dtype/provenance and refuse or flag unknown methods; superseded by the sparse dispatcher where applicable. |
+| 8 | open | medium | `gitm/planner/moe_graph.py:498-528`; `gitm/serve/model_config.py:274-276` | Expert weights, often the dominant term | silent substitution | Missing `expert_dtype` inherits linear `weight_dtype`; on mixed-precision checkpoints this can misprice most resident and fetched bytes. Official V4 Flash configs checked so far explicitly declare the field (Flash=`fp4`, Base=`fp8`), but foreign/uniform MoEs may omit it. | Record whether expert dtype was explicit or inherited; require it for model families/quantization layouts where mixed precision is possible, otherwise surface the inheritance in provenance. |
+| 9 | open | medium | `gitm/planner/graph.py:123-133` | Zero-time byte-moving nodes | flag exists but name can lose coverage | `has_unpriced_collectives` scans all nodes, not only collectives; a future “cleanup” to match the name would silently remove the general zero-pricing net. | Rename general predicate (with compatibility alias if needed) or split general and collective-specific intent; trace all consumers. |
+| 10 | open | high | `gitm/scheduler/loop.py:587-592` | Prediction trust diagnostics | unconsumed | `predicted_graph.json` writes only node count, total time, and hardware; peak fallback, byte fallback, unpriced nodes, estimates, default batch/model, and provenance do not reach the loop artifact/report. | Serialize machine-readable graph diagnostics and propagate them into the human report and any claim gate. |
+
+Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`.
+
+## Seed-finding verification
+
+| Seed | Verification | Status |
+|---:|---|---|
+| 1 | Unknown weight dtype falls back to bf16 without a bytes-side flag. | Confirmed; incidental peak fallback is insufficient for scalar/mixed-byte paths. Advisor design recorded. |
+| 2 | Loop MoE dispatch lacks the attach path's dtype priceability gate. | Confirmed and broader: current main has no sparse loop dispatcher/caller at all. |
+| 3 | Missing `expert_dtype` inherits `weight_dtype`, potentially mispricing the dominant expert term. | Confirmed in parser. Official DeepSeek V4 Flash and Base configs explicitly declare differing expert dtypes; omission risk remains for other MoEs. |
+| 4 | Residual percentage clamps at ±100% and loses raw error magnitude. | Confirmed; advisor design recorded. |
+| 5 | Residual classification drops unmatched kernels without loop-path coverage. | Confirmed. |
+| 6 | Unknown SKU becomes A100 while recording A100 as if observed. | Confirmed. |
+| 7 | Broad model-config parse rescue becomes an unmarked default model. | Confirmed. |
+| 8 | `has_unpriced_collectives` scans all nodes despite its narrow name. | Confirmed. |
+
+## Sweep coverage
+
+| Area | Phase 1 fallback sweep | Phase 2 wiring sweep | Notes |
+|---|---|---|---|
+| top-level runtime / API / CLI / workloads | Pending | Pending | |
+| agents | Pending | Pending | |
+| bench | Pending | Pending | |
+| benchmarks | Pending | Pending | |
+| deploy | Pending | Pending | |
+| importers | Pending | Pending | |
+| kernels | Pending | Pending | |
+| optimizer | Pending | Pending | |
+| planner | Pending | Pending | |
+| routing | Pending | Pending | |
+| safety | Pending | Pending | |
+| scheduler | Pending | Pending | |
+| serve | Pending | Pending | |
+| telemetry | Pending | Pending | |
+| tracer | Pending | Pending | |
+| scripts | Pending | Pending | |
+
+## Diagnostic-consumer trace
+
+Every boolean flag, warning list, `estimated`, provenance field, and `has_*` or
+`*_fallback` property discovered during the sweep is listed here and traced to a
+human- or gate-visible consumer.
+
+| Producer | Diagnostic | Downstream consumer | User/gate boundary | Status |
+|---|---|---|---|---|
+| — | — | — | — | Inventory pending |
+
+## Sibling-path validation matrix
+
+| Capability | Path A | Path B | Guard parity | Status |
+|---|---|---|---|---|
+| MoE config pricing | attach sidecar validates some raw config dtypes | scheduler has no sparse dispatcher and silently uses legacy dense defaults | Asymmetric | Open finding #1/#2/#7 |
+| execution lifecycle | launch | attach | To inventory | Pending |
+| model family | dense | MoE | To inventory | Pending |
+| workloads | each dispatch branch | sibling branches | To inventory | Pending |
+
+## Artifact-consumer trace
+
+| Artifact writer | Artifact | Production reader / boundary | Status |
+|---|---|---|---|
+| — | — | — | Inventory pending |
+
+## Completeness pass
+
+This section must be empty before completion.
+
+- Subpackages not swept: top-level runtime/API/CLI/workloads, agents, bench,
+  benchmarks, deploy, importers, kernels, optimizer, planner, routing, safety,
+  scheduler, serve, telemetry, tracer, scripts.
+- Diagnostic flags/warnings not traced: inventory not yet complete.
+- Asymmetric validation gates: inventory not yet complete.
+- Fallbacks judged acceptable without confirming REFUSE/FLAG/WARN: inventory not
+  yet complete.
diff --git a/gitm/planner/graph.py b/gitm/planner/graph.py
index 147b96f..92e5e07 100644
--- a/gitm/planner/graph.py
+++ b/gitm/planner/graph.py
@@ -141,6 +141,15 @@ def has_fallback_peaks(self) -> bool:
         """
         return any(n.prediction.peak_is_fallback for n in self.nodes)
 
+    @property
+    def has_fallback_bytes(self) -> bool:
+        """True if any node substituted bf16 for an unknown byte-width dtype.
+
+        Kept separate from :attr:`has_fallback_peaks`: compute peak and byte
+        width are independent inputs, and decode is commonly memory-bound.
+        """
+        return any(n.prediction.bytes_are_fallback for n in self.nodes)
+
 
 def predict_graph(
     model: ModelSpec | None = None,
diff --git a/gitm/planner/moe_graph.py b/gitm/planner/moe_graph.py
index 0c98102..5ffb219 100644
--- a/gitm/planner/moe_graph.py
+++ b/gitm/planner/moe_graph.py
@@ -72,6 +72,7 @@
     distinct_experts,
     roofline,
     weight_bytes,
+    weight_bytes_is_fallback,
 )
 
 
@@ -259,12 +260,36 @@ def _emit_layer(
     ww = weight_bytes(spec.weight_dtype)
     ew = weight_bytes(spec.expert_dtype)
     kw = weight_bytes(spec.kv_dtype)
+    af = weight_bytes_is_fallback(spec.act_dtype)
+    wf = weight_bytes_is_fallback(spec.weight_dtype)
+    ef = weight_bytes_is_fallback(spec.expert_dtype)
+    kf = weight_bytes_is_fallback(spec.kv_dtype)
     wd, ed = spec.weight_dtype, spec.expert_dtype
 
-    def add(op: str, flops: float, byts: float, dtype: str, *, estimated: bool = False) -> None:
+    def add(
+        op: str,
+        flops: float,
+        byts: float,
+        dtype: str,
+        *,
+        byte_fallback: bool,
+        estimated: bool = False,
+    ) -> None:
         name = f"{prefix}{op}"
         g.nodes.append(
-            PredictedNode(name, layer, roofline(name, flops, byts, hw, dtype, estimated=estimated))
+            PredictedNode(
+                name,
+                layer,
+                roofline(
+                    name,
+                    flops,
+                    byts,
+                    hw,
+                    dtype,
+                    estimated=estimated,
+                    bytes_are_fallback=byte_fallback,
+                ),
+            )
         )
 
     tp = max(1, sh.tp)
@@ -276,14 +301,15 @@ def add(op: str, flops: float, byts: float, dtype: str, *, estimated: bool = Fal
     # head. Every rank pays for them in full, so TP's speedup on attention is
     # strictly less than ``tp``.
     f, b = _linear(positions, h, spec.q_lora_rank, aw, ww)
-    add("attn_q_a", f, b, wd)
+    add("attn_q_a", f, b, wd, byte_fallback=af or wf)
 
     f, b = _linear(positions, spec.q_lora_rank, spec.n_heads * spec.q_head_dim // tp, aw, ww)
-    add("attn_q_b", f, b, wd)
+    add("attn_q_b", f, b, wd, byte_fallback=af or wf)
 
     # KV down-projection, plus the cache write for the positions just computed.
     f, b = _linear(positions, h, spec.kv_latent_dim, aw, ww)
-    add("attn_kv_a", f, b + positions * spec.kv_latent_dim * kw, wd)
+    add("attn_kv_a", f, b + positions * spec.kv_latent_dim * kw, wd,
+        byte_fallback=af or wf or kf)
 
     # ── indexer: project the query, then scan the compressed candidate set ───
     # Sliding-window layers have no indexer — a fixed recent window needs no
@@ -293,7 +319,7 @@ def add(op: str, flops: float, byts: float, dtype: str, *, estimated: bool = Fal
     cand = index_candidates(spec, layer, kv_len)
     if cand > 0:
         f, b = _linear(positions, h, spec.index_n_heads * spec.index_head_dim // tp, aw, ww)
-        add("attn_index_proj", f, b, wd)
+        add("attn_index_proj", f, b, wd, byte_fallback=af or wf)
 
         add(
             "attn_index_score",
@@ -302,6 +328,7 @@ def add(op: str, flops: float, byts: float, dtype: str, *, estimated: bool = Fal
             # position, and replicated across TP ranks alongside the KV latent.
             sequences * cand * spec.index_head_dim * kw,
             wd,
+            byte_fallback=kf,
         )
 
     # ── attention core over the selected positions ──────────────────────────
@@ -322,6 +349,7 @@ def add(op: str, flops: float, byts: float, dtype: str, *, estimated: bool = Fal
         # waiting to happen if the graph divided here.
         sequences * t_eff * spec.kv_latent_dim * kw,
         wd,
+        byte_fallback=kf,
     )
 
     # Output projection, grouped and low-rank. ``o_groups`` partitions the head
@@ -333,13 +361,14 @@ def add(op: str, flops: float, byts: float, dtype: str, *, estimated: bool = Fal
     f_b, b_b = _linear(positions, spec.o_lora_rank, h, aw, ww)
     # Named to match the dense graph's output projection: it is the same op in
     # the same place, so residuals for it stay comparable across model families.
-    add("attn_out_proj", f_a + f_b, b_a + b_b, wd, estimated=True)
+    add("attn_out_proj", f_a + f_b, b_a + b_b, wd,
+        byte_fallback=af or wf, estimated=True)
 
     # ── mixture of experts ──────────────────────────────────────────────────
     # Routing is replicated: every rank scores every expert so it knows what to
     # keep and what to ship.
     f, b = _linear(positions, h, spec.n_routed_experts, aw, ww)
-    add("moe_router", f, b, wd)
+    add("moe_router", f, b, wd, byte_fallback=af or wf)
 
     inter = spec.moe_intermediate_size
     # gate + up + down == three h x inter matrices per expert.
@@ -353,6 +382,7 @@ def add(op: str, flops: float, byts: float, dtype: str, *, estimated: bool = Fal
             per_expert_weights * spec.n_shared_experts * ew / tp
             + aw * (positions * h * 2 + positions * inter * 2 * spec.n_shared_experts / tp),
             ed,
+            byte_fallback=ef or af,
         )
 
     # Shared with the dense MoE roofline — one owner for the union term. Note the
@@ -374,6 +404,7 @@ def add(op: str, flops: float, byts: float, dtype: str, *, estimated: bool = Fal
         per_expert_weights * distinct * ew * skew / es
         + aw * (positions * h * 2 + positions * inter * 2 * spec.num_experts_per_tok / es),
         ed,
+        byte_fallback=ef or af,
     )
 
     # ── low-rank state update on the tail layers ────────────────────────────
@@ -384,7 +415,8 @@ def add(op: str, flops: float, byts: float, dtype: str, *, estimated: bool = Fal
         # Coarse: a down-up low-rank pair. The published shape isn't public, so
         # this establishes an order of magnitude and flags itself as an estimate
         # rather than sitting silently in the total.
-        add("dspark", f_d + f_u, b_d + b_u, wd, estimated=True)
+        add("dspark", f_d + f_u, b_d + b_u, wd,
+            byte_fallback=af or wf, estimated=True)
 
     # ── cross-rank traffic ──────────────────────────────────────────────────
     # Priced against the interconnect, not HBM, by swapping the bandwidth term.
@@ -400,7 +432,15 @@ def add_link(op: str, byts: float) -> None:
             g.nodes.append(
                 PredictedNode(
                     name, layer,
-                    roofline(name, 0.0, byts, link, spec.act_dtype, estimated=True),
+                    roofline(
+                        name,
+                        0.0,
+                        byts,
+                        link,
+                        spec.act_dtype,
+                        estimated=True,
+                        bytes_are_fallback=af,
+                    ),
                 )
             )
 
@@ -476,7 +516,21 @@ def predict_moe_graph(
     ww = weight_bytes(spec.weight_dtype)
     f, b = _linear(positions, spec.hidden, spec.vocab // max(1, sh.tp), aw, ww)
     g.nodes.append(
-        PredictedNode("lm_head", None, roofline("lm_head", f, b, hw, spec.weight_dtype))
+        PredictedNode(
+            "lm_head",
+            None,
+            roofline(
+                "lm_head",
+                f,
+                b,
+                hw,
+                spec.weight_dtype,
+                bytes_are_fallback=(
+                    weight_bytes_is_fallback(spec.act_dtype)
+                    or weight_bytes_is_fallback(spec.weight_dtype)
+                ),
+            ),
+        )
     )
 
     return g
diff --git a/gitm/planner/roofline.py b/gitm/planner/roofline.py
index 3b1a1ea..0e71ee1 100644
--- a/gitm/planner/roofline.py
+++ b/gitm/planner/roofline.py
@@ -49,6 +49,11 @@ def weight_bytes(dtype: str) -> float:
     return _WEIGHT_BYTES.get(dtype.lower(), 2.0)
 
 
+def weight_bytes_is_fallback(dtype: str) -> bool:
+    """Whether :func:`weight_bytes` substitutes bf16 for an unknown dtype."""
+    return dtype.lower() not in _WEIGHT_BYTES
+
+
 @dataclass(frozen=True)
 class HardwareSpec:
     """Peak achievable rates for a target GPU.
@@ -498,6 +503,11 @@ class RooflinePrediction:
     # derivation from published shapes — carried through to the report so an
     # estimate is never read as a measurement.
     estimated: bool = False
+    # Set when any dtype contributing to ``bytes`` was unknown and therefore
+    # priced at :func:`weight_bytes`'s fail-open bf16 width. This cannot be
+    # inferred from ``dtype``: a node's compute dtype and its activation, weight,
+    # or KV-cache byte contributors may differ.
+    bytes_are_fallback: bool = False
 
     @property
     def peak_is_fallback(self) -> bool:
@@ -555,6 +565,7 @@ def roofline(
     dtype: str = "fp16",
     *,
     estimated: bool = False,
+    bytes_are_fallback: bool = False,
 ) -> RooflinePrediction:
     """Compute the roofline prediction for a single op."""
     peak_flops, peak_dtype = resolve_peak(hw, dtype)
@@ -573,4 +584,5 @@ def roofline(
         peak_dtype=peak_dtype,
         peak_flops_per_s=peak_flops,
         estimated=estimated,
+        bytes_are_fallback=bytes_are_fallback,
     )
diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py
index 7de5e0b..c31e056 100644
--- a/tests/test_moe_graph.py
+++ b/tests/test_moe_graph.py
@@ -15,6 +15,8 @@
 
 from __future__ import annotations
 
+from dataclasses import replace
+
 import pytest
 
 from gitm.optimizer.deviation import classify_op
@@ -34,6 +36,7 @@
     ShardingConfig,
     resolve_peak,
     weight_bytes,
+    weight_bytes_is_fallback,
 )
 
 # The shape of DeepSeek-V4-Flash-0731, trimmed to the keys the planner reads.
@@ -256,6 +259,37 @@ def test_fp4_weight_bytes_include_the_block_scales():
     assert weight_bytes("bf16") == 2.0
 
 
+def test_unknown_weight_dtype_is_fail_open_but_explicitly_flagged(spec, b200):
+    """The bf16 byte-width fallback must ride with the memory term it changes."""
+    unknown = replace(spec, expert_dtype="future_fp3")
+    g = predict_moe_graph(unknown, b200, BatchConfig(batch=1, kv_cache_len=1024))
+
+    assert weight_bytes("future_fp3") == weight_bytes("bf16")
+    assert weight_bytes_is_fallback("future_fp3")
+    assert g.has_fallback_bytes
+    assert {
+        n.op for n in g.nodes if n.prediction.bytes_are_fallback
+    } == {"moe_shared", "moe_routed"}
+
+
+def test_known_weight_dtypes_leave_bytes_fallback_clean(spec, b200):
+    g = predict_moe_graph(spec, b200, BatchConfig(batch=1, kv_cache_len=1024))
+
+    assert not weight_bytes_is_fallback("fp4")
+    assert not g.has_fallback_bytes
+    assert not any(n.prediction.bytes_are_fallback for n in g.nodes)
+
+
+def test_mixed_byte_node_flags_unknown_kv_dtype(spec, b200):
+    """A node is flagged when any byte contributor is unknown, not just compute dtype."""
+    unknown = replace(spec, kv_dtype="future_kv3")
+    g = predict_moe_graph(unknown, b200, BatchConfig(batch=1, kv_cache_len=1024))
+
+    flagged = {n.op for n in g.nodes if n.prediction.bytes_are_fallback}
+    assert {"attn_kv_a", "attn_score_value"} <= flagged
+    assert "moe_router" not in flagged
+
+
 # ── config parsing ──────────────────────────────────────────────────────────
 
 

From 8c6fc3827b28d8a17667fe6392c140ceab454763 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 12:02:50 -0700
Subject: [PATCH 02/70] fix: surface saturated residual magnitude

---
 AUDIT.md                              |  6 +--
 gitm/optimizer/report.py              | 17 ++++++++
 gitm/optimizer/templates/report.md.j2 |  2 +-
 gitm/scheduler/loop.py                | 12 ++----
 tests/test_report_snapshot.py         | 59 +++++++++++++++++++++++++++
 tests/test_vllm_knobs_and_restart.py  | 10 +++--
 6 files changed, 90 insertions(+), 16 deletions(-)

diff --git a/AUDIT.md b/AUDIT.md
index b1e16f8..d2bceb1 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -7,7 +7,7 @@ Status: **in progress**. This ledger is the primary deliverable for the audit of
 turn missing knowledge into a confident wrong result, with answer-deciding byte
 traffic and dominant expert terms ranked above non-binding estimates.
 
-Highest-severity masks closed: **0 so far**. Wiring gaps confirmed: **3 so far**.
+Highest-severity masks closed: **1 so far**. Wiring gaps confirmed: **3 so far**.
 Deferred findings: **none so far**.
 
 The worktree already contained uncommitted scheduler/serve changes and two new
@@ -23,7 +23,7 @@ they will not be silently absorbed into an audit commit.
 | 2 | open | critical | `gitm/scheduler/loop.py:165-204` | Model identity and every residual | silent | Any parse bug or version-drift attribute error is caught by `except Exception`, returns `None`, and becomes the plausible Llama-2-7B default graph. | Narrow expected absence handling; return a typed resolution result/diagnostic and degrade to measurement-only rather than a default model. |
 | 3 | open | high | `gitm/planner/context.py:156-169`; `gitm/scheduler/loop.py:576-590`; `gitm/serve/attach.py:477-487` | Compute and HBM denominators for every node | silent | Unknown/no GPU SKU makes `hardware_spec_for(None)` return an A100 spec; artifacts then write `hardware: A100-SXM4-80GB` as though detected on a different or absent GPU. | Preserve fail-open only with explicit hardware-fallback provenance and warnings; graph-based claims should not present the fallback SKU as observed hardware. |
 | 4 | open (planner flag fixed; boundary surfacing pending) | high | `gitm/planner/roofline.py:42-54`; sparse graph byte builders | Weight/KV/activation bytes; dominant decode-binding term | planner FLAG added; production consumers pending | Unknown dtype returns 2-byte bf16. Graph peak fallback may incidentally flag an unknown compute dtype, but scalar sizing calls and mixed-byte nodes cannot identify that their byte width was substituted. | Added `weight_bytes_is_fallback`, per-node `bytes_are_fallback`, and `Graph.has_fallback_bytes` with known/unknown/mixed regression tests. Still must share final-spec validation across attach/loop and surface flags in artifacts/reports before closing. |
-| 5 | open | high | `gitm/scheduler/loop.py:322-354`; `gitm/optimizer/report.py:20-29`; report template | Residual magnitude shipped on every Claim | silent clamp | A 10x/18x model error and a 2x error both render `+100%`, hiding that the model is broken and repeating one aggregate as if claim-specific. | Advisor-approved contract: preserve raw residual, derive capped display + saturation, render capped and raw values; label run-level versus target-op scope; reject/surface non-finite values. |
+| 5 | fixed | high | `gitm/scheduler/loop.py`; `gitm/optimizer/report.py`; report template | Residual magnitude shipped on every Claim | FLAG | A 10x/18x model error and a 2x error both rendered `+100%`, hiding that the model was broken and repeating one aggregate as if claim-specific. | Raw residuals now remain on `Claim`; display capping and saturation are derived, saturated rows print raw magnitude, scopes distinguish run/target-op, and non-finite residuals are refused. Report, scheduler sibling, and ruff checks pass. |
 | 6 | open | high | `gitm/optimizer/monitor.py:83-184` | Residual population / coverage | silent drop | Unclassified kernels and classified ops absent from the graph are skipped, so the loop can report clean residuals over a small, biased fraction without matched/total coverage. | Add total/classified/matched kernel and duration coverage to `Residuals`; serialize and print/report warnings when incomplete. Test unknown and good paths. |
 | 7 | open | high | `gitm/scheduler/loop.py:309-319` | Dense-path MoE expert weight bytes | silent | An unknown quantization method is ignored, leaving `weight_dtype_bytes` at the bf16 default; dominant expert traffic can be overstated while claims look fully priced. | Replace numeric-only extraction with named dtype/provenance and refuse or flag unknown methods; superseded by the sparse dispatcher where applicable. |
 | 8 | open | medium | `gitm/planner/moe_graph.py:498-528`; `gitm/serve/model_config.py:274-276` | Expert weights, often the dominant term | silent substitution | Missing `expert_dtype` inherits linear `weight_dtype`; on mixed-precision checkpoints this can misprice most resident and fetched bytes. Official V4 Flash configs checked so far explicitly declare the field (Flash=`fp4`, Base=`fp8`), but foreign/uniform MoEs may omit it. | Record whether expert dtype was explicit or inherited; require it for model families/quantization layouts where mixed precision is possible, otherwise surface the inheritance in provenance. |
@@ -39,7 +39,7 @@ Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`.
 | 1 | Unknown weight dtype falls back to bf16 without a bytes-side flag. | Confirmed; incidental peak fallback is insufficient for scalar/mixed-byte paths. Advisor design recorded. |
 | 2 | Loop MoE dispatch lacks the attach path's dtype priceability gate. | Confirmed and broader: current main has no sparse loop dispatcher/caller at all. |
 | 3 | Missing `expert_dtype` inherits `weight_dtype`, potentially mispricing the dominant expert term. | Confirmed in parser. Official DeepSeek V4 Flash and Base configs explicitly declare differing expert dtypes; omission risk remains for other MoEs. |
-| 4 | Residual percentage clamps at ±100% and loses raw error magnitude. | Confirmed; advisor design recorded. |
+| 4 | Residual percentage clamps at ±100% and loses raw error magnitude. | Fixed with raw + capped-display contract, saturation note, scope, and non-finite refusal. |
 | 5 | Residual classification drops unmatched kernels without loop-path coverage. | Confirmed. |
 | 6 | Unknown SKU becomes A100 while recording A100 as if observed. | Confirmed. |
 | 7 | Broad model-config parse rescue becomes an unmarked default model. | Confirmed. |
diff --git a/gitm/optimizer/report.py b/gitm/optimizer/report.py
index 9eae9b9..b80e8e8 100644
--- a/gitm/optimizer/report.py
+++ b/gitm/optimizer/report.py
@@ -7,6 +7,7 @@
 
 from __future__ import annotations
 
+import math
 import subprocess
 import time
 from dataclasses import dataclass, field
@@ -25,8 +26,24 @@ class Claim:
     intervention_name: str
     predicted_delta: float
     measured_delta: float | None
+    # Whether the residual is specific to this claim, aggregated over the run,
+    # or tied to an autoresearch target op. The report hides the default label.
+    residual_scope: str = "claim"
     rolled_back: bool = False
 
+    def __post_init__(self) -> None:
+        if not math.isfinite(self.residual_value):
+            raise ValueError("residual_value must be finite")
+
+    @property
+    def residual_display_value(self) -> float:
+        """Value used in the compact table cell; raw truth remains available."""
+        return max(-1.0, min(1.0, self.residual_value))
+
+    @property
+    def residual_saturated(self) -> bool:
+        return abs(self.residual_value) > 1.0
+
 
 @dataclass
 class Provenance:
diff --git a/gitm/optimizer/templates/report.md.j2 b/gitm/optimizer/templates/report.md.j2
index 8269e1f..499c07c 100644
--- a/gitm/optimizer/templates/report.md.j2
+++ b/gitm/optimizer/templates/report.md.j2
@@ -25,7 +25,7 @@ _No claims._
 | # | Claim | Residual | Causal evidence | Intervention | Predicted Δ | Measured Δ |
 |---|---|---|---|---|---|---|
 {% for c in claims -%}
-| {{ loop.index }} | {{ c.summary }} | `{{ c.residual_invariant }}`: {{ '{:+.1%}'.format(c.residual_value) }} | {{ c.causal_evidence }} | `{{ c.intervention_name }}` | {{ '{:+.1%}'.format(c.predicted_delta) }} | {% if c.measured_delta is none %}—{% else %}{{ '{:+.1%}'.format(c.measured_delta) }}{% endif %}{% if c.rolled_back %} (rolled back){% endif %} |
+| {{ loop.index }} | {{ c.summary }} | `{{ c.residual_invariant }}`{% if c.residual_scope != 'claim' %} ({{ c.residual_scope }}){% endif %}: {{ '{:+.1%}'.format(c.residual_display_value) }}{% if c.residual_saturated %} (display capped; raw {{ '{:+.1%}'.format(c.residual_value) }}){% endif %} | {{ c.causal_evidence }} | `{{ c.intervention_name }}` | {{ '{:+.1%}'.format(c.predicted_delta) }} | {% if c.measured_delta is none %}—{% else %}{{ '{:+.1%}'.format(c.measured_delta) }}{% endif %}{% if c.rolled_back %} (rolled back){% endif %} |
 {% endfor %}
 {%- endif %}
 
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 370debb..ee484ef 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -317,12 +317,6 @@ def _moe_fields_from_hf(hf: Any) -> dict[str, Any]:
     return out
 
 
-def _clamp_pct(value: float) -> float:
-    """Bound a residual ratio to +/-100% so a bad/misaligned prediction (or a
-    small-sample outlier) can't blow up a report row into an absurd 18x."""
-    return max(-1.0, min(1.0, value))
-
-
 def _agg_kt_residual(res: Any) -> float:
     """Run-level kernel-time residual for the report: duration-weighted
     ``sum(obs - pred) / sum(pred)`` when timings are available, else the
@@ -339,7 +333,7 @@ def _agg_kt_residual(res: Any) -> float:
         kts = sorted(float(kr.r_kt) for kr in rows)
         mid = len(kts) // 2
         value = kts[mid] if len(kts) % 2 else (kts[mid - 1] + kts[mid]) / 2.0
-    return _clamp_pct(value)
+    return value
 
 
 def _ar_target_residual(ar_run: AutoresearchRun, fallback: float = 0.0) -> float:
@@ -349,7 +343,7 @@ def _ar_target_residual(ar_run: AutoresearchRun, fallback: float = 0.0) -> float
     target, fall back to the run-level kernel-time residual so generated claims
     do not all display a misleading +0.0% gap.
     """
-    return _clamp_pct(ar_run.target.residual) if ar_run.target is not None else fallback
+    return ar_run.target.residual if ar_run.target is not None else fallback
 
 
 def run_loop(cfg: LoopConfig) -> dict[str, Any]:
@@ -764,6 +758,7 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
                 summary=c.spec.summary,
                 residual_invariant="kernel_time",
                 residual_value=kt_residual,
+                residual_scope="run",
                 causal_evidence=causal_evidence,
                 intervention_name=c.spec.name,
                 predicted_delta=c.predicted_delta,
@@ -825,6 +820,7 @@ def _unenactable(spec: Any) -> str | None:
                 summary=r.spec.summary,
                 residual_invariant="kernel_time",
                 residual_value=ar_residual,
+                residual_scope=("target_op" if ar_run.target is not None else "run"),
                 causal_evidence=evidence,
                 intervention_name=r.spec.name,
                 predicted_delta=r.predicted_delta,
diff --git a/tests/test_report_snapshot.py b/tests/test_report_snapshot.py
index 30a2579..8f5d89b 100644
--- a/tests/test_report_snapshot.py
+++ b/tests/test_report_snapshot.py
@@ -19,9 +19,12 @@
 
 from __future__ import annotations
 
+import math
 import os
 from pathlib import Path
 
+import pytest
+
 from gitm.optimizer.report import Claim, Provenance, write_report
 
 GOLDEN_PATH = Path(__file__).parent / "golden" / "report_basic.md"
@@ -109,3 +112,59 @@ def test_report_renders_byte_equal_to_golden(monkeypatch):
         "    UPDATE_GOLDENS=1 .venv/bin/pytest tests/test_report_snapshot.py\n"
         "and review the diff with `git diff tests/golden/` before committing."
     )
+
+
+@pytest.mark.parametrize(
+    ("raw", "display", "raw_text"),
+    [(17.8, "+100.0%", "+1780.0%"), (-3.0, "-100.0%", "-300.0%")],
+)
+def test_saturated_residual_renders_capped_display_and_raw_magnitude(raw, display, raw_text):
+    claim = Claim(
+        summary="model gap",
+        residual_invariant="kernel_time",
+        residual_value=raw,
+        residual_scope="run",
+        causal_evidence="trace",
+        intervention_name="candidate",
+        predicted_delta=0.01,
+        measured_delta=None,
+    )
+
+    rendered = write_report([claim], _fixed_provenance())
+
+    assert claim.residual_value == raw
+    assert claim.residual_saturated
+    assert display in rendered
+    assert f"raw {raw_text}" in rendered
+    assert "`kernel_time` (run)" in rendered
+
+
+@pytest.mark.parametrize("raw", [-1.0, 0.42, 1.0])
+def test_unsaturated_residual_has_no_raw_magnitude_note(raw):
+    claim = Claim(
+        summary="model gap",
+        residual_invariant="kernel_time",
+        residual_value=raw,
+        causal_evidence="trace",
+        intervention_name="candidate",
+        predicted_delta=0.01,
+        measured_delta=None,
+    )
+
+    assert claim.residual_display_value == raw
+    assert not claim.residual_saturated
+    assert "display capped" not in write_report([claim], _fixed_provenance())
+
+
+@pytest.mark.parametrize("raw", [math.nan, math.inf, -math.inf])
+def test_non_finite_residual_is_refused(raw):
+    with pytest.raises(ValueError, match="residual_value must be finite"):
+        Claim(
+            summary="model gap",
+            residual_invariant="kernel_time",
+            residual_value=raw,
+            causal_evidence="trace",
+            intervention_name="candidate",
+            predicted_delta=0.01,
+            measured_delta=None,
+        )
diff --git a/tests/test_vllm_knobs_and_restart.py b/tests/test_vllm_knobs_and_restart.py
index a4bbd44..a7951e7 100644
--- a/tests/test_vllm_knobs_and_restart.py
+++ b/tests/test_vllm_knobs_and_restart.py
@@ -464,7 +464,7 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
     # Scheduler summary surfaced in the run summary (synchronous first sample).
     assert out["summary"]["scheduler_stats"] is not None
 
-def test_report_kernel_time_residual_uses_weighted_total_and_clamps():
+def test_report_kernel_time_residual_preserves_raw_weighted_total():
     from gitm.optimizer.monitor import KernelResidual, Residuals
     from gitm.scheduler.loop import _agg_kt_residual
 
@@ -475,7 +475,9 @@ def test_report_kernel_time_residual_uses_weighted_total_and_clamps():
         ]
     )
 
-    assert _agg_kt_residual(res) == 1.0
+    assert _agg_kt_residual(res) == pytest.approx(
+        (211e-6 - 10.01e-6) / 10.01e-6
+    )
 
     sane = Residuals(
         per_kernel=[
@@ -494,7 +496,7 @@ def test_ar_target_residual_uses_the_search_target_not_a_hardcoded_zero():
     empty = AutoresearchRun(bottleneck_class="idle_stall", results=[], target=None)
     assert _ar_target_residual(empty) == 0.0
 
-    # A real target -> its residual surfaces, clamped like every other residual.
+    # A real target -> its raw residual surfaces; display capping belongs to Claim.
     modest = AutoresearchRun(
         bottleneck_class="idle_stall", results=[],
         target=ResidualTarget(op="attn_score_value", residual=0.42, n_kernels=8),
@@ -505,4 +507,4 @@ def test_ar_target_residual_uses_the_search_target_not_a_hardcoded_zero():
         bottleneck_class="idle_stall", results=[],
         target=ResidualTarget(op="attn_score_value", residual=17.8, n_kernels=8),
     )
-    assert _ar_target_residual(huge) == 1.0
+    assert _ar_target_residual(huge) == 17.8

From ee1c58cd579eddde3217bdbf03fc96b5889fb79a Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 12:08:21 -0700
Subject: [PATCH 03/70] fix: surface residual matching coverage

---
 AUDIT.md                              |  6 +--
 gitm/optimizer/monitor.py             | 60 ++++++++++++++++++++++++++-
 gitm/optimizer/report.py              |  2 +
 gitm/optimizer/templates/report.md.j2 |  8 ++++
 gitm/scheduler/loop.py                | 13 ++++++
 tests/test_report_snapshot.py         | 15 +++++++
 tests/test_run_loop_workload.py       | 34 +++++++++++++++
 tests/test_runtime_on_trace.py        | 43 +++++++++++++++++++
 8 files changed, 176 insertions(+), 5 deletions(-)

diff --git a/AUDIT.md b/AUDIT.md
index d2bceb1..0ff6ccb 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -7,7 +7,7 @@ Status: **in progress**. This ledger is the primary deliverable for the audit of
 turn missing knowledge into a confident wrong result, with answer-deciding byte
 traffic and dominant expert terms ranked above non-binding estimates.
 
-Highest-severity masks closed: **1 so far**. Wiring gaps confirmed: **3 so far**.
+Highest-severity masks closed: **2 so far**. Wiring gaps confirmed: **3 so far**.
 Deferred findings: **none so far**.
 
 The worktree already contained uncommitted scheduler/serve changes and two new
@@ -24,7 +24,7 @@ they will not be silently absorbed into an audit commit.
 | 3 | open | high | `gitm/planner/context.py:156-169`; `gitm/scheduler/loop.py:576-590`; `gitm/serve/attach.py:477-487` | Compute and HBM denominators for every node | silent | Unknown/no GPU SKU makes `hardware_spec_for(None)` return an A100 spec; artifacts then write `hardware: A100-SXM4-80GB` as though detected on a different or absent GPU. | Preserve fail-open only with explicit hardware-fallback provenance and warnings; graph-based claims should not present the fallback SKU as observed hardware. |
 | 4 | open (planner flag fixed; boundary surfacing pending) | high | `gitm/planner/roofline.py:42-54`; sparse graph byte builders | Weight/KV/activation bytes; dominant decode-binding term | planner FLAG added; production consumers pending | Unknown dtype returns 2-byte bf16. Graph peak fallback may incidentally flag an unknown compute dtype, but scalar sizing calls and mixed-byte nodes cannot identify that their byte width was substituted. | Added `weight_bytes_is_fallback`, per-node `bytes_are_fallback`, and `Graph.has_fallback_bytes` with known/unknown/mixed regression tests. Still must share final-spec validation across attach/loop and surface flags in artifacts/reports before closing. |
 | 5 | fixed | high | `gitm/scheduler/loop.py`; `gitm/optimizer/report.py`; report template | Residual magnitude shipped on every Claim | FLAG | A 10x/18x model error and a 2x error both rendered `+100%`, hiding that the model was broken and repeating one aggregate as if claim-specific. | Raw residuals now remain on `Claim`; display capping and saturation are derived, saturated rows print raw magnitude, scopes distinguish run/target-op, and non-finite residuals are refused. Report, scheduler sibling, and ruff checks pass. |
-| 6 | open | high | `gitm/optimizer/monitor.py:83-184` | Residual population / coverage | silent drop | Unclassified kernels and classified ops absent from the graph are skipped, so the loop can report clean residuals over a small, biased fraction without matched/total coverage. | Add total/classified/matched kernel and duration coverage to `Residuals`; serialize and print/report warnings when incomplete. Test unknown and good paths. |
+| 6 | fixed | high | `gitm/optimizer/monitor.py`; `gitm/scheduler/loop.py`; report template | Residual population / coverage | WARN | Unclassified kernels and classified ops absent from the graph were skipped, so the loop could report clean residuals over a small, biased fraction without matched/total coverage. | `Residuals` now records total/classified/matched launch and kernel-time coverage. Incomplete coverage emits warnings into `residuals.json`, the run summary, and a printed Runtime diagnostics section; fully matched traces stay clean. Monitor/report/loop sibling tests and ruff pass. |
 | 7 | open | high | `gitm/scheduler/loop.py:309-319` | Dense-path MoE expert weight bytes | silent | An unknown quantization method is ignored, leaving `weight_dtype_bytes` at the bf16 default; dominant expert traffic can be overstated while claims look fully priced. | Replace numeric-only extraction with named dtype/provenance and refuse or flag unknown methods; superseded by the sparse dispatcher where applicable. |
 | 8 | open | medium | `gitm/planner/moe_graph.py:498-528`; `gitm/serve/model_config.py:274-276` | Expert weights, often the dominant term | silent substitution | Missing `expert_dtype` inherits linear `weight_dtype`; on mixed-precision checkpoints this can misprice most resident and fetched bytes. Official V4 Flash configs checked so far explicitly declare the field (Flash=`fp4`, Base=`fp8`), but foreign/uniform MoEs may omit it. | Record whether expert dtype was explicit or inherited; require it for model families/quantization layouts where mixed precision is possible, otherwise surface the inheritance in provenance. |
 | 9 | open | medium | `gitm/planner/graph.py:123-133` | Zero-time byte-moving nodes | flag exists but name can lose coverage | `has_unpriced_collectives` scans all nodes, not only collectives; a future “cleanup” to match the name would silently remove the general zero-pricing net. | Rename general predicate (with compatibility alias if needed) or split general and collective-specific intent; trace all consumers. |
@@ -40,7 +40,7 @@ Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`.
 | 2 | Loop MoE dispatch lacks the attach path's dtype priceability gate. | Confirmed and broader: current main has no sparse loop dispatcher/caller at all. |
 | 3 | Missing `expert_dtype` inherits `weight_dtype`, potentially mispricing the dominant expert term. | Confirmed in parser. Official DeepSeek V4 Flash and Base configs explicitly declare differing expert dtypes; omission risk remains for other MoEs. |
 | 4 | Residual percentage clamps at ±100% and loses raw error magnitude. | Fixed with raw + capped-display contract, saturation note, scope, and non-finite refusal. |
-| 5 | Residual classification drops unmatched kernels without loop-path coverage. | Confirmed. |
+| 5 | Residual classification drops unmatched kernels without loop-path coverage. | Fixed with count/time coverage and JSON/summary/report warnings. |
 | 6 | Unknown SKU becomes A100 while recording A100 as if observed. | Confirmed. |
 | 7 | Broad model-config parse rescue becomes an unmarked default model. | Confirmed. |
 | 8 | `has_unpriced_collectives` scans all nodes despite its narrow name. | Confirmed. |
diff --git a/gitm/optimizer/monitor.py b/gitm/optimizer/monitor.py
index ceab2fb..f86bbf6 100644
--- a/gitm/optimizer/monitor.py
+++ b/gitm/optimizer/monitor.py
@@ -52,6 +52,54 @@ class Residuals:
 
     per_kernel: list[KernelResidual] = field(default_factory=list)
     serialized_concurrency_fraction: float = 0.0
+    total_kernels: int = 0
+    classified_kernels: int = 0
+    matched_kernels: int = 0
+    total_kernel_time_ns: int = 0
+    classified_kernel_time_ns: int = 0
+    matched_kernel_time_ns: int = 0
+
+    @staticmethod
+    def _ratio(part: int, whole: int) -> float:
+        return part / whole if whole > 0 else 0.0
+
+    @property
+    def classification_coverage(self) -> float:
+        return self._ratio(self.classified_kernels, self.total_kernels)
+
+    @property
+    def match_coverage(self) -> float:
+        return self._ratio(self.matched_kernels, self.total_kernels)
+
+    @property
+    def classified_time_coverage(self) -> float:
+        return self._ratio(self.classified_kernel_time_ns, self.total_kernel_time_ns)
+
+    @property
+    def matched_time_coverage(self) -> float:
+        return self._ratio(self.matched_kernel_time_ns, self.total_kernel_time_ns)
+
+    @property
+    def coverage_warnings(self) -> list[str]:
+        """Human-facing notes for work excluded from residual conclusions."""
+        if self.total_kernels == 0:
+            return ["residual coverage unavailable: trace contains no kernels"]
+        warnings: list[str] = []
+        if self.classified_kernels < self.total_kernels:
+            warnings.append(
+                "residual coverage: classified "
+                f"{self.classification_coverage:.1%} of launches and "
+                f"{self.classified_time_coverage:.1%} of kernel time "
+                f"({self.classified_kernels}/{self.total_kernels} kernels)"
+            )
+        if self.matched_kernels < self.total_kernels:
+            warnings.append(
+                "residual coverage: matched to the predicted graph "
+                f"{self.match_coverage:.1%} of launches and "
+                f"{self.matched_time_coverage:.1%} of kernel time "
+                f"({self.matched_kernels}/{self.total_kernels} kernels)"
+            )
+        return warnings
 
 
 def _class_key(pn: PredictedNode) -> tuple[float, float]:
@@ -119,7 +167,11 @@ def residuals(trace: Trace, graph: Graph) -> Residuals:
     obs = trace.kernels()
     pred = graph.nodes
 
-    res = Residuals()
+    durations = [max(ok.end_ns - ok.start_ns, 0) for ok in obs]
+    res = Residuals(
+        total_kernels=len(obs),
+        total_kernel_time_ns=sum(durations),
+    )
     by_op_layer: dict[tuple[str, int], PredictedNode] = {}
     classes: dict[str, dict[tuple[float, float], PredictedNode]] = {}
     for pn in pred:
@@ -127,13 +179,17 @@ def residuals(trace: Trace, graph: Graph) -> Residuals:
             by_op_layer.setdefault((pn.op, pn.layer), pn)
         classes.setdefault(pn.op, {}).setdefault(_class_key(pn), pn)
 
-    for ok in obs:
+    for ok, duration_ns in zip(obs, durations, strict=True):
         op = ok.range_op or classify_op(ok.name)
         if op is None:
             continue
+        res.classified_kernels += 1
+        res.classified_kernel_time_ns += duration_ns
         cls = list(classes.get(op, {}).values())
         if not cls:
             continue
+        res.matched_kernels += 1
+        res.matched_kernel_time_ns += duration_ns
 
         t_obs = max((ok.end_ns - ok.start_ns) / 1e9, 1e-12)
         b_obs = (
diff --git a/gitm/optimizer/report.py b/gitm/optimizer/report.py
index b80e8e8..5be890f 100644
--- a/gitm/optimizer/report.py
+++ b/gitm/optimizer/report.py
@@ -78,6 +78,7 @@ def write_report(
     provenance: Provenance,
     *,
     qualification_diagnostic: str = "",
+    runtime_diagnostics: list[str] | None = None,
     summary: str | None = None,
 ) -> str:
     """Render the provenance report as markdown."""
@@ -91,6 +92,7 @@ def write_report(
         "claims": claims,
         "provenance": provenance,
         "qualification_diagnostic": qualification_diagnostic,
+        "runtime_diagnostics": runtime_diagnostics or [],
         "summary": summary or _default_summary(claims),
         "now_ns": time.time_ns(),
     }
diff --git a/gitm/optimizer/templates/report.md.j2 b/gitm/optimizer/templates/report.md.j2
index 499c07c..98dd103 100644
--- a/gitm/optimizer/templates/report.md.j2
+++ b/gitm/optimizer/templates/report.md.j2
@@ -13,6 +13,14 @@
 
 > **Qualification gate diagnostic.** {{ qualification_diagnostic }}
 {%- endif %}
+{%- if runtime_diagnostics %}
+
+## Runtime diagnostics
+
+{% for diagnostic in runtime_diagnostics -%}
+- {{ diagnostic }}
+{% endfor %}
+{%- endif %}
 
 ## Claims
 
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index ee484ef..2e78a77 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -577,6 +577,16 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
     violations = check_invariants(res)  # multi-basis confirmed
     hypotheses = attribute(res, graph)  # Granger
     dr_hypotheses = attribute_dr(res, graph)  # doubly-robust, corroborating
+    coverage = {
+        "total_kernels": res.total_kernels,
+        "classified_kernels": res.classified_kernels,
+        "matched_kernels": res.matched_kernels,
+        "classification_coverage": res.classification_coverage,
+        "match_coverage": res.match_coverage,
+        "classified_time_coverage": res.classified_time_coverage,
+        "matched_time_coverage": res.matched_time_coverage,
+        "warnings": res.coverage_warnings,
+    }
 
     (run_dir / "violations.json").write_text(
         json.dumps(
@@ -599,6 +609,7 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
                 "n_kernel_residuals": len(res.per_kernel),
                 "n_violations": len(violations),
                 "serialized_concurrency_fraction": res.serialized_concurrency_fraction,
+                "coverage": coverage,
                 "top_hypotheses_granger": [
                     {"cause": h.cause_op, "effect": h.effect_op, "p_value": h.p_value}
                     for h in hypotheses.top(5)
@@ -885,6 +896,7 @@ def _unenactable(spec: Any) -> str | None:
         claims=claims,
         provenance=provenance,
         qualification_diagnostic=qual.diagnostic,
+        runtime_diagnostics=res.coverage_warnings,
         summary=(
             f"vLLM decode on {pctx.sku or 'unknown SKU'}: {len(claims)} candidate(s) "
             f"evaluated, {len(rolled_back)} rolled back. {sched_note}"
@@ -908,6 +920,7 @@ def _unenactable(spec: Any) -> str | None:
         "bottleneck_class": ar_run.bottleneck_class,
         "n_autoresearch": len(ar_run.results),
         "scheduler_stats": asdict(sched_summary) if sched_stats.samples else None,
+        "residual_coverage": coverage,
         "report_path": str(run_dir / "report.md"),
     }
     return {"summary": summary, "report_md": report_md, "run_dir": str(run_dir)}
diff --git a/tests/test_report_snapshot.py b/tests/test_report_snapshot.py
index 8f5d89b..b0006d8 100644
--- a/tests/test_report_snapshot.py
+++ b/tests/test_report_snapshot.py
@@ -168,3 +168,18 @@ def test_non_finite_residual_is_refused(raw):
             predicted_delta=0.01,
             measured_delta=None,
         )
+
+
+def test_runtime_diagnostics_are_printed_when_present():
+    rendered = write_report(
+        [],
+        _fixed_provenance(),
+        runtime_diagnostics=["residual coverage: matched 40.0% of kernel time"],
+    )
+
+    assert "## Runtime diagnostics" in rendered
+    assert "matched 40.0% of kernel time" in rendered
+
+
+def test_runtime_diagnostics_section_is_absent_when_clean():
+    assert "## Runtime diagnostics" not in write_report([], _fixed_provenance())
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index ca919e9..93bd60e 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -252,6 +252,40 @@ def test_vllm_workload_still_uses_intervention_path(tmp_path: Path, monkeypatch)
     assert result["summary"]["mode"] == "intervention"
 
 
+def test_vllm_loop_surfaces_residual_coverage(tmp_path: Path, monkeypatch):
+    """Unclassified work must be visible in both the JSON and human report."""
+    import json
+
+    import gitm.scheduler.loop as loop
+
+    @contextmanager
+    def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
+        kernels = [
+            make_kernel("flash_attn_kernel", start_ns=0, end_ns=900),
+            make_kernel("triton_rms_norm_kernel", start_ns=900, end_ns=1000),
+        ]
+        yield make_trace(events=kernels, vendor="nvidia", run_id=run_id or "r")
+
+    monkeypatch.setattr(loop, "capture", fake_capture)
+    monkeypatch.setattr(loop, "sync_device", lambda: None)
+
+    from gitm import optimize
+
+    result = optimize(
+        workload="vllm-decode",
+        budget="1s",
+        scratch=str(tmp_path),
+        workload_runner=lambda: {},
+    )
+    payload = json.loads((Path(result["run_dir"]) / "residuals.json").read_text())
+
+    assert payload["coverage"]["total_kernels"] == 2
+    assert payload["coverage"]["matched_kernels"] == 1
+    assert payload["coverage"]["warnings"]
+    assert "## Runtime diagnostics" in result["report_md"]
+    assert "matched to the predicted graph" in result["report_md"]
+
+
 def test_vllm_loop_runs_autoresearch(tmp_path: Path, monkeypatch):
     """The vllm path runs agentic autoresearch: it classifies the bottleneck via
     trace telemetry (the serialized same-stream "paged_attention" kernels are
diff --git a/tests/test_runtime_on_trace.py b/tests/test_runtime_on_trace.py
index 780f071..11a0ceb 100644
--- a/tests/test_runtime_on_trace.py
+++ b/tests/test_runtime_on_trace.py
@@ -84,6 +84,49 @@ def test_residuals_skip_unmodeled_kernels():
     trace = _trace([_kernel("triton_rms_norm_kernel", 0, 100)])
     res = residuals(trace, predict_graph())
     assert res.per_kernel == []
+    assert res.total_kernels == 1
+    assert res.classified_kernels == 0
+    assert res.matched_kernels == 0
+    assert res.classification_coverage == 0.0
+    assert res.match_coverage == 0.0
+    assert res.coverage_warnings
+
+
+def test_residual_coverage_distinguishes_unclassified_from_graph_miss():
+    from gitm.optimizer.monitor import residuals
+    from gitm.planner.graph import predict_graph
+
+    # FlashAttention classifies and matches; NCCL classifies but a whole-model
+    # dense graph has no collective node; RMSNorm does not classify at all.
+    trace = _trace(
+        [
+            _kernel("flash_attn_kernel", 0, 700),
+            _kernel("ncclDevKernel_AllReduce", 700, 900),
+            _kernel("triton_rms_norm_kernel", 900, 1000),
+        ]
+    )
+    res = residuals(trace, predict_graph())
+
+    assert (res.total_kernels, res.classified_kernels, res.matched_kernels) == (3, 2, 1)
+    assert res.classification_coverage == pytest.approx(2 / 3)
+    assert res.match_coverage == pytest.approx(1 / 3)
+    assert res.classified_time_coverage == pytest.approx(0.9)
+    assert res.matched_time_coverage == pytest.approx(0.7)
+    assert any("classified" in warning for warning in res.coverage_warnings)
+    assert any("matched" in warning for warning in res.coverage_warnings)
+
+
+def test_fully_matched_residual_coverage_is_clean():
+    from gitm.optimizer.monitor import residuals
+    from gitm.planner.graph import predict_graph
+
+    res = residuals(_trace([_kernel("flash_attn_kernel", 0, 100)]), predict_graph())
+
+    assert res.classification_coverage == 1.0
+    assert res.match_coverage == 1.0
+    assert res.classified_time_coverage == 1.0
+    assert res.matched_time_coverage == 1.0
+    assert res.coverage_warnings == []
 
 
 # --- multi-basis filter ------------------------------------------------------

From c6227cd7509fff9c42c43b871f41aee7c12061c6 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 12:30:30 -0700
Subject: [PATCH 04/70] fix: refuse unpriceable execution graphs

---
 AUDIT.md                               |  32 +--
 gitm/planner/context.py                |   1 +
 gitm/planner/graph.py                  |  26 +-
 gitm/planner/roofline.py               |   4 +
 gitm/scheduler/loop.py                 | 340 +++++++++++++++++++++++--
 gitm/serve/attach.py                   |  37 ++-
 gitm/serve/model_config.py             |  60 +++++
 tests/test_execution_graph_dispatch.py | 107 ++++++++
 tests/test_moe_graph.py                |  19 ++
 tests/test_run_loop_workload.py        |  77 +++++-
 tests/test_serve_attach.py             |  97 +++++++
 tests/test_serve_model_config.py       |  42 +++
 tests/test_vllm_embodiment.py          |  40 ++-
 13 files changed, 810 insertions(+), 72 deletions(-)
 create mode 100644 tests/test_execution_graph_dispatch.py

diff --git a/AUDIT.md b/AUDIT.md
index 0ff6ccb..2a7114a 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -7,7 +7,7 @@ Status: **in progress**. This ledger is the primary deliverable for the audit of
 turn missing knowledge into a confident wrong result, with answer-deciding byte
 traffic and dominant expert terms ranked above non-binding estimates.
 
-Highest-severity masks closed: **2 so far**. Wiring gaps confirmed: **3 so far**.
+Highest-severity masks closed: **9 so far**. Wiring gaps confirmed: **3 so far**.
 Deferred findings: **none so far**.
 
 The worktree already contained uncommitted scheduler/serve changes and two new
@@ -19,16 +19,16 @@ they will not be silently absorbed into an audit commit.
 
 | Rank | Status | Severity | Location | Distorted term / contract | Surfacing state | Failure scenario | Disposition |
 |---:|---|---|---|---|---|---|---|
-| 1 | open | critical | `gitm/scheduler/loop.py:165-204, 576-586` | Entire sparse-MoE execution graph; especially dominant expert weight bytes and hybrid-attention/KV terms | silent | A live V4 engine is converted into the legacy `ModelSpec` and sent to `predict_graph`; the production loop never calls `spec_from_hf_config`/`predict_moe_graph`, so it can issue optimization claims against the wrong architecture. | Add one execution-graph dispatcher shared in intent with attach: recognize/normalize/validate the final config, build the sparse graph, and refuse graph-based claims with named diagnostics when it cannot be priced. |
-| 2 | open | critical | `gitm/scheduler/loop.py:165-204` | Model identity and every residual | silent | Any parse bug or version-drift attribute error is caught by `except Exception`, returns `None`, and becomes the plausible Llama-2-7B default graph. | Narrow expected absence handling; return a typed resolution result/diagnostic and degrade to measurement-only rather than a default model. |
-| 3 | open | high | `gitm/planner/context.py:156-169`; `gitm/scheduler/loop.py:576-590`; `gitm/serve/attach.py:477-487` | Compute and HBM denominators for every node | silent | Unknown/no GPU SKU makes `hardware_spec_for(None)` return an A100 spec; artifacts then write `hardware: A100-SXM4-80GB` as though detected on a different or absent GPU. | Preserve fail-open only with explicit hardware-fallback provenance and warnings; graph-based claims should not present the fallback SKU as observed hardware. |
-| 4 | open (planner flag fixed; boundary surfacing pending) | high | `gitm/planner/roofline.py:42-54`; sparse graph byte builders | Weight/KV/activation bytes; dominant decode-binding term | planner FLAG added; production consumers pending | Unknown dtype returns 2-byte bf16. Graph peak fallback may incidentally flag an unknown compute dtype, but scalar sizing calls and mixed-byte nodes cannot identify that their byte width was substituted. | Added `weight_bytes_is_fallback`, per-node `bytes_are_fallback`, and `Graph.has_fallback_bytes` with known/unknown/mixed regression tests. Still must share final-spec validation across attach/loop and surface flags in artifacts/reports before closing. |
+| 1 | fixed | critical | `gitm/scheduler/loop.py` execution-graph dispatcher | Entire sparse-MoE execution graph; especially dominant expert weight bytes and hybrid-attention/KV terms | REFUSE/FLAG | A live V4 engine was converted into the legacy `ModelSpec` and sent to `predict_graph`; the production loop never called `spec_from_hf_config`/`predict_moe_graph`. | Added `_execution_graph`: partial sparse configs reach the sparse gate, valid configs dispatch to `predict_moe_graph`, and unpriceable configs refuse claims into a named measurement-only result. The full graph provenance/flags are serialized. |
+| 2 | fixed | critical | `gitm/scheduler/loop.py` config resolution and refusal artifact | Model identity and every residual | REFUSE | Parse/version-drift failures returned `None` and became the plausible Llama-2-7B default graph. | Typed resolution preserves exception class/message in `prediction_refusal.json` and the report; missing engine/config refuses graph-based claims instead of defaulting. |
+| 3 | fixed | high | `gitm/planner/context.py`; `gitm/planner/roofline.py`; loop and attach artifacts | Compute and HBM denominators for every node | REFUSE/FLAG/WARN | Unknown/no GPU SKU became A100 and artifacts recorded A100 as if detected. | `HardwareSpec.is_fallback`/`Graph.hardware_is_fallback` distinguish substituted pricing. The loop refuses unknown hardware; attach records observed hardware separately from fallback pricing and prints a warning. |
+| 4 | fixed | high | roofline, sparse graph, shared final-spec dtype gate, loop/attach boundaries | Weight/KV/activation bytes; dominant decode-binding term | FLAG/REFUSE | Unknown dtype returned 2-byte bf16; mixed-byte nodes and scalar sizing could not identify the substituted width. | Per-node `bytes_are_fallback` and `Graph.has_fallback_bytes` now cover the graph, while shared `validate_priceable_dtypes` refuses final resolved live/command-line dtypes before customer-facing predictions. JSON and CLI/report consumers surface both directions. |
 | 5 | fixed | high | `gitm/scheduler/loop.py`; `gitm/optimizer/report.py`; report template | Residual magnitude shipped on every Claim | FLAG | A 10x/18x model error and a 2x error both rendered `+100%`, hiding that the model was broken and repeating one aggregate as if claim-specific. | Raw residuals now remain on `Claim`; display capping and saturation are derived, saturated rows print raw magnitude, scopes distinguish run/target-op, and non-finite residuals are refused. Report, scheduler sibling, and ruff checks pass. |
 | 6 | fixed | high | `gitm/optimizer/monitor.py`; `gitm/scheduler/loop.py`; report template | Residual population / coverage | WARN | Unclassified kernels and classified ops absent from the graph were skipped, so the loop could report clean residuals over a small, biased fraction without matched/total coverage. | `Residuals` now records total/classified/matched launch and kernel-time coverage. Incomplete coverage emits warnings into `residuals.json`, the run summary, and a printed Runtime diagnostics section; fully matched traces stay clean. Monitor/report/loop sibling tests and ruff pass. |
-| 7 | open | high | `gitm/scheduler/loop.py:309-319` | Dense-path MoE expert weight bytes | silent | An unknown quantization method is ignored, leaving `weight_dtype_bytes` at the bf16 default; dominant expert traffic can be overstated while claims look fully priced. | Replace numeric-only extraction with named dtype/provenance and refuse or flag unknown methods; superseded by the sparse dispatcher where applicable. |
-| 8 | open | medium | `gitm/planner/moe_graph.py:498-528`; `gitm/serve/model_config.py:274-276` | Expert weights, often the dominant term | silent substitution | Missing `expert_dtype` inherits linear `weight_dtype`; on mixed-precision checkpoints this can misprice most resident and fetched bytes. Official V4 Flash configs checked so far explicitly declare the field (Flash=`fp4`, Base=`fp8`), but foreign/uniform MoEs may omit it. | Record whether expert dtype was explicit or inherited; require it for model families/quantization layouts where mixed precision is possible, otherwise surface the inheritance in provenance. |
-| 9 | open | medium | `gitm/planner/graph.py:123-133` | Zero-time byte-moving nodes | flag exists but name can lose coverage | `has_unpriced_collectives` scans all nodes, not only collectives; a future “cleanup” to match the name would silently remove the general zero-pricing net. | Rename general predicate (with compatibility alias if needed) or split general and collective-specific intent; trace all consumers. |
-| 10 | open | high | `gitm/scheduler/loop.py:587-592` | Prediction trust diagnostics | unconsumed | `predicted_graph.json` writes only node count, total time, and hardware; peak fallback, byte fallback, unpriced nodes, estimates, default batch/model, and provenance do not reach the loop artifact/report. | Serialize machine-readable graph diagnostics and propagate them into the human report and any claim gate. |
+| 7 | fixed | high | loop dispatcher dense/sparse quantization gates | Dense-path MoE expert weight bytes | REFUSE | Unknown quantization methods were ignored, leaving bf16 byte defaults. | Sparse configs no longer use the legacy dense extraction; both sparse and dense dispatch refuse unknown quantization methods with the method named. |
+| 8 | fixed | medium | sparse config resolution in loop and attach | Expert weights, often the dominant term | WARN | Missing `expert_dtype` inherited linear `weight_dtype`. Official V4 Flash configs declare it (Flash=`fp4`, Base=`fp8`), but uniform foreign MoEs may omit it legitimately. | Accepted inheritance now rides in `LiveSpec.warnings` / loop diagnostics and reaches artifacts plus human output; unpriceable inherited dtypes still refuse. |
+| 9 | fixed | medium | `gitm/planner/graph.py` | Zero-time byte-moving nodes | FLAG | `has_unpriced_collectives` scanned all nodes despite its narrow name. | Split `has_unpriced_nodes` (general safety net) from the genuinely collective-specific property; production trust consumers use the general flag and artifacts retain both. |
+| 10 | fixed | high | loop `predicted_graph.json`, summary, and Markdown diagnostics | Prediction trust diagnostics | FLAG/WARN | Loop artifacts omitted fallback, estimate, default, model, batch, sharding, and hardware provenance. | Artifact now carries model source, observed/pricing hardware, batch/sharding, graph flags, per-node diagnostics, and warnings; the report and run summary consume them. |
 
 Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`.
 
@@ -36,14 +36,14 @@ Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`.
 
 | Seed | Verification | Status |
 |---:|---|---|
-| 1 | Unknown weight dtype falls back to bf16 without a bytes-side flag. | Confirmed; incidental peak fallback is insufficient for scalar/mixed-byte paths. Advisor design recorded. |
-| 2 | Loop MoE dispatch lacks the attach path's dtype priceability gate. | Confirmed and broader: current main has no sparse loop dispatcher/caller at all. |
-| 3 | Missing `expert_dtype` inherits `weight_dtype`, potentially mispricing the dominant expert term. | Confirmed in parser. Official DeepSeek V4 Flash and Base configs explicitly declare differing expert dtypes; omission risk remains for other MoEs. |
+| 1 | Unknown weight dtype falls back to bf16 without a bytes-side flag. | Fixed with per-node/graph flags, shared final dtype refusal, and boundary consumers. |
+| 2 | Loop MoE dispatch lacks the attach path's dtype priceability gate. | Fixed with shared predicates and typed sparse dispatcher/refusal. |
+| 3 | Missing `expert_dtype` inherits `weight_dtype`, potentially mispricing the dominant expert term. | Fixed by explicit inheritance warnings; official V4 configs were confirmed to declare the field. |
 | 4 | Residual percentage clamps at ±100% and loses raw error magnitude. | Fixed with raw + capped-display contract, saturation note, scope, and non-finite refusal. |
 | 5 | Residual classification drops unmatched kernels without loop-path coverage. | Fixed with count/time coverage and JSON/summary/report warnings. |
-| 6 | Unknown SKU becomes A100 while recording A100 as if observed. | Confirmed. |
-| 7 | Broad model-config parse rescue becomes an unmarked default model. | Confirmed. |
-| 8 | `has_unpriced_collectives` scans all nodes despite its narrow name. | Confirmed. |
+| 6 | Unknown SKU becomes A100 while recording A100 as if observed. | Fixed with hardware fallback provenance, loop refusal, and attach warning/separate fields. |
+| 7 | Broad model-config parse rescue becomes an unmarked default model. | Fixed in production dispatch with typed named refusal; no default graph. |
+| 8 | `has_unpriced_collectives` scans all nodes despite its narrow name. | Fixed by splitting general and collective-specific predicates. |
 
 ## Sweep coverage
 
@@ -80,7 +80,7 @@ human- or gate-visible consumer.
 
 | Capability | Path A | Path B | Guard parity | Status |
 |---|---|---|---|---|
-| MoE config pricing | attach sidecar validates some raw config dtypes | scheduler has no sparse dispatcher and silently uses legacy dense defaults | Asymmetric | Open finding #1/#2/#7 |
+| MoE config pricing | attach resolves raw config then validates final dtypes | scheduler recognizes partial sparse configs, validates, and dispatches sparse graph | Shared predicates; path-specific input adapters | Fixed |
 | execution lifecycle | launch | attach | To inventory | Pending |
 | model family | dense | MoE | To inventory | Pending |
 | workloads | each dispatch branch | sibling branches | To inventory | Pending |
diff --git a/gitm/planner/context.py b/gitm/planner/context.py
index 70fd7df..d8e5f13 100644
--- a/gitm/planner/context.py
+++ b/gitm/planner/context.py
@@ -170,6 +170,7 @@ def hardware_spec_for(peak: HardwarePeak | None) -> HardwareSpec:
     quant = quant_peaks_for_sku(peak.name)
     return HardwareSpec(
         name=peak.name,
+        is_fallback=False,
         peak_flops_fp16_per_s=peak.peak_flops,
         peak_flops_bf16_per_s=peak.peak_flops,
         peak_flops_fp8_per_s=quant.get("fp8", 0.0),
diff --git a/gitm/planner/graph.py b/gitm/planner/graph.py
index 92e5e07..f7dd0ca 100644
--- a/gitm/planner/graph.py
+++ b/gitm/planner/graph.py
@@ -120,18 +120,27 @@ def total_pred_s(self) -> float:
         return sum(n.prediction.t_pred_s for n in self.nodes)
 
     @property
-    def has_unpriced_collectives(self) -> bool:
-        """True if a collective moves bytes but predicts zero time.
+    def has_unpriced_nodes(self) -> bool:
+        """True if any node moves bytes but predicts zero time.
 
-        Happens when the SKU has no interconnect bandwidth in the catalogue. The
-        node is still in the graph — it just costs nothing, which would quietly
-        credit a sharded deployment with a free all-to-all. Louder to ask than to
-        discover it in a report.
+        This is the general trust net: any missing bandwidth denominator can make
+        real work look free, whether or not the node is a collective.
         """
         return any(
             n.prediction.bytes > 0 and n.prediction.t_pred_s == 0.0 for n in self.nodes
         )
 
+    @property
+    def has_unpriced_collectives(self) -> bool:
+        """True if a collective moves bytes but predicts zero time."""
+        collective_ops = {"moe_all_to_all", "tp_all_reduce"}
+        return any(
+            n.op in collective_ops
+            and n.prediction.bytes > 0
+            and n.prediction.t_pred_s == 0.0
+            for n in self.nodes
+        )
+
     @property
     def has_fallback_peaks(self) -> bool:
         """True if any node was priced against a dtype it doesn't run in.
@@ -150,6 +159,11 @@ def has_fallback_bytes(self) -> bool:
         """
         return any(n.prediction.bytes_are_fallback for n in self.nodes)
 
+    @property
+    def hardware_is_fallback(self) -> bool:
+        """True when the graph uses substituted rather than detected SKU peaks."""
+        return self.hw.is_fallback
+
 
 def predict_graph(
     model: ModelSpec | None = None,
diff --git a/gitm/planner/roofline.py b/gitm/planner/roofline.py
index 0e71ee1..957d253 100644
--- a/gitm/planner/roofline.py
+++ b/gitm/planner/roofline.py
@@ -68,6 +68,10 @@ class HardwareSpec:
     """
 
     name: str = "A100-SXM4-80GB"
+    # True when these catalogue numbers were substituted because the runtime did
+    # not identify a priceable SKU. Direct ``HardwareSpec()`` construction is the
+    # same documented A100 fallback; catalogue-backed builders set this false.
+    is_fallback: bool = True
     peak_flops_fp16_per_s: float = 312e12
     peak_flops_bf16_per_s: float = 312e12
     peak_flops_fp32_per_s: float = 19.5e12
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 2e78a77..02fc528 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -54,8 +54,16 @@
     unmet_prerequisite,
 )
 from gitm.planner.context import build_planner_context, hardware_spec_for
-from gitm.planner.graph import predict_graph
+from gitm.planner.graph import Graph, predict_graph
+from gitm.planner.moe_graph import predict_moe_graph, spec_from_hf_config
+from gitm.planner.roofline import BatchConfig, ModelSpec, ShardingConfig
 from gitm.safety.audit import AuditLog, _write_report
+from gitm.serve.model_config import (
+    is_sparse_moe_config,
+    normalize_moe_config,
+    validate_moe_config,
+    validate_priceable_dtypes,
+)
 from gitm.tracer.capture import capture
 from gitm.tracer.vllm_stats import sample_scheduler_stats, summarize_requests
 from gitm.workloads import WorkloadRunner, get_factory, sync_device
@@ -267,6 +275,234 @@ def _batch_config_from_stats(sched: Any):
     return BatchConfig(batch=max(int(round(float(running))), 1))
 
 
+@dataclass
+class ExecutionGraphResolution:
+    """A trustworthy loop graph, or the named reason graph-based claims refuse."""
+
+    graph: Graph | None
+    diagnostics: list[str]
+    refusal_reason: str = ""
+    model_source: str = ""
+
+    @property
+    def ok(self) -> bool:
+        return self.graph is not None and not self.refusal_reason
+
+
+def _engine_hf_config(engine: Any) -> tuple[Any | None, str]:
+    if engine is None:
+        return None, "no live engine was supplied"
+    for path in (
+        "llm_engine.model_config.hf_config",
+        "llm_engine.vllm_config.model_config.hf_config",
+        "model_config.hf_config",
+    ):
+        obj: Any = engine
+        for attr in path.split("."):
+            obj = getattr(obj, attr, None)
+            if obj is None:
+                break
+        if obj is not None:
+            return obj, path
+    return None, "the live engine exposes no HuggingFace config"
+
+
+def _config_dict(hf: Any) -> tuple[dict[str, Any] | None, str]:
+    try:
+        if isinstance(hf, dict):
+            return dict(hf), ""
+        to_dict = getattr(hf, "to_dict", None)
+        raw = to_dict() if callable(to_dict) else vars(hf)
+        if not isinstance(raw, dict):
+            return None, f"config conversion returned {type(raw).__name__}, not dict"
+        return dict(raw), ""
+    except Exception as exc:
+        # Fail open at the runtime boundary, but preserve the parser failure as
+        # the refusal reason instead of turning it into a default model.
+        return None, f"config conversion failed ({type(exc).__name__}: {exc})"
+
+
+def _engine_value(engine: Any, paths: tuple[str, ...]) -> Any:
+    for path in paths:
+        obj: Any = engine
+        for attr in path.split("."):
+            obj = getattr(obj, attr, None)
+            if obj is None:
+                break
+        if obj is not None:
+            return obj
+    return None
+
+
+def _loop_batch(engine: Any, pctx: Any, sched: Any) -> tuple[BatchConfig, list[str]]:
+    diagnostics: list[str] = []
+    observed = _batch_config_from_stats(sched)
+    batch = observed.batch if observed is not None else 1
+    if observed is None:
+        diagnostics.append("decode concurrency was not observed; using batch=1")
+
+    kv_len = getattr(getattr(pctx, "gate", None), "kv_cache_len", None)
+    if not isinstance(kv_len, int) or kv_len <= 0:
+        kv_len = 4096
+        diagnostics.append("KV cache length was not exposed; using kv_cache_len=4096")
+
+    speculative = _engine_value(
+        engine,
+        (
+            "speculative_config.num_speculative_tokens",
+            "vllm_config.speculative_config.num_speculative_tokens",
+        ),
+    )
+    speculative_tokens = int(speculative) if isinstance(speculative, int) and speculative > 0 else 0
+    return BatchConfig(
+        batch=batch,
+        kv_cache_len=kv_len,
+        speculative_tokens=speculative_tokens,
+    ), diagnostics
+
+
+def _loop_sharding(engine: Any) -> tuple[ShardingConfig, list[str]]:
+    tp = _engine_value(
+        engine,
+        ("parallel_config.tensor_parallel_size", "vllm_config.parallel_config.tensor_parallel_size"),
+    )
+    dp = _engine_value(
+        engine,
+        ("parallel_config.data_parallel_size", "vllm_config.parallel_config.data_parallel_size"),
+    )
+    ep = _engine_value(
+        engine,
+        ("parallel_config.enable_expert_parallel", "vllm_config.parallel_config.enable_expert_parallel"),
+    )
+    if not isinstance(tp, int) or tp < 1:
+        return ShardingConfig(), [
+            "sharding topology was not exposed; using whole-model tp=1 ep=1 dp=1"
+        ]
+    dp_i = dp if isinstance(dp, int) and dp > 0 else 1
+    return ShardingConfig(tp=tp, ep=tp if bool(ep) else 1, dp=dp_i), []
+
+
+def _dense_spec_from_config(cfg: dict[str, Any]) -> tuple[ModelSpec | None, str]:
+    required = (
+        "hidden_size",
+        "num_hidden_layers",
+        "num_attention_heads",
+        "intermediate_size",
+        "vocab_size",
+    )
+    missing = [key for key in required if cfg.get(key) is None]
+    if missing:
+        return None, "dense model config is missing answer-deciding fields: " + ", ".join(missing)
+    try:
+        hidden = int(cfg["hidden_size"])
+        n_heads = int(cfg["num_attention_heads"])
+        n_kv = int(cfg.get("num_key_value_heads", n_heads) or n_heads)
+        head_dim = int(cfg.get("head_dim", 0) or (hidden // n_heads))
+        act = str(cfg.get("torch_dtype", "bf16")).lower()
+        dtype_bytes = 4 if act in ("fp32", "float32") else 2
+        quant = cfg.get("quantization_config") or {}
+        method = quant.get("quant_method") if isinstance(quant, dict) else None
+        if method is not None and str(method).lower() not in _QUANT_WEIGHT_BYTES:
+            return None, f"dense quantization method {method!r} is not priceable"
+        return ModelSpec(
+            name=str(cfg.get("_name_or_path") or cfg.get("model_type") or "live-dense"),
+            hidden=hidden,
+            n_layers=int(cfg["num_hidden_layers"]),
+            n_heads=n_heads,
+            num_kv_heads=n_kv,
+            head_dim=head_dim,
+            intermediate=int(cfg["intermediate_size"]),
+            dtype_bytes=dtype_bytes,
+            weight_dtype_bytes=_QUANT_WEIGHT_BYTES.get(str(method).lower()) if method else None,
+            vocab=int(cfg["vocab_size"]),
+        ), ""
+    except (TypeError, ValueError, ZeroDivisionError) as exc:
+        return None, f"dense model config could not be parsed ({type(exc).__name__}: {exc})"
+
+
+def _execution_graph(engine: Any, pctx: Any, sched: Any) -> ExecutionGraphResolution:
+    """Resolve the live engine to the matching graph, or refuse namedly."""
+    hf, source = _engine_hf_config(engine)
+    if hf is None:
+        return ExecutionGraphResolution(None, [], source)
+    cfg, error = _config_dict(hf)
+    if cfg is None:
+        return ExecutionGraphResolution(None, [], error)
+
+    if getattr(pctx, "peak", None) is None:
+        sku = getattr(pctx, "sku", None) or "unknown"
+        return ExecutionGraphResolution(
+            None,
+            [],
+            f"GPU SKU {sku!r} is not in the hardware catalogue; refusing A100 substitution",
+        )
+    hw = hardware_spec_for(pctx.peak)
+    batch, diagnostics = _loop_batch(engine, pctx, sched)
+
+    if is_sparse_moe_config(cfg):
+        invalid = validate_moe_config(cfg)
+        if invalid:
+            return ExecutionGraphResolution(
+                None,
+                diagnostics,
+                "sparse-MoE config cannot be priced without guessing: " + "; ".join(invalid),
+            )
+        try:
+            spec = spec_from_hf_config(normalize_moe_config(cfg), name=str(cfg.get("_name_or_path") or "live-moe"))
+        except (TypeError, ValueError, ZeroDivisionError) as exc:
+            return ExecutionGraphResolution(
+                None,
+                diagnostics,
+                f"sparse-MoE config could not be parsed ({type(exc).__name__}: {exc})",
+            )
+        act_dtype = getattr(getattr(pctx, "gate", None), "dtype", None)
+        kv_dtype = _engine_value(
+            engine,
+            ("cache_config.cache_dtype", "cache_config.kv_cache_dtype", "vllm_config.cache_config.cache_dtype"),
+        )
+        changes: dict[str, Any] = {}
+        if act_dtype:
+            changes["act_dtype"] = str(act_dtype).lower()
+        if kv_dtype:
+            changes["kv_dtype"] = str(kv_dtype).lower().replace("fp8_e4m3", "fp8")
+        else:
+            diagnostics.append("KV cache dtype was not exposed; using planner default kv_dtype='fp8'")
+        if cfg.get("expert_dtype") is None:
+            diagnostics.append(
+                f"expert_dtype absent; inherited weight_dtype={spec.weight_dtype!r} for expert bytes"
+            )
+        if changes:
+            from dataclasses import replace
+
+            spec = replace(spec, **changes)
+        unpriceable = validate_priceable_dtypes(spec)
+        if unpriceable:
+            return ExecutionGraphResolution(None, diagnostics, "; ".join(unpriceable))
+        sharding, sharding_diagnostics = _loop_sharding(engine)
+        diagnostics.extend(sharding_diagnostics)
+        graph = predict_moe_graph(spec, hw, batch, sharding)
+    else:
+        spec, error = _dense_spec_from_config(cfg)
+        if spec is None:
+            return ExecutionGraphResolution(None, diagnostics, error)
+        graph = predict_graph(model=spec, hw=hw, batch=batch)
+
+    if graph.has_fallback_peaks:
+        diagnostics.append("one or more nodes use fallback compute peaks")
+    if graph.has_fallback_bytes:
+        diagnostics.append("one or more nodes use fallback byte widths")
+    if graph.has_unpriced_nodes:
+        diagnostics.append("one or more byte-moving nodes have no priceable bandwidth")
+    n_estimated = sum(1 for node in graph.nodes if node.prediction.estimated)
+    if n_estimated:
+        diagnostics.append(f"{n_estimated} predicted node(s) use estimated cost models")
+    return ExecutionGraphResolution(
+        graph,
+        diagnostics,
+        model_source="live_hf_config",
+    )
+
+
 def _read_int_aliases(hf: Any, table: dict[str, tuple[str, ...]]) -> dict[str, Any]:
     """First positive int found for each field across its aliases. Duck-typed."""
     out: dict[str, Any] = {}
@@ -543,31 +779,76 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
             trace_path=trace_path,
         )
 
-    # Predict against the model that ACTUALLY ran (read from the live engine),
-    # not the Llama-2-7B default — otherwise residuals/deviation score the real
-    # kernels against the wrong graph. Falls back to the default graph when there
-    # is no engine or its config can't be read (CPU boxes, tests, dry-run).
-    #
-    # Same for hardware: predict_graph's own default is A100-SXM4-80GB peaks,
-    # which silently over-predicts on anything weaker (T4/L4/...) and
-    # produces a run-level kernel-time residual that saturates the report's
-    # +/-100% clamp on every claim. pctx is built here (moved up from Phase 3)
-    # so its NVML-detected SKU peak feeds the graph before residuals are ever
-    # computed against it.
+    # Resolve model, hardware, serving batch, and sharding as one trust gate. A
+    # missing/partial live config or unknown SKU refuses graph-based claims and
+    # falls through to an honest measurement report; it never becomes a plausible
+    # Llama/A100 default prediction.
     pctx = build_planner_context(cfg.engine, workload=workload)
-    _spec = _model_spec_from_engine(cfg.engine)
-    _hw = hardware_spec_for(pctx.peak)
-    # Batch matters for the same reason the model does — and more so on a
-    # mixture, where weight traffic follows the *distinct* experts a batch
-    # activates: distinct(1)=top_k but distinct(16) is an order of magnitude
-    # larger, so predicting a batch-16 step at the batch-1 default understates
-    # expert traffic ~12x. Read the real concurrency off the sampled scheduler
-    # rather than defaulting.
-    _batch = _batch_config_from_stats(sched_summary)
-    graph = predict_graph(model=_spec, hw=_hw, batch=_batch)
+    graph_resolution = _execution_graph(cfg.engine, pctx, sched_summary)
+    if not graph_resolution.ok:
+        (run_dir / "prediction_refusal.json").write_text(
+            json.dumps(
+                {
+                    "reason": graph_resolution.refusal_reason,
+                    "diagnostics": graph_resolution.diagnostics,
+                    "hardware": pctx.sku,
+                },
+                indent=2,
+            )
+        )
+        return _measurement_result(
+            run_dir=run_dir,
+            run_id=run_id,
+            workload=workload,
+            trace=trace,
+            qual=qual,
+            started_ns=started_ns,
+            trace_path=trace_path,
+            diagnostic=(
+                "Prediction gate refused graph-based optimization claims: "
+                f"{graph_resolution.refusal_reason}"
+            ),
+            runtime_diagnostics=graph_resolution.diagnostics,
+            status="prediction_refused",
+        )
+    graph = graph_resolution.graph
+    assert graph is not None
     (run_dir / "predicted_graph.json").write_text(
         json.dumps(
-            {"nodes": len(graph.nodes), "total_pred_s": graph.total_pred_s, "hardware": _hw.name},
+            {
+                "model": graph.model.name,
+                "model_source": graph_resolution.model_source,
+                "nodes": len(graph.nodes),
+                "total_pred_s": graph.total_pred_s,
+                "hardware": pctx.sku,
+                "hardware_pricing": graph.hw.name,
+                "hardware_is_fallback": graph.hardware_is_fallback,
+                "batch": {
+                    "batch": graph.batch.batch,
+                    "kv_cache_len": graph.batch.kv_cache_len,
+                    "speculative_tokens": graph.batch.speculative_tokens,
+                },
+                "sharding": {
+                    "tp": graph.sharding.tp,
+                    "ep": graph.sharding.ep,
+                    "dp": graph.sharding.dp,
+                },
+                "has_unpriced_collectives": graph.has_unpriced_collectives,
+                "has_unpriced_nodes": graph.has_unpriced_nodes,
+                "has_fallback_peaks": graph.has_fallback_peaks,
+                "has_fallback_bytes": graph.has_fallback_bytes,
+                "diagnostics": graph_resolution.diagnostics,
+                "predictions": [
+                    {
+                        "op": node.op,
+                        "layer": node.layer,
+                        "estimated": node.prediction.estimated,
+                        "peak_is_fallback": node.prediction.peak_is_fallback,
+                        "bytes_are_fallback": node.prediction.bytes_are_fallback,
+                    }
+                    for node in graph.nodes
+                ],
+            },
             indent=2,
         )
     )
@@ -896,7 +1177,7 @@ def _unenactable(spec: Any) -> str | None:
         claims=claims,
         provenance=provenance,
         qualification_diagnostic=qual.diagnostic,
-        runtime_diagnostics=res.coverage_warnings,
+        runtime_diagnostics=graph_resolution.diagnostics + res.coverage_warnings,
         summary=(
             f"vLLM decode on {pctx.sku or 'unknown SKU'}: {len(claims)} candidate(s) "
             f"evaluated, {len(rolled_back)} rolled back. {sched_note}"
@@ -921,6 +1202,7 @@ def _unenactable(spec: Any) -> str | None:
         "n_autoresearch": len(ar_run.results),
         "scheduler_stats": asdict(sched_summary) if sched_stats.samples else None,
         "residual_coverage": coverage,
+        "prediction_diagnostics": graph_resolution.diagnostics,
         "report_path": str(run_dir / "report.md"),
     }
     return {"summary": summary, "report_md": report_md, "run_dir": str(run_dir)}
@@ -935,6 +1217,9 @@ def _measurement_result(
     qual: Any,
     started_ns: int,
     trace_path: Path,
+    diagnostic: str | None = None,
+    runtime_diagnostics: list[str] | None = None,
+    status: str = "ok",
 ) -> dict[str, Any]:
     """Honest measurement report for a workload with no intervention library.
 
@@ -972,10 +1257,11 @@ def _measurement_result(
     report_md = write_report(
         claims=claims,
         provenance=provenance,
-        qualification_diagnostic=(
+        qualification_diagnostic=diagnostic or (
             "Measurement-only run: the runtime observed the workload and reports "
             "its real kernels. No intervention library applies to this workload."
         ),
+        runtime_diagnostics=runtime_diagnostics,
         summary=measurement_summary(workload, result),
     )
     _write_report(run_dir, report_md)
@@ -983,7 +1269,7 @@ def _measurement_result(
     summary = {
         "run_id": run_id,
         "workload": workload,
-        "status": "ok",
+        "status": status,
         "mode": "measurement",
         "fingerprint": qual.fingerprint,
         "commit": False,
@@ -992,6 +1278,8 @@ def _measurement_result(
         "n_claims": 0,
         "n_rolled_back": 0,
         "n_rejected": 0,
+        "prediction_refusal": diagnostic,
+        "runtime_diagnostics": runtime_diagnostics or [],
         "report_path": str(run_dir / "report.md"),
     }
     return {"summary": summary, "report_md": report_md, "run_dir": str(run_dir)}
diff --git a/gitm/serve/attach.py b/gitm/serve/attach.py
index 0abd046..ca28ef8 100644
--- a/gitm/serve/attach.py
+++ b/gitm/serve/attach.py
@@ -424,6 +424,7 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
     prints the named-key refusal and writes nothing — a defaulted DeepSeek prediction
     next to a real trace would be read as a measurement, which is the one outcome the
     gate exists to prevent. Never raises into the capture path.
+
     """
     from gitm.serve.model_config import LiveSpec, live_moe_spec
 
@@ -442,14 +443,36 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
     from gitm.planner.context import build_planner_context, hardware_spec_for
     from gitm.planner.moe_graph import predict_moe_graph
 
-    hw = hardware_spec_for(build_planner_context().peak)
+    planner_ctx = build_planner_context()
+    hw = hardware_spec_for(planner_ctx.peak)
     g = predict_moe_graph(resolved.spec, hw, resolved.batch, resolved.sharding)
     sh, spec = resolved.sharding, resolved.spec
+    warnings = list(resolved.warnings)
+    if planner_ctx.peak is None:
+        warnings.append(
+            f"GPU SKU {planner_ctx.sku or 'unknown'!r} is not in the hardware catalogue; "
+            f"pricing uses fallback {hw.name!r}"
+        )
+    if g.has_unpriced_nodes:
+        warnings.append(
+            "byte-moving nodes are unpriced (a required bandwidth is absent from the catalogue)"
+        )
+    if g.has_fallback_peaks:
+        warnings.append("priced against fallback compute peaks; the ceiling is low")
+    if g.has_fallback_bytes:
+        warnings.append(
+            "byte widths include an unknown-dtype bf16 fallback; the memory floor is approximate"
+        )
+    n_estimated = sum(1 for n in g.nodes if n.prediction.estimated)
+    if n_estimated:
+        warnings.append(f"{n_estimated} predicted node(s) use documented estimated cost models")
 
     payload = {
         "model_ref": resolved.model_ref,
         "config_source": str(resolved.source_path),
-        "hardware": hw.name,
+        "hardware": planner_ctx.sku,
+        "hardware_pricing": hw.name,
+        "hardware_is_fallback": planner_ctx.peak is None,
         "sharding": {"tp": sh.tp, "ep": sh.ep, "dp": sh.dp},
         "dtypes": {
             "weight": spec.weight_dtype,
@@ -461,7 +484,10 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
         "applied_overrides": resolved.applied_overrides,
         "total_pred_s": g.total_pred_s,
         "has_unpriced_collectives": g.has_unpriced_collectives,
+        "has_unpriced_nodes": g.has_unpriced_nodes,
         "has_fallback_peaks": g.has_fallback_peaks,
+        "has_fallback_bytes": g.has_fallback_bytes,
+        "warnings": warnings,
         "nodes": [
             {
                 "op": n.op,
@@ -472,6 +498,7 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
                 "flops": n.prediction.flops,
                 "bytes": n.prediction.bytes,
                 "estimated": n.prediction.estimated,
+                "bytes_are_fallback": n.prediction.bytes_are_fallback,
             }
             for n in g.nodes
         ],
@@ -484,10 +511,8 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
         f"(TP={sh.tp} EP={sh.ep} DP={sh.dp}, "
         f"w={spec.weight_dtype}/e={spec.expert_dtype}/kv={spec.kv_dtype})"
     )
-    if g.has_unpriced_collectives:
-        print("    - collectives are unpriced (SKU has no interconnect bandwidth in the catalogue)")
-    if g.has_fallback_peaks:
-        print("    - priced against fallback peaks — the ceiling is low in a known direction")
+    for warning in warnings:
+        print(f"    - {warning}")
 
 
 def describe_targets(proc: Path = discover.PROC) -> list[dict]:
diff --git a/gitm/serve/model_config.py b/gitm/serve/model_config.py
index 770c69b..4bc6da2 100644
--- a/gitm/serve/model_config.py
+++ b/gitm/serve/model_config.py
@@ -209,6 +209,22 @@ def _first_present(cfg: dict[str, Any], keys: tuple[str, ...]) -> Any:
     return None
 
 
+def is_sparse_moe_config(cfg: dict[str, Any]) -> bool:
+    """True when any routed-expert shape field makes this a sparse candidate.
+
+    This deliberately recognizes partial configs. A config that declares an
+    expert count but omits top-k must reach :func:`validate_moe_config` and be
+    refused, never fall through to the dense graph because it was incomplete.
+    """
+    # ``intermediate_size`` is also the standard dense-MLP field, so only the
+    # explicitly MoE spelling is a sparse signal on that axis.
+    return (
+        _first_present(cfg, _EXPERT_COUNT_ALIASES) is not None
+        or _first_present(cfg, _EXPERT_TOPK_ALIASES) is not None
+        or cfg.get("moe_intermediate_size") is not None
+    )
+
+
 def validate_moe_config(cfg: dict[str, Any]) -> list[str]:
     """Dominant-term fields this config fails to declare usably. Empty == usable.
 
@@ -241,6 +257,19 @@ def validate_moe_config(cfg: dict[str, Any]) -> list[str]:
     return missing
 
 
+def validate_priceable_dtypes(spec: SparseMoEModelSpec) -> list[str]:
+    """Final resolved dtypes that would make byte-width pricing fall back."""
+    missing: list[str] = []
+    for field_name in ("weight_dtype", "expert_dtype", "kv_dtype", "act_dtype"):
+        value = getattr(spec, field_name)
+        if str(value).lower() not in KNOWN_DTYPES:
+            missing.append(
+                f"{field_name}={value!r} (not priceable; known: "
+                f"{', '.join(sorted(KNOWN_DTYPES))})"
+            )
+    return missing
+
+
 def normalize_moe_config(cfg: dict[str, Any]) -> dict[str, Any]:
     """Copy of ``cfg`` with alias keys mapped onto the canonical DeepSeek names.
 
@@ -272,6 +301,7 @@ class LiveSpec:
     source_path: Path
     model_ref: str
     applied_overrides: dict[str, Any] = field(default_factory=dict)
+    warnings: list[str] = field(default_factory=list)
     ok: bool = True
 
 
@@ -310,6 +340,7 @@ def live_moe_spec(
     supplied (tests). Returns :class:`LiveSpec` on success or :class:`LiveSpecError`
     with the reason — never a defaulted spec, so a report can only ever show a
     prediction against the model that is genuinely loaded.
+
     """
     model_ref = model_ref_from_cmdline(target.cmdline)
     if not model_ref:
@@ -359,10 +390,38 @@ def live_moe_spec(
 
         spec = replace(spec, **spec_changes)
 
+    unpriceable = validate_priceable_dtypes(spec)
+    if unpriceable:
+        return LiveSpecError(
+            reason=(
+                f"{model_ref!r} at {cfg_path} resolves to dtypes this planner cannot "
+                "price without substituting bf16 byte widths."
+            ),
+            model_ref=model_ref,
+            missing_keys=unpriceable,
+        )
+
     sharding = ShardingConfig(
         tp=overrides.get("tp", 1), ep=overrides.get("ep", 1), dp=overrides.get("dp", 1)
     )
     batch = BatchConfig(kv_cache_len=overrides.get("max_model_len", default_kv_cache_len))
+    warnings: list[str] = []
+    if cfg.get("expert_dtype") is None:
+        warnings.append(
+            "expert_dtype absent; inherited "
+            f"weight_dtype={spec.weight_dtype!r} for routed and shared expert weights"
+        )
+    warnings.append("decode batch was not observed; using batch=1 single-sequence floor")
+    if "max_model_len" not in overrides:
+        warnings.append(
+            f"KV cache length was not declared on the launch command; using "
+            f"kv_cache_len={default_kv_cache_len}"
+        )
+    if overrides.get("ep_enabled") and overrides.get("dp", 1) > 1:
+        warnings.append(
+            "expert parallelism with data_parallel_size>1 is approximated over the "
+            "tensor-parallel group"
+        )
 
     return LiveSpec(
         spec=spec,
@@ -371,4 +430,5 @@ def live_moe_spec(
         source_path=cfg_path,
         model_ref=model_ref,
         applied_overrides=overrides,
+        warnings=warnings,
     )
diff --git a/tests/test_execution_graph_dispatch.py b/tests/test_execution_graph_dispatch.py
new file mode 100644
index 0000000..7047bf9
--- /dev/null
+++ b/tests/test_execution_graph_dispatch.py
@@ -0,0 +1,107 @@
+"""The loop must dispatch the live model to a priceable execution graph or refuse."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+from gitm.planner.context import peak_for_sku
+from gitm.planner.roofline import SparseMoEModelSpec
+from gitm.scheduler.loop import _execution_graph
+
+
+def _pctx(sku: str | None = "NVIDIA B200", *, kv_cache_len: int | None = None):
+    return SimpleNamespace(
+        peak=peak_for_sku(sku),
+        sku=sku,
+        gate=SimpleNamespace(kv_cache_len=kv_cache_len),
+    )
+
+
+def _engine(cfg: dict):
+    hf = SimpleNamespace(**cfg)
+    return SimpleNamespace(model_config=SimpleNamespace(hf_config=hf))
+
+
+def _moe_cfg(**over) -> dict:
+    cfg = {
+        "model_type": "deepseek_v4",
+        "hidden_size": 64,
+        "num_hidden_layers": 2,
+        "num_attention_heads": 1,
+        "num_key_value_heads": 1,
+        "head_dim": 64,
+        "n_routed_experts": 4,
+        "num_experts_per_tok": 1,
+        "moe_intermediate_size": 32,
+        "expert_dtype": "fp4",
+        "quantization_config": {"quant_method": "fp8"},
+        "torch_dtype": "bfloat16",
+    }
+    cfg.update(over)
+    return cfg
+
+
+def test_sparse_engine_dispatches_to_sparse_graph():
+    resolved = _execution_graph(_engine(_moe_cfg()), _pctx(), sched=None)
+
+    assert resolved.ok
+    assert isinstance(resolved.graph.model, SparseMoEModelSpec)
+    assert any(node.op == "moe_routed" for node in resolved.graph.nodes)
+    assert not resolved.graph.has_fallback_bytes
+    assert resolved.model_source == "live_hf_config"
+
+
+def test_sparse_dispatch_refuses_unpriceable_final_dtype():
+    resolved = _execution_graph(
+        _engine(_moe_cfg(expert_dtype="future_fp3")), _pctx(), sched=None
+    )
+
+    assert not resolved.ok and resolved.graph is None
+    assert "expert_dtype='future_fp3'" in resolved.refusal_reason
+
+
+def test_partial_sparse_config_is_refused_not_sent_to_dense_graph():
+    resolved = _execution_graph(
+        _engine({"hidden_size": 64, "n_routed_experts": 4}), _pctx(), sched=None
+    )
+
+    assert not resolved.ok and resolved.graph is None
+    assert "experts per token" in resolved.refusal_reason
+
+
+def test_missing_live_model_refuses_instead_of_defaulting_to_llama():
+    resolved = _execution_graph(None, _pctx(), sched=None)
+
+    assert not resolved.ok and resolved.graph is None
+    assert "no live engine" in resolved.refusal_reason
+
+
+def test_unknown_hardware_refuses_instead_of_recording_a100_as_observed():
+    resolved = _execution_graph(_engine(_moe_cfg()), _pctx("Unknown GPU"), sched=None)
+
+    assert not resolved.ok and resolved.graph is None
+    assert "Unknown GPU" in resolved.refusal_reason
+    assert "hardware catalogue" in resolved.refusal_reason
+
+
+def test_config_conversion_bug_is_a_named_refusal():
+    class BrokenConfig:
+        def to_dict(self):
+            raise RuntimeError("parser exploded")
+
+    engine = SimpleNamespace(model_config=SimpleNamespace(hf_config=BrokenConfig()))
+    resolved = _execution_graph(engine, _pctx(), sched=None)
+
+    assert not resolved.ok and resolved.graph is None
+    assert "RuntimeError" in resolved.refusal_reason
+    assert "parser exploded" in resolved.refusal_reason
+
+
+def test_accepted_batch_and_kv_defaults_are_diagnostics():
+    resolved = _execution_graph(_engine(_moe_cfg()), _pctx(), sched=None)
+
+    assert resolved.ok
+    assert resolved.graph.batch.batch == 1
+    assert resolved.graph.batch.kv_cache_len == 4096
+    assert any("batch=1" in note for note in resolved.diagnostics)
+    assert any("kv_cache_len=4096" in note for note in resolved.diagnostics)
diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py
index c31e056..5a409cd 100644
--- a/tests/test_moe_graph.py
+++ b/tests/test_moe_graph.py
@@ -21,6 +21,7 @@
 
 from gitm.optimizer.deviation import classify_op
 from gitm.planner.context import hardware_spec_for, peak_for_sku
+from gitm.planner.graph import Graph, PredictedNode
 from gitm.planner.moe_graph import (
     effective_kv_tokens,
     index_candidates,
@@ -35,6 +36,7 @@
     HardwareSpec,
     ShardingConfig,
     resolve_peak,
+    roofline,
     weight_bytes,
     weight_bytes_is_fallback,
 )
@@ -246,6 +248,12 @@ def test_graph_flags_when_any_node_ran_on_a_fallback_peak(spec):
 def test_b200_graph_needs_no_fallback(spec, b200):
     g = predict_moe_graph(spec, b200, BatchConfig(batch=1, kv_cache_len=1024))
     assert not g.has_fallback_peaks
+    assert not g.hardware_is_fallback
+
+
+def test_default_hardware_graph_flags_catalogue_fallback(spec):
+    g = predict_moe_graph(spec, HardwareSpec(), BatchConfig(batch=1, kv_cache_len=1024))
+    assert g.hardware_is_fallback
 
 
 def test_fp4_weight_bytes_include_the_block_scales():
@@ -553,6 +561,7 @@ def test_unknown_interconnect_surfaces_rather_than_pricing_collectives_free(spec
         spec, no_link, BatchConfig(batch=64, kv_cache_len=4096), ShardingConfig(tp=8, ep=8)
     )
     assert g.has_unpriced_collectives
+    assert g.has_unpriced_nodes
 
 
 def test_priced_interconnect_is_not_flagged(spec, b200):
@@ -560,6 +569,16 @@ def test_priced_interconnect_is_not_flagged(spec, b200):
         spec, b200, BatchConfig(batch=64, kv_cache_len=4096), ShardingConfig(tp=8, ep=8)
     )
     assert not g.has_unpriced_collectives
+    assert not g.has_unpriced_nodes
+
+
+def test_general_unpriced_node_net_is_not_collective_specific(spec):
+    hw = HardwareSpec(peak_mem_bw_bytes_per_s=0.0)
+    node = PredictedNode("memory_only", None, roofline("memory_only", 0.0, 1024.0, hw))
+    g = Graph(model=spec, hw=hw, batch=BatchConfig(), nodes=[node])
+
+    assert g.has_unpriced_nodes
+    assert not g.has_unpriced_collectives
 
 
 def test_expert_imbalance_only_applies_under_expert_parallelism(spec, b200):
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index 93bd60e..6af729e 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -9,6 +9,7 @@
 
 from contextlib import contextmanager
 from pathlib import Path
+from types import SimpleNamespace
 
 import pytest
 
@@ -215,6 +216,33 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
     return fake_capture
 
 
+def _priceable_moe_engine():
+    cfg = SimpleNamespace(
+        model_type="deepseek_v4",
+        hidden_size=64,
+        num_hidden_layers=2,
+        num_attention_heads=1,
+        num_key_value_heads=1,
+        head_dim=64,
+        n_routed_experts=4,
+        num_experts_per_tok=1,
+        moe_intermediate_size=32,
+        expert_dtype="fp4",
+        quantization_config={"quant_method": "fp8"},
+        torch_dtype="bfloat16",
+        vocab_size=128,
+    )
+    return SimpleNamespace(
+        model_config=SimpleNamespace(hf_config=cfg, dtype="bf16", max_model_len=4096),
+        cache_config=SimpleNamespace(max_model_len=4096, cache_dtype="fp8"),
+        parallel_config=SimpleNamespace(
+            tensor_parallel_size=1,
+            data_parallel_size=1,
+            enable_expert_parallel=False,
+        ),
+    )
+
+
 def test_non_vllm_workload_emits_measurement_not_vllm_claims(tmp_path: Path, monkeypatch):
     """HFT (no intervention library) must report real kernels, never vLLM knobs."""
     import gitm.scheduler.loop as loop
@@ -243,15 +271,41 @@ def test_vllm_workload_still_uses_intervention_path(tmp_path: Path, monkeypatch)
 
     monkeypatch.setattr(loop, "capture", _fake_capture_with_kernels("paged_attention"))
     monkeypatch.setattr(loop, "sync_device", lambda: None)
+    monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200")
 
     from gitm import optimize
 
     result = optimize(
-        workload="vllm-decode", budget="1s", scratch=str(tmp_path), workload_runner=lambda: {}
+        _priceable_moe_engine(),
+        workload="vllm-decode",
+        budget="1s",
+        scratch=str(tmp_path),
+        workload_runner=lambda: {},
     )
     assert result["summary"]["mode"] == "intervention"
 
 
+def test_vllm_without_live_model_refuses_prediction_claims(tmp_path: Path, monkeypatch):
+    import gitm.scheduler.loop as loop
+
+    monkeypatch.setattr(loop, "capture", _fake_capture_with_kernels("paged_attention"))
+    monkeypatch.setattr(loop, "sync_device", lambda: None)
+    monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200")
+
+    from gitm import optimize
+
+    result = optimize(
+        workload="vllm-decode", budget="1s", scratch=str(tmp_path), workload_runner=lambda: {}
+    )
+
+    assert result["summary"]["status"] == "prediction_refused"
+    assert result["summary"]["mode"] == "measurement"
+    assert result["summary"]["n_claims"] == 0
+    assert "no live engine" in result["report_md"]
+    refusal = Path(result["run_dir"]) / "prediction_refusal.json"
+    assert refusal.exists() and "no live engine" in refusal.read_text()
+
+
 def test_vllm_loop_surfaces_residual_coverage(tmp_path: Path, monkeypatch):
     """Unclassified work must be visible in both the JSON and human report."""
     import json
@@ -268,10 +322,12 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
 
     monkeypatch.setattr(loop, "capture", fake_capture)
     monkeypatch.setattr(loop, "sync_device", lambda: None)
+    monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200")
 
     from gitm import optimize
 
     result = optimize(
+        _priceable_moe_engine(),
         workload="vllm-decode",
         budget="1s",
         scratch=str(tmp_path),
@@ -286,7 +342,9 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
     assert "matched to the predicted graph" in result["report_md"]
 
 
-def test_vllm_loop_runs_autoresearch(tmp_path: Path, monkeypatch):
+def test_vllm_loop_without_model_does_not_run_autoresearch_on_default_graph(
+    tmp_path: Path, monkeypatch
+):
     """The vllm path runs agentic autoresearch: it classifies the bottleneck via
     trace telemetry (the serialized same-stream "paged_attention" kernels are
     also roofline-predicted memory-bound at batch=1, so classification is
@@ -309,16 +367,13 @@ def test_vllm_loop_runs_autoresearch(tmp_path: Path, monkeypatch):
         workload="vllm-decode", budget="30s", scratch=str(tmp_path), workload_runner=lambda: {}
     )
     s = result["summary"]
-    assert s["bottleneck_class"] == "memory_bound"
-    assert s["n_autoresearch"] == 0
-
-    ar_json = (Path(result["run_dir"]) / "autoresearch.json").read_text(encoding="utf-8")
-    # Denylisted knobs with unchecked prerequisites must never surface, regardless
-    # of which bottleneck class the run classifies as.
-    assert "max_num_partial_prefills=" not in ar_json
-    assert "long_prefill_token_threshold=" not in ar_json
+    assert s["status"] == "prediction_refused"
+    assert s["n_claims"] == 0
+    assert "bottleneck_class" not in s
 
-    # Dry-run (no live engine) mutates nothing, so no safety trail is written.
+    # The prediction gate fires before candidate ranking/autoresearch, so neither
+    # artifact can imply that a default graph supported an optimization.
+    assert not (Path(result["run_dir"]) / "autoresearch.json").exists()
     assert not (Path(result["run_dir"]) / "audit.jsonl").exists()
 
 
diff --git a/tests/test_serve_attach.py b/tests/test_serve_attach.py
index 6bddb56..3a848a7 100644
--- a/tests/test_serve_attach.py
+++ b/tests/test_serve_attach.py
@@ -13,6 +13,8 @@
 
 import json
 import os
+from dataclasses import replace
+from types import SimpleNamespace
 
 import pytest
 
@@ -221,6 +223,101 @@ def test_base_url_prefers_explicit_then_flag_then_cmdline():
     )
 
 
+def test_predicted_graph_surfaces_resolved_warnings_and_bytes_fallback(
+    tmp_path, monkeypatch, capsys
+):
+    from gitm.planner.context import peak_for_sku
+    from gitm.planner.moe_graph import spec_from_hf_config
+    from gitm.planner.roofline import BatchConfig, ShardingConfig
+    from gitm.serve import model_config as mc
+
+    cfg = {
+        "model_type": "deepseek_v4",
+        "hidden_size": 64,
+        "num_hidden_layers": 1,
+        "num_attention_heads": 1,
+        "num_key_value_heads": 1,
+        "head_dim": 64,
+        "n_routed_experts": 2,
+        "num_experts_per_tok": 1,
+        "moe_intermediate_size": 32,
+        "expert_dtype": "fp4",
+        "quantization_config": {"quant_method": "fp8"},
+        "torch_dtype": "bfloat16",
+    }
+    spec = replace(spec_from_hf_config(cfg), expert_dtype="future_fp3")
+    resolved = mc.LiveSpec(
+        spec=spec,
+        sharding=ShardingConfig(),
+        batch=BatchConfig(batch=1, kv_cache_len=128),
+        source_path=tmp_path / "config.json",
+        model_ref="org/model",
+        warnings=["decode batch was not observed; using batch=1 single-sequence floor"],
+    )
+    monkeypatch.setattr(mc, "live_moe_spec", lambda _target, **_kwargs: resolved)
+
+    import gitm.planner.context as planner_context
+
+    monkeypatch.setattr(
+        planner_context,
+        "build_planner_context",
+        lambda: SimpleNamespace(peak=peak_for_sku("NVIDIA B200"), sku="NVIDIA B200"),
+    )
+
+    att._emit_predicted_graph(discover.Target(pid=1, cmdline=[]), tmp_path)
+
+    payload = json.loads((tmp_path / "predicted_moe_graph.json").read_text())
+    assert payload["has_fallback_bytes"] is True
+    assert any(node["bytes_are_fallback"] for node in payload["nodes"])
+    assert payload["warnings"]
+    stdout = capsys.readouterr().out
+    assert "single-sequence floor" in stdout
+    assert "unknown-dtype bf16 fallback" in stdout
+
+
+def test_predicted_graph_known_dtypes_leave_bytes_fallback_clean(tmp_path, monkeypatch):
+    from gitm.planner.context import peak_for_sku
+    from gitm.planner.moe_graph import spec_from_hf_config
+    from gitm.planner.roofline import BatchConfig, ShardingConfig
+    from gitm.serve import model_config as mc
+
+    cfg = {
+        "model_type": "deepseek_v4",
+        "hidden_size": 64,
+        "num_hidden_layers": 1,
+        "num_attention_heads": 1,
+        "num_key_value_heads": 1,
+        "head_dim": 64,
+        "n_routed_experts": 2,
+        "num_experts_per_tok": 1,
+        "moe_intermediate_size": 32,
+        "expert_dtype": "fp4",
+        "quantization_config": {"quant_method": "fp8"},
+        "torch_dtype": "bfloat16",
+    }
+    resolved = mc.LiveSpec(
+        spec=spec_from_hf_config(cfg),
+        sharding=ShardingConfig(),
+        batch=BatchConfig(batch=1, kv_cache_len=128),
+        source_path=tmp_path / "config.json",
+        model_ref="org/model",
+    )
+    monkeypatch.setattr(mc, "live_moe_spec", lambda _target, **_kwargs: resolved)
+
+    import gitm.planner.context as planner_context
+
+    monkeypatch.setattr(
+        planner_context,
+        "build_planner_context",
+        lambda: SimpleNamespace(peak=peak_for_sku("NVIDIA B200"), sku="NVIDIA B200"),
+    )
+
+    att._emit_predicted_graph(discover.Target(pid=1, cmdline=[]), tmp_path)
+    payload = json.loads((tmp_path / "predicted_moe_graph.json").read_text())
+    assert payload["has_fallback_bytes"] is False
+    assert not any(node["bytes_are_fallback"] for node in payload["nodes"])
+
+
 # --- preflight ---------------------------------------------------------------
 
 
diff --git a/tests/test_serve_model_config.py b/tests/test_serve_model_config.py
index faab167..7bc99d8 100644
--- a/tests/test_serve_model_config.py
+++ b/tests/test_serve_model_config.py
@@ -91,6 +91,11 @@ def test_deepseek_config_is_usable():
     assert mc.validate_moe_config(_deepseek_cfg()) == []
 
 
+def test_sparse_candidate_with_partial_expert_shape_is_not_misread_as_dense():
+    assert mc.is_sparse_moe_config({"n_routed_experts": 8})
+    assert not mc.is_sparse_moe_config({"model_type": "llama"})
+
+
 def test_mixtral_aliases_are_recognized_not_rejected():
     mixtral = {
         "model_type": "mixtral",
@@ -118,6 +123,18 @@ def test_unpriceable_quant_method_is_refused_not_defaulted():
     assert any("awq" in m for m in missing)
 
 
+def test_final_spec_dtype_validation_covers_every_byte_contributor():
+    from dataclasses import replace
+
+    spec = mc.spec_from_hf_config(_deepseek_cfg())
+    assert mc.validate_priceable_dtypes(spec) == []
+
+    bad = replace(spec, kv_dtype="future_kv3", act_dtype="future_act3")
+    missing = mc.validate_priceable_dtypes(bad)
+    assert any("kv_dtype='future_kv3'" in item for item in missing)
+    assert any("act_dtype='future_act3'" in item for item in missing)
+
+
 # --- serving overrides -------------------------------------------------------
 
 
@@ -173,6 +190,31 @@ def test_live_moe_spec_refuses_unpredictable_config_with_named_keys(tmp_path):
     assert "routed expert count" in r.render()
 
 
+def test_unpriceable_command_line_dtype_is_refused_after_overrides(tmp_path):
+    ckpt = tmp_path / "ckpt"
+    _write_config(ckpt, _deepseek_cfg())
+    target = _target(["vllm", "serve", str(ckpt), "--kv-cache-dtype", "future_kv3"])
+
+    r = mc.live_moe_spec(target, environ={})
+
+    assert isinstance(r, mc.LiveSpecError)
+    assert any("kv_dtype='future_kv3'" in item for item in r.missing_keys)
+
+
+def test_accepted_default_substitutions_are_named_on_live_spec(tmp_path):
+    ckpt = tmp_path / "ckpt"
+    cfg = _deepseek_cfg()
+    cfg.pop("expert_dtype")
+    _write_config(ckpt, cfg)
+
+    r = mc.live_moe_spec(_target(["vllm", "serve", str(ckpt)]), environ={})
+
+    assert isinstance(r, mc.LiveSpec)
+    assert any("expert_dtype absent" in warning for warning in r.warnings)
+    assert any("batch=1" in warning for warning in r.warnings)
+    assert any("kv_cache_len=4096" in warning for warning in r.warnings)
+
+
 def test_live_moe_spec_refuses_when_no_config_found(tmp_path):
     r = mc.live_moe_spec(
         _target(["vllm", "serve", "org/Missing"]),
diff --git a/tests/test_vllm_embodiment.py b/tests/test_vllm_embodiment.py
index 6fa6a22..afd154b 100644
--- a/tests/test_vllm_embodiment.py
+++ b/tests/test_vllm_embodiment.py
@@ -9,6 +9,7 @@
 from __future__ import annotations
 
 from pathlib import Path
+from types import SimpleNamespace
 
 import pytest
 
@@ -115,6 +116,7 @@ def test_hardware_spec_for_uses_detected_peak_not_default():
     l4_peak = HardwarePeak(name="NVIDIA L4", peak_flops=121e12, peak_bw_bytes_s=300e9)
     hw = hardware_spec_for(l4_peak)
     assert hw.name == "NVIDIA L4"
+    assert not hw.is_fallback
     assert hw.peak_flops_fp16_per_s == 121e12
     assert hw.peak_flops_bf16_per_s == 121e12
     assert hw.peak_mem_bw_bytes_per_s == 300e9
@@ -124,7 +126,9 @@ def test_hardware_spec_for_falls_back_to_default_on_unknown_sku():
     from gitm.planner.context import hardware_spec_for
     from gitm.planner.roofline import HardwareSpec
 
-    assert hardware_spec_for(None) == HardwareSpec()
+    hw = hardware_spec_for(None)
+    assert hw == HardwareSpec()
+    assert hw.is_fallback
 
 
 def test_predict_graph_on_l4_predicts_slower_than_default_a100():
@@ -318,7 +322,27 @@ class _LiveEngine:
     """Duck-typed vLLM engine with a restartable max_num_seqs lever."""
 
     def __init__(self, max_num_seqs: int = 32):
-        self.model_config = _ModelConfig()
+        self.model_config = SimpleNamespace(
+            dtype="torch.bfloat16",
+            max_model_len=4096,
+            hf_config=SimpleNamespace(
+                model_type="llama",
+                hidden_size=64,
+                num_hidden_layers=2,
+                num_attention_heads=1,
+                num_key_value_heads=1,
+                head_dim=64,
+                intermediate_size=128,
+                vocab_size=128,
+                torch_dtype="bfloat16",
+            ),
+        )
+        self.cache_config = SimpleNamespace(max_model_len=4096, cache_dtype="bf16")
+        self.parallel_config = SimpleNamespace(
+            tensor_parallel_size=1,
+            data_parallel_size=1,
+            enable_expert_parallel=False,
+        )
         self.max_num_seqs = max_num_seqs
         # Decode throughput scales with batch width, so raising max_num_seqs is a
         # measurable win and is kept. Other structural knobs raise from restart
@@ -351,6 +375,7 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
 
     monkeypatch.setattr(loop, "capture", fake_capture)
     monkeypatch.setattr(loop, "sync_device", lambda: None)
+    monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200")
 
     engine = _LiveEngine()
     cfg = LoopConfig(
@@ -384,8 +409,8 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
     assert summary["n_rolled_back"] >= 1
 
 
-def test_run_loop_no_engine_is_predict_only(tmp_path: Path, monkeypatch):
-    """Without an engine, candidates are unverified (no measured delta), never won."""
+def test_run_loop_no_engine_refuses_default_prediction(tmp_path: Path, monkeypatch):
+    """Without an engine, measurement survives but graph-based claims refuse."""
     from contextlib import contextmanager
 
     import gitm.scheduler.loop as loop
@@ -401,6 +426,7 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
     monkeypatch.setattr(loop, "sync_device", lambda: None)
 
     out = run_loop(LoopConfig(workload="vllm-decode", budget="5s", scratch=str(tmp_path)))
-    assert out["summary"]["status"] == "ok"
-    # No scheduler stats without an engine.
-    assert out["summary"]["scheduler_stats"] is None
+    assert out["summary"]["status"] == "prediction_refused"
+    assert out["summary"]["mode"] == "measurement"
+    assert out["summary"]["n_claims"] == 0
+    assert "no live engine" in out["report_md"]

From 78dac33817a0682169cb35fd3b5c5e5c7ec936b7 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 12:55:49 -0700
Subject: [PATCH 05/70] fix: flag each unpriced roofline dimension

---
 gitm/planner/graph.py          | 22 +++++++++++++---------
 gitm/planner/roofline.py       |  9 +++++++++
 tests/test_moe_graph.py        | 12 ++++++++++++
 tests/test_planner_roofline.py | 11 +++++++++++
 4 files changed, 45 insertions(+), 9 deletions(-)

diff --git a/gitm/planner/graph.py b/gitm/planner/graph.py
index f7dd0ca..59607c8 100644
--- a/gitm/planner/graph.py
+++ b/gitm/planner/graph.py
@@ -121,23 +121,27 @@ def total_pred_s(self) -> float:
 
     @property
     def has_unpriced_nodes(self) -> bool:
-        """True if any node moves bytes but predicts zero time.
+        """True when a positive compute or byte term lacks its denominator.
 
-        This is the general trust net: any missing bandwidth denominator can make
-        real work look free, whether or not the node is a collective.
+        A priced memory term must not hide missing compute throughput (or vice
+        versa), so this cannot be inferred from total predicted time alone.
         """
-        return any(
-            n.prediction.bytes > 0 and n.prediction.t_pred_s == 0.0 for n in self.nodes
-        )
+        return self.has_unpriced_compute or self.has_unpriced_memory
+
+    @property
+    def has_unpriced_compute(self) -> bool:
+        return any(n.prediction.compute_is_unpriced for n in self.nodes)
+
+    @property
+    def has_unpriced_memory(self) -> bool:
+        return any(n.prediction.memory_is_unpriced for n in self.nodes)
 
     @property
     def has_unpriced_collectives(self) -> bool:
         """True if a collective moves bytes but predicts zero time."""
         collective_ops = {"moe_all_to_all", "tp_all_reduce"}
         return any(
-            n.op in collective_ops
-            and n.prediction.bytes > 0
-            and n.prediction.t_pred_s == 0.0
+            n.op in collective_ops and n.prediction.memory_is_unpriced
             for n in self.nodes
         )
 
diff --git a/gitm/planner/roofline.py b/gitm/planner/roofline.py
index 957d253..328b485 100644
--- a/gitm/planner/roofline.py
+++ b/gitm/planner/roofline.py
@@ -512,6 +512,11 @@ class RooflinePrediction:
     # inferred from ``dtype``: a node's compute dtype and its activation, weight,
     # or KV-cache byte contributors may differ.
     bytes_are_fallback: bool = False
+    # Zero catalogue rates retain fail-open arithmetic, but positive work must
+    # never look free. Keep the missing denominator even if the sibling term
+    # still produces a nonzero prediction.
+    compute_is_unpriced: bool = False
+    memory_is_unpriced: bool = False
 
     @property
     def peak_is_fallback(self) -> bool:
@@ -573,6 +578,8 @@ def roofline(
 ) -> RooflinePrediction:
     """Compute the roofline prediction for a single op."""
     peak_flops, peak_dtype = resolve_peak(hw, dtype)
+    compute_is_unpriced = flops > 0 and peak_flops <= 0
+    memory_is_unpriced = bytes_moved > 0 and hw.peak_mem_bw_bytes_per_s <= 0
     t_c = flops / peak_flops if peak_flops > 0 else 0.0
     t_m = bytes_moved / hw.peak_mem_bw_bytes_per_s if hw.peak_mem_bw_bytes_per_s > 0 else 0.0
     bound = "compute" if t_c >= t_m else "memory"
@@ -589,4 +596,6 @@ def roofline(
         peak_flops_per_s=peak_flops,
         estimated=estimated,
         bytes_are_fallback=bytes_are_fallback,
+        compute_is_unpriced=compute_is_unpriced,
+        memory_is_unpriced=memory_is_unpriced,
     )
diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py
index 5a409cd..6bc697a 100644
--- a/tests/test_moe_graph.py
+++ b/tests/test_moe_graph.py
@@ -578,9 +578,21 @@ def test_general_unpriced_node_net_is_not_collective_specific(spec):
     g = Graph(model=spec, hw=hw, batch=BatchConfig(), nodes=[node])
 
     assert g.has_unpriced_nodes
+    assert g.has_unpriced_memory
+    assert not g.has_unpriced_compute
     assert not g.has_unpriced_collectives
 
 
+def test_graph_preserves_each_missing_roofline_denominator(spec):
+    hw = HardwareSpec(peak_flops_fp16_per_s=0.0, peak_mem_bw_bytes_per_s=1e9)
+    node = PredictedNode("partial", None, roofline("partial", 1e12, 1024.0, hw))
+    g = Graph(model=spec, hw=hw, batch=BatchConfig(), nodes=[node])
+
+    assert g.has_unpriced_nodes
+    assert g.has_unpriced_compute
+    assert not g.has_unpriced_memory
+
+
 def test_expert_imbalance_only_applies_under_expert_parallelism(spec, b200):
     """TP is balanced by construction; EP waits for the unluckiest rank."""
     bc = BatchConfig(batch=64, kv_cache_len=4096)
diff --git a/tests/test_planner_roofline.py b/tests/test_planner_roofline.py
index abcd318..ca42e2c 100644
--- a/tests/test_planner_roofline.py
+++ b/tests/test_planner_roofline.py
@@ -111,3 +111,14 @@ def test_roofline_zero_peak_rates_dont_divide_by_zero():
     assert pred.t_compute_s == 0.0
     assert pred.t_memory_s == 0.0
     assert pred.t_pred_s == 0.0
+    assert pred.compute_is_unpriced
+    assert pred.memory_is_unpriced
+
+
+def test_nonzero_memory_time_does_not_hide_missing_compute_rate():
+    hw = HardwareSpec(peak_flops_fp16_per_s=0.0, peak_mem_bw_bytes_per_s=1e9)
+    pred = roofline("partial", flops=1e12, bytes_moved=1e6, hw=hw)
+
+    assert pred.t_pred_s > 0.0
+    assert pred.compute_is_unpriced
+    assert not pred.memory_is_unpriced

From 693bb5ca607a703b014f8f5244b55c466aaf2ddc Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 12:55:53 -0700
Subject: [PATCH 06/70] fix: refuse unqualified benchmark baselines

---
 gitm/bench/baseline.py             | 38 ++++++++++++++++++++++++++----
 gitm/bench/runner.py               | 16 ++++++++++++-
 gitm/bench/schema.py               |  5 ++--
 gitm/bench/templates/results.md.j2 |  6 ++++-
 tests/test_bench.py                | 35 ++++++++++++++++++++++++++-
 5 files changed, 90 insertions(+), 10 deletions(-)

diff --git a/gitm/bench/baseline.py b/gitm/bench/baseline.py
index 2a16f2d..aa2ab2b 100644
--- a/gitm/bench/baseline.py
+++ b/gitm/bench/baseline.py
@@ -47,7 +47,7 @@ class BaselineSummary:
     mean: float
     stddev: float
     spread: float  # (max - min) / mean
-    gpu_active_overall: float  # worst (max) across runs
+    gpu_active_overall: float | None  # worst (max) across runs; None = unmeasured
     recorded: float  # the number we publish = mean
     gates: list[GateResult] = field(default_factory=list)
     seeds: list[int] = field(default_factory=list)
@@ -96,7 +96,9 @@ def aggregate(runs: list[BaselineRun], config: BenchConfig) -> BaselineSummary:
     mean = statistics.fmean(values)
     stddev = statistics.pstdev(values) if len(values) > 1 else 0.0
     spread = (max(values) - min(values)) / mean if mean else float("inf")
-    gpu_overall = max(r.gpu_active_overall() for r in runs)
+    gpu_values = [r.gpu_active_overall() for r in runs]
+    gpu_coverage_ok = all(value is not None for value in gpu_values)
+    gpu_overall = max(value for value in gpu_values if value is not None) if gpu_coverage_ok else None
 
     gates: list[GateResult] = []
 
@@ -119,13 +121,39 @@ def aggregate(runs: list[BaselineRun], config: BenchConfig) -> BaselineSummary:
     )
 
     # Gate 2: saturation / swap rule.
-    sat_ok = gpu_overall < config.gpu_active_ceiling
+    sat_ok = gpu_overall is not None and gpu_overall < config.gpu_active_ceiling
     gates.append(
         GateResult(
             "saturation",
             sat_ok,
-            f"GPU active {gpu_overall:.1%} vs ceiling {config.gpu_active_ceiling:.0%}"
-            + ("" if sat_ok else " — trips swap rule, shard same day"),
+            (
+                f"GPU active {gpu_overall:.1%} vs ceiling {config.gpu_active_ceiling:.0%}"
+                + ("" if sat_ok else " — trips swap rule, shard same day")
+                if gpu_overall is not None
+                else "GPU-active coverage unavailable: at least one run has no positive-wall-time "
+                "stall breakdown; saturation sign-off refused"
+            ),
+        )
+    )
+
+    provenance_issues: list[str] = []
+    for run in runs:
+        if not run.git_sha or run.git_sha == "unknown":
+            provenance_issues.append(f"seed {run.seed}: git SHA unavailable")
+        if not run.manifest_sha256:
+            provenance_issues.append(f"seed {run.seed}: dataset manifest digest unavailable")
+        if not run.gpu_name or run.gpu_name.lower() in {"cpu", "gpu-unknown"}:
+            provenance_issues.append(f"seed {run.seed}: GPU identity unavailable ({run.gpu_name!r})")
+        if run.device_count < 1:
+            provenance_issues.append(f"seed {run.seed}: no GPU devices reported")
+        provenance_issues.extend(f"seed {run.seed}: {note}" for note in run.provenance_warnings)
+    gates.append(
+        GateResult(
+            "provenance",
+            not provenance_issues,
+            "all runs pin code, dataset, and GPU identity"
+            if not provenance_issues
+            else "; ".join(provenance_issues),
         )
     )
 
diff --git a/gitm/bench/runner.py b/gitm/bench/runner.py
index d8637a1..382a353 100644
--- a/gitm/bench/runner.py
+++ b/gitm/bench/runner.py
@@ -92,10 +92,23 @@ def run_seed(
 
     payload = _last_json_line(proc.stdout)
 
+    provenance_warnings: list[str] = []
     manifest_sha = None
     resolved_manifest = _resolve_manifest(config, manifest_path, config_dir)
     if resolved_manifest and Path(resolved_manifest).exists():
         manifest_sha = manifest_digest(resolved_manifest)
+    else:
+        provenance_warnings.append(
+            f"dataset manifest unavailable at {resolved_manifest!s}; baseline cannot be signed off"
+        )
+
+    git_sha = _git_sha()
+    if git_sha == "unknown":
+        provenance_warnings.append("git SHA unavailable; baseline cannot be signed off")
+    if not payload.get("gpu_name"):
+        provenance_warnings.append("harness omitted gpu_name")
+    if payload.get("device_count") is None:
+        provenance_warnings.append("harness omitted device_count; defaulted to 1")
 
     breakdown = [StallPhase.model_validate(p) for p in payload.get("stall_breakdown", [])]
 
@@ -106,10 +119,11 @@ def run_seed(
         metric=config.metric,
         metric_value=float(payload["metric_value"]),
         warm_window_s=config.warm_window_s,
-        git_sha=_git_sha(),
+        git_sha=git_sha,
         gitm_version=__version__,
         harness_commit=payload.get("harness_commit"),
         manifest_sha256=manifest_sha,
+        provenance_warnings=provenance_warnings,
         gpu_name=payload.get("gpu_name", ""),
         device_count=int(payload.get("device_count", 1)),
         started_at_ns=started,
diff --git a/gitm/bench/schema.py b/gitm/bench/schema.py
index da372b8..5a565a8 100644
--- a/gitm/bench/schema.py
+++ b/gitm/bench/schema.py
@@ -148,6 +148,7 @@ class BaselineRun(BaseModel):
     gitm_version: str
     harness_commit: str | None = None
     manifest_sha256: str | None = None  # sha256 of the dataset manifest itself
+    provenance_warnings: list[str] = Field(default_factory=list)
 
     gpu_name: str = ""
     device_count: int = 1
@@ -156,7 +157,7 @@ class BaselineRun(BaseModel):
 
     stall_breakdown: list[StallPhase] = Field(default_factory=list)
 
-    def gpu_active_overall(self) -> float:
+    def gpu_active_overall(self) -> float | None:
         """Wall-clock-weighted GPU active fraction across phases.
 
         This is the number checked against ``gpu_active_ceiling`` — a single
@@ -164,5 +165,5 @@ def gpu_active_overall(self) -> float:
         """
         total = sum(p.wall_clock_s for p in self.stall_breakdown)
         if total <= 0.0:
-            return 0.0
+            return None
         return sum(p.gpu_active * p.wall_clock_s for p in self.stall_breakdown) / total
diff --git a/gitm/bench/templates/results.md.j2 b/gitm/bench/templates/results.md.j2
index 491653a..aa98a84 100644
--- a/gitm/bench/templates/results.md.j2
+++ b/gitm/bench/templates/results.md.j2
@@ -8,7 +8,7 @@
 | Mean | {{ "%.4g"|format(s.mean) }} |
 | Std dev | {{ "%.4g"|format(s.stddev) }} |
 | Spread (max−min)/mean | {{ pct(s.spread) }} |
-| GPU active (worst run) | {{ pct(s.gpu_active_overall) }} |
+| GPU active (worst run) | {{ pct(s.gpu_active_overall) if s.gpu_active_overall is not none else "unmeasured" }} |
 
 ## Sign-off gates
 
@@ -19,7 +19,11 @@
 {% endfor %}
 **Overall: {{ "✅ SIGNED OFF" if s.passed else "❌ NOT SIGNED OFF" }}**
 
+{% if s.gpu_active_overall is none %}
+GPU active coverage is **unmeasured; saturation sign-off refused**.
+{% else %}
 GPU active % {{ pct(s.gpu_active_overall) }} {{ "<" if s.gpu_active_overall < gpu_active_ceiling else "≥" }} ceiling {{ pct(gpu_active_ceiling) }}{% if s.gpu_active_overall >= gpu_active_ceiling %} — **saturated, swap rule applies**{% endif %}.
+{% endif %}
 
 ## Stall breakdown
 {% if breakdown %}
diff --git a/tests/test_bench.py b/tests/test_bench.py
index 96f567a..da4bd36 100644
--- a/tests/test_bench.py
+++ b/tests/test_bench.py
@@ -124,6 +124,9 @@ def _run(seed: int, value: float, gpu: float = 0.7) -> object:
         warm_window_s=60,
         git_sha="abc1234",
         gitm_version="0.0.1",
+        manifest_sha256="manifest123",
+        gpu_name="A100",
+        device_count=1,
         stall_breakdown=[
             StallPhase(phase="all", cpu=0.03, data_stall=max(0.0, 1 - gpu - 0.03 - 0.05),
                        sync=0.05, gpu_active=gpu, throughput=value, wall_clock_s=60.0)
@@ -168,6 +171,36 @@ def test_baseline_fails_on_saturation():
     assert any(g.name == "saturation" and not g.passed for g in summary.gates)
 
 
+def test_baseline_refuses_missing_gpu_activity_instead_of_substituting_zero():
+    from gitm.bench.baseline import aggregate
+
+    runs = [_run(42, 26e6), _run(43, 26e6), _run(44, 26e6)]
+    for run in runs:
+        run.stall_breakdown = []
+
+    summary = aggregate(runs, _hft_config())
+
+    assert summary.gpu_active_overall is None
+    gate = next(g for g in summary.gates if g.name == "saturation")
+    assert not gate.passed
+    assert "coverage unavailable" in gate.detail
+
+
+def test_baseline_refuses_cpu_or_unpinned_runs():
+    from gitm.bench.baseline import aggregate
+
+    runs = [_run(42, 26e6), _run(43, 26e6), _run(44, 26e6)]
+    runs[0].gpu_name = "cpu"
+    runs[1].manifest_sha256 = None
+
+    summary = aggregate(runs, _hft_config())
+
+    gate = next(g for g in summary.gates if g.name == "provenance")
+    assert not gate.passed
+    assert "GPU identity unavailable" in gate.detail
+    assert "manifest digest unavailable" in gate.detail
+
+
 def test_baseline_fails_on_too_few_runs_and_below_target():
     from gitm.bench.baseline import aggregate
 
@@ -326,7 +359,7 @@ def test_run_seed_end_to_end_with_echo_harness(tmp_path: Path, monkeypatch):
         'name="hft"\nvendor="nvidia"\nmetric="events_per_second"\n'
         'warm_window_s=60\nseeds=[42]\n'
         '[dataset]\nroot="hft"\n'
-        f'[work_unit]\ncommand="python {harness} --seed {{seed}}"\n'
+        f'[work_unit]\ncommand="python {harness.as_posix()} --seed {{seed}}"\n'
         '[expected_stall]\ncpu={lo=0,hi=0.05}\ndata_stall={lo=0.1,hi=0.25}\n'
         'sync={lo=0.05,hi=0.15}\ngpu_active={lo=0.6,hi=0.8}\n'
     )

From db892d4405555a4f60cc36971eb03693b2586141 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 12:55:57 -0700
Subject: [PATCH 07/70] fix: distinguish estimated serving token counts

---
 gitm/serve/vllm.py            |  6 ++++-
 gitm/tracer/vllm_stats.py     | 48 ++++++++++++++++++++++++++++++++++-
 tests/test_serve_capture.py   |  1 +
 tests/test_serving_latency.py | 48 +++++++++++++++++++++++++++++++++++
 4 files changed, 101 insertions(+), 2 deletions(-)

diff --git a/gitm/serve/vllm.py b/gitm/serve/vllm.py
index e46007d..5e7a352 100644
--- a/gitm/serve/vllm.py
+++ b/gitm/serve/vllm.py
@@ -544,6 +544,7 @@ def one_request(base: str, model: str, prompt: str, max_tokens: int,
     # usage is authoritative; the chunk count is the fallback when a build doesn't
     # emit it, and it undercounts whenever a chunk carries several tokens.
     rec.n_output_tokens = usage_tokens if usage_tokens is not None else chunks
+    rec.token_count_source = "usage" if usage_tokens is not None else "chunks"
     return rec
 
 
@@ -759,8 +760,11 @@ def launch_and_capture(args, serve_argv: list[str] | None = None):
 
     print(f"\n==> {len(records)} ok / {failures} failed in {wall:.1f}s")
     if summary.ttft_p50_s is not None:
+        tpot = f"{summary.tpot_p50_s * 1e3:.1f} ms" if summary.tpot_p50_s is not None else "n/a"
         print(f"    TTFT p50/p95 {summary.ttft_p50_s * 1e3:.0f}/{summary.ttft_p95_s * 1e3:.0f} ms"
-              f"   TPOT p50 {(summary.tpot_p50_s or 0) * 1e3:.1f} ms")
+              f"   TPOT p50 {tpot}")
+    for warning in summary.warnings:
+        print(f"    WARN: {warning}")
     print_result(result)
 
     if result.status == "no_kernels":
diff --git a/gitm/tracer/vllm_stats.py b/gitm/tracer/vllm_stats.py
index 0fc4418..8f2bf22 100644
--- a/gitm/tracer/vllm_stats.py
+++ b/gitm/tracer/vllm_stats.py
@@ -89,6 +89,14 @@ class RequestRecord:
     first_token_wall_s: float | None = None
     finished_wall_s: float | None = None
     n_output_tokens: int = 0
+    # ``authoritative`` means usage metadata or engine token_ids; ``chunks`` is
+    # the SSE-event fallback (one event can carry multiple tokens); ``unknown``
+    # means no trustworthy denominator exists.
+    token_count_source: str = "unknown"
+
+    @property
+    def token_count_authoritative(self) -> bool:
+        return self.token_count_source in {"usage", "engine"}
 
     @property
     def ttft_s(self) -> float | None:
@@ -106,6 +114,8 @@ def tpot_s(self) -> float | None:
         inter-token interval at all and yields ``None`` rather than a number
         derived from a zero-length gap.
         """
+        if not self.token_count_authoritative:
+            return None
         if self.first_token_wall_s is None or self.finished_wall_s is None:
             return None
         if self.n_output_tokens < 2:
@@ -113,6 +123,16 @@ def tpot_s(self) -> float | None:
         span = max(self.finished_wall_s - self.first_token_wall_s, 0.0)
         return span / (self.n_output_tokens - 1)
 
+    @property
+    def estimated_tpot_s(self) -> float | None:
+        """Chunk-count TPOT estimate, never admitted to SLO goodput."""
+        if self.token_count_source != "chunks" or self.n_output_tokens < 2:
+            return None
+        if self.first_token_wall_s is None or self.finished_wall_s is None:
+            return None
+        span = max(self.finished_wall_s - self.first_token_wall_s, 0.0)
+        return span / (self.n_output_tokens - 1)
+
     def meets_slo(self, ttft_slo_s: float, tpot_slo_s: float) -> bool:
         """True if every *measurable* latency component cleared its SLO.
 
@@ -124,8 +144,14 @@ def meets_slo(self, ttft_slo_s: float, tpot_slo_s: float) -> bool:
         ttft = self.ttft_s
         if ttft is None or ttft > ttft_slo_s:
             return False
+        if not self.token_count_authoritative:
+            return False
+        if self.n_output_tokens == 1:
+            return True
+        if self.n_output_tokens < 1:
+            return False
         tpot = self.tpot_s
-        return tpot is None or tpot <= tpot_slo_s
+        return tpot is not None and tpot <= tpot_slo_s
 
 
 @dataclass
@@ -141,17 +167,20 @@ class ServingSummary:
     n_requests: int
     n_ttft: int
     n_tpot: int
+    n_tpot_estimated: int
     ttft_p50_s: float | None
     ttft_p95_s: float | None
     ttft_p99_s: float | None
     tpot_p50_s: float | None
     tpot_p95_s: float | None
     tpot_p99_s: float | None
+    tpot_estimated_p50_s: float | None
     n_met_slo: int
     goodput_rps: float | None  # SLO-meeting requests per second over the window
     window_s: float | None
     ttft_slo_s: float
     tpot_slo_s: float
+    warnings: list[str]
 
 
 def _percentile(vals: list[float], q: float) -> float | None:
@@ -182,6 +211,7 @@ def request_records_from_outputs(outputs: Any) -> list[RequestRecord]:
         if outs:
             try:
                 rec.n_output_tokens = len(outs[0].token_ids)
+                rec.token_count_source = "engine"
             except (AttributeError, TypeError, IndexError):
                 pass
         metrics = getattr(o, "metrics", None)
@@ -212,6 +242,7 @@ def summarize_requests(
     """
     ttfts = [t for t in (r.ttft_s for r in records) if t is not None]
     tpots = [t for t in (r.tpot_s for r in records) if t is not None]
+    estimated_tpots = [t for t in (r.estimated_tpot_s for r in records) if t is not None]
     met = [r for r in records if r.meets_slo(ttft_slo_s, tpot_slo_s)]
 
     arrivals = [r.arrival_wall_s for r in records if r.arrival_wall_s is not None]
@@ -223,21 +254,36 @@ def summarize_requests(
     # has no meaningful rate — report the count, not a division by ~0.
     goodput = (len(met) / window_s) if window_s else None
 
+    warnings: list[str] = []
+    if estimated_tpots:
+        warnings.append(
+            f"TPOT coverage: {len(estimated_tpots)} request(s) use SSE chunk-count estimates; "
+            "excluded from authoritative TPOT percentiles and SLO goodput"
+        )
+    missing_counts = sum(1 for r in records if r.token_count_source == "unknown")
+    if missing_counts:
+        warnings.append(
+            f"TPOT coverage: {missing_counts} request(s) have no authoritative output-token count"
+        )
+
     return ServingSummary(
         n_requests=len(records),
         n_ttft=len(ttfts),
         n_tpot=len(tpots),
+        n_tpot_estimated=len(estimated_tpots),
         ttft_p50_s=_percentile(ttfts, 0.50),
         ttft_p95_s=_percentile(ttfts, 0.95),
         ttft_p99_s=_percentile(ttfts, 0.99),
         tpot_p50_s=_percentile(tpots, 0.50),
         tpot_p95_s=_percentile(tpots, 0.95),
         tpot_p99_s=_percentile(tpots, 0.99),
+        tpot_estimated_p50_s=_percentile(estimated_tpots, 0.50),
         n_met_slo=len(met),
         goodput_rps=goodput,
         window_s=window_s,
         ttft_slo_s=ttft_slo_s,
         tpot_slo_s=tpot_slo_s,
+        warnings=warnings,
     )
 
 
diff --git a/tests/test_serve_capture.py b/tests/test_serve_capture.py
index bc656ae..6cc253c 100644
--- a/tests/test_serve_capture.py
+++ b/tests/test_serve_capture.py
@@ -94,6 +94,7 @@ def test_chunk_count_is_the_fallback_when_usage_is_absent(sse_server):
     rec = sc.one_request(base, "m", "p", max_tokens=4, ignore_eos=True, timeout_s=30)
 
     assert rec is not None and rec.n_output_tokens == 3
+    assert rec.token_count_source == "chunks"
 
 
 def test_unreachable_server_returns_none_not_a_fake_record():
diff --git a/tests/test_serving_latency.py b/tests/test_serving_latency.py
index 89db662..e0c2b6a 100644
--- a/tests/test_serving_latency.py
+++ b/tests/test_serving_latency.py
@@ -17,6 +17,7 @@ def _rec(arrival, first, finished, n_tokens):
         first_token_wall_s=first,
         finished_wall_s=finished,
         n_output_tokens=n_tokens,
+        token_count_source="engine",
     )
 
 
@@ -113,6 +114,52 @@ def test_single_token_request_not_penalised_for_absent_tpot():
     assert s.n_met_slo == 1
 
 
+def test_chunk_count_tpot_is_flagged_and_excluded_from_goodput():
+    record = RequestRecord(
+        arrival_wall_s=0.0,
+        first_token_wall_s=0.1,
+        finished_wall_s=1.1,
+        n_output_tokens=3,
+        token_count_source="chunks",
+    )
+
+    summary = summarize_requests([record], ttft_slo_s=1.0, tpot_slo_s=1.0)
+
+    assert record.tpot_s is None
+    assert record.estimated_tpot_s == 0.5
+    assert summary.n_tpot == 0
+    assert summary.n_tpot_estimated == 1
+    assert summary.n_met_slo == 0
+    assert summary.goodput_rps == 0.0
+    assert any("excluded" in warning for warning in summary.warnings)
+
+
+def test_unknown_token_count_cannot_pass_tpot_slo_by_absence():
+    record = RequestRecord(
+        arrival_wall_s=0.0,
+        first_token_wall_s=0.1,
+        finished_wall_s=1.0,
+        n_output_tokens=0,
+    )
+
+    summary = summarize_requests([record])
+
+    assert summary.n_met_slo == 0
+    assert any("no authoritative" in warning for warning in summary.warnings)
+
+
+def test_authoritative_zero_token_count_cannot_pass_slo():
+    record = RequestRecord(
+        arrival_wall_s=0.0,
+        first_token_wall_s=0.1,
+        finished_wall_s=0.1,
+        n_output_tokens=0,
+        token_count_source="usage",
+    )
+
+    assert not record.meets_slo(ttft_slo_s=1.0, tpot_slo_s=1.0)
+
+
 # --------------------------------------------------------------------------- #
 # vLLM RequestOutput adaptation                                               #
 # --------------------------------------------------------------------------- #
@@ -123,6 +170,7 @@ def test_records_from_vllm_outputs():
     )
     (rec,) = request_records_from_outputs([out])
     assert rec.n_output_tokens == 3
+    assert rec.token_count_source == "engine"
     assert rec.ttft_s == 0.5
 
 

From 9d19721c3fd68e0d742b3ab270fadf0bd32498ab Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 12:56:07 -0700
Subject: [PATCH 08/70] fix: surface degraded runtime measurements

---
 gitm/benchmarks/edge/baseline.py       |  18 ++-
 gitm/benchmarks/kitti/baseline.py      |  79 +++++++++----
 gitm/optimizer/attribution.py          |  33 +++++-
 gitm/optimizer/dr.py                   |  42 ++++++-
 gitm/optimizer/headroom_kernel_rank.py |  48 +++++---
 gitm/optimizer/measure.py              |  49 ++++++--
 gitm/runtime_driver.py                 | 151 ++++++++-----------------
 gitm/telemetry/collector.py            |  34 +++++-
 tests/test_headroom_kernel_rank.py     |  41 +++++++
 tests/test_runtime_driver.py           |  47 ++++++++
 tests/test_runtime_on_trace.py         |  36 ++++++
 tests/test_telemetry_fallbacks.py      |  36 ++++++
 12 files changed, 438 insertions(+), 176 deletions(-)
 create mode 100644 tests/test_headroom_kernel_rank.py
 create mode 100644 tests/test_runtime_driver.py
 create mode 100644 tests/test_telemetry_fallbacks.py

diff --git a/gitm/benchmarks/edge/baseline.py b/gitm/benchmarks/edge/baseline.py
index cfaf57a..54c2e56 100644
--- a/gitm/benchmarks/edge/baseline.py
+++ b/gitm/benchmarks/edge/baseline.py
@@ -46,7 +46,9 @@
 GPU_ACTIVE_WARN_PCT = 85.0
 
 
-def _sample_nvml(samples: list[float], stop_event: threading.Event) -> None:
+def _sample_nvml(
+    samples: list[float], stop_event: threading.Event, diagnostics: list[str]
+) -> None:
     """Background thread: sample GPU 0 utilization at NVML_SAMPLE_HZ."""
     try:
         import pynvml
@@ -59,8 +61,10 @@ def _sample_nvml(samples: list[float], stop_event: threading.Event) -> None:
             samples.append(float(util))
             time.sleep(interval)
         pynvml.nvmlShutdown()
-    except Exception:
-        pass  # no GPU or pynvml absent -> nvml_mean_pct null in output
+    except Exception as exc:
+        diagnostics.append(
+            f"NVML telemetry unavailable: {type(exc).__name__}: {exc}"
+        )
 
 
 def _convergence_check(results: list[dict]) -> tuple[bool, float]:
@@ -148,9 +152,12 @@ def run_baseline(
 
     warm_indices = run_indices[WARM_FRAMES:]
     nvml_samples: list[float] = []
+    telemetry_diagnostics: list[str] = []
     stop_event = threading.Event()
     nvml_thread = threading.Thread(
-        target=_sample_nvml, args=(nvml_samples, stop_event), daemon=True
+        target=_sample_nvml,
+        args=(nvml_samples, stop_event, telemetry_diagnostics),
+        daemon=True,
     )
     nvml_thread.start()
 
@@ -194,6 +201,7 @@ def run_baseline(
         "sync_stall_pct": round(sync_stall_pct, 2),
         "cpu_pct": round(cpu_pct, 2),
         "nvml_mean_util_pct": round(nvml_mean, 2) if nvml_mean is not None else None,
+        "telemetry_diagnostics": telemetry_diagnostics,
         "total_detections": total_detections,
         "elapsed_s": round(elapsed, 3),
         "max_sweeps": max_sweeps,
@@ -222,6 +230,8 @@ def run_baseline(
         else f"\nResult: {fps:.1f} fps | GPU active {gpu_active_pct:.1f}%"
     )
     print(f"Wrote {output_path}")
+    for diagnostic in telemetry_diagnostics:
+        print(f"WARNING: {diagnostic}")
 
     if nvml_mean is not None and nvml_mean > GPU_ACTIVE_WARN_PCT:
         print(
diff --git a/gitm/benchmarks/kitti/baseline.py b/gitm/benchmarks/kitti/baseline.py
index 2971d36..0f2281b 100644
--- a/gitm/benchmarks/kitti/baseline.py
+++ b/gitm/benchmarks/kitti/baseline.py
@@ -124,7 +124,9 @@ def _load_frame_paths(data_root: Path) -> list[Path]:
     return paths
 
 
-def _sample_nvml(samples: list[dict], stop_event: threading.Event) -> None:
+def _sample_nvml(
+    samples: list[dict], stop_event: threading.Event, diagnostics: list[str]
+) -> None:
     """Background thread: sample GPU 0 util + memory at NVML_SAMPLE_HZ.
 
     Each sample is a dict with util_pct, mem_used_bytes, mem_total_bytes —
@@ -146,8 +148,10 @@ def _sample_nvml(samples: list[dict], stop_event: threading.Event) -> None:
             })
             time.sleep(interval)
         pynvml.nvmlShutdown()
-    except Exception:
-        pass  # no GPU or pynvml absent — headroom fields will be null in output
+    except Exception as exc:
+        diagnostics.append(
+            f"NVML telemetry unavailable: {type(exc).__name__}: {exc}"
+        )
 
 
 def _convergence_check(results: list[dict]) -> tuple[bool, float]:
@@ -230,9 +234,12 @@ def run_baseline(
     # Timed warm window
     warm_paths = run_paths[WARM_FRAMES:]
     nvml_samples: list[dict] = []
+    headroom_diagnostics: list[str] = []
     stop_event = threading.Event()
     nvml_thread = threading.Thread(
-        target=_sample_nvml, args=(nvml_samples, stop_event), daemon=True
+        target=_sample_nvml,
+        args=(nvml_samples, stop_event, headroom_diagnostics),
+        daemon=True,
     )
     nvml_thread.start()
 
@@ -286,17 +293,35 @@ def run_baseline(
         from gitm.optimizer.headroom_kernel_rank import gpu_headroom
 
         h = gpu_headroom(nvml_samples)
-        if h is not None:
-            headroom_dict = {
-                "compute_headroom_pct": round(h.compute_headroom_pct, 2),
-                "mean_util_pct": round(h.mean_util_pct, 2),
-                "peak_util_pct": round(h.peak_util_pct, 2),
-                "mem_free_at_peak_gb": round(h.mem_free_at_peak_bytes / 1e9, 3),
-                "mem_total_gb": round(h.mem_total_bytes / 1e9, 3),
-                "nvml_n_samples": h.n_samples,
-            }
-    except Exception:
-        pass  # graceful degradation if headroom_kernel_rank unavailable
+        headroom_dict = {
+            "compute_headroom_pct": (
+                round(h.compute_headroom_pct, 2)
+                if h.compute_headroom_pct is not None
+                else None
+            ),
+            "mean_util_pct": (
+                round(h.mean_util_pct, 2) if h.mean_util_pct is not None else None
+            ),
+            "peak_util_pct": (
+                round(h.peak_util_pct, 2) if h.peak_util_pct is not None else None
+            ),
+            "mem_free_at_peak_gb": (
+                round(h.mem_free_at_peak_bytes / 1e9, 3)
+                if h.mem_free_at_peak_bytes is not None
+                else None
+            ),
+            "mem_total_gb": (
+                round(h.mem_total_bytes / 1e9, 3)
+                if h.mem_total_bytes is not None
+                else None
+            ),
+            "nvml_n_samples": h.n_samples,
+        }
+        headroom_diagnostics.extend(h.diagnostics)
+    except Exception as exc:
+        headroom_diagnostics.append(
+            f"GPU headroom analysis unavailable: {type(exc).__name__}: {exc}"
+        )
 
     import platform
     import socket
@@ -310,6 +335,7 @@ def run_baseline(
         "sync_stall_pct": round(sync_stall_pct, 2),
         "cpu_pct": round(cpu_pct, 2),
         **headroom_dict,
+        "headroom_diagnostics": headroom_diagnostics,
         "stage_spread": spread,
         "total_detections": total_detections,
         "elapsed_s": round(elapsed, 3),
@@ -329,13 +355,24 @@ def run_baseline(
         f"\nResult: {fps:.1f} fps | GPU active {gpu_active_pct:.1f}% "
         f"| data stall {data_stall_pct:.1f}% | wrote {output_path}"
     )
-    if headroom_dict:
-        print(
-            f"Headroom: compute {headroom_dict['compute_headroom_pct']:.1f}% free "
-            f"| mem {headroom_dict['mem_free_at_peak_gb']:.1f} GB free at peak"
+    headroom_parts: list[str] = []
+    if headroom_dict.get("compute_headroom_pct") is not None:
+        headroom_parts.append(
+            f"compute {headroom_dict['compute_headroom_pct']:.1f}% free"
         )
-
-    if headroom_dict.get("mean_util_pct", 0) > 85.0:
+    if headroom_dict.get("mem_free_at_peak_gb") is not None:
+        headroom_parts.append(
+            f"mem {headroom_dict['mem_free_at_peak_gb']:.1f} GB free at peak"
+        )
+    if headroom_parts:
+        print(f"Headroom: {' | '.join(headroom_parts)}")
+    for diagnostic in headroom_diagnostics:
+        print(f"WARNING: {diagnostic}")
+
+    if (
+        headroom_dict.get("mean_util_pct") is not None
+        and headroom_dict["mean_util_pct"] > 85.0
+    ):
         print(
             f"\nWARNING: NVML util={headroom_dict['mean_util_pct']:.1f}% > 85% — "
             "flag for review: workload may be near-saturated. Consider the 500-frame fallback."
diff --git a/gitm/optimizer/attribution.py b/gitm/optimizer/attribution.py
index db41cf3..d50f110 100644
--- a/gitm/optimizer/attribution.py
+++ b/gitm/optimizer/attribution.py
@@ -10,7 +10,7 @@
 from __future__ import annotations
 
 import warnings
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 
 import numpy as np
 
@@ -30,6 +30,7 @@ class Hypothesis:
 @dataclass
 class RankedHypotheses:
     hypotheses: list[Hypothesis]
+    diagnostics: list[str] = field(default_factory=list)
 
     def top(self, n: int = 5) -> list[Hypothesis]:
         return self.hypotheses[:n]
@@ -54,8 +55,14 @@ def attribute(
         from statsmodels.tsa.stattools import (
             grangercausalitytests,  # type: ignore[import-not-found]
         )
-    except Exception:
-        return RankedHypotheses(hypotheses=[])
+    except Exception as exc:
+        return RankedHypotheses(
+            hypotheses=[],
+            diagnostics=[
+                f"Granger attribution unavailable: statsmodels import failed "
+                f"({type(exc).__name__}: {exc})"
+            ],
+        )
 
     # Group residuals by op into ordered time series (per layer-position step)
     series: dict[str, list[float]] = {}
@@ -64,10 +71,17 @@ def attribute(
 
     ops = [op for op, vals in series.items() if len(vals) >= max_lag + 2]
     if len(ops) < 2:
-        return RankedHypotheses(hypotheses=[])
+        return RankedHypotheses(
+            hypotheses=[],
+            diagnostics=[
+                f"Granger attribution not run: need at least 2 op series with "
+                f"{max_lag + 2}+ samples; found {len(ops)}"
+            ],
+        )
 
     n = min(len(series[op]) for op in ops)
     hypotheses: list[Hypothesis] = []
+    fit_failures: list[str] = []
     for cause in ops:
         for effect in ops:
             if cause == effect:
@@ -79,7 +93,8 @@ def attribute(
                     result = grangercausalitytests(arr, maxlag=max_lag, verbose=False)
                 pvals = [result[lag][0]["ssr_ftest"][1] for lag in range(1, max_lag + 1)]
                 p = float(min(pvals))
-            except Exception:
+            except Exception as exc:
+                fit_failures.append(f"{cause}->{effect}: {type(exc).__name__}: {exc}")
                 continue
             direction = "+ slower" if np.mean(series[cause]) > 0 else "- faster"
             hypotheses.append(
@@ -87,4 +102,10 @@ def attribute(
             )
 
     hypotheses.sort(key=lambda h: h.p_value)
-    return RankedHypotheses(hypotheses=hypotheses)
+    diagnostics = []
+    if fit_failures:
+        diagnostics.append(
+            f"Granger attribution skipped {len(fit_failures)} failed pair fit(s); "
+            f"first: {fit_failures[0]}"
+        )
+    return RankedHypotheses(hypotheses=hypotheses, diagnostics=diagnostics)
diff --git a/gitm/optimizer/dr.py b/gitm/optimizer/dr.py
index 0311403..7bd0b6e 100644
--- a/gitm/optimizer/dr.py
+++ b/gitm/optimizer/dr.py
@@ -52,7 +52,13 @@ class DREffect:
     n_treated: int
 
 
-def doubly_robust_ate(y: np.ndarray, t: np.ndarray, X: np.ndarray) -> tuple[float, float]:
+def doubly_robust_ate(
+    y: np.ndarray,
+    t: np.ndarray,
+    X: np.ndarray,
+    *,
+    diagnostics: list[str] | None = None,
+) -> tuple[float, float]:
     """AIPW estimate of the ATE of ``t`` on ``y`` given covariates ``X``.
 
     Returns ``(ate, se)``. Robust to a misspecified outcome *or* propensity model.
@@ -81,7 +87,11 @@ def doubly_robust_ate(y: np.ndarray, t: np.ndarray, X: np.ndarray) -> tuple[floa
         # Propensity e = P(T=1|X); clip away from 0/1 to bound the IPW weights.
         try:
             e = sm.Logit(t, Xc).fit(disp=0).predict(Xc)
-        except Exception:
+        except Exception as exc:
+            if diagnostics is not None:
+                diagnostics.append(
+                    f"propensity model fell back to treated mean ({type(exc).__name__})"
+                )
             e = np.full(n, t.mean())
         e = np.clip(e, 0.05, 0.95)
 
@@ -91,7 +101,11 @@ def _outcome(mask: np.ndarray) -> np.ndarray:
                 return np.full(n, float(y[mask].mean()) if mask.any() else 0.0)
             try:
                 return sm.OLS(y[mask], Xc[mask]).fit().predict(Xc)
-            except Exception:
+            except Exception as exc:
+                if diagnostics is not None:
+                    diagnostics.append(
+                        f"outcome model fell back to group mean ({type(exc).__name__})"
+                    )
                 return np.full(n, float(y[mask].mean()))
 
         m1 = _outcome(t == 1)
@@ -118,11 +132,18 @@ def attribute_dr(residuals: Residuals, graph: Graph, *, band: float = _KT_BAND)
 
     ops = [op for op, v in series.items() if len(v) >= 4]
     if len(ops) < 2:
-        return RankedHypotheses(hypotheses=[])
+        return RankedHypotheses(
+            hypotheses=[],
+            diagnostics=[
+                f"doubly-robust attribution not run: need at least 2 op series with "
+                f"4+ samples; found {len(ops)}"
+            ],
+        )
     n = min(len(series[op]) for op in ops)
     pos = np.arange(n, dtype=float)
 
     effects: list[DREffect] = []
+    diagnostics: list[str] = []
     for cause in ops:
         t = (np.abs(np.asarray(series[cause][:n])) > band).astype(float)
         n_t = int(t.sum())
@@ -132,7 +153,11 @@ def attribute_dr(residuals: Residuals, graph: Graph, *, band: float = _KT_BAND)
             if effect == cause:
                 continue
             y = np.asarray(series[effect][:n], dtype=float)
-            ate, se = doubly_robust_ate(y, t, pos)
+            pair_diagnostics: list[str] = []
+            ate, se = doubly_robust_ate(y, t, pos, diagnostics=pair_diagnostics)
+            diagnostics.extend(
+                f"{cause}->{effect}: {note}" for note in pair_diagnostics
+            )
             z = ate / se if se not in (0.0, float("inf")) else 0.0
             effects.append(DREffect(cause, effect, ate, se, z, int(t.sum())))
 
@@ -151,4 +176,9 @@ def attribute_dr(residuals: Residuals, graph: Graph, *, band: float = _KT_BAND)
         )
         for d in effects
     ]
-    return RankedHypotheses(hypotheses=hyps)
+    if diagnostics:
+        diagnostics = [
+            f"doubly-robust attribution used {len(diagnostics)} nuisance-model fallback(s); "
+            f"first: {diagnostics[0]}"
+        ]
+    return RankedHypotheses(hypotheses=hyps, diagnostics=diagnostics)
diff --git a/gitm/optimizer/headroom_kernel_rank.py b/gitm/optimizer/headroom_kernel_rank.py
index aff48da..dfbc9db 100644
--- a/gitm/optimizer/headroom_kernel_rank.py
+++ b/gitm/optimizer/headroom_kernel_rank.py
@@ -19,7 +19,7 @@
 from __future__ import annotations
 
 import re
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 
 # Mangled CUDA kernel names -> a small set of stable families, so durations
 # aggregate per kernel *type* rather than per template instantiation. Shared
@@ -116,39 +116,53 @@ def render_roi_table(rows: list[KernelROI], *, floor_pct: float = 10.0, top: int
 
 @dataclass
 class GpuHeadroom:
-    mean_util_pct: float
-    peak_util_pct: float
-    compute_headroom_pct: float # 100 - mean util (coarse, time-based)
-    peak_mem_used_bytes: int
-    mem_total_bytes: int
-    mem_free_at_peak_bytes: int
+    mean_util_pct: float | None
+    peak_util_pct: float | None
+    compute_headroom_pct: float | None # 100 - mean util (coarse, time-based)
+    peak_mem_used_bytes: int | None
+    mem_total_bytes: int | None
+    mem_free_at_peak_bytes: int | None
     serialized_concurrency_fraction: float # fraction of kernel-time with no overlap
     n_samples: int
+    diagnostics: list[str] = field(default_factory=list)
 
-def gpu_headroom(samples, serialized_concurrency_fraction: float = 0.0) -> GpuHeadroom | None:
+def gpu_headroom(samples, serialized_concurrency_fraction: float = 0.0) -> GpuHeadroom:
     """Summarise GPU headroom from a list of telemetry sample dicts.
 
     Each sample is a dict with ``util_pct`` / ``mem_used_bytes`` /
     ``mem_total_bytes`` (the canonical :class:`gitm.telemetry.Sample` fields).
-    Returns ``None`` if there are no usable samples.
+    Missing metric families remain ``None`` and are named in ``diagnostics``;
+    memory-only telemetry must not become 100% compute headroom, or vice versa.
     """
     utils = [float(s["util_pct"]) for s in samples if s.get("util_pct") is not None]
     mems = [int(s["mem_used_bytes"]) for s in samples if s.get("mem_used_bytes") is not None]
-    total = next((int(s["mem_total_bytes"]) for s in samples if s.get("mem_total_bytes")), 0)
-    if not utils and not mems:
-        return None
-    mean_u = sum(utils) / len(utils) if utils else 0.0
-    peak_u = max(utils) if utils else 0.0
-    peak_m = max(mems) if mems else 0
+    totals = [int(s["mem_total_bytes"]) for s in samples if s.get("mem_total_bytes")]
+    diagnostics: list[str] = []
+    if not samples:
+        diagnostics.append("GPU headroom unavailable: telemetry contains no samples")
+    if not utils:
+        diagnostics.append("compute headroom unavailable: utilization telemetry is absent")
+    if not mems:
+        diagnostics.append("memory headroom unavailable: used-memory telemetry is absent")
+    if not totals:
+        diagnostics.append("memory headroom unavailable: total-memory telemetry is absent")
+    mean_u = sum(utils) / len(utils) if utils else None
+    peak_u = max(utils) if utils else None
+    peak_m = max(mems) if mems else None
+    total = max(totals) if totals else None
+    mem_free = (
+        max(0, total - peak_m) if total is not None and peak_m is not None else None
+    )
     return GpuHeadroom(
         mean_util_pct=mean_u,
         peak_util_pct=peak_u,
-        compute_headroom_pct=max(0.0, 100.0 - mean_u),
+        compute_headroom_pct=max(0.0, 100.0 - mean_u) if mean_u is not None else None,
         peak_mem_used_bytes=peak_m,
         mem_total_bytes=total,
-        mem_free_at_peak_bytes=max(0, total - peak_m),
+        mem_free_at_peak_bytes=mem_free,
         serialized_concurrency_fraction=serialized_concurrency_fraction,
         n_samples=len(samples),
+        diagnostics=diagnostics,
     )
 
 def live_gpu_headroom():
diff --git a/gitm/optimizer/measure.py b/gitm/optimizer/measure.py
index 574a31b..d08ec63 100644
--- a/gitm/optimizer/measure.py
+++ b/gitm/optimizer/measure.py
@@ -59,6 +59,8 @@ class MeasureResult:
     violations: list = field(default_factory=list)
     top_hypotheses: list = field(default_factory=list)
     families: list[str] = field(default_factory=list)
+    n_invalid_duration: int = 0
+    diagnostics: list[str] = field(default_factory=list)
 
 
 def measure_trace(trace: Trace, *, min_attr: int = 16) -> MeasureResult:
@@ -71,42 +73,68 @@ def measure_trace(trace: Trace, *, min_attr: int = 16) -> MeasureResult:
     kernels = trace.kernels()
     memcpys = [e for e in trace.events if e.kind == "memcpy"]
     if not kernels:
-        return MeasureResult(0, len(memcpys), 0.0)
+        return MeasureResult(
+            0,
+            len(memcpys),
+            0.0,
+            diagnostics=["measurement coverage unavailable: trace contains no kernels"],
+        )
+
+    valid_kernels = [k for k in kernels if k.end_ns - k.start_ns > 0]
+    n_invalid = len(kernels) - len(valid_kernels)
+    diagnostics: list[str] = []
+    if n_invalid:
+        diagnostics.append(
+            f"measurement coverage: excluded {n_invalid}/{len(kernels)} kernel(s) "
+            "with non-positive duration"
+        )
+    if not valid_kernels:
+        return MeasureResult(
+            len(kernels),
+            len(memcpys),
+            0.0,
+            n_invalid_duration=n_invalid,
+            diagnostics=diagnostics,
+        )
 
-    sc = _serialized_fraction(kernels)
+    sc = _serialized_fraction(valid_kernels)
     by_name: dict[str, list[int]] = {}
-    for k in kernels:
+    for k in valid_kernels:
         by_name.setdefault(k.name, []).append(k.end_ns - k.start_ns)
     med = {nm: float(np.median(v)) for nm, v in by_name.items()}
 
     res = Residuals()
     res.serialized_concurrency_fraction = sc
-    for k in kernels:
-        m = med[k.name] or 1.0
+    for k in valid_kernels:
+        m = med[k.name]
         res.per_kernel.append(
             KernelResidual(op=k.name[:40], layer=None, r_kt=((k.end_ns - k.start_ns) - m) / m, r_mt=None)
         )
     violations = check_invariants(res, multi_basis=True)
 
     fam_of = {nm: kernel_family(nm) for nm in by_name}
-    fam_counts = Counter(fam_of[k.name] for k in kernels)
+    fam_counts = Counter(fam_of[k.name] for k in valid_kernels)
     res_attr = Residuals()
     res_attr.serialized_concurrency_fraction = sc
-    for k in kernels:
+    for k in valid_kernels:
         fam = fam_of[k.name]
         if fam_counts[fam] < min_attr:
             continue
-        m = med[k.name] or 1.0
+        m = med[k.name]
         res_attr.per_kernel.append(
             KernelResidual(op=fam, layer=None, r_kt=((k.end_ns - k.start_ns) - m) / m, r_mt=None)
         )
     families = sorted({kr.op for kr in res_attr.per_kernel})
-    top_hyps = list(attribute(res_attr, predict_graph()).top(5)) if res_attr.per_kernel else []
+    ranked = attribute(res_attr, predict_graph())
+    top_hyps = list(ranked.top(5)) if res_attr.per_kernel else []
+    diagnostics.extend(ranked.diagnostics)
 
     return MeasureResult(
         n_kernels=len(kernels),
         n_memcpy=len(memcpys),
         serialized_fraction=sc,
+        n_invalid_duration=n_invalid,
+        diagnostics=diagnostics,
         violations=violations,
         top_hypotheses=top_hyps,
         families=families,
@@ -143,10 +171,11 @@ def measurement_claims(result: MeasureResult, *, limit: int = 5) -> list[Claim]:
 
 def measurement_summary(workload: str, result: MeasureResult) -> str:
     fams = ", ".join(result.families[:6]) or "none with enough samples"
+    diagnostic = f" Coverage diagnostics: {'; '.join(result.diagnostics)}." if result.diagnostics else ""
     return (
         f"Measurement run for {workload!r}: {result.n_kernels:,} kernels "
         f"({result.n_memcpy:,} memcpy) captured, {len(result.violations)} invariant "
         f"deviation(s), serialized-concurrency={result.serialized_fraction:.3f}. "
         f"Kernel families: {fams}. No interventions applied — this workload has no "
-        f"tuned intervention library, so the runtime reports what it measured."
+        f"tuned intervention library, so the runtime reports what it measured.{diagnostic}"
     )
diff --git a/gitm/runtime_driver.py b/gitm/runtime_driver.py
index bd8749f..9cd43d9 100644
--- a/gitm/runtime_driver.py
+++ b/gitm/runtime_driver.py
@@ -27,7 +27,6 @@
 import argparse
 import json
 import os
-import re
 import time
 from contextlib import closing
 from pathlib import Path
@@ -42,25 +41,6 @@ def _sync():
         pass
 
 
-# Mangled CUDA kernel names → a small set of stable "families" so the residual
-# series feeding Granger are well-populated (a variable = a kernel *type*, not
-# every template instantiation). Noise tokens are dropped; the first distinctive
-# identifier after the library prefix names the family.
-_NOISE = {
-    "detail", "kernel", "void", "const", "unsigned", "int", "long", "float",
-    "double", "global", "device", "functor", "impl", "internal", "type",
-    "types", "common", "native", "operator", "policy", "dispatch", "agent",
-}
-
-
-def _kernel_family(name: str) -> str:
-    lib = ("cub" if "cub" in name else "cudf" if "cudf" in name
-           else "thrust" if "thrust" in name else "k")
-    toks = [t for t in re.findall(r"[a-z][a-z_]{3,}", name) if t not in _NOISE]
-    fn = toks[0] if toks else "anon"
-    return f"{lib}.{fn}"
-
-
 def _load_hft(stage: Path, seed: int, max_events: int | None):
     from gitm.benchmarks.hft.harness import _gpu_name, load_events, run_pipeline, select_backend
     from gitm.benchmarks.hft.optimize import run_pipeline_fast, verify_equivalent
@@ -307,18 +287,9 @@ def main(argv: list[str] | None = None) -> int:
             raise SystemExit("--optimize currently supports --workload hft")
         return _optimize_hft_cmd(args)
 
-    import numpy as np
-
     from gitm import __version__
-    from gitm.optimizer.attribution import attribute
-    from gitm.optimizer.monitor import (
-        KernelResidual,
-        Residuals,
-        _serialized_fraction,
-        check_invariants,
-    )
-    from gitm.optimizer.report import Claim, Provenance, write_report
-    from gitm.planner.graph import predict_graph
+    from gitm.optimizer.measure import measure_trace, measurement_claims
+    from gitm.optimizer.report import Provenance, write_report
     from gitm.tracer import capture
 
     stage = args.stage or Path(os.environ.get("GITM_BENCH_STAGE", "/workspace/hft/staging/hft"))
@@ -347,15 +318,19 @@ def main(argv: list[str] | None = None) -> int:
     trace_path = args.outdir / f"{args.workload}_seed{args.seed}_trace.jsonl"
     tele_path = args.outdir / f"{args.workload}_seed{args.seed}_telemetry.jsonl"
 
-    # State telemetry — best-effort; never block the run on it.
+    # State telemetry remains fail-open, but every degradation is carried into
+    # the measurement artifact and report.
     tele = None
+    telemetry_diagnostics: list[str] = []
     try:
         from gitm.telemetry import Collector, CollectorConfig
         from gitm.telemetry.sinks import build_sink
 
         tele = Collector(CollectorConfig(interval_s=0.25, sinks=[build_sink(f"jsonl:{tele_path}")]))
     except Exception as exc:
-        print(f"telemetry disabled (best-effort): {exc}")
+        diagnostic = f"telemetry unavailable: {type(exc).__name__}: {exc}"
+        telemetry_diagnostics.append(diagnostic)
+        print(f"WARN: {diagnostic}")
 
     started_ns = time.time_ns()
     if tele:
@@ -368,6 +343,7 @@ def main(argv: list[str] | None = None) -> int:
         elapsed = max(time.perf_counter() - t0, 1e-9)
     if tele:
         tele.stop()
+        telemetry_diagnostics.extend(tele.diagnostics)
     ended_ns = time.time_ns()
 
     units = summary.get("events", summary.get("frames", 0))
@@ -403,60 +379,27 @@ def main(argv: list[str] | None = None) -> int:
         "trace_path": str(trace_path),
     }
 
-    violations = []
-    top_hyps: list = []
-    sc = 0.0
-    if kernels:
-        sc = _serialized_fraction(kernels)
-        by_name: dict[str, list[int]] = {}
-        for k in kernels:
-            by_name.setdefault(k.name, []).append(k.end_ns - k.start_ns)
-        med = {nm: float(np.median(v)) for nm, v in by_name.items()}
-        res = Residuals()
-        res.serialized_concurrency_fraction = sc
-        for k in kernels:
-            m = med[k.name] or 1.0
-            res.per_kernel.append(
-                KernelResidual(op=k.name[:40], layer=None, r_kt=((k.end_ns - k.start_ns) - m) / m, r_mt=None)
-            )
-        v_mb = check_invariants(res, multi_basis=True)
-        v_raw = check_invariants(res, multi_basis=False)
-        violations = v_mb
-        print(
-            f"serialized_concurrency_fraction = {sc:.3f}  |  "
-            f"violations multi-basis={len(v_mb)} raw={len(v_raw)} "
-            f"(filter dropped {len(v_raw) - len(v_mb)})"
-        )
-        # Attribution: group kernels into families (see _kernel_family) and keep
-        # only families with enough samples, so Granger's per-op residual series
-        # are well-formed. attribute() truncates all series to the shortest, so a
-        # sparse op would otherwise collapse every pair to too few points.
-        from collections import Counter
-
-        fam_of = {nm: _kernel_family(nm) for nm in by_name}
-        fam_counts = Counter(fam_of[k.name] for k in kernels)
-        MIN_ATTR = 16
-        res_attr = Residuals()
-        res_attr.serialized_concurrency_fraction = sc
-        for k in kernels:
-            fam = fam_of[k.name]
-            if fam_counts[fam] < MIN_ATTR:
-                continue
-            m = med[k.name] or 1.0
-            res_attr.per_kernel.append(
-                KernelResidual(op=fam, layer=None, r_kt=((k.end_ns - k.start_ns) - m) / m, r_mt=None)
-            )
-        fams = sorted({kr.op for kr in res_attr.per_kernel})
-        print(f"attribution families (>= {MIN_ATTR} samples): {fams}")
-        ranked = attribute(res_attr, predict_graph())
-        top_hyps = ranked.top(5)
-        print(
-            "top Granger hypotheses:",
-            [(h.cause_op, h.effect_op, round(h.p_value, 4)) for h in top_hyps] or "none",
-        )
+    measured = measure_trace(tr)
+    measured.diagnostics.extend(telemetry_diagnostics)
+    violations = measured.violations
+    top_hyps = measured.top_hypotheses
+    sc = measured.serialized_fraction
+    print(
+        f"serialized_concurrency_fraction = {sc:.3f}  |  "
+        f"violations multi-basis={len(violations)}"
+    )
+    print(f"attribution families (>= 16 samples): {measured.families}")
+    print(
+        "top Granger hypotheses:",
+        [(h.cause_op, h.effect_op, round(h.p_value, 4)) for h in top_hyps] or "none",
+    )
+    for diagnostic in measured.diagnostics:
+        print(f"WARN: {diagnostic}")
 
     measure["serialized_concurrency_fraction"] = sc
     measure["n_violations"] = len(violations)
+    measure["n_invalid_duration"] = measured.n_invalid_duration
+    measure["diagnostics"] = measured.diagnostics
     measure["top_hypotheses"] = [
         {"cause": h.cause_op, "effect": h.effect_op, "p_value": h.p_value} for h in top_hyps
     ]
@@ -480,25 +423,8 @@ def main(argv: list[str] | None = None) -> int:
         ended_at_ns=ended_ns,
         trace_path=str(trace_path),
     )
-    claims: list[Claim] = []
-    for v in violations[:5]:
-        ev = (
-            f"top hypothesis: {top_hyps[0].cause_op[:30]} -> {top_hyps[0].effect_op[:30]} "
-            f"(p={top_hyps[0].p_value:.3g})"
-            if top_hyps
-            else "no ranked hypothesis"
-        )
-        claims.append(
-            Claim(
-                summary=f"{v.invariant} deviation on {v.node_op}",
-                residual_invariant=v.invariant,
-                residual_value=float(v.residual),
-                causal_evidence=ev,
-                intervention_name="(none — measurement run)",
-                predicted_delta=0.0,
-                measured_delta=None,
-            )
-        )
+    claims = measurement_claims(measured)
+    kernel_coverage_available = measured.n_kernels > measured.n_invalid_duration
     if args.workload == "hft":
         run_summary = (
             f"HFT cuDF/CuPy on {gpu_name}: {events_per_second:,.0f} events/s over {n:,} events; "
@@ -515,15 +441,28 @@ def main(argv: list[str] | None = None) -> int:
     report_md = write_report(
         claims,
         prov,
-        qualification_diagnostic="Measurement-only run: runtime observed the workload; "
-        "no intervention library applied.",
+        qualification_diagnostic=(
+            "NO DATA — tracer captured no positive-duration GPU kernels; "
+            "workload throughput was measured, "
+            "but runtime kernel details were not."
+            if not kernel_coverage_available
+            else "Measurement-only run: runtime observed the workload; "
+            "no intervention library applied."
+        ),
+        runtime_diagnostics=measured.diagnostics,
         summary=run_summary,
     )
     report_path = args.outdir / f"{args.workload}_seed{args.seed}_report.md"
     report_path.write_text(report_md)
     print(f"\nwrote: {trace_path}\n       {tele_path}\n       {report_path}\n       "
           f"{args.outdir / f'{args.workload}_seed{args.seed}_measure.json'}")
-    print("PASS: workload ran under the runtime; all details measured.")
+    if not kernel_coverage_available:
+        print(
+            "FAIL: workload ran, but no positive-duration GPU kernels were captured; "
+            "runtime details unavailable."
+        )
+        return 3
+    print("PASS: workload ran under the runtime; kernel details measured.")
     return 0
 
 
diff --git a/gitm/telemetry/collector.py b/gitm/telemetry/collector.py
index 4af45f1..18f3eb3 100644
--- a/gitm/telemetry/collector.py
+++ b/gitm/telemetry/collector.py
@@ -7,6 +7,7 @@
 
 import threading
 import time
+import warnings
 from dataclasses import dataclass, field
 
 from gitm.telemetry.backends import Backend, discover_backends
@@ -34,6 +35,10 @@ class Collector:
     def __init__(self, cfg: CollectorConfig) -> None:
         self._cfg = cfg
         self._backends: list[Backend] = cfg.backends if cfg.backends is not None else discover_backends()
+        self.diagnostics: list[str] = []
+        self._diagnostic_keys: set[str] = set()
+        if not self._backends:
+            self._record_failure("backend-discovery", "no live GPU telemetry backend found")
         self._stop = threading.Event()
         self._thread: threading.Thread | None = None
 
@@ -50,13 +55,22 @@ def stop(self) -> None:
         for s in self._cfg.sinks:
             try:
                 s.close()
-            except Exception:
-                pass
+            except Exception as exc:
+                self._record_failure(type(s).__name__, f"sink close failed: {exc}")
         for b in self._backends:
             try:
                 b.close()
-            except Exception:
-                pass
+            except Exception as exc:
+                self._record_failure(type(b).__name__, f"backend close failed: {exc}")
+
+    def _record_failure(self, key: str, detail: str) -> None:
+        """Record and warn once per failing component/path."""
+        if key in self._diagnostic_keys:
+            return
+        self._diagnostic_keys.add(key)
+        message = f"telemetry degraded [{key}]: {detail}"
+        self.diagnostics.append(message)
+        warnings.warn(message, RuntimeWarning, stacklevel=2)
 
     def __enter__(self) -> Collector:
         self.start()
@@ -72,12 +86,20 @@ def _run(self) -> None:
                 for idx in range(backend.device_count()):
                     try:
                         sample = backend.sample(idx, labels=self._cfg.labels)
-                    except Exception:
+                    except Exception as exc:
+                        self._record_failure(
+                            f"sample:{type(backend).__name__}:{idx}",
+                            f"sample failed ({type(exc).__name__}: {exc})",
+                        )
                         continue
                     for sink in self._cfg.sinks:
                         try:
                             sink.emit(sample)
-                        except Exception:
+                        except Exception as exc:
+                            self._record_failure(
+                                f"sink:{type(sink).__name__}",
+                                f"emit failed ({type(exc).__name__}: {exc})",
+                            )
                             continue
             next_tick += self._cfg.interval_s
             sleep_for = max(0.0, next_tick - time.monotonic())
diff --git a/tests/test_headroom_kernel_rank.py b/tests/test_headroom_kernel_rank.py
new file mode 100644
index 0000000..21b36e2
--- /dev/null
+++ b/tests/test_headroom_kernel_rank.py
@@ -0,0 +1,41 @@
+from __future__ import annotations
+
+from gitm.optimizer.headroom_kernel_rank import gpu_headroom
+
+
+def test_memory_only_telemetry_does_not_fabricate_compute_headroom():
+    result = gpu_headroom(
+        [{"mem_used_bytes": 4_000, "mem_total_bytes": 10_000}]
+    )
+
+    assert result.compute_headroom_pct is None
+    assert result.mean_util_pct is None
+    assert result.mem_free_at_peak_bytes == 6_000
+    assert any("utilization telemetry is absent" in note for note in result.diagnostics)
+
+
+def test_utilization_only_telemetry_does_not_fabricate_memory_headroom():
+    result = gpu_headroom([{"util_pct": 65.0}])
+
+    assert result.compute_headroom_pct == 35.0
+    assert result.mem_free_at_peak_bytes is None
+    assert result.mem_total_bytes is None
+    assert any("memory headroom unavailable" in note for note in result.diagnostics)
+
+
+def test_complete_headroom_sample_stays_clean():
+    result = gpu_headroom(
+        [{"util_pct": 65.0, "mem_used_bytes": 4_000, "mem_total_bytes": 10_000}]
+    )
+
+    assert result.compute_headroom_pct == 35.0
+    assert result.mem_free_at_peak_bytes == 6_000
+    assert result.diagnostics == []
+
+
+def test_empty_headroom_input_is_explicitly_unavailable():
+    result = gpu_headroom([])
+
+    assert result.compute_headroom_pct is None
+    assert result.mem_free_at_peak_bytes is None
+    assert any("no samples" in note for note in result.diagnostics)
diff --git a/tests/test_runtime_driver.py b/tests/test_runtime_driver.py
new file mode 100644
index 0000000..812da60
--- /dev/null
+++ b/tests/test_runtime_driver.py
@@ -0,0 +1,47 @@
+from __future__ import annotations
+
+from contextlib import contextmanager
+
+
+def test_runtime_driver_refuses_empty_trace_instead_of_printing_pass(tmp_path, monkeypatch, capsys):
+    from gitm import runtime_driver
+    from gitm.tracer.schema import Trace
+
+    monkeypatch.setattr(
+        runtime_driver,
+        "_load_hft",
+        lambda *_args, **_kwargs: (
+            lambda: {"events": 10, "vwap_buckets": 1},
+            10,
+            "cpu",
+            "cpu",
+            0,
+        ),
+    )
+
+    @contextmanager
+    def empty_capture(*_args, **_kwargs):
+        yield Trace(
+            workload_id="hft",
+            fingerprint="f",
+            run_id="r",
+            device_count=0,
+            vendor="none",
+            captured_at_ns=0,
+            duration_ns=1,
+            source="none",
+            events=[],
+        )
+
+    monkeypatch.setattr("gitm.tracer.capture", empty_capture)
+
+    rc = runtime_driver.main(
+        ["--workload", "hft", "--stage", str(tmp_path), "--outdir", str(tmp_path)]
+    )
+
+    output = capsys.readouterr().out
+    assert rc == 3
+    assert "runtime details unavailable" in output
+    assert "all details measured" not in output
+    report = (tmp_path / "hft_seed42_report.md").read_text()
+    assert "NO DATA" in report
diff --git a/tests/test_runtime_on_trace.py b/tests/test_runtime_on_trace.py
index 11a0ceb..fd311db 100644
--- a/tests/test_runtime_on_trace.py
+++ b/tests/test_runtime_on_trace.py
@@ -205,6 +205,42 @@ def test_doubly_robust_degenerate_inputs():
     assert ate == 0.0 and se == float("inf")
 
 
+def test_attribution_abstention_is_diagnostic_not_no_causal_signal():
+    from gitm.optimizer.attribution import attribute
+    from gitm.optimizer.dr import attribute_dr
+    from gitm.optimizer.monitor import KernelResidual, Residuals
+    from gitm.planner.graph import predict_graph
+
+    res = Residuals(per_kernel=[KernelResidual("only", None, 0.5, None)])
+
+    granger = attribute(res, predict_graph())
+    dr = attribute_dr(res, predict_graph())
+
+    assert granger.hypotheses == []
+    assert any("not run" in note for note in granger.diagnostics)
+    assert dr.hypotheses == []
+    assert any("not run" in note for note in dr.diagnostics)
+
+
+def test_measure_trace_excludes_zero_duration_kernels_with_diagnostic():
+    from gitm.optimizer.measure import measure_trace
+
+    trace = _trace(
+        [
+            _kernel("bad", start=10, end=10),
+            _kernel("good", start=20, end=30),
+            _kernel("good", start=40, end=50),
+        ]
+    )
+
+    result = measure_trace(trace, min_attr=1)
+
+    assert result.n_kernels == 3
+    assert result.n_invalid_duration == 1
+    assert any("excluded 1/3" in note for note in result.diagnostics)
+    assert all(v.node_op != "bad" for v in result.violations)
+
+
 def test_attribute_dr_ranks_pairs():
     from gitm.optimizer.dr import attribute_dr
     from gitm.optimizer.monitor import KernelResidual, Residuals
diff --git a/tests/test_telemetry_fallbacks.py b/tests/test_telemetry_fallbacks.py
new file mode 100644
index 0000000..99531c4
--- /dev/null
+++ b/tests/test_telemetry_fallbacks.py
@@ -0,0 +1,36 @@
+from __future__ import annotations
+
+import time
+
+import pytest
+
+from gitm.telemetry.collector import Collector, CollectorConfig
+
+
+class _BrokenBackend:
+    def device_count(self):
+        return 1
+
+    def sample(self, _index, labels=None):
+        raise RuntimeError("nvml read failed")
+
+    def close(self):
+        return None
+
+
+def test_collector_surfaces_background_sample_failure_once():
+    collector = Collector(CollectorConfig(interval_s=0.001, backends=[_BrokenBackend()]))
+
+    with pytest.warns(RuntimeWarning, match="sample failed"):
+        collector.start()
+        time.sleep(0.02)
+        collector.stop()
+
+    assert len([d for d in collector.diagnostics if "sample failed" in d]) == 1
+
+
+def test_collector_names_missing_backend_instead_of_looking_idle():
+    with pytest.warns(RuntimeWarning, match="no live GPU telemetry backend"):
+        collector = Collector(CollectorConfig(backends=[]))
+
+    assert collector.diagnostics

From 7c2b81a38e14aa1a932a3930aa6650c8ec03ef36 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 12:58:32 -0700
Subject: [PATCH 09/70] fix: surface runtime fallback diagnostics

---
 AUDIT.md                        |  36 +++++++---
 gitm/kernels/library.py         |   4 +-
 gitm/scheduler/loop.py          | 124 +++++++++++++++++++++++++++-----
 gitm/serve/attach.py            |  13 +++-
 tests/test_edge_optimize.py     |  53 ++++++++++++--
 tests/test_gate_wiring.py       |  10 +++
 tests/test_run_loop_workload.py |  33 +++++++++
 7 files changed, 235 insertions(+), 38 deletions(-)

diff --git a/AUDIT.md b/AUDIT.md
index 2a7114a..80def0a 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -7,7 +7,7 @@ Status: **in progress**. This ledger is the primary deliverable for the audit of
 turn missing knowledge into a confident wrong result, with answer-deciding byte
 traffic and dominant expert terms ranked above non-binding estimates.
 
-Highest-severity masks closed: **9 so far**. Wiring gaps confirmed: **3 so far**.
+Highest-severity masks closed: **19 so far**. Wiring gaps confirmed: **7 so far**.
 Deferred findings: **none so far**.
 
 The worktree already contained uncommitted scheduler/serve changes and two new
@@ -29,6 +29,16 @@ they will not be silently absorbed into an audit commit.
 | 8 | fixed | medium | sparse config resolution in loop and attach | Expert weights, often the dominant term | WARN | Missing `expert_dtype` inherited linear `weight_dtype`. Official V4 Flash configs declare it (Flash=`fp4`, Base=`fp8`), but uniform foreign MoEs may omit it legitimately. | Accepted inheritance now rides in `LiveSpec.warnings` / loop diagnostics and reaches artifacts plus human output; unpriceable inherited dtypes still refuse. |
 | 9 | fixed | medium | `gitm/planner/graph.py` | Zero-time byte-moving nodes | FLAG | `has_unpriced_collectives` scanned all nodes despite its narrow name. | Split `has_unpriced_nodes` (general safety net) from the genuinely collective-specific property; production trust consumers use the general flag and artifacts retain both. |
 | 10 | fixed | high | loop `predicted_graph.json`, summary, and Markdown diagnostics | Prediction trust diagnostics | FLAG/WARN | Loop artifacts omitted fallback, estimate, default, model, batch, sharding, and hardware provenance. | Artifact now carries model source, observed/pricing hardware, batch/sharding, graph flags, per-node diagnostics, and warnings; the report and run summary consume them. |
+| 11 | fixed | critical | `gitm/bench/schema.py`; `gitm/bench/baseline.py` | Benchmark saturation/sign-off gate | REFUSE | A missing `stall_breakdown` became 0% GPU active and could sign off a CPU/no-telemetry run as unsaturated. | Missing coverage is now `None` and fails saturation; code, manifest, and GPU identity have a separate provenance gate and safe report rendering. |
+| 12 | fixed | high | `gitm/tracer/vllm_stats.py`; `gitm/serve/vllm.py` | TPOT percentiles and SLO goodput | FLAG/WARN | SSE chunk counts undercounted multi-token chunks but entered TPOT and goodput as authoritative; missing counts passed the TPOT half of the SLO. | Usage/engine counts are authoritative, chunk estimates are excluded from TPOT/goodput, and coverage warnings reach CLI and loop reports. |
+| 13 | fixed | high | `gitm/planner/roofline.py`; `gitm/planner/graph.py` | Compute/HBM denominator | FLAG/WARN | Positive work with a zero catalogue rate was priced at zero and a sibling term could hide the missing denominator. | Per-dimension unpriced flags survive on each prediction, aggregate on the graph, and surface through loop/attach artifacts and diagnostics. |
+| 14 | fixed | high | `gitm/runtime_driver.py` | Trace coverage and every claimed runtime detail | REFUSE/WARN | Zero captured kernels still produced `PASS: ... all details measured`. | The driver now uses canonical measurement, emits NO DATA, records diagnostics, and exits 3 without any positive-duration kernel. |
+| 15 | fixed | high | `gitm/optimizer/attribution.py`; `gitm/optimizer/dr.py`; loop/driver/report consumers | Causal evidence | WARN | Import/fit failure became `no strong causal signal`; DR nuisance-model fallbacks emitted estimates silently. | Attribution carries import, sample-coverage, pair-fit, and nuisance-model diagnostics into JSON, CLI, and Markdown. |
+| 16 | fixed | high | `gitm/kernels/library.py`; scheduler caller | Intervention availability / candidate coverage | REFUSE/WARN | A missing library returned `[]`, indistinguishable from no applicable levers. | The loader refuses with the path; the scheduler emits a named candidate-coverage-unavailable measurement report and no optimization claims. |
+| 17 | fixed | medium | `gitm/telemetry/collector.py`; benchmark samplers and runtime-driver consumer | State-telemetry coverage | WARN | Sampling, sink, and close failures were swallowed, making empty telemetry look like a quiet GPU. | Collector failures are deduplicated warnings and report diagnostics; benchmark sampler failures ride into JSON and stdout. |
+| 18 | fixed | high | scheduler specialized HFT/OpenFold/edge intervention result paths | Residual and intervention status | FLAG/REFUSE | No CUPTI trace attached A/B speedup to fabricated `stream_concurrency=0.0`; missing A/B still reported `ok`. | All siblings use measured throughput delta without trace coverage; missing/non-finite A/B emits no claim and returns `intervention_failed`. |
+| 19 | fixed | medium | `gitm/optimizer/headroom_kernel_rank.py` | Compute and memory headroom | FLAG/WARN | Memory-only samples fabricated 100% compute headroom; utilization-only samples fabricated zero memory capacity. | Each dimension is optional, absent families stay `None`, and diagnostics name missing telemetry. |
+| 20 | fixed | medium | `gitm/optimizer/measure.py`; duplicated runtime-driver measurement | Kernel residual denominator | WARN/REFUSE | Zero-duration kernels used a fabricated 1 ns median and attribution filtering had no coverage diagnostic. | Invalid durations are excluded with counts, attribution abstention is diagnostic, and both consumers use canonical measurement. |
 
 Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`.
 
@@ -51,19 +61,19 @@ Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`.
 |---|---|---|---|
 | top-level runtime / API / CLI / workloads | Pending | Pending | |
 | agents | Pending | Pending | |
-| bench | Pending | Pending | |
-| benchmarks | Pending | Pending | |
+| bench | In progress | In progress | Saturation and provenance sign-off gates swept/fixed; remaining CLI/results paths under review. |
+| benchmarks | In progress | In progress | KITTI/edge telemetry fallbacks fixed; remaining harnesses under review. |
 | deploy | Pending | Pending | |
 | importers | Pending | Pending | |
 | kernels | Pending | Pending | |
-| optimizer | Pending | Pending | |
-| planner | Pending | Pending | |
+| optimizer | In progress | In progress | Attribution, headroom, and measurement masks fixed; apply/safety-audit paths remain under review. |
+| planner | In progress | In progress | Seed and denominator paths swept/fixed; dead KITTI planner path remains under review. |
 | routing | Pending | Pending | |
 | safety | Pending | Pending | |
-| scheduler | Pending | Pending | |
-| serve | Pending | Pending | |
-| telemetry | Pending | Pending | |
-| tracer | Pending | Pending | |
+| scheduler | In progress | In progress | Main vLLM and specialized intervention siblings swept/fixed; remaining orchestration fallbacks under review. |
+| serve | In progress | In progress | Launch/attach gates and token provenance swept/fixed; remaining CLI paths under review. |
+| telemetry | In progress | In progress | Optional fields and collector/backend/sink failures now surface; remaining call-site consumers under review. |
+| tracer | In progress | In progress | Capture backend failures warn/source-flag; scheduler sampling and request-summary fallbacks under review. |
 | scripts | Pending | Pending | |
 
 ## Diagnostic-consumer trace
@@ -74,7 +84,13 @@ human- or gate-visible consumer.
 
 | Producer | Diagnostic | Downstream consumer | User/gate boundary | Status |
 |---|---|---|---|---|
-| — | — | — | — | Inventory pending |
+| `RooflinePrediction` / `Graph` | peak, bytes, hardware fallback; estimated; per-dimension unpriced nodes | loop and attach serializers/diagnostics | JSON + Markdown/CLI | Traced/fixed |
+| `Residuals` | coverage counts/warnings | loop residual JSON, summary, report diagnostics | JSON + Markdown | Fixed |
+| `ImportStats` / importer rollup | warnings, drops, caveats, SKU/time provenance | analyze summary + customer report | JSON + Markdown | Traced |
+| `CaptureResult` / kernel taxonomy | warnings and capture status | serve artifacts + CLI | JSON + CLI exit | Traced |
+| `ServingSummary` | TTFT/TPOT sample counts and token-provenance warnings | serve/loop artifacts | JSON + CLI/Markdown | Traced/fixed |
+| `Collector` / `GpuHeadroom` | component failures and missing metric-family diagnostics | runtime driver and benchmark artifacts | warning + JSON/Markdown/stdout | Traced/fixed |
+| `FailOpenGuard` | revert failures | `failures` attribute + audit log | programmatic/audit artifact | Revert failures traced; broken audit-sink fallback under review |
 
 ## Sibling-path validation matrix
 
diff --git a/gitm/kernels/library.py b/gitm/kernels/library.py
index d0fae97..f3bd4d3 100644
--- a/gitm/kernels/library.py
+++ b/gitm/kernels/library.py
@@ -17,7 +17,9 @@ def load_library(path: Path | str | None = None, *, workload: str | None = None)
     """Load and validate every entry in the library."""
     p = Path(path) if path is not None else _library_path()
     if not p.exists():
-        return []
+        raise FileNotFoundError(
+            f"intervention library not found at {p}; candidate coverage is unavailable"
+        )
     with p.open() as fh:
         raw = yaml.safe_load(fh) or {}
     entries = raw.get("interventions", [])
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 02fc528..40f3221 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -9,6 +9,7 @@
 from __future__ import annotations
 
 import json
+import math
 import os
 import re
 import time
@@ -492,7 +493,12 @@ def _execution_graph(engine: Any, pctx: Any, sched: Any) -> ExecutionGraphResolu
     if graph.has_fallback_bytes:
         diagnostics.append("one or more nodes use fallback byte widths")
     if graph.has_unpriced_nodes:
-        diagnostics.append("one or more byte-moving nodes have no priceable bandwidth")
+        missing = []
+        if graph.has_unpriced_compute:
+            missing.append("compute throughput")
+        if graph.has_unpriced_memory:
+            missing.append("memory bandwidth")
+        diagnostics.append(f"one or more predicted nodes have unpriced {' and '.join(missing)}")
     n_estimated = sum(1 for node in graph.nodes if node.prediction.estimated)
     if n_estimated:
         diagnostics.append(f"{n_estimated} predicted node(s) use estimated cost models")
@@ -835,6 +841,8 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
                 },
                 "has_unpriced_collectives": graph.has_unpriced_collectives,
                 "has_unpriced_nodes": graph.has_unpriced_nodes,
+                "has_unpriced_compute": graph.has_unpriced_compute,
+                "has_unpriced_memory": graph.has_unpriced_memory,
                 "has_fallback_peaks": graph.has_fallback_peaks,
                 "has_fallback_bytes": graph.has_fallback_bytes,
                 "diagnostics": graph_resolution.diagnostics,
@@ -845,6 +853,8 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
                         "estimated": node.prediction.estimated,
                         "peak_is_fallback": node.prediction.peak_is_fallback,
                         "bytes_are_fallback": node.prediction.bytes_are_fallback,
+                        "compute_is_unpriced": node.prediction.compute_is_unpriced,
+                        "memory_is_unpriced": node.prediction.memory_is_unpriced,
                     }
                     for node in graph.nodes
                 ],
@@ -900,6 +910,9 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
                      "notes": h.notes}
                     for h in dr_hypotheses.top(5)
                 ],
+                "attribution_diagnostics": (
+                    hypotheses.diagnostics + dr_hypotheses.diagnostics
+                ),
                 # Engine-scheduler causes (from the vLLM stats adapter) ranked
                 # alongside the kernel-level hypotheses (the engine-signal causal link).
                 "scheduler_causes": [
@@ -936,9 +949,25 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
     # pctx was built earlier (Phase 1) so its hardware peak could feed predict_graph.
     # Relative/swept levers resolve against the live engine here, once, before
     # ranking. See expand_relative_candidates.
+    try:
+        raw_library = load_library(workload=workload)
+    except (FileNotFoundError, ValueError) as exc:
+        diagnostic = f"intervention candidate coverage unavailable: {type(exc).__name__}: {exc}"
+        return _measurement_result(
+            run_dir=run_dir,
+            run_id=run_id,
+            workload=workload,
+            trace=trace,
+            qual=qual,
+            started_ns=started_ns,
+            trace_path=trace_path,
+            diagnostic=diagnostic,
+            runtime_diagnostics=graph_resolution.diagnostics + [diagnostic],
+            status="candidate_coverage_unavailable",
+        )
     library = [
         resolved
-        for s in load_library(workload=workload)
+        for s in raw_library
         for resolved in expand_relative_candidates(s, cfg.engine)
     ]
     policy = Policy(require_qualification_commit=qual.commit, skip_high_risk=not qual.commit)
@@ -1177,7 +1206,13 @@ def _unenactable(spec: Any) -> str | None:
         claims=claims,
         provenance=provenance,
         qualification_diagnostic=qual.diagnostic,
-        runtime_diagnostics=graph_resolution.diagnostics + res.coverage_warnings,
+        runtime_diagnostics=(
+            graph_resolution.diagnostics
+            + res.coverage_warnings
+            + (serving_summary.warnings if serving_summary is not None else [])
+            + hypotheses.diagnostics
+            + dr_hypotheses.diagnostics
+        ),
         summary=(
             f"vLLM decode on {pctx.sku or 'unknown SKU'}: {len(claims)} candidate(s) "
             f"evaluated, {len(rolled_back)} rolled back. {sched_note}"
@@ -1235,6 +1270,8 @@ def _measurement_result(
             {
                 "n_kernels": result.n_kernels,
                 "n_memcpy": result.n_memcpy,
+                "n_invalid_duration": result.n_invalid_duration,
+                "diagnostics": result.diagnostics,
                 "serialized_concurrency_fraction": result.serialized_fraction,
                 "n_violations": len(result.violations),
                 "families": result.families,
@@ -1261,7 +1298,7 @@ def _measurement_result(
             "Measurement-only run: the runtime observed the workload and reports "
             "its real kernels. No intervention library applies to this workload."
         ),
-        runtime_diagnostics=runtime_diagnostics,
+        runtime_diagnostics=(runtime_diagnostics or []) + result.diagnostics,
         summary=measurement_summary(workload, result),
     )
     _write_report(run_dir, report_md)
@@ -1285,6 +1322,31 @@ def _measurement_result(
     return {"summary": summary, "report_md": report_md, "run_dir": str(run_dir)}
 
 
+def _specialized_claim_basis(
+    mres: Any, ab: Any
+) -> tuple[tuple[str, float] | None, float | None, list[str]]:
+    """Choose an observed residual basis without fabricating trace coverage."""
+    diagnostics = list(mres.diagnostics)
+    if ab is None:
+        diagnostics.append("intervention A/B produced no result; no performance claim emitted")
+        return None, None, diagnostics
+    try:
+        measured_delta = float(ab.speedup) - 1.0
+    except (AttributeError, TypeError, ValueError):
+        diagnostics.append("intervention A/B result has no numeric speedup; no claim emitted")
+        return None, None, diagnostics
+    if not math.isfinite(measured_delta):
+        diagnostics.append("intervention A/B speedup is non-finite; no claim emitted")
+        return None, None, diagnostics
+    if mres.n_kernels > mres.n_invalid_duration:
+        return ("stream_concurrency", float(mres.serialized_fraction)), measured_delta, diagnostics
+    diagnostics.append(
+        "no positive-duration CUPTI kernels; claim residual uses the measured A/B "
+        "throughput delta instead of a fabricated stream-concurrency value"
+    )
+    return ("throughput_delta", measured_delta), measured_delta, diagnostics
+
+
 def _hft_intervention_result(
     *,
     run_dir: Path,
@@ -1330,6 +1392,9 @@ def _hft_intervention_result(
         spec, applicator, min_keep_delta=0.0, audit=AuditLog(run_dir / "audit.jsonl")
     )
     ab = applicator.last_result
+    basis, measured_delta, runtime_diagnostics = _specialized_claim_basis(mres, ab)
+    if apply_res.error:
+        runtime_diagnostics.append(f"intervention apply failed: {apply_res.error}")
 
     # Prove: one claim carrying the measured delta, gated on identical output.
     top = mres.top_hypotheses
@@ -1351,16 +1416,18 @@ def _hft_intervention_result(
 
     claims: list[Claim] = []
     rolled_back: list[str] = []
-    if ab is not None:
+    if basis is not None and ab is not None:
+        residual_invariant, residual_value = basis
         claims.append(
             Claim(
                 summary=spec.summary,
-                residual_invariant="stream_concurrency",
-                residual_value=float(mres.serialized_fraction),
+                residual_invariant=residual_invariant,
+                residual_value=residual_value,
+                residual_scope="run",
                 causal_evidence=evidence,
                 intervention_name=spec.name,
                 predicted_delta=predicted,
-                measured_delta=(ab.speedup - 1.0) if ab.identical else None,
+                measured_delta=measured_delta if ab.identical else None,
                 rolled_back=apply_res.rolled_back,
             )
         )
@@ -1383,6 +1450,8 @@ def _hft_intervention_result(
                 "speedup": getattr(ab, "speedup", None),
                 "serialized_concurrency_fraction": mres.serialized_fraction,
                 "families": mres.families,
+                "residual_basis": basis[0] if basis is not None else None,
+                "diagnostics": runtime_diagnostics,
             },
             indent=2,
         )
@@ -1401,6 +1470,7 @@ def _hft_intervention_result(
         claims=claims,
         provenance=provenance,
         qualification_diagnostic=qual.diagnostic,
+        runtime_diagnostics=runtime_diagnostics,
         summary=(
             f"HFT intervention {spec.name!r}: {verdict}. "
             f"{mres.n_kernels:,} kernels observed, serialized-concurrency="
@@ -1412,7 +1482,7 @@ def _hft_intervention_result(
     summary = {
         "run_id": run_id,
         "workload": workload,
-        "status": "ok",
+        "status": "ok" if basis is not None else "intervention_failed",
         "mode": "intervention",
         "fingerprint": qual.fingerprint,
         "commit": qual.commit,
@@ -1465,6 +1535,9 @@ def _openfold_intervention_result(
         spec, applicator, min_keep_delta=0.0, audit=AuditLog(run_dir / "audit.jsonl")
     )
     ab = applicator.last_result  # AF2ABResult
+    basis, measured_delta, runtime_diagnostics = _specialized_claim_basis(mres, ab)
+    if apply_res.error:
+        runtime_diagnostics.append(f"intervention apply failed: {apply_res.error}")
 
     top = mres.top_hypotheses
     if top:
@@ -1485,16 +1558,18 @@ def _openfold_intervention_result(
 
     claims: list[Claim] = []
     rolled_back: list[str] = []
-    if ab is not None:
+    if basis is not None and ab is not None:
+        residual_invariant, residual_value = basis
         claims.append(
             Claim(
                 summary=spec.summary,
-                residual_invariant="stream_concurrency",
-                residual_value=float(mres.serialized_fraction),
+                residual_invariant=residual_invariant,
+                residual_value=residual_value,
+                residual_scope="run",
                 causal_evidence=evidence,
                 intervention_name=spec.name,
                 # plDDT-equivalence is the AF2 correctness gate (vs byte-identical).
-                measured_delta=(ab.speedup - 1.0) if ab.equivalent else None,
+                measured_delta=measured_delta if ab.equivalent else None,
                 predicted_delta=predicted,
                 rolled_back=apply_res.rolled_back,
             )
@@ -1520,6 +1595,8 @@ def _openfold_intervention_result(
                 "speedup": getattr(ab, "speedup", None),
                 "serialized_concurrency_fraction": mres.serialized_fraction,
                 "families": mres.families,
+                "residual_basis": basis[0] if basis is not None else None,
+                "diagnostics": runtime_diagnostics,
             },
             indent=2,
         )
@@ -1538,6 +1615,7 @@ def _openfold_intervention_result(
         claims=claims,
         provenance=provenance,
         qualification_diagnostic=qual.diagnostic,
+        runtime_diagnostics=runtime_diagnostics,
         summary=(
             f"AF2 intervention {spec.name!r}: {verdict}. "
             f"{mres.n_kernels:,} kernels observed, serialized-concurrency="
@@ -1549,7 +1627,7 @@ def _openfold_intervention_result(
     summary = {
         "run_id": run_id,
         "workload": workload,
-        "status": "ok",
+        "status": "ok" if basis is not None else "intervention_failed",
         "mode": "intervention",
         "fingerprint": qual.fingerprint,
         "commit": qual.commit,
@@ -1604,6 +1682,9 @@ def _edge_intervention_result(
         spec, applicator, min_keep_delta=0.0, audit=AuditLog(run_dir / "audit.jsonl")
     )
     ab = applicator.last_result  # EdgeABResult
+    basis, measured_delta, runtime_diagnostics = _specialized_claim_basis(mres, ab)
+    if apply_res.error:
+        runtime_diagnostics.append(f"intervention apply failed: {apply_res.error}")
 
     top = mres.top_hypotheses
     if top:
@@ -1624,16 +1705,18 @@ def _edge_intervention_result(
 
     claims: list[Claim] = []
     rolled_back: list[str] = []
-    if ab is not None:
+    if basis is not None and ab is not None:
+        residual_invariant, residual_value = basis
         claims.append(
             Claim(
                 summary=spec.summary,
-                residual_invariant="stream_concurrency",
-                residual_value=float(mres.serialized_fraction),
+                residual_invariant=residual_invariant,
+                residual_value=residual_value,
+                residual_scope="run",
                 causal_evidence=evidence,
                 intervention_name=spec.name,
                 # detection-equivalence is the edge correctness gate.
-                measured_delta=(ab.speedup - 1.0) if ab.identical else None,
+                measured_delta=measured_delta if ab.identical else None,
                 predicted_delta=predicted,
                 rolled_back=apply_res.rolled_back,
             )
@@ -1657,6 +1740,8 @@ def _edge_intervention_result(
                 "speedup": getattr(ab, "speedup", None),
                 "serialized_concurrency_fraction": mres.serialized_fraction,
                 "families": mres.families,
+                "residual_basis": basis[0] if basis is not None else None,
+                "diagnostics": runtime_diagnostics,
             },
             indent=2,
         )
@@ -1675,6 +1760,7 @@ def _edge_intervention_result(
         claims=claims,
         provenance=provenance,
         qualification_diagnostic=qual.diagnostic,
+        runtime_diagnostics=runtime_diagnostics,
         summary=(
             f"edge intervention {spec.name!r}: {verdict}. "
             f"{mres.n_kernels:,} kernels observed, serialized-concurrency="
@@ -1686,7 +1772,7 @@ def _edge_intervention_result(
     summary = {
         "run_id": run_id,
         "workload": workload,
-        "status": "ok",
+        "status": "ok" if basis is not None else "intervention_failed",
         "mode": "intervention",
         "fingerprint": qual.fingerprint,
         "commit": qual.commit,
diff --git a/gitm/serve/attach.py b/gitm/serve/attach.py
index ca28ef8..acfe7c8 100644
--- a/gitm/serve/attach.py
+++ b/gitm/serve/attach.py
@@ -454,9 +454,12 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
             f"pricing uses fallback {hw.name!r}"
         )
     if g.has_unpriced_nodes:
-        warnings.append(
-            "byte-moving nodes are unpriced (a required bandwidth is absent from the catalogue)"
-        )
+        missing = []
+        if g.has_unpriced_compute:
+            missing.append("compute throughput")
+        if g.has_unpriced_memory:
+            missing.append("memory bandwidth")
+        warnings.append(f"predicted nodes have unpriced {' and '.join(missing)}")
     if g.has_fallback_peaks:
         warnings.append("priced against fallback compute peaks; the ceiling is low")
     if g.has_fallback_bytes:
@@ -485,6 +488,8 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
         "total_pred_s": g.total_pred_s,
         "has_unpriced_collectives": g.has_unpriced_collectives,
         "has_unpriced_nodes": g.has_unpriced_nodes,
+        "has_unpriced_compute": g.has_unpriced_compute,
+        "has_unpriced_memory": g.has_unpriced_memory,
         "has_fallback_peaks": g.has_fallback_peaks,
         "has_fallback_bytes": g.has_fallback_bytes,
         "warnings": warnings,
@@ -499,6 +504,8 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
                 "bytes": n.prediction.bytes,
                 "estimated": n.prediction.estimated,
                 "bytes_are_fallback": n.prediction.bytes_are_fallback,
+                "compute_is_unpriced": n.prediction.compute_is_unpriced,
+                "memory_is_unpriced": n.prediction.memory_is_unpriced,
             }
             for n in g.nodes
         ],
diff --git a/tests/test_edge_optimize.py b/tests/test_edge_optimize.py
index 4f8d9a3..115dd7a 100644
--- a/tests/test_edge_optimize.py
+++ b/tests/test_edge_optimize.py
@@ -161,10 +161,8 @@ def test_applicator_runs_through_the_apply_gate():
     assert apply_intervention(drift.spec, drift, min_keep_delta=0.0).rolled_back
 
 
-def test_edge_report_claim_labels_serialized_concurrency_not_kernel_time(tmp_path):
-    """The edge claim's residual is a serialized-concurrency fraction (from
-    measure_trace), not a kernel-time ratio — the report must label it
-    `stream_concurrency`, matching the HFT/AF2 claims, not `kernel_time`."""
+def test_edge_report_uses_ab_delta_when_trace_has_no_kernel_coverage(tmp_path):
+    """An empty trace must not become a plausible zero-concurrency residual."""
     from gitm.optimizer.qualification import QualificationResult
     from gitm.scheduler.loop import _edge_intervention_result
     from gitm.tracer.schema import Trace
@@ -187,5 +185,50 @@ def test_edge_report_claim_labels_serialized_concurrency_not_kernel_time(tmp_pat
         trace_path=tmp_path / "trace.jsonl",
     )
     md = result["report_md"]
-    assert "`stream_concurrency`" in md
+    assert "`throughput_delta`" in md
+    assert "`stream_concurrency`" not in md
     assert "`kernel_time`" not in md
+
+
+def test_edge_report_names_missing_ab_result_as_intervention_failure(
+    tmp_path, monkeypatch
+):
+    from types import SimpleNamespace
+
+    from gitm.benchmarks.edge.optimize import edge_intervention_spec
+    from gitm.optimizer.qualification import QualificationResult
+    from gitm.scheduler.loop import _edge_intervention_result
+    from gitm.tracer.schema import Trace
+
+    trace = Trace(
+        workload_id="kitti",
+        fingerprint="f",
+        run_id="r",
+        device_count=0,
+        vendor="none",
+        captured_at_ns=0,
+        duration_ns=1,
+        events=[],
+    )
+    applicator = SimpleNamespace(last_result=None, spec=edge_intervention_spec())
+    monkeypatch.setattr(
+        "gitm.optimizer.apply.apply_intervention",
+        lambda *_args, **_kwargs: SimpleNamespace(
+            applied=False, rolled_back=True, measured_delta=None, error="measure failed"
+        ),
+    )
+
+    result = _edge_intervention_result(
+        run_dir=tmp_path,
+        run_id="r",
+        workload="kitti",
+        trace=trace,
+        qual=QualificationResult(commit=False, floor=0.0, fingerprint="f"),
+        applicator=applicator,
+        started_ns=0,
+        trace_path=tmp_path / "trace.jsonl",
+    )
+
+    assert result["summary"]["status"] == "intervention_failed"
+    assert result["summary"]["n_claims"] == 0
+    assert "produced no result" in result["report_md"]
diff --git a/tests/test_gate_wiring.py b/tests/test_gate_wiring.py
index d9eb026..3afb9c1 100644
--- a/tests/test_gate_wiring.py
+++ b/tests/test_gate_wiring.py
@@ -57,3 +57,13 @@ def test_load_library_filters_by_workload(tmp_path):
     )
     assert [s.name for s in load_library(p, workload="vllm-decode")] == ["v"]
     assert {s.name for s in load_library(p)} == {"v", "e"}  # unfiltered
+
+
+def test_missing_library_refuses_with_named_path(tmp_path):
+    import pytest
+
+    from gitm.kernels.library import load_library
+
+    missing = tmp_path / "missing.yaml"
+    with pytest.raises(FileNotFoundError, match="candidate coverage is unavailable"):
+        load_library(missing)
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index 6af729e..d46be6e 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -285,6 +285,39 @@ def test_vllm_workload_still_uses_intervention_path(tmp_path: Path, monkeypatch)
     assert result["summary"]["mode"] == "intervention"
 
 
+def test_vllm_missing_intervention_library_degrades_with_named_coverage_refusal(
+    tmp_path: Path, monkeypatch
+):
+    import gitm.scheduler.loop as loop
+
+    monkeypatch.setattr(loop, "capture", _fake_capture_with_kernels("paged_attention"))
+    monkeypatch.setattr(loop, "sync_device", lambda: None)
+    monkeypatch.setattr(
+        loop,
+        "load_library",
+        lambda **_kwargs: (_ for _ in ()).throw(
+            FileNotFoundError("missing.yaml; candidate coverage is unavailable")
+        ),
+    )
+    monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200")
+
+    from gitm import optimize
+
+    result = optimize(
+        _priceable_moe_engine(),
+        workload="vllm-decode",
+        budget="1s",
+        scratch=str(tmp_path),
+        workload_runner=lambda: {},
+    )
+
+    assert result["summary"]["status"] == "candidate_coverage_unavailable"
+    assert result["summary"]["mode"] == "measurement"
+    assert result["summary"]["n_claims"] == 0
+    assert "missing.yaml" in result["report_md"]
+    assert "candidate coverage" in result["report_md"]
+
+
 def test_vllm_without_live_model_refuses_prediction_claims(tmp_path: Path, monkeypatch):
     import gitm.scheduler.loop as loop
 

From 13ba794a517b60fe66509a4d6e1c88da0e67e882 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:14:50 -0700
Subject: [PATCH 10/70] fix: refuse invalid benchmark timing

---
 gitm/_timing.py                  | 22 +++++++++++++
 gitm/benchmarks/edge/optimize.py |  5 ++-
 gitm/benchmarks/hft/harness.py   | 22 +++++++++++--
 gitm/benchmarks/hft/optimize.py  | 13 ++++++--
 gitm/runtime_driver.py           | 16 ++++++++--
 gitm/workloads.py                | 53 ++++++++++++++++++++++++++------
 scripts/demo_improve_gpu.py      |  6 +++-
 tests/test_timing.py             | 22 +++++++++++++
 tests/test_workload_bootstrap.py |  3 +-
 9 files changed, 141 insertions(+), 21 deletions(-)
 create mode 100644 gitm/_timing.py
 create mode 100644 tests/test_timing.py

diff --git a/gitm/_timing.py b/gitm/_timing.py
new file mode 100644
index 0000000..455e922
--- /dev/null
+++ b/gitm/_timing.py
@@ -0,0 +1,22 @@
+"""Shared timing trust predicates."""
+
+from __future__ import annotations
+
+import math
+
+
+def require_positive_duration(duration_s: float, *, context: str) -> float:
+    """Return a usable duration or refuse to fabricate a throughput denominator."""
+    if not math.isfinite(duration_s) or duration_s <= 0.0:
+        raise RuntimeError(
+            f"{context} timing unavailable: expected a finite positive duration, "
+            f"got {duration_s!r}"
+        )
+    return duration_s
+
+
+def require_positive_work(value: int | float, *, context: str) -> int | float:
+    """Refuse throughput or speedup claims over an empty work unit."""
+    if not math.isfinite(float(value)) or value <= 0:
+        raise RuntimeError(f"{context} work coverage unavailable: expected > 0, got {value!r}")
+    return value
diff --git a/gitm/benchmarks/edge/optimize.py b/gitm/benchmarks/edge/optimize.py
index a76d195..1dff1f4 100644
--- a/gitm/benchmarks/edge/optimize.py
+++ b/gitm/benchmarks/edge/optimize.py
@@ -27,6 +27,7 @@
 from collections.abc import Callable
 from dataclasses import dataclass, field
 
+from gitm._timing import require_positive_duration, require_positive_work
 from gitm.kernels.spec import Applicability, InterventionSpec, SafetyGate
 
 # A run_mode runs N frames in a given mode and returns a per-frame summary dict:
@@ -145,7 +146,9 @@ def _timed(mode: str) -> tuple[dict, float]:
             sync()
             best = min(best, time.perf_counter() - t0)
         n = max(int(summary.get("n_frames", 0)), 0)
-        return summary, n / max(best, 1e-9)
+        require_positive_work(n, context=f"edge A/B {mode}")
+        best = require_positive_duration(best, context=f"edge A/B {mode}")
+        return summary, n / best
 
     base_summary, base_eps = _timed(baseline_mode)
     cand_summary, cand_eps = _timed(candidate_mode)
diff --git a/gitm/benchmarks/hft/harness.py b/gitm/benchmarks/hft/harness.py
index 9e0a15f..6ff1bf0 100644
--- a/gitm/benchmarks/hft/harness.py
+++ b/gitm/benchmarks/hft/harness.py
@@ -25,8 +25,11 @@
 import json
 import os
 import time
+import warnings
 from pathlib import Path
 
+from gitm._timing import require_positive_duration
+
 
 def select_backend():
     """Return ``(kind, df_module, array_module)`` — cuDF/CuPy if available, else pandas/NumPy."""
@@ -35,7 +38,13 @@ def select_backend():
         import cupy
 
         return "gpu", cudf, cupy
-    except Exception:
+    except Exception as exc:
+        warnings.warn(
+            f"cuDF/CuPy backend unavailable; HFT harness is using the CPU fallback "
+            f"and cannot produce a GPU baseline ({type(exc).__name__}: {exc})",
+            RuntimeWarning,
+            stacklevel=2,
+        )
         import numpy
         import pandas
 
@@ -52,7 +61,12 @@ def _gpu_name(kind: str) -> tuple[str, int]:
         props = cupy.cuda.runtime.getDeviceProperties(0)
         name = props["name"].decode() if isinstance(props["name"], bytes) else str(props["name"])
         return name, n
-    except Exception:
+    except Exception as exc:
+        warnings.warn(
+            f"GPU identity probe failed ({type(exc).__name__}: {exc})",
+            RuntimeWarning,
+            stacklevel=2,
+        )
         return "gpu-unknown", 1
 
 
@@ -162,7 +176,9 @@ def main(argv: list[str] | None = None) -> int:
     # Warm window: replay the loaded events, measuring sustained throughput.
     t0 = time.perf_counter()
     summary = run_pipeline(df, dflib)
-    elapsed = max(time.perf_counter() - t0, 1e-9)
+    elapsed = require_positive_duration(
+        time.perf_counter() - t0, context="HFT harness"
+    )
 
     events_per_second = summary["events"] / elapsed
     print(f"[hft harness:{kind}] {summary['events']} events in {elapsed:.3f}s "
diff --git a/gitm/benchmarks/hft/optimize.py b/gitm/benchmarks/hft/optimize.py
index 1fc4eed..76cd1c5 100644
--- a/gitm/benchmarks/hft/optimize.py
+++ b/gitm/benchmarks/hft/optimize.py
@@ -25,6 +25,7 @@
 import time
 from dataclasses import dataclass
 
+from gitm._timing import require_positive_duration, require_positive_work
 from gitm.benchmarks.hft.harness import microprice, run_pipeline, vwap_1s
 from gitm.kernels.spec import Applicability, InterventionSpec, SafetyGate
 
@@ -117,7 +118,9 @@ def _timed(fn) -> tuple[dict, float]:
             summary = fn(df, dflib)
             sync()
             best = min(best, time.perf_counter() - t0)
-        return summary, summary["events"] / max(best, 1e-9)
+        best = require_positive_duration(best, context=f"HFT A/B {fn.__name__}")
+        events = require_positive_work(summary["events"], context=f"HFT A/B {fn.__name__}")
+        return summary, events / best
 
     base_summary, base_eps = _timed(run_pipeline)
     cand_summary, cand_eps = _timed(run_pipeline_fast)
@@ -198,8 +201,12 @@ def optimize_hft_streaming(batches, dflib, *, sync=None, on_batch=None) -> ABRes
     if n_batches == 0:
         raise ValueError("optimize_hft_streaming: no batches to process (empty iterable)")
 
-    base_eps = base_events / max(base_t, 1e-9)
-    cand_eps = cand_events / max(cand_t, 1e-9)
+    require_positive_work(base_events, context="HFT streaming baseline")
+    require_positive_work(cand_events, context="HFT streaming candidate")
+    base_t = require_positive_duration(base_t, context="HFT streaming baseline")
+    cand_t = require_positive_duration(cand_t, context="HFT streaming candidate")
+    base_eps = base_events / base_t
+    cand_eps = cand_events / cand_t
     speedup = cand_eps / base_eps if base_eps else 0.0
     kept = "candidate" if (identical and cand_eps > base_eps) else "baseline"
     # Per-pipeline count aggregates (kept separate so a divergent run never reports
diff --git a/gitm/runtime_driver.py b/gitm/runtime_driver.py
index 9cd43d9..2c0bea5 100644
--- a/gitm/runtime_driver.py
+++ b/gitm/runtime_driver.py
@@ -28,17 +28,25 @@
 import json
 import os
 import time
+import warnings
 from contextlib import closing
 from pathlib import Path
 
+from gitm._timing import require_positive_duration
+
 
 def _sync():
     try:
         import cupy
 
         cupy.cuda.runtime.deviceSynchronize()
-    except Exception:
-        pass
+    except Exception as exc:
+        warnings.warn(
+            f"CuPy device synchronization unavailable; trace completeness is not "
+            f"guaranteed ({type(exc).__name__}: {exc})",
+            RuntimeWarning,
+            stacklevel=2,
+        )
 
 
 def _load_hft(stage: Path, seed: int, max_events: int | None):
@@ -340,7 +348,9 @@ def main(argv: list[str] | None = None) -> int:
         t0 = time.perf_counter()
         summary = work()
         _sync()
-        elapsed = max(time.perf_counter() - t0, 1e-9)
+        elapsed = require_positive_duration(
+            time.perf_counter() - t0, context=f"{args.workload} runtime driver"
+        )
     if tele:
         tele.stop()
         telemetry_diagnostics.extend(tele.diagnostics)
diff --git a/gitm/workloads.py b/gitm/workloads.py
index 7fd3275..a569831 100644
--- a/gitm/workloads.py
+++ b/gitm/workloads.py
@@ -29,10 +29,13 @@
 
 import os
 import socket
+import warnings
 from collections.abc import Callable
 from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
+from gitm._timing import require_positive_duration
+
 if TYPE_CHECKING:
     from gitm.scheduler.loop import LoopConfig
 
@@ -134,8 +137,13 @@ def _free_gpu_pool() -> None:
         import cupy
 
         cupy.get_default_memory_pool().free_all_blocks()
-    except Exception:
-        pass
+    except Exception as exc:
+        warnings.warn(
+            f"GPU memory-pool cleanup unavailable; streaming may retain cached blocks "
+            f"({type(exc).__name__}: {exc})",
+            RuntimeWarning,
+            stacklevel=2,
+        )
 
 
 def _hft_batches_factory(stage: Path, seed: int, dflib, shards_per_batch: int,
@@ -193,6 +201,12 @@ def _ensure_hft_data(stage: Path, seed: int) -> None:
 
     events = int(os.environ.get("GITM_BENCH_EVENTS", "200000"))
     out = stage / f"hft_smoke_seed{seed}"
+    warnings.warn(
+        f"no staged HFT dataset found; generating {events:,}-event smoke data at {out}. "
+        "This is synthetic smoke coverage, not a publishable baseline dataset.",
+        RuntimeWarning,
+        stacklevel=2,
+    )
     generate(
         GenConfig(events=events, seed=seed, events_per_file=min(events, 100_000)),
         out,
@@ -392,7 +406,9 @@ def run() -> dict[str, Any]:
         t0 = time.perf_counter()
         for idx in run_indices:
             total_dets += unit.run(idx).n_detections
-        elapsed = max(time.perf_counter() - t0, 1e-9)
+        elapsed = require_positive_duration(
+            time.perf_counter() - t0, context="nuScenes workload"
+        )
         return {
             "frames": len(run_indices),
             "detections": total_dets,
@@ -440,7 +456,9 @@ def run() -> dict[str, Any]:
         t0 = time.perf_counter()
         for p in run_paths:
             total_dets += unit.run(p).n_detections
-        elapsed = max(time.perf_counter() - t0, 1e-9)
+        elapsed = require_positive_duration(
+            time.perf_counter() - t0, context="KITTI workload"
+        )
         return {
             "frames": len(run_paths),
             "detections": total_dets,
@@ -760,7 +778,10 @@ def _throughput(eng: Any) -> float:
         outs = eng.generate(prompts, params)
         toks = sum(len(o.outputs[0].token_ids) for o in outs)
         sync_device()
-        return toks / max(time.perf_counter() - t0, 1e-9)
+        elapsed = require_positive_duration(
+            time.perf_counter() - t0, context="vLLM throughput probe"
+        )
+        return toks / elapsed
 
     def _restart(_old_engine: Any, knob_values: dict[str, Any]) -> Any:
         """Rebuild a fresh vLLM engine with one or more structural knobs changed.
@@ -883,17 +904,31 @@ def set_decode_run_defaults() -> dict[str, str]:
 def sync_device() -> None:
     """Block until queued GPU work completes, so all kernels land in the trace
     before capture stops. Best-effort — a no-op without CuPy/torch."""
+    cupy_error: Exception | None = None
     try:
         import cupy
 
         cupy.cuda.runtime.deviceSynchronize()
         return
-    except Exception:
-        pass
+    except Exception as exc:
+        cupy_error = exc
     try:
         import torch
 
         if torch.cuda.is_available():
             torch.cuda.synchronize()
-    except Exception:
-        pass
+            return
+    except Exception as exc:
+        warnings.warn(
+            f"GPU device synchronization failed; trace completeness is not guaranteed "
+            f"(CuPy: {type(cupy_error).__name__ if cupy_error else 'unavailable'}; "
+            f"torch: {type(exc).__name__}: {exc})",
+            RuntimeWarning,
+            stacklevel=2,
+        )
+        return
+    warnings.warn(
+        "GPU device synchronization unavailable; trace completeness is not guaranteed",
+        RuntimeWarning,
+        stacklevel=2,
+    )
diff --git a/scripts/demo_improve_gpu.py b/scripts/demo_improve_gpu.py
index 0790e0d..1f0ec4e 100644
--- a/scripts/demo_improve_gpu.py
+++ b/scripts/demo_improve_gpu.py
@@ -25,6 +25,8 @@
 import time
 from pathlib import Path
 
+from gitm._timing import require_positive_duration
+
 
 def _serialized(trace) -> float:
     from gitm.optimizer.monitor import _serialized_fraction
@@ -71,7 +73,9 @@ def _run_observed(fn, label: str, outdir: Path) -> tuple[object, dict]:
         t0 = time.perf_counter()
         result = fn()
         cupy.cuda.runtime.deviceSynchronize()
-        elapsed = max(time.perf_counter() - t0, 1e-9)
+        elapsed = require_positive_duration(
+            time.perf_counter() - t0, context=f"demo observation {label}"
+        )
     if tele:
         tele.stop()
 
diff --git a/tests/test_timing.py b/tests/test_timing.py
new file mode 100644
index 0000000..524014e
--- /dev/null
+++ b/tests/test_timing.py
@@ -0,0 +1,22 @@
+from __future__ import annotations
+
+import math
+
+import pytest
+
+from gitm._timing import require_positive_duration, require_positive_work
+
+
+@pytest.mark.parametrize("duration", [0.0, -1.0, math.inf, -math.inf, math.nan])
+def test_nonpositive_or_nonfinite_timing_is_refused(duration):
+    with pytest.raises(RuntimeError, match="timing unavailable"):
+        require_positive_duration(duration, context="test workload")
+
+
+def test_positive_timing_is_preserved():
+    assert require_positive_duration(0.125, context="test workload") == 0.125
+
+
+def test_empty_work_unit_is_refused():
+    with pytest.raises(RuntimeError, match="work coverage unavailable"):
+        require_positive_work(0, context="test workload")
diff --git a/tests/test_workload_bootstrap.py b/tests/test_workload_bootstrap.py
index bb0bd38..08e7e0b 100644
--- a/tests/test_workload_bootstrap.py
+++ b/tests/test_workload_bootstrap.py
@@ -20,7 +20,8 @@ def test_ensure_hft_data_generates_then_reuses(tmp_path: Path, monkeypatch):
     monkeypatch.setenv("GITM_BENCH_EVENTS", "2000")
     stage = tmp_path / "stage"
 
-    _ensure_hft_data(stage, 42)
+    with pytest.warns(RuntimeWarning, match="synthetic smoke coverage"):
+        _ensure_hft_data(stage, 42)
     seed_dir = stage / "hft_smoke_seed42"
     shards = sorted(seed_dir.glob("part-*.parquet"))
     assert shards, "smoke data should have been generated"

From aae4903e59d8abd88b4b359ec2cdca04fbe22882 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:14:58 -0700
Subject: [PATCH 11/70] fix: surface degraded control paths

---
 gitm/agents/autoresearch.py     | 50 +++++++++++++++++++++++++++------
 gitm/bench/cli.py               |  8 +++++-
 gitm/bench/profile.py           | 24 ++++++++++++----
 gitm/deploy/attach.py           | 11 ++++++--
 gitm/kernels/library.py         | 12 ++++++--
 gitm/routing/scorer_v0.py       | 19 ++++++++++++-
 gitm/safety/failopen.py         | 23 +++++++++++----
 tests/test_autoresearch.py      | 20 +++++++++----
 tests/test_bench.py             | 25 +++++++++++++++++
 tests/test_cli_attach.py        | 12 ++++++++
 tests/test_gate_wiring.py       | 11 ++++++++
 tests/test_routing.py           | 19 +++++++++++++
 tests/test_safety_primitives.py | 14 +++++++++
 13 files changed, 215 insertions(+), 33 deletions(-)
 create mode 100644 tests/test_routing.py

diff --git a/gitm/agents/autoresearch.py b/gitm/agents/autoresearch.py
index 8ba7d3c..807d711 100644
--- a/gitm/agents/autoresearch.py
+++ b/gitm/agents/autoresearch.py
@@ -38,6 +38,7 @@
 from __future__ import annotations
 
 import random
+import warnings
 from collections.abc import Callable
 from dataclasses import dataclass, field
 from typing import TYPE_CHECKING, Protocol
@@ -63,6 +64,7 @@
     "IDLE_STALL",
     "MEMORY_BOUND",
     "COMPUTE_BOUND",
+    "UNCLASSIFIED",
     "classify_bottleneck",
     "ResidualTarget",
     "largest_residual",
@@ -97,7 +99,8 @@
 IDLE_STALL = "idle_stall"
 MEMORY_BOUND = "memory_bound"
 COMPUTE_BOUND = "compute_bound"
-BOTTLENECK_CLASSES = (IDLE_STALL, MEMORY_BOUND, COMPUTE_BOUND)
+UNCLASSIFIED = "unclassified"
+BOTTLENECK_CLASSES = (IDLE_STALL, MEMORY_BOUND, COMPUTE_BOUND, UNCLASSIFIED)
 
 #: Serialized-concurrency fraction above this ⇒ kernels ran back-to-back on one
 #: stream instead of overlapping: scheduling gaps / launch-bound idle time.
@@ -125,18 +128,18 @@ def _roofline_memory_fraction(residuals: Residuals | None) -> float | None:
 
 
 def classify_bottleneck(trace: Trace, residuals: Residuals | None = None) -> str:
-    """Map a captured trace to one of ``idle_stall`` / ``memory_bound`` / ``compute_bound``.
+    """Classify a measured trace, or return ``unclassified`` without valid kernels.
 
     Two signals, scored against a threshold each; the stronger wins, neither
-    crossing defaults to compute bound: serialized-concurrency fraction (poor
+    crossing selects compute bound: serialized-concurrency fraction (poor
     kernel overlap ⇒ idle/scheduling gaps), and memory pressure (memcpy share
     of GPU-op time, widened by the roofline-predicted memory-bound fraction of
     matched kernel time when ``residuals`` is passed). Without ``residuals``
     this is the memcpy-only heuristic.
     """
-    kernels = trace.kernels()
+    kernels = [k for k in trace.kernels() if k.end_ns > k.start_ns]
     if not kernels:
-        return COMPUTE_BOUND
+        return UNCLASSIFIED
 
     memcpys = [e for e in trace.events if e.kind == "memcpy"]
     sc = _serialized_fraction(kernels)
@@ -213,6 +216,7 @@ def _op_present(trace: Trace, op: str) -> bool:
         ("compilation_config", 3,
          "raise torch.compile to level 3 for kernel fusion + piecewise CUDA graphs"),
     ],
+    "unclassified": [],
 }
 
 
@@ -393,6 +397,7 @@ class Knob:
     "idle_stall": ("prefill", "partial", "schedul", "chunk", "overlap"),
     "memory_bound": ("cache", "swap", "offload", "block", "gpu_memory", "kv", "preempt", "cpu"),
     "compute_bound": ("compil", "cudagraph", "cuda_graph", "graph", "quant", "fus", "eager"),
+    "unclassified": (),
 }
 
 
@@ -585,7 +590,13 @@ def _argparse_domains(engine_args_cls: object) -> dict[str, _ArgDomain]:
         import argparse
 
         parser = engine_args_cls.add_cli_args(argparse.ArgumentParser())  # type: ignore[attr-defined]
-    except Exception:
+    except Exception as exc:
+        warnings.warn(
+            "vLLM CLI-domain introspection unavailable; candidate grids will use "
+            f"coarser dataclass annotations ({type(exc).__name__}: {exc})",
+            RuntimeWarning,
+            stacklevel=2,
+        )
         return {}
     out: dict[str, _ArgDomain] = {}
     for action in getattr(parser, "_actions", []):
@@ -668,11 +679,32 @@ def _engine_arg_knobs(*, gpu_count: int | None = None) -> list[Knob]:
     """
     try:
         from vllm import EngineArgs  # type: ignore
-    except Exception:
+    except Exception as exc:
+        warnings.warn(
+            "vLLM EngineArgs unavailable; autoresearch is using the frozen fallback "
+            f"knob catalog ({type(exc).__name__}: {exc})",
+            RuntimeWarning,
+            stacklevel=2,
+        )
         return list(_FALLBACK_KNOBS)
     try:
-        return _knobs_from_engine_args(EngineArgs, gpu_count=gpu_count) or list(_FALLBACK_KNOBS)
-    except Exception:
+        knobs = _knobs_from_engine_args(EngineArgs, gpu_count=gpu_count)
+        if knobs:
+            return knobs
+        warnings.warn(
+            "vLLM EngineArgs introspection produced no searchable knobs; autoresearch "
+            "is using the frozen fallback knob catalog",
+            RuntimeWarning,
+            stacklevel=2,
+        )
+        return list(_FALLBACK_KNOBS)
+    except Exception as exc:
+        warnings.warn(
+            "vLLM EngineArgs introspection failed; autoresearch is using the frozen "
+            f"fallback knob catalog ({type(exc).__name__}: {exc})",
+            RuntimeWarning,
+            stacklevel=2,
+        )
         return list(_FALLBACK_KNOBS)
 
 
diff --git a/gitm/bench/cli.py b/gitm/bench/cli.py
index dd567ef..a4a657f 100644
--- a/gitm/bench/cli.py
+++ b/gitm/bench/cli.py
@@ -145,6 +145,12 @@ def _cmd_edge_manifest(args) -> int:
     n_nusc = sum(1 for r in rows if r.source == "nuscenes")
     n_kitti = sum(1 for r in rows if r.source == "kitti")
     print(f"wrote {args.out}: {len(rows)} keyframes ({n_nusc} nuScenes, {n_kitti} KITTI)")
+    if not rows:
+        print(
+            "ERROR: no dataset keyframes were discovered; empty manifest is not usable",
+            file=sys.stderr,
+        )
+        return 1
     return 0
 
 
@@ -187,7 +193,7 @@ def _cmd_profile(args) -> int:
             f"detected: {ProfilerTools.detect()}",
             file=sys.stderr,
         )
-    return 0
+    return 0 if bundle.complete else 1
 
 
 def _cmd_baseline(args) -> int:
diff --git a/gitm/bench/profile.py b/gitm/bench/profile.py
index 4b6b293..6cd72cf 100644
--- a/gitm/bench/profile.py
+++ b/gitm/bench/profile.py
@@ -161,16 +161,21 @@ def run_profile(
     if not tools.py_spy:
         bundle.missing.append("py-spy")
 
-    subprocess.run(argv, check=False)
+    completed = subprocess.run(argv, check=False)
+    if completed.returncode != 0:
+        bundle.missing.append(f"workload command failed (exit {completed.returncode})")
 
     for hp in host_procs:
         try:
             hp.wait(timeout=host_capture_s + 5)
         except subprocess.TimeoutExpired:
             hp.terminate()
+            bundle.missing.append("host sampler timed out")
 
     if config.vendor == "nvidia" and tools.nsys and bundle.gpu_report:
         bundle.gpu_csv = _export_nsys_csv(tools.nsys, bundle.gpu_report, out_dir)
+        if bundle.gpu_csv is None:
+            bundle.missing.append("nsys GPU CSV export")
 
     return bundle
 
@@ -284,14 +289,21 @@ def build_breakdown(phases: list[PhaseTiming]) -> list[StallPhase]:
         gpu = p.gpu_busy_s / wall
         sync = p.sync_s / wall
         cpu = p.cpu_s / wall
-        data_stall = max(0.0, 1.0 - gpu - sync - cpu)
+        components = {"gpu_busy_s": gpu, "sync_s": sync, "cpu_s": cpu}
+        invalid = {name: value for name, value in components.items() if value < 0 or value > 1}
+        if invalid or gpu + sync + cpu > 1.0 + 1e-9:
+            raise ValueError(
+                f"phase {p.phase!r} has contradictory/overlapping timing attribution: "
+                f"gpu={gpu:.3f}, sync={sync:.3f}, cpu={cpu:.3f}; refusing to clamp"
+            )
+        data_stall = 1.0 - gpu - sync - cpu
         out.append(
             StallPhase(
                 phase=p.phase,
-                cpu=min(1.0, cpu),
-                data_stall=min(1.0, data_stall),
-                sync=min(1.0, sync),
-                gpu_active=min(1.0, gpu),
+                cpu=cpu,
+                data_stall=data_stall,
+                sync=sync,
+                gpu_active=gpu,
                 throughput=p.throughput,
                 wall_clock_s=wall,
             )
diff --git a/gitm/deploy/attach.py b/gitm/deploy/attach.py
index 41c91b3..5ec0718 100644
--- a/gitm/deploy/attach.py
+++ b/gitm/deploy/attach.py
@@ -27,7 +27,7 @@ class AttachPlan:
     job_id: str
     workload: str | None
     mode: str  # always "user-space"
-    status: str  # "planned" | "attached" | "no_target"
+    status: str  # "planned" | "unsupported" | "no_target"
     pid: int | None
     steps: list[str] = field(default_factory=list)
     reason: str = ""
@@ -101,12 +101,17 @@ def attach_job(
             reason=f"PID {resolved} is not live.",
         ).to_dict()
 
+    # PID resolution is wired, but no injector or telemetry-shim installation is.
+    # Refuse rather than turning a validated target into a false attach success.
     return AttachPlan(
         job_id=job_id,
         workload=workload,
         mode="user-space",
-        status="attached",
+        status="unsupported",
         pid=resolved,
         steps=steps,
-        reason="attached (user-space, fail-open).",
+        reason=(
+            "target is live, but standalone PID injection is not implemented; "
+            "no telemetry shim was installed"
+        ),
     ).to_dict()
diff --git a/gitm/kernels/library.py b/gitm/kernels/library.py
index f3bd4d3..845b356 100644
--- a/gitm/kernels/library.py
+++ b/gitm/kernels/library.py
@@ -21,8 +21,16 @@ def load_library(path: Path | str | None = None, *, workload: str | None = None)
             f"intervention library not found at {p}; candidate coverage is unavailable"
         )
     with p.open() as fh:
-        raw = yaml.safe_load(fh) or {}
-    entries = raw.get("interventions", [])
+        raw = yaml.safe_load(fh)
+    if not isinstance(raw, dict) or not isinstance(raw.get("interventions"), list):
+        raise ValueError(
+            f"intervention library {p} has no interventions list; candidate coverage is unavailable"
+        )
+    entries = raw["interventions"]
+    if not entries:
+        raise ValueError(
+            f"intervention library {p} is empty; candidate coverage is unavailable"
+        )
     specs = [InterventionSpec.model_validate(e) for e in entries]
     if workload is not None:
         specs = [s for s in specs if workload in s.applicability.workloads]
diff --git a/gitm/routing/scorer_v0.py b/gitm/routing/scorer_v0.py
index 85101c9..6cbb155 100644
--- a/gitm/routing/scorer_v0.py
+++ b/gitm/routing/scorer_v0.py
@@ -34,9 +34,26 @@ def score_prospect(
     W_ENGAGEMENT = 0.05
     W_PRIOR = 0.05
 
+    unit_values = {
+        "warmth": warmth,
+        "signal_recency": signal_recency,
+        "engagement_score": engagement_score,
+    }
+    invalid_units = {name: value for name, value in unit_values.items() if not 0 <= value <= 1}
+    if invalid_units:
+        raise ValueError(f"routing inputs must be within [0, 1], got {invalid_units}")
+    for name, value in {
+        "pain_acknowledged": pain_acknowledged,
+        "prior_engagement": prior_engagement,
+    }.items():
+        if value not in (0, 1):
+            raise ValueError(f"{name} must be 0 or 1, got {value!r}")
+
     # Company tier score
     tier_score_map = {1: 1.0, 2: 0.6, 3: 0.2}
-    tier_score = tier_score_map.get(company_tier, 0.2)
+    if company_tier not in tier_score_map:
+        raise ValueError(f"company_tier must be 1, 2, or 3, got {company_tier!r}")
+    tier_score = tier_score_map[company_tier]
 
     score = (
         warmth * W_WARMTH +
diff --git a/gitm/safety/failopen.py b/gitm/safety/failopen.py
index 4327333..a747bbf 100644
--- a/gitm/safety/failopen.py
+++ b/gitm/safety/failopen.py
@@ -14,6 +14,7 @@
 from __future__ import annotations
 
 import signal
+import warnings
 from collections.abc import Callable
 from typing import Any
 
@@ -35,6 +36,8 @@ def __init__(
         #: Names whose revert raised during :meth:`fire`. A non-empty list means
         #: fail-open did NOT fully clean up — the workload may still be mutated.
         self.failures: list[str] = []
+        self.audit_failures: list[str] = []
+        self.signal_failures: list[str] = []
 
     def register(self, name: str, revert_fn: Callable[[], None], *, cause: str = "") -> None:
         """Register a revert to run if we exit before it is disarmed."""
@@ -74,8 +77,10 @@ def _record(self, event: str, name: str, cause: str, **detail: Any) -> None:
             return
         try:
             self._audit.record(event, name, cause, **detail)
-        except Exception:
-            pass
+        except Exception as exc:
+            message = f"audit sink failed while recording {event} for {name}: {exc}"
+            self.audit_failures.append(message)
+            warnings.warn(message, RuntimeWarning, stacklevel=2)
 
     def _signal_handler(self, signum: int, frame: Any) -> None:
         self.fire()
@@ -86,14 +91,18 @@ def _signal_handler(self, signum: int, frame: Any) -> None:
     def __enter__(self) -> FailOpenGuard:
         self._fired = False
         self.failures = []
+        self.audit_failures = []
+        self.signal_failures = []
         if self._install:
             for sig in (signal.SIGTERM, signal.SIGINT):
                 try:
                     self._prev_handlers[sig] = signal.getsignal(sig)
                     signal.signal(sig, self._signal_handler)
-                except (ValueError, OSError):
+                except (ValueError, OSError) as exc:
                     # not in the main thread (e.g. tests) — context exit still covers us
-                    pass
+                    message = f"signal handler installation unavailable for {sig}: {exc}"
+                    self.signal_failures.append(message)
+                    warnings.warn(message, RuntimeWarning, stacklevel=2)
         return self
 
     def __exit__(self, exc_type: Any, exc: Any, tb: Any) -> None:
@@ -101,6 +110,8 @@ def __exit__(self, exc_type: Any, exc: Any, tb: Any) -> None:
         for sig, prev in self._prev_handlers.items():
             try:
                 signal.signal(sig, prev)
-            except (ValueError, OSError):
-                pass
+            except (ValueError, OSError) as exc:
+                message = f"signal handler restoration failed for {sig}: {exc}"
+                self.signal_failures.append(message)
+                warnings.warn(message, RuntimeWarning, stacklevel=2)
         self._prev_handlers.clear()
diff --git a/tests/test_autoresearch.py b/tests/test_autoresearch.py
index 08dec27..96f3566 100644
--- a/tests/test_autoresearch.py
+++ b/tests/test_autoresearch.py
@@ -2,6 +2,8 @@
 
 from __future__ import annotations
 
+import pytest
+
 import gitm.agents.autoresearch as ar
 from gitm.agents.autoresearch import (
     AutoresearchRun,
@@ -106,8 +108,11 @@ def test_classify_compute_bound_when_overlapped_and_no_memcpy() -> None:
     assert classify_bottleneck(make_trace(events=events)) == "compute_bound"
 
 
-def test_classify_empty_trace_defaults_to_compute() -> None:
-    assert classify_bottleneck(make_trace(events=[])) == "compute_bound"
+def test_classify_empty_or_invalid_trace_is_unclassified() -> None:
+    assert classify_bottleneck(make_trace(events=[])) == ar.UNCLASSIFIED
+    assert classify_bottleneck(
+        make_trace(events=[make_kernel("zero", start_ns=10, end_ns=10)])
+    ) == ar.UNCLASSIFIED
 
 
 # --- classify_bottleneck: roofline-weighted memory signal ---------------------
@@ -615,9 +620,13 @@ def test_engineargs_proposer_is_a_vllm_bound_generative_proposer() -> None:
     assert specs and all(s.applicability.workloads == ["vllm-decode"] for s in specs)
 
 
-def test_vllm_knob_source_yields_offline_fallback_without_vllm() -> None:
+def test_vllm_knob_source_warns_when_using_offline_fallback(monkeypatch) -> None:
     # vLLM isn't importable in CI → the source yields the frozen fallback catalog.
-    knobs = VLLMKnobSource().knobs()
+    import sys
+
+    monkeypatch.setitem(sys.modules, "vllm", None)
+    with pytest.warns(RuntimeWarning, match="frozen fallback knob catalog"):
+        knobs = VLLMKnobSource().knobs()
     assert knobs and all(isinstance(k, Knob) for k in knobs)
     names = {k.name for k in knobs}
     assert "cpu_offload_gb" in names
@@ -722,7 +731,8 @@ def test_argparse_domains_empty_when_no_cli_builder() -> None:
     class _Bare:
         pass
 
-    assert _argparse_domains(_Bare) == {}
+    with pytest.warns(RuntimeWarning, match="CLI-domain introspection unavailable"):
+        assert _argparse_domains(_Bare) == {}
 
 
 # --- hardware-applicability: skip multi-GPU knobs on a single-GPU box --------
diff --git a/tests/test_bench.py b/tests/test_bench.py
index da4bd36..c1308a3 100644
--- a/tests/test_bench.py
+++ b/tests/test_bench.py
@@ -262,6 +262,31 @@ def test_wrap_command_marks_missing_profiler():
     assert "nsys" in bundle.missing
 
 
+def test_profile_marks_failed_workload_command(tmp_path):
+    import sys
+
+    from gitm.bench.profile import ProfilerTools, run_profile
+
+    bundle = run_profile(
+        _hft_config(),
+        [sys.executable, "-c", "raise SystemExit(7)"],
+        tmp_path,
+        tools=ProfilerTools(nsys=None, rocprof=None, py_spy=None, sar=None),
+    )
+
+    assert any("exit 7" in item for item in bundle.missing)
+    assert not bundle.complete
+
+
+def test_breakdown_refuses_to_clamp_overlapping_timings():
+    from gitm.bench.profile import PhaseTiming, build_breakdown
+
+    with pytest.raises(ValueError, match="refusing to clamp"):
+        build_breakdown(
+            [PhaseTiming("overlap", wall_clock_s=1.0, gpu_busy_s=0.8, cpu_s=0.4)]
+        )
+
+
 # --- edge manifest ----------------------------------------------------------
 
 
diff --git a/tests/test_cli_attach.py b/tests/test_cli_attach.py
index 34dd2e9..1190e3a 100644
--- a/tests/test_cli_attach.py
+++ b/tests/test_cli_attach.py
@@ -35,3 +35,15 @@ def test_main_attach_returns_zero_on_plan(capsys):
     assert rc == 0
     out = json.loads(capsys.readouterr().out)
     assert out["job_id"] == "j" and out["status"] == "planned"
+
+
+def test_live_pid_is_not_falsely_reported_attached(monkeypatch):
+    import gitm.deploy.attach as attach
+
+    monkeypatch.setattr(attach, "_pid_is_live", lambda _pid: True)
+
+    plan = attach_job("job-live", pid=4321, dry_run=False)
+
+    assert plan["status"] == "unsupported"
+    assert "not implemented" in plan["reason"]
+    assert "no telemetry shim was installed" in plan["reason"]
diff --git a/tests/test_gate_wiring.py b/tests/test_gate_wiring.py
index 3afb9c1..82146a7 100644
--- a/tests/test_gate_wiring.py
+++ b/tests/test_gate_wiring.py
@@ -67,3 +67,14 @@ def test_missing_library_refuses_with_named_path(tmp_path):
     missing = tmp_path / "missing.yaml"
     with pytest.raises(FileNotFoundError, match="candidate coverage is unavailable"):
         load_library(missing)
+
+
+def test_empty_library_refuses_instead_of_looking_like_no_applicable_levers(tmp_path):
+    import pytest
+
+    from gitm.kernels.library import load_library
+
+    empty = tmp_path / "empty.yaml"
+    empty.write_text("interventions: []\n")
+    with pytest.raises(ValueError, match="candidate coverage is unavailable"):
+        load_library(empty)
diff --git a/tests/test_routing.py b/tests/test_routing.py
new file mode 100644
index 0000000..5c2c7fe
--- /dev/null
+++ b/tests/test_routing.py
@@ -0,0 +1,19 @@
+from __future__ import annotations
+
+import pytest
+
+from gitm.routing.scorer_v0 import score_prospect
+
+
+def test_unknown_company_tier_is_not_silently_scored_as_tier_three():
+    with pytest.raises(ValueError, match="company_tier"):
+        score_prospect(0.5, 0.5, 99, 1, 0.5, 0)
+
+
+def test_out_of_range_inputs_are_refused():
+    with pytest.raises(ValueError, match=r"within \[0, 1\]"):
+        score_prospect(1.5, 0.5, 1, 1, 0.5, 0)
+
+
+def test_valid_score_contract_is_unchanged():
+    assert score_prospect(1.0, 1.0, 1, 1, 1.0, 1) == 100.0
diff --git a/tests/test_safety_primitives.py b/tests/test_safety_primitives.py
index 5d49a91..9d8a77f 100644
--- a/tests/test_safety_primitives.py
+++ b/tests/test_safety_primitives.py
@@ -75,6 +75,20 @@ def test_failopen_failures_reset_on_reentry():
     assert g.failures == []
 
 
+def test_failopen_surfaces_broken_audit_sink():
+    class BrokenAudit:
+        def record(self, *_args, **_kwargs):
+            raise OSError("disk full")
+
+    g = FailOpenGuard(audit=BrokenAudit(), install_signal_handlers=False)
+    with pytest.warns(RuntimeWarning, match="audit sink failed"):
+        with g:
+            g.register("x", lambda: None)
+
+    assert g.audit_failures
+    assert "disk full" in g.audit_failures[0]
+
+
 # --------- auto-revert ---------------------------------------------------------
 def test_autorevert_warms_up_then_holds_within_tolerance():
     ar = AutoRevert(baseline=100.0, tolerance=0.05, window=3)

From 1a684252900a3690b9c22c607d78e84debe00608 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:15:02 -0700
Subject: [PATCH 12/70] fix: refuse invalid importer rollups

---
 gitm/importers/analyze.py      | 11 ++++++++++-
 gitm/importers/node_rollup.py  | 11 +++++++----
 tests/test_importers.py        |  2 +-
 tests/test_node_rollup_math.py | 13 +++++++++++++
 4 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/gitm/importers/analyze.py b/gitm/importers/analyze.py
index 178799f..aa5a111 100644
--- a/gitm/importers/analyze.py
+++ b/gitm/importers/analyze.py
@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import json
+import re
 import sys
 from dataclasses import dataclass, field
 from datetime import datetime, timezone
@@ -161,6 +162,12 @@ def _device_id_of(trace: Trace) -> int:
     return trace.events[0].device_id if trace.events else 0
 
 
+def _artifact_stem(value: str) -> str:
+    """Return a portable filename stem without changing the report identifier."""
+    stem = re.sub(r"[^A-Za-z0-9_.-]+", "_", value).strip("._")
+    return stem or "trace"
+
+
 def analyze_paths(
     paths: list[str | Path],
     *,
@@ -256,7 +263,9 @@ def analyze_paths(
                 if keep_traces is not None:
                     keep_dir = Path(keep_traces)
                     keep_dir.mkdir(parents=True, exist_ok=True)
-                    tpath = keep_dir / f"{internal_id}_{tr.run_id}.jsonl"
+                    tpath = keep_dir / (
+                        f"{_artifact_stem(internal_id)}_{_artifact_stem(tr.run_id)}.jsonl"
+                    )
                     write_trace_jsonl(tpath, tr)
                     eng_bits.append(render_headroom_md(headroom))
 
diff --git a/gitm/importers/node_rollup.py b/gitm/importers/node_rollup.py
index c30bfd4..4239a02 100644
--- a/gitm/importers/node_rollup.py
+++ b/gitm/importers/node_rollup.py
@@ -10,6 +10,7 @@
 from dataclasses import dataclass, field
 from typing import Any
 
+from gitm._timing import require_positive_duration
 from gitm.optimizer.metrics import _merge_intervals
 from gitm.tracer.schema import Trace
 
@@ -149,7 +150,9 @@ def device_comm_stats(trace: Trace) -> DeviceCommStats:
     kernels = [e for e in trace.events if getattr(e, "kind", None) == "kernel"]
     # Prefer the device_id on events; fall back to 0.
     dev = kernels[0].device_id if kernels else 0
-    wall = max(trace.duration_ns, 1)
+    wall = require_positive_duration(
+        float(trace.duration_ns), context=f"device {dev} communication rollup"
+    )
     comm = [(k.start_ns, k.end_ns) for k in kernels if is_comm_kernel(k.name)]
     non_comm = [(k.start_ns, k.end_ns) for k in kernels if not is_comm_kernel(k.name)]
     busy_ns = _interval_len([(k.start_ns, k.end_ns) for k in kernels])
@@ -210,8 +213,8 @@ def build_node_rollup(
         comm_stats.append(cs)
         if cs.comm_ns > 0:
             any_comm = True
-        weight_sum += max(wall_s, 0.0)
-        weighted_ceiling += ceiling * max(wall_s, 0.0)
+        weight_sum += wall_s
+        weighted_ceiling += ceiling * wall_s
 
     busies = list(device_busy.values())
     skew = (max(busies) - min(busies)) if busies else 0.0
@@ -219,7 +222,7 @@ def build_node_rollup(
     n_devices = len(device_busy)
     multi = multi_device_file or n_devices > 1
     comm_inconclusive = multi and not any_comm
-    node_ceiling = (weighted_ceiling / weight_sum) if weight_sum > 0 else 0.0
+    node_ceiling = weighted_ceiling / weight_sum
     mean_exposed = (
         sum(c.exposed_comm_share_of_wall for c in comm_stats) / len(comm_stats)
         if comm_stats
diff --git a/tests/test_importers.py b/tests/test_importers.py
index ce22b95..03bbf12 100644
--- a/tests/test_importers.py
+++ b/tests/test_importers.py
@@ -447,7 +447,7 @@ def test_golden_customer_report(tmp_path):
 
     if os.environ.get("GITM_UPDATE_GOLDEN") == "1":
         golden.write_text(md)
-    assert md == golden.read_text(), (
+    assert md == golden.read_text(encoding="utf-8"), (
         "customer report drifted from golden — set GITM_UPDATE_GOLDEN=1 if intentional"
     )
 
diff --git a/tests/test_node_rollup_math.py b/tests/test_node_rollup_math.py
index d99a4f5..d42191b 100644
--- a/tests/test_node_rollup_math.py
+++ b/tests/test_node_rollup_math.py
@@ -125,6 +125,13 @@ def test_device_comm_no_comm_kernels():
     assert cs.comm_share_of_busy == 0.0
 
 
+@pytest.mark.parametrize("duration", [0, -1])
+def test_device_comm_refuses_nonpositive_wall_time(duration: int):
+    tr = _trace([], duration=duration)
+    with pytest.raises(RuntimeError, match="communication rollup timing unavailable"):
+        device_comm_stats(tr)
+
+
 # ── rollup skew / collective flags ───────────────────────────────────────────
 
 
@@ -164,6 +171,12 @@ def test_rollup_weighted_ceiling():
     assert r.node_ceiling_distance == pytest.approx(0.325)
 
 
+def test_rollup_refuses_nonpositive_device_wall_time():
+    invalid = _trace([], duration=0)
+    with pytest.raises(RuntimeError, match="communication rollup timing unavailable"):
+        build_node_rollup([(invalid, 0.0, 0.4)], multi_device_file=False)
+
+
 def test_rollup_comm_inconclusive():
     t0 = _trace([_k("gemm", 0, 50, device=0)], duration=100)
     t1 = _trace([_k("gemm", 0, 50, device=1)], duration=100)

From 199715932bb1c18ce462c489a9122b74167c814c Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:15:35 -0700
Subject: [PATCH 13/70] fix: make degraded loop outcomes fail visibly

---
 gitm/cli.py                     | 22 ++++++++++---
 tests/test_run_loop_workload.py | 58 +++++++++++++++++++++++++++++++++
 2 files changed, 75 insertions(+), 5 deletions(-)

diff --git a/gitm/cli.py b/gitm/cli.py
index 47c042a..24804a5 100644
--- a/gitm/cli.py
+++ b/gitm/cli.py
@@ -309,14 +309,26 @@ def main(argv: list[str] | None = None) -> int:
             target=_parse_target(args.target),
             scratch=args.scratch,
         )
-        summary = result.get("summary", {})
+        summary = result.get("summary")
+        if not isinstance(summary, dict):
+            summary = {
+                "status": "invalid_result",
+                "diagnostic": "optimization loop returned no machine-readable summary",
+            }
         if args.report is not None:
-            args.report.write_text(result.get("report_md", ""))
+            report_md = result.get("report_md")
+            if not isinstance(report_md, str) or not report_md.strip():
+                report_md = (
+                    "# Runtime result unavailable\n\n"
+                    f"{summary.get('diagnostic', 'optimization loop returned no report')}\n"
+                )
+            args.report.write_text(report_md, encoding="utf-8")
         else:
             print(json.dumps(summary, indent=2))
-        # Non-zero so automation notices a run that measured nothing (no GPU /
-        # CUPTI shim, or the workload never ran) instead of seeing a fake pass.
-        return 3 if summary.get("status") == "no_data" else 0
+        # Only the explicit success state is a shell success. Prediction/candidate
+        # refusals and failed A/Bs may still have useful measurement reports, but
+        # automation must not read those degraded outcomes as a completed run.
+        return 0 if summary.get("status") == "ok" else 3
 
     if args.cmd == "replay":
         from gitm.optimizer.replay import predict_delta_from_files
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index d46be6e..646ab09 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -35,6 +35,34 @@ def test_no_data_guard_does_not_fabricate_claims(tmp_path: Path):
     assert "NO DATA" in result["report_md"]
 
 
+def test_zero_duration_kernel_does_not_bypass_no_data_guard(tmp_path, monkeypatch):
+    import gitm.scheduler.loop as loop
+
+    @contextmanager
+    def invalid_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
+        yield make_trace(
+            events=[make_kernel("invalid", start_ns=10, end_ns=10)],
+            vendor="nvidia",
+            run_id=run_id or "r",
+        )
+
+    monkeypatch.setattr(loop, "capture", invalid_capture)
+    monkeypatch.setattr(loop, "sync_device", lambda: None)
+
+    from gitm import optimize
+
+    result = optimize(
+        workload="custom",
+        budget="1s",
+        scratch=str(tmp_path),
+        workload_runner=lambda: {"events": 1},
+    )
+
+    assert result["summary"]["status"] == "no_data"
+    assert result["summary"]["n_claims"] == 0
+    assert "positive-duration" in result["report_md"]
+
+
 _UNSET = object()  # identity sentinel — a real workload_id could legitimately be any string
 
 
@@ -418,6 +446,36 @@ def test_cli_run_returns_nonzero_on_no_data(tmp_path: Path, capsys):
     assert rc == 3
 
 
+@pytest.mark.parametrize(
+    "status",
+    ["prediction_refused", "candidate_coverage_unavailable", "intervention_failed"],
+)
+def test_cli_run_returns_nonzero_for_every_degraded_status(status, monkeypatch):
+    import gitm
+    from gitm.cli import main
+
+    monkeypatch.setattr(
+        gitm,
+        "optimize",
+        lambda **_kwargs: {"summary": {"status": status}, "report_md": "diagnostic"},
+    )
+
+    assert main(["run", "--workload", "vllm-decode", "--budget", "1s"]) == 3
+
+
+def test_cli_run_refuses_malformed_loop_result(monkeypatch, tmp_path):
+    import gitm
+    from gitm.cli import main
+
+    monkeypatch.setattr(gitm, "optimize", lambda **_kwargs: {})
+    report = tmp_path / "report.md"
+
+    rc = main(["run", "--workload", "vllm-decode", "--report", str(report)])
+
+    assert rc == 3
+    assert "returned no machine-readable summary" in report.read_text()
+
+
 def test_hft_harness_importable_from_package():
     """The harness must ship in the wheel, i.e. be importable from the package."""
     from gitm.benchmarks.hft import harness

From 4d4f250c506fc44264dde9d5c3bbee0ce540bc61 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:15:55 -0700
Subject: [PATCH 14/70] fix: reject invalid kernel timestamps

---
 gitm/scheduler/loop.py | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 40f3221..6d32c11 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -752,13 +752,13 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
                 trace_path=trace_path,
             )
 
-    # Guard: if the tracer captured nothing (no GPU/shim, or the workload never
-    # ran), do NOT proceed to attribution + emit claims — that fabricates a
-    # result from an empty trace. Report no-data honestly instead.
-    if trace.vendor == "none" or not trace.kernels():
-        diagnostic = runner_error or qual.diagnostic or (
-            "Tracer captured no GPU kernels. Either no GPU/CUPTI shim is present, "
-            "or the workload did not run under the runtime."
+    # Guard: a kernel launch with no positive duration is not measurement
+    # coverage. Do not classify it or emit claims from a fabricated denominator.
+    valid_kernels = [k for k in trace.kernels() if k.end_ns > k.start_ns]
+    if trace.vendor == "none" or not valid_kernels:
+        diagnostic = runner_error or (
+            "Tracer captured no positive-duration GPU kernels. Either no GPU/CUPTI "
+            "shim is present, the workload did not run, or kernel timestamps are invalid."
         )
         return _no_data_result(
             run_dir=run_dir,
@@ -1799,8 +1799,8 @@ def _no_data_result(
 ) -> dict[str, Any]:
     """Write an honest no-data report and return its summary (status=no_data).
 
-    Used when the trace has no kernels — a misconfigured box or a workload that
-    never ran. We emit zero claims rather than fabricating results from nothing.
+    Used when the trace has no positive-duration kernels — a misconfigured box,
+    a workload that never ran, or invalid timestamps. We emit zero claims.
     """
     provenance = build_provenance(
         workload_id=workload,
@@ -1813,7 +1813,7 @@ def _no_data_result(
         claims=[],
         provenance=provenance,
         qualification_diagnostic=diagnostic,
-        summary="NO DATA — tracer captured no GPU kernels; nothing was measured.",
+        summary="NO DATA — tracer captured no positive-duration GPU kernels; nothing was measured.",
     )
     _write_report(run_dir, report_md)
 

From c12cc6fd097bbaea7f6925cc344c00de385a2b30 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:34:17 -0700
Subject: [PATCH 15/70] fix: surface partial serving telemetry

---
 gitm/serve/metrics.py             | 35 +++++++++++-
 gitm/telemetry/backends/nvidia.py | 75 +++++++++++++++++++------
 gitm/telemetry/collector.py       |  5 ++
 gitm/telemetry/schema.py          |  4 ++
 gitm/tracer/vllm_stats.py         | 92 ++++++++++++++++++++++++-------
 tests/test_serve_metrics.py       | 11 ++++
 tests/test_serving_latency.py     | 14 +++++
 tests/test_telemetry_fallbacks.py | 30 ++++++++++
 tests/test_vllm_stress.py         | 17 ++++--
 9 files changed, 237 insertions(+), 46 deletions(-)

diff --git a/gitm/serve/metrics.py b/gitm/serve/metrics.py
index 45bbb11..afd6288 100644
--- a/gitm/serve/metrics.py
+++ b/gitm/serve/metrics.py
@@ -20,9 +20,11 @@
 from __future__ import annotations
 
 import json
+import math
 import threading
 import time
 import urllib.request
+import warnings
 from dataclasses import asdict, dataclass, field
 from pathlib import Path
 
@@ -88,7 +90,7 @@ def parse_prometheus(text: str) -> dict[str, float]:
             value = float(value_str.split()[0])
         except (ValueError, IndexError):
             continue
-        if value != value:  # NaN: an untouched histogram, not a zero
+        if not math.isfinite(value):  # untouched/invalid metric, not an observed zero
             continue
         out[name] = out.get(name, 0.0) + value
     return out
@@ -170,8 +172,17 @@ def window_from_snapshots(
     w.ttft_mean_s = _mean(before, after, "ttft_sum", "ttft_count")
     w.tpot_mean_s = _mean(before, after, "tpot_sum", "tpot_count")
     w.e2e_mean_s = _mean(before, after, "e2e_sum", "e2e_count")
-    if w.generation_tokens is not None and window_s:
+    if (
+        w.generation_tokens is not None
+        and window_s is not None
+        and math.isfinite(window_s)
+        and window_s > 0.0
+    ):
         w.output_tokens_per_s = w.generation_tokens / window_s
+    elif window_s is not None and (not math.isfinite(window_s) or window_s <= 0.0):
+        w.notes.append(
+            f"throughput unavailable: capture window must be finite and positive, got {window_s!r}"
+        )
 
     if samples:
         w.n_samples = len(samples)
@@ -225,7 +236,7 @@ def snapshot_metrics(base_url: str, dest: Path, timeout: float = 15.0) -> str:
     """
     text = fetch_metrics(base_url, timeout)
     dest.parent.mkdir(parents=True, exist_ok=True)
-    dest.write_text(text)
+    dest.write_text(text, encoding="utf-8")
     return text
 
 
@@ -240,6 +251,13 @@ class MetricsSampler:
 
     def __init__(self, base_url: str, *, interval_s: float = 1.0) -> None:
         self.base_url = base_url
+        if not math.isfinite(interval_s) or interval_s <= 0.0:
+            raise ValueError(f"metrics sampling interval must be finite and positive, got {interval_s!r}")
+        self.diagnostics: list[str] = []
+        if interval_s < 0.05:
+            self._record_failure(
+                f"requested {interval_s!r}s; using the 0.05s minimum sampling interval"
+            )
         self.interval_s = max(interval_s, 0.05)
         self.samples: list[dict] = []
         self._stop = threading.Event()
@@ -254,6 +272,8 @@ def start(self) -> None:
     def _run(self) -> None:
         while not self._stop.is_set():
             text = fetch_metrics(self.base_url, timeout=min(self.interval_s * 4, 10.0))
+            if text.startswith("# unavailable:"):
+                self._record_failure(text.removeprefix("# unavailable:").strip())
             snap = parse_prometheus(text)
             if snap:
                 self.samples.append(
@@ -270,9 +290,18 @@ def stop(self) -> list[dict]:
         self._stop.set()
         if self._thread is not None:
             self._thread.join(timeout=5.0)
+            if self._thread.is_alive():
+                self._record_failure("metrics sampler did not stop within 5 seconds")
             self._thread = None
         return self.samples
 
+    def _record_failure(self, detail: str) -> None:
+        message = f"metrics sampling degraded: {detail}"
+        if message in self.diagnostics:
+            return
+        self.diagnostics.append(message)
+        warnings.warn(message, RuntimeWarning, stacklevel=2)
+
     def write(self, dest: Path) -> None:
         dest.parent.mkdir(parents=True, exist_ok=True)
         with dest.open("w", encoding="utf-8") as fh:
diff --git a/gitm/telemetry/backends/nvidia.py b/gitm/telemetry/backends/nvidia.py
index 46e5652..c90f18d 100644
--- a/gitm/telemetry/backends/nvidia.py
+++ b/gitm/telemetry/backends/nvidia.py
@@ -32,14 +32,22 @@
 }
 
 
-def _try(fn: Callable[[], T], default: T | None = None) -> T | None:
+def _try(
+    fn: Callable[[], T],
+    default: T | None = None,
+    *,
+    diagnostics: list[str] | None = None,
+    field: str = "NVML field",
+) -> T | None:
     """Call ``fn``; return ``default`` on any exception.
 
     Used per NVML call so a single failure doesn't drop the whole sample.
     """
     try:
         return fn()
-    except Exception:
+    except Exception as exc:
+        if diagnostics is not None:
+            diagnostics.append(f"{field} unavailable ({type(exc).__name__}: {exc})")
         return default
 
 
@@ -85,33 +93,68 @@ def device_count(self) -> int:
     def sample(self, gpu_index: int, labels: WorkloadLabels | None = None) -> Sample:
         nv = self._pynvml
         h = self._handle(gpu_index)
+        diagnostics: list[str] = []
 
-        uuid = _try(lambda: nv.nvmlDeviceGetUUID(h))
+        uuid = _try(lambda: nv.nvmlDeviceGetUUID(h), diagnostics=diagnostics, field="GPU UUID")
         if isinstance(uuid, bytes):
             uuid = uuid.decode()
 
-        util = _try(lambda: nv.nvmlDeviceGetUtilizationRates(h))
-        mem = _try(lambda: nv.nvmlDeviceGetMemoryInfo(h))
-        power_mw = _try(lambda: nv.nvmlDeviceGetPowerUsage(h))
-        temp = _try(lambda: nv.nvmlDeviceGetTemperature(h, nv.NVML_TEMPERATURE_GPU))
-        sm_clock = _try(lambda: nv.nvmlDeviceGetClockInfo(h, nv.NVML_CLOCK_SM))
-        mem_clock = _try(lambda: nv.nvmlDeviceGetClockInfo(h, nv.NVML_CLOCK_MEM))
-        throttle_bits = _try(lambda: nv.nvmlDeviceGetCurrentClocksThrottleReasons(h), 0) or 0
+        util = _try(
+            lambda: nv.nvmlDeviceGetUtilizationRates(h),
+            diagnostics=diagnostics,
+            field="GPU utilization",
+        )
+        mem = _try(
+            lambda: nv.nvmlDeviceGetMemoryInfo(h), diagnostics=diagnostics, field="GPU memory"
+        )
+        power_mw = _try(
+            lambda: nv.nvmlDeviceGetPowerUsage(h), diagnostics=diagnostics, field="GPU power"
+        )
+        temp = _try(
+            lambda: nv.nvmlDeviceGetTemperature(h, nv.NVML_TEMPERATURE_GPU),
+            diagnostics=diagnostics,
+            field="GPU temperature",
+        )
+        sm_clock = _try(
+            lambda: nv.nvmlDeviceGetClockInfo(h, nv.NVML_CLOCK_SM),
+            diagnostics=diagnostics,
+            field="SM clock",
+        )
+        mem_clock = _try(
+            lambda: nv.nvmlDeviceGetClockInfo(h, nv.NVML_CLOCK_MEM),
+            diagnostics=diagnostics,
+            field="memory clock",
+        )
+        throttle_bits = _try(
+            lambda: nv.nvmlDeviceGetCurrentClocksThrottleReasons(h),
+            0,
+            diagnostics=diagnostics,
+            field="clock throttle reasons",
+        ) or 0
 
         per_proc: dict[int, float] = {}
-        procs = _try(lambda: nv.nvmlDeviceGetComputeRunningProcesses(h), []) or []
+        procs = _try(
+            lambda: nv.nvmlDeviceGetComputeRunningProcesses(h),
+            [],
+            diagnostics=diagnostics,
+            field="compute process list",
+        ) or []
         for p in procs:
             per_proc[int(p.pid)] = 0.0  # NVML doesn't expose per-process util directly
 
         ecc_sbe = _try(
             lambda: nv.nvmlDeviceGetTotalEccErrors(
                 h, nv.NVML_MEMORY_ERROR_TYPE_CORRECTED, nv.NVML_VOLATILE_ECC
-            )
+            ),
+            diagnostics=diagnostics,
+            field="corrected ECC counter",
         )
         ecc_dbe = _try(
             lambda: nv.nvmlDeviceGetTotalEccErrors(
                 h, nv.NVML_MEMORY_ERROR_TYPE_UNCORRECTED, nv.NVML_VOLATILE_ECC
-            )
+            ),
+            diagnostics=diagnostics,
+            field="uncorrected ECC counter",
         )
 
         return Sample(
@@ -132,11 +175,9 @@ def sample(self, gpu_index: int, labels: WorkloadLabels | None = None) -> Sample
             per_process=per_proc,
             ecc_volatile_sbe=int(ecc_sbe) if ecc_sbe is not None else None,
             ecc_volatile_dbe=int(ecc_dbe) if ecc_dbe is not None else None,
+            diagnostics=diagnostics,
             labels=labels,
         )
 
     def close(self) -> None:
-        try:
-            self._pynvml.nvmlShutdown()
-        except Exception:
-            pass
+        self._pynvml.nvmlShutdown()
diff --git a/gitm/telemetry/collector.py b/gitm/telemetry/collector.py
index 18f3eb3..4955eac 100644
--- a/gitm/telemetry/collector.py
+++ b/gitm/telemetry/collector.py
@@ -92,6 +92,11 @@ def _run(self) -> None:
                             f"sample failed ({type(exc).__name__}: {exc})",
                         )
                         continue
+                    for diagnostic in sample.diagnostics:
+                        self._record_failure(
+                            f"sample-field:{type(backend).__name__}:{idx}:{diagnostic.split(' ', 1)[0]}",
+                            diagnostic,
+                        )
                     for sink in self._cfg.sinks:
                         try:
                             sink.emit(sample)
diff --git a/gitm/telemetry/schema.py b/gitm/telemetry/schema.py
index ad2a2d9..acd5088 100644
--- a/gitm/telemetry/schema.py
+++ b/gitm/telemetry/schema.py
@@ -75,4 +75,8 @@ class Sample(BaseModel):
     # keep working unchanged.
     extra: dict[str, float] = Field(default_factory=dict)
 
+    # Per-field backend failures. A partial sample remains usable, but consumers
+    # must not read a missing throttle/process field as an observed zero.
+    diagnostics: list[str] = Field(default_factory=list)
+
     labels: WorkloadLabels | None = None
diff --git a/gitm/tracer/vllm_stats.py b/gitm/tracer/vllm_stats.py
index 8f2bf22..30ce319 100644
--- a/gitm/tracer/vllm_stats.py
+++ b/gitm/tracer/vllm_stats.py
@@ -25,11 +25,13 @@
 
 from __future__ import annotations
 
+import math
 import threading
 import time
+import warnings
 from collections.abc import Iterator
 from contextlib import contextmanager
-from dataclasses import asdict, dataclass
+from dataclasses import asdict, dataclass, field
 from typing import Any
 
 
@@ -66,6 +68,7 @@ class SchedulerStatsSummary:
     # series on the same wall clock as vLLM's per-request timestamps and as
     # ``Trace.captured_at_ns`` — the join across the three views. 0 when unset.
     t0_wall_ns: int = 0
+    diagnostics: list[str] = field(default_factory=list)
 
 
 # SLO defaults for goodput. Serving-shaped starting points, not tuned: a request
@@ -103,7 +106,8 @@ def ttft_s(self) -> float | None:
         """Time to first token — the queue+prefill wait the user actually feels."""
         if self.arrival_wall_s is None or self.first_token_wall_s is None:
             return None
-        return max(self.first_token_wall_s - self.arrival_wall_s, 0.0)
+        span = self.first_token_wall_s - self.arrival_wall_s
+        return span if math.isfinite(span) and span >= 0.0 else None
 
     @property
     def tpot_s(self) -> float | None:
@@ -120,7 +124,9 @@ def tpot_s(self) -> float | None:
             return None
         if self.n_output_tokens < 2:
             return None
-        span = max(self.finished_wall_s - self.first_token_wall_s, 0.0)
+        span = self.finished_wall_s - self.first_token_wall_s
+        if not math.isfinite(span) or span < 0.0:
+            return None
         return span / (self.n_output_tokens - 1)
 
     @property
@@ -130,7 +136,9 @@ def estimated_tpot_s(self) -> float | None:
             return None
         if self.first_token_wall_s is None or self.finished_wall_s is None:
             return None
-        span = max(self.finished_wall_s - self.first_token_wall_s, 0.0)
+        span = self.finished_wall_s - self.first_token_wall_s
+        if not math.isfinite(span) or span < 0.0:
+            return None
         return span / (self.n_output_tokens - 1)
 
     def meets_slo(self, ttft_slo_s: float, tpot_slo_s: float) -> bool:
@@ -245,16 +253,45 @@ def summarize_requests(
     estimated_tpots = [t for t in (r.estimated_tpot_s for r in records) if t is not None]
     met = [r for r in records if r.meets_slo(ttft_slo_s, tpot_slo_s)]
 
-    arrivals = [r.arrival_wall_s for r in records if r.arrival_wall_s is not None]
-    finishes = [r.finished_wall_s for r in records if r.finished_wall_s is not None]
+    arrivals = [
+        r.arrival_wall_s
+        for r in records
+        if r.arrival_wall_s is not None and math.isfinite(r.arrival_wall_s)
+    ]
+    finishes = [
+        r.finished_wall_s
+        for r in records
+        if r.finished_wall_s is not None and math.isfinite(r.finished_wall_s)
+    ]
     window_s: float | None = None
     if arrivals and finishes:
-        window_s = max(max(finishes) - min(arrivals), 0.0)
+        span = max(finishes) - min(arrivals)
+        window_s = span if span >= 0.0 else None
     # A zero-length window (single instantaneous request, or clock granularity)
     # has no meaningful rate — report the count, not a division by ~0.
     goodput = (len(met) / window_s) if window_s else None
 
     warnings: list[str] = []
+    def _has_invalid_timestamps(r: RequestRecord) -> bool:
+        pairs = (
+            (r.arrival_wall_s, r.first_token_wall_s),
+            (r.first_token_wall_s, r.finished_wall_s),
+        )
+        return any(
+            start is not None
+            and end is not None
+            and (not math.isfinite(start) or not math.isfinite(end) or end < start)
+            for start, end in pairs
+        )
+
+    invalid_timestamps = sum(_has_invalid_timestamps(r) for r in records)
+    if invalid_timestamps:
+        warnings.append(
+            f"latency coverage: {invalid_timestamps} request(s) have non-finite or "
+            "non-monotonic timestamps; excluded from latency and SLO goodput"
+        )
+    if records and arrivals and finishes and not window_s:
+        warnings.append("goodput unavailable: request window has no positive duration")
     if estimated_tpots:
         warnings.append(
             f"TPOT coverage: {len(estimated_tpots)} request(s) use SSE chunk-count estimates; "
@@ -419,14 +456,6 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non
                     setattr(sample, field_name, val)
                     saw_any = True
 
-        # vLLM V1: fill running / waiting / cache from the scheduler's stat object
-        # where the VO deques weren't exposed (they read empty on V1)
-        for sch in schedulers:
-            for field_name, val in _v1_scheduler_stats(sch).items():
-                if getattr(sample, field_name) is None:
-                    setattr(sample, field_name, val)
-                    saw_any = True
-
     # Total unfinished — a stable public method on LLMEngine across versions.
     getter = _first_attr(
         engine,
@@ -481,6 +510,17 @@ class SchedulerStatsSampler:
 
     def __init__(self, engine: Any, *, interval_s: float = 0.05) -> None:
         self.engine = engine
+        if not math.isfinite(interval_s) or interval_s <= 0.0:
+            raise ValueError(
+                f"scheduler sampling interval must be finite and positive, got {interval_s!r}"
+            )
+        self.diagnostics: list[str] = []
+        self._diagnostic_keys: set[str] = set()
+        if interval_s < 1e-3:
+            self._record_failure(
+                "interval-clamped",
+                f"requested {interval_s!r}s; using the 0.001s minimum",
+            )
         self.interval_s = max(interval_s, 1e-3)
         self.samples: list[SchedulerSample] = []
         self._stop = threading.Event()
@@ -504,8 +544,8 @@ def start(self) -> None:
             s0 = read_scheduler_stats(self.engine, t_ns=0)
             if s0 is not None:
                 self.samples.append(s0)
-        except Exception:
-            pass
+        except Exception as exc:
+            self._record_failure("initial-read", f"scheduler stats read failed: {exc}")
         self._stop.clear()
         self._thread = threading.Thread(target=self._run, name="gitm-vllm-stats", daemon=True)
         self._thread.start()
@@ -516,8 +556,8 @@ def _run(self) -> None:
                 s = read_scheduler_stats(self.engine, t_ns=time.perf_counter_ns() - self._t0_ns)
                 if s is not None:
                     self.samples.append(s)
-            except Exception:
-                pass  # best-effort; never let sampling crash the run
+            except Exception as exc:
+                self._record_failure("background-read", f"scheduler stats read failed: {exc}")
             self._stop.wait(self.interval_s)
 
     def stop(self) -> None:
@@ -525,13 +565,25 @@ def stop(self) -> None:
             return
         self._stop.set()
         self._thread.join(timeout=2.0)
+        if self._thread.is_alive():
+            self._record_failure("join-timeout", "scheduler sampler did not stop within 2 seconds")
         self._thread = None
 
+    def _record_failure(self, key: str, detail: str) -> None:
+        if key in self._diagnostic_keys:
+            return
+        self._diagnostic_keys.add(key)
+        message = f"scheduler telemetry degraded [{key}]: {detail}"
+        self.diagnostics.append(message)
+        warnings.warn(message, RuntimeWarning, stacklevel=2)
+
     def summary(self) -> SchedulerStatsSummary:
         # Snapshot first: stop() joins with a timeout, so in the pathological case
         # where the daemon thread is still alive, summarize must iterate a stable
         # copy rather than a list being appended to concurrently.
-        return summarize(list(self.samples), t0_wall_ns=self._t0_wall_ns)
+        result = summarize(list(self.samples), t0_wall_ns=self._t0_wall_ns)
+        result.diagnostics.extend(self.diagnostics)
+        return result
 
     def to_records(self) -> list[dict[str, Any]]:
         """Samples as plain dicts, ready for JSONL alongside the kernel trace."""
diff --git a/tests/test_serve_metrics.py b/tests/test_serve_metrics.py
index 6920c0d..348216c 100644
--- a/tests/test_serve_metrics.py
+++ b/tests/test_serve_metrics.py
@@ -62,6 +62,11 @@ def test_parse_ignores_comments_and_nan():
     assert snap["vllm:num_requests_running"] == 3.0
 
 
+def test_parse_ignores_infinite_values():
+    snap = metrics.parse_prometheus("vllm:num_requests_running +Inf\n")
+    assert "vllm:num_requests_running" not in snap
+
+
 def test_window_is_the_difference_not_the_lifetime():
     w = metrics.window_from_snapshots(BEFORE, AFTER, window_s=10.0)
 
@@ -124,6 +129,12 @@ def test_missing_window_length_leaves_throughput_unset():
     assert w.output_tokens_per_s is None
 
 
+def test_nonpositive_window_refuses_throughput_with_note():
+    w = metrics.window_from_snapshots(BEFORE, AFTER, window_s=0.0)
+    assert w.output_tokens_per_s is None
+    assert any("finite and positive" in note for note in w.notes)
+
+
 def test_fetch_never_raises_on_a_dead_endpoint():
     text = metrics.fetch_metrics("http://127.0.0.1:1", timeout=0.5)
     assert text.startswith("# unavailable")
diff --git a/tests/test_serving_latency.py b/tests/test_serving_latency.py
index e0c2b6a..9cf91ca 100644
--- a/tests/test_serving_latency.py
+++ b/tests/test_serving_latency.py
@@ -45,6 +45,20 @@ def test_missing_timestamps_yield_none_not_zero():
     assert r.ttft_s is None and r.tpot_s is None
 
 
+def test_nonmonotonic_timestamps_are_excluded_and_warned():
+    record = _rec(10.0, 9.0, 8.0, 3)
+
+    summary = summarize_requests([record])
+
+    assert record.ttft_s is None
+    assert record.tpot_s is None
+    assert summary.n_ttft == 0
+    assert summary.n_tpot == 0
+    assert summary.n_met_slo == 0
+    assert summary.goodput_rps is None
+    assert any("non-monotonic" in warning for warning in summary.warnings)
+
+
 # --------------------------------------------------------------------------- #
 # summary: percentiles exclude unmeasurable requests                          #
 # --------------------------------------------------------------------------- #
diff --git a/tests/test_telemetry_fallbacks.py b/tests/test_telemetry_fallbacks.py
index 99531c4..88c62c2 100644
--- a/tests/test_telemetry_fallbacks.py
+++ b/tests/test_telemetry_fallbacks.py
@@ -5,6 +5,7 @@
 import pytest
 
 from gitm.telemetry.collector import Collector, CollectorConfig
+from gitm.telemetry.schema import Sample
 
 
 class _BrokenBackend:
@@ -34,3 +35,32 @@ def test_collector_names_missing_backend_instead_of_looking_idle():
         collector = Collector(CollectorConfig(backends=[]))
 
     assert collector.diagnostics
+
+
+class _PartialBackend:
+    def device_count(self):
+        return 1
+
+    def sample(self, _index, labels=None):
+        return Sample(
+            ts_ns=1,
+            node="n",
+            gpu_uuid="g",
+            gpu_index=0,
+            vendor="nvidia",
+            diagnostics=["clock throttle reasons unavailable (NVMLError: denied)"],
+        )
+
+    def close(self):
+        return None
+
+
+def test_collector_surfaces_partial_sample_field_failure():
+    collector = Collector(CollectorConfig(interval_s=0.001, backends=[_PartialBackend()]))
+
+    with pytest.warns(RuntimeWarning, match="clock throttle reasons unavailable"):
+        collector.start()
+        time.sleep(0.01)
+        collector.stop()
+
+    assert any("clock throttle reasons unavailable" in d for d in collector.diagnostics)
diff --git a/tests/test_vllm_stress.py b/tests/test_vllm_stress.py
index adde9cf..44fc38d 100644
--- a/tests/test_vllm_stress.py
+++ b/tests/test_vllm_stress.py
@@ -9,6 +9,8 @@
 
 import threading
 
+import pytest
+
 from gitm.kernels.spec import InterventionSpec
 from gitm.optimizer.apply import LiveEngineApplicator, apply_intervention
 from gitm.optimizer.deviation import deviating_kernel_indices, deviation_summary, deviation_trace
@@ -93,14 +95,17 @@ def get_num_unfinished_requests(self):
         raise RuntimeError("engine busy")
 
 
-def test_sampler_swallows_engine_exceptions():
+def test_sampler_surfaces_engine_exceptions_without_crashing():
     sampler = SchedulerStatsSampler(_RaisingEngine(), interval_s=0.002)
-    sampler.start()
-    for _ in range(500):
-        pass
-    sampler.stop()  # must not raise
+    with pytest.warns(RuntimeWarning, match="scheduler telemetry degraded"):
+        sampler.start()
+        for _ in range(500):
+            pass
+        sampler.stop()  # must not raise
     # A raising engine yields no usable samples, summary is the empty shape.
-    assert sampler.summary().n_samples == 0
+    summary = sampler.summary()
+    assert summary.n_samples == 0
+    assert summary.diagnostics
 
 
 def test_sampler_repeated_start_stop_is_safe():

From 517c8b1d81a6e534ee63600f7defc8fce7d1dfe5 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:34:23 -0700
Subject: [PATCH 16/70] fix: refuse invalid optimization evidence

---
 gitm/optimizer/apply.py           | 44 ++++++++++++++++++++++++-------
 gitm/optimizer/deviation.py       |  9 +++++--
 gitm/optimizer/metrics.py         | 21 +++++++++++----
 gitm/optimizer/monitor.py         | 34 ++++++++++++++++++++----
 tests/test_apply_rollback.py      | 16 ++++++++++-
 tests/test_deviation_alignment.py |  9 +++++++
 tests/test_metrics.py             | 12 +++++++++
 tests/test_runtime_on_trace.py    | 18 +++++++++++++
 8 files changed, 140 insertions(+), 23 deletions(-)

diff --git a/gitm/optimizer/apply.py b/gitm/optimizer/apply.py
index 9c9704c..be8d87e 100644
--- a/gitm/optimizer/apply.py
+++ b/gitm/optimizer/apply.py
@@ -21,6 +21,7 @@
 
 import copy
 import gc
+import warnings
 from collections.abc import Callable
 from dataclasses import dataclass
 from pathlib import Path
@@ -98,6 +99,13 @@ def apply_intervention(
                            error=f"measure failed, restored: {exc}")
 
     # Step 4: keep-or-rollback on the regression threshold.
+    if delta is None:
+        warnings.warn(
+            f"intervention {spec.name!r} was applied without a measurement; "
+            "the change is unverified",
+            RuntimeWarning,
+            stacklevel=2,
+        )
     if delta is not None and delta < min_keep_delta:
         applicator.restore(snapshot)
         _audit(audit, "revert", spec, knobs=_knob_values(spec),
@@ -117,8 +125,12 @@ def _audit(
         return
     try:
         audit.record(event, spec.name, cause, **detail)
-    except Exception:
-        pass
+    except Exception as exc:
+        warnings.warn(
+            f"intervention safety audit failed for {event} {spec.name!r}: {exc}",
+            RuntimeWarning,
+            stacklevel=2,
+        )
 
 
 def _knob_values(spec: Any) -> dict[str, Any]:
@@ -430,8 +442,10 @@ def _activate(engine: Any) -> None:
         if callable(fn):
             try:
                 fn(engine)
-            except Exception:
-                pass
+            except Exception as exc:
+                warnings.warn(
+                    f"engine activation hook failed: {exc}", RuntimeWarning, stacklevel=2
+                )
 
     @staticmethod
     def _shutdown(engine: Any) -> None:
@@ -440,8 +454,10 @@ def _shutdown(engine: Any) -> None:
         if callable(custom):
             try:
                 custom(engine)
-            except Exception:
-                pass
+            except Exception as exc:
+                warnings.warn(
+                    f"engine shutdown hook failed: {exc}", RuntimeWarning, stacklevel=2
+                )
 
         for path in ("shutdown", "llm_engine.shutdown", "engine.shutdown"):
             obj: Any = engine
@@ -452,8 +468,12 @@ def _shutdown(engine: Any) -> None:
             if callable(obj):
                 try:
                     obj()
-                except Exception:
-                    pass
+                except Exception as exc:
+                    warnings.warn(
+                        f"engine shutdown path {path!r} failed: {exc}",
+                        RuntimeWarning,
+                        stacklevel=2,
+                    )
                 break
         try:
             gc.collect()
@@ -462,8 +482,12 @@ def _shutdown(engine: Any) -> None:
             if torch.cuda.is_available():
                 torch.cuda.empty_cache()
                 torch.cuda.ipc_collect()
-        except Exception:
-            pass
+        except Exception as exc:
+            warnings.warn(
+                f"GPU cleanup after engine shutdown unavailable: {exc}",
+                RuntimeWarning,
+                stacklevel=2,
+            )
 
     def measure(self, spec: InterventionSpec) -> float | None:
         baseline = self._baseline_tps if self._baseline_tps is not None else self._bench_stats()[0]
diff --git a/gitm/optimizer/deviation.py b/gitm/optimizer/deviation.py
index 8bd656b..7ff1ae0 100644
--- a/gitm/optimizer/deviation.py
+++ b/gitm/optimizer/deviation.py
@@ -19,6 +19,7 @@
 
 from __future__ import annotations
 
+import math
 from dataclasses import dataclass
 from pathlib import Path
 
@@ -122,8 +123,12 @@ def _departs(
     inv_mt: Invariant | None,
 ) -> bool:
     """True if observed kernel ``ok`` is out-of-band vs its predicted node."""
-    t_obs = max((ok.end_ns - ok.start_ns) / 1e9, 1e-12)
-    t_pred = max(node_pred_s, 1e-12)
+    t_obs = (ok.end_ns - ok.start_ns) / 1e9
+    if not math.isfinite(t_obs) or t_obs <= 0.0:
+        return True
+    if not math.isfinite(node_pred_s) or node_pred_s <= 0.0:
+        return True
+    t_pred = node_pred_s
     r_kt = (t_obs - t_pred) / t_pred
     if inv_kt is not None and abs(r_kt) > inv_kt.band_width:
         return True
diff --git a/gitm/optimizer/metrics.py b/gitm/optimizer/metrics.py
index 8d93542..ba0d700 100644
--- a/gitm/optimizer/metrics.py
+++ b/gitm/optimizer/metrics.py
@@ -26,6 +26,7 @@
 from collections.abc import Callable, Iterable
 from dataclasses import dataclass
 
+from gitm._timing import require_positive_duration
 from gitm.tracer.schema import KernelEvent, Trace
 
 FlopsModel = Callable[[KernelEvent], float]
@@ -169,8 +170,10 @@ def compute_metrics(
     kernels = trace.kernels()
     memcpys = [e for e in trace.events if getattr(e, "kind", None) == "memcpy"]
     syncs = [e for e in trace.events if getattr(e, "kind", None) == "sync"]
-    wall_s = trace.duration_ns / 1e9
-    busy_fraction = (_merged_busy_ns(kernels) / trace.duration_ns) if trace.duration_ns else 0.0
+    wall_s = require_positive_duration(
+        trace.duration_ns / 1e9, context=f"{trace.workload_id} utilization metrics"
+    )
+    busy_fraction = _merged_busy_ns(kernels) / trace.duration_ns
     gaps = _idle_gaps(kernels, trace.duration_ns)
     stall_breakdown = _classify_stalls(gaps, memcpys, syncs, trace.duration_ns)
 
@@ -179,7 +182,11 @@ def compute_metrics(
     mfu: float | None = None
     if flops_model is not None:
         total_flops = sum(flops_model(k) for k in kernels)
-        achieved_flops = total_flops / wall_s if wall_s else 0.0
+        achieved_flops = total_flops / wall_s
+        if total_flops > 0 and peak.peak_flops <= 0:
+            raise RuntimeError(
+                f"{peak.name} compute utilization unavailable: peak FLOP/s must be positive"
+            )
         if peak.peak_flops > 0:
             hfu = achieved_flops / peak.peak_flops
             mfu = hfu * (1.0 - recompute_fraction)
@@ -187,8 +194,12 @@ def compute_metrics(
     bytes_moved = sum(e.bytes for e in memcpys)
     for k in kernels:
         bytes_moved += (k.bytes_read or 0) + (k.bytes_written or 0)
-    achieved_bw = bytes_moved / wall_s if wall_s else 0.0
-    mbu = achieved_bw / peak.peak_bw_bytes_s if peak.peak_bw_bytes_s > 0 else 0.0
+    achieved_bw = bytes_moved / wall_s
+    if peak.peak_bw_bytes_s <= 0:
+        raise RuntimeError(
+            f"{peak.name} memory utilization unavailable: peak bandwidth must be positive"
+        )
+    mbu = achieved_bw / peak.peak_bw_bytes_s
 
     return MetricsResult(
         n_kernels=len(kernels),
diff --git a/gitm/optimizer/monitor.py b/gitm/optimizer/monitor.py
index f86bbf6..5afb087 100644
--- a/gitm/optimizer/monitor.py
+++ b/gitm/optimizer/monitor.py
@@ -9,6 +9,7 @@
 
 from __future__ import annotations
 
+import math
 from dataclasses import dataclass, field
 
 import numpy as np
@@ -58,6 +59,8 @@ class Residuals:
     total_kernel_time_ns: int = 0
     classified_kernel_time_ns: int = 0
     matched_kernel_time_ns: int = 0
+    invalid_duration_kernels: int = 0
+    unpriced_prediction_kernels: int = 0
 
     @staticmethod
     def _ratio(part: int, whole: int) -> float:
@@ -85,6 +88,18 @@ def coverage_warnings(self) -> list[str]:
         if self.total_kernels == 0:
             return ["residual coverage unavailable: trace contains no kernels"]
         warnings: list[str] = []
+        if self.invalid_duration_kernels:
+            warnings.append(
+                "residual coverage: excluded "
+                f"{self.invalid_duration_kernels}/{self.total_kernels} kernel(s) "
+                "with non-positive timestamps"
+            )
+        if self.unpriced_prediction_kernels:
+            warnings.append(
+                "residual coverage: excluded "
+                f"{self.unpriced_prediction_kernels} matched kernel(s) because "
+                "their predicted duration is non-positive or non-finite"
+            )
         if self.classified_kernels < self.total_kernels:
             warnings.append(
                 "residual coverage: classified "
@@ -167,10 +182,11 @@ def residuals(trace: Trace, graph: Graph) -> Residuals:
     obs = trace.kernels()
     pred = graph.nodes
 
-    durations = [max(ok.end_ns - ok.start_ns, 0) for ok in obs]
+    durations = [ok.end_ns - ok.start_ns for ok in obs]
     res = Residuals(
         total_kernels=len(obs),
-        total_kernel_time_ns=sum(durations),
+        total_kernel_time_ns=sum(d for d in durations if d > 0),
+        invalid_duration_kernels=sum(d <= 0 for d in durations),
     )
     by_op_layer: dict[tuple[str, int], PredictedNode] = {}
     classes: dict[str, dict[tuple[float, float], PredictedNode]] = {}
@@ -180,6 +196,8 @@ def residuals(trace: Trace, graph: Graph) -> Residuals:
         classes.setdefault(pn.op, {}).setdefault(_class_key(pn), pn)
 
     for ok, duration_ns in zip(obs, durations, strict=True):
+        if duration_ns <= 0:
+            continue
         op = ok.range_op or classify_op(ok.name)
         if op is None:
             continue
@@ -191,7 +209,7 @@ def residuals(trace: Trace, graph: Graph) -> Residuals:
         res.matched_kernels += 1
         res.matched_kernel_time_ns += duration_ns
 
-        t_obs = max((ok.end_ns - ok.start_ns) / 1e9, 1e-12)
+        t_obs = duration_ns / 1e9
         b_obs = (
             ok.bytes_read + ok.bytes_written
             if ok.bytes_read is not None and ok.bytes_written is not None
@@ -205,7 +223,10 @@ def residuals(trace: Trace, graph: Graph) -> Residuals:
             pn = cls[0]
 
         if pn is not None:
-            t_pred = max(pn.prediction.t_pred_s, 1e-12)
+            t_pred = pn.prediction.t_pred_s
+            if not math.isfinite(t_pred) or t_pred <= 0.0:
+                res.unpriced_prediction_kernels += 1
+                continue
             r_kt = (t_obs - t_pred) / t_pred
             r_mt = (
                 (b_obs - pn.prediction.bytes) / pn.prediction.bytes
@@ -214,7 +235,10 @@ def residuals(trace: Trace, graph: Graph) -> Residuals:
             )
             layer, bound = ok.range_layer, pn.prediction.bound
         else:
-            ts = sorted(max(c.prediction.t_pred_s, 1e-12) for c in cls)
+            ts = sorted(c.prediction.t_pred_s for c in cls)
+            if any(not math.isfinite(t) or t <= 0.0 for t in ts):
+                res.unpriced_prediction_kernels += 1
+                continue
             r_kt = _interval_residual(t_obs, ts[0], ts[-1])
             bs = sorted(c.prediction.bytes for c in cls if c.prediction.bytes > 0)
             r_mt = _interval_residual(b_obs, bs[0], bs[-1]) if b_obs is not None and bs else None
diff --git a/tests/test_apply_rollback.py b/tests/test_apply_rollback.py
index 46aa2a0..63490ca 100644
--- a/tests/test_apply_rollback.py
+++ b/tests/test_apply_rollback.py
@@ -45,12 +45,26 @@ def test_apply_keeps_on_positive_delta():
 
 def test_apply_only_when_no_measurement_keeps():
     cfg = {"block_size": 8}
-    res = apply_intervention(_spec(), DictApplicator(cfg))  # measure -> None
+    with pytest.warns(RuntimeWarning, match="applied without a measurement"):
+        res = apply_intervention(_spec(), DictApplicator(cfg))  # measure -> None
     assert res.applied and not res.rolled_back
     assert res.measured_delta is None
     assert cfg["block_size"] == 16
 
 
+def test_broken_apply_audit_sink_warns():
+    class BrokenAudit:
+        def record(self, *_args, **_kwargs):
+            raise OSError("disk full")
+
+    with pytest.warns(RuntimeWarning, match="safety audit failed"):
+        result = apply_intervention(
+            _spec(), DictApplicator({"block_size": 8}, measure_fn=lambda _s: 0.1), audit=BrokenAudit()
+        )
+
+    assert result.applied and not result.rolled_back
+
+
 # --- rollback case 1: bad value (apply raises) ------------------------------
 
 
diff --git a/tests/test_deviation_alignment.py b/tests/test_deviation_alignment.py
index a82cd29..6ab15e0 100644
--- a/tests/test_deviation_alignment.py
+++ b/tests/test_deviation_alignment.py
@@ -86,6 +86,15 @@ def test_summary_keys_by_the_observed_kernels_op():
     assert summary["kept_ops"] == {"attn_score_value": 1, "<unmodeled>": 1}
 
 
+def test_zero_duration_kernel_is_kept_as_a_departure():
+    g = predict_graph()
+    tr = _trace([_k("flash_attn_kernel", 0.0)])
+
+    dev = deviating_kernel_indices(tr, g)
+
+    assert dev.kept_indices == [0]
+
+
 def test_range_identity_classifies_a_bare_gemm_that_name_matching_cannot():
     """The dominant real-world gap: bare cuBLAS/cutlass GEMMs carry no
     projection tag in their name (test_classify_op_matches_real_vllm_kernel_names
diff --git a/tests/test_metrics.py b/tests/test_metrics.py
index 3ae6831..455239c 100644
--- a/tests/test_metrics.py
+++ b/tests/test_metrics.py
@@ -65,6 +65,18 @@ def test_rejects_bad_recompute_fraction():
         compute_metrics(_trace(), PEAK, recompute_fraction=1.0)
 
 
+def test_refuses_nonpositive_trace_duration():
+    trace = _trace().model_copy(update={"duration_ns": 0})
+    with pytest.raises(RuntimeError, match="utilization metrics timing unavailable"):
+        compute_metrics(trace, PEAK)
+
+
+def test_refuses_unpriced_memory_bandwidth():
+    peak = HardwarePeak(name="UNKNOWN", peak_flops=1e14, peak_bw_bytes_s=0.0)
+    with pytest.raises(RuntimeError, match="peak bandwidth must be positive"):
+        compute_metrics(_trace(), peak)
+
+
 def test_stall_breakdown_transfer_and_idle():
     # _trace() gaps: [50,100]us and [150,200]us; the memcpy [60,65]us sits in the
     # first gap -> 5us transfer-bound, the remaining 95us of idle is a long stall.
diff --git a/tests/test_runtime_on_trace.py b/tests/test_runtime_on_trace.py
index fd311db..84a9da6 100644
--- a/tests/test_runtime_on_trace.py
+++ b/tests/test_runtime_on_trace.py
@@ -241,6 +241,24 @@ def test_measure_trace_excludes_zero_duration_kernels_with_diagnostic():
     assert all(v.node_op != "bad" for v in result.violations)
 
 
+def test_residuals_exclude_zero_duration_kernels_with_diagnostic():
+    from gitm.optimizer.monitor import residuals
+    from gitm.planner.graph import predict_graph
+
+    trace = _trace(
+        [
+            _kernel("flash_attn_kernel", start=10, end=10),
+            _kernel("flash_attn_kernel", start=20, end=30),
+        ]
+    )
+
+    result = residuals(trace, predict_graph())
+
+    assert result.invalid_duration_kernels == 1
+    assert len(result.per_kernel) == 1
+    assert any("non-positive timestamps" in note for note in result.coverage_warnings)
+
+
 def test_attribute_dr_ranks_pairs():
     from gitm.optimizer.dr import attribute_dr
     from gitm.optimizer.monitor import KernelResidual, Residuals

From 7a4e8340970da885ef690737188ca7ba32c5f71c Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:34:30 -0700
Subject: [PATCH 17/70] fix: gate edge timing and wire kitti planner

---
 gitm/benchmarks/edge/baseline.py  |  8 ++++++--
 gitm/benchmarks/edge/workunit.py  | 23 ++++++++++++++---------
 gitm/benchmarks/kitti/baseline.py | 10 +++++++---
 gitm/benchmarks/kitti/workunit.py | 17 ++++++++---------
 gitm/cli.py                       | 27 +++++++++++++++++++++++++++
 gitm/planner/kitti_graph.py       | 14 ++++++++++----
 scripts/demo_improve_gpu.py       | 15 ++++++++++++---
 scripts/emit_report.py            |  2 +-
 scripts/run_on_real_trace.py      | 15 +++++++++------
 scripts/serve_headroom.py         | 18 +++++++++++++-----
 tests/test_kitti_benchmark.py     | 23 +++++++++++++++++++----
 11 files changed, 126 insertions(+), 46 deletions(-)

diff --git a/gitm/benchmarks/edge/baseline.py b/gitm/benchmarks/edge/baseline.py
index 54c2e56..8f71f0d 100644
--- a/gitm/benchmarks/edge/baseline.py
+++ b/gitm/benchmarks/edge/baseline.py
@@ -41,6 +41,8 @@
 from pathlib import Path
 from typing import Any
 
+from gitm._timing import require_positive_duration
+
 WARM_FRAMES = 100        # discarded before the timing window
 NVML_SAMPLE_HZ = 5
 GPU_ACTIVE_WARN_PCT = 85.0
@@ -181,7 +183,9 @@ def run_baseline(
     nvml_thread.join(timeout=5)
 
     n_warm = len(warm_indices)
-    elapsed = t_wall_end - t_wall_start
+    elapsed = require_positive_duration(
+        t_wall_end - t_wall_start, context="nuScenes baseline warm window"
+    )
     fps = n_warm / elapsed
 
     data_stall_pct = sum(data_stall_fracs) / n_warm * 100
@@ -223,7 +227,7 @@ def run_baseline(
         "captured_at_iso": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
     }
 
-    output_path.write_text(json.dumps(output, indent=2))
+    output_path.write_text(json.dumps(output, indent=2), encoding="utf-8")
     print(
         f"\nResult: {fps:.1f} fps | GPU active {gpu_active_pct:.1f}% "
         f"(NVML {nvml_mean:.1f}%)" if nvml_mean is not None
diff --git a/gitm/benchmarks/edge/workunit.py b/gitm/benchmarks/edge/workunit.py
index fa0b1bc..1fb7142 100644
--- a/gitm/benchmarks/edge/workunit.py
+++ b/gitm/benchmarks/edge/workunit.py
@@ -36,6 +36,8 @@
 from pathlib import Path
 from typing import Any
 
+from gitm._timing import require_positive_duration
+
 # SHA256 of cbgs_pp_centerpoint_nds6070.pth (OpenPCDet nuScenes
 # CenterPoint-PointPillar checkpoint). Confirm with:
 #   sha256sum cbgs_pp_centerpoint_nds6070.pth
@@ -73,21 +75,24 @@ def data_stall_frac(self) -> float:
         With the dyn config, voxelization is on the GPU, so this reflects
         only the multi-sweep load. See the module STAGE-TIMING CAVEAT.
         """
-        if self.t_total_s <= 0:
-            return 0.0
-        return (self.t_load_s + self.t_preprocess_s) / self.t_total_s
+        wall = require_positive_duration(
+            self.t_total_s, context=f"nuScenes frame {self.frame_id}"
+        )
+        return (self.t_load_s + self.t_preprocess_s) / wall
 
     @property
     def sync_stall_frac(self) -> float:
-        if self.t_total_s <= 0:
-            return 0.0
-        return self.t_postprocess_s / self.t_total_s
+        wall = require_positive_duration(
+            self.t_total_s, context=f"nuScenes frame {self.frame_id}"
+        )
+        return self.t_postprocess_s / wall
 
     @property
     def gpu_active_frac(self) -> float:
-        if self.t_total_s <= 0:
-            return 0.0
-        return self.t_inference_s / self.t_total_s
+        wall = require_positive_duration(
+            self.t_total_s, context=f"nuScenes frame {self.frame_id}"
+        )
+        return self.t_inference_s / wall
 
 
 class NuScenesWorkUnit:
diff --git a/gitm/benchmarks/kitti/baseline.py b/gitm/benchmarks/kitti/baseline.py
index 0f2281b..676f54d 100644
--- a/gitm/benchmarks/kitti/baseline.py
+++ b/gitm/benchmarks/kitti/baseline.py
@@ -38,6 +38,8 @@
 from pathlib import Path
 from typing import Any
 
+from gitm._timing import require_positive_duration
+
 WARM_FRAMES = 100  # discard before timing window
 NVML_SAMPLE_HZ = 5
 
@@ -274,7 +276,9 @@ def run_baseline(
     nvml_thread.join(timeout=5)
 
     n_warm = len(warm_paths)
-    elapsed = t_wall_end - t_wall_start
+    elapsed = require_positive_duration(
+        t_wall_end - t_wall_start, context="KITTI baseline warm window"
+    )
     fps = n_warm / elapsed
 
     data_stall_pct = sum(data_stall_fracs) / n_warm * 100
@@ -347,10 +351,10 @@ def run_baseline(
         "captured_at_iso": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
     }
 
-    output_path.write_text(json.dumps(output, indent=2))
+    output_path.write_text(json.dumps(output, indent=2), encoding="utf-8")
     # Save stage spread report alongside the JSON for easy review
     report_path = output_path.with_name(output_path.stem + "_stage_spread.txt")
-    report_path.write_text(spread_report)
+    report_path.write_text(spread_report, encoding="utf-8")
     print(
         f"\nResult: {fps:.1f} fps | GPU active {gpu_active_pct:.1f}% "
         f"| data stall {data_stall_pct:.1f}% | wrote {output_path}"
diff --git a/gitm/benchmarks/kitti/workunit.py b/gitm/benchmarks/kitti/workunit.py
index 39ac4d7..7a82b33 100644
--- a/gitm/benchmarks/kitti/workunit.py
+++ b/gitm/benchmarks/kitti/workunit.py
@@ -27,6 +27,8 @@
 from pathlib import Path
 from typing import Any
 
+from gitm._timing import require_positive_duration
+
 # SHA256 of pointpillar_7728.pth (OpenPCDet KITTI PointPillars checkpoint).
 # Confirm with: sha256sum pointpillar_7728.pth
 CHECKPOINT_SHA256 = "4c83fc0fa02575b9b3e9dec676f698e7a70bb5a795e89f91df8a96b916fa19e2"
@@ -50,23 +52,20 @@ class WorkUnitResult:
     @property
     def data_stall_frac(self) -> float:
         """Fraction of frame time spent on data loading + voxelization."""
-        if self.t_total_s <= 0:
-            return 0.0
-        return (self.t_load_s + self.t_preprocess_s) / self.t_total_s
+        wall = require_positive_duration(self.t_total_s, context=f"KITTI frame {self.frame_id}")
+        return (self.t_load_s + self.t_preprocess_s) / wall
 
     @property
     def sync_stall_frac(self) -> float:
         """Fraction of frame time spent on NMS (CPU-serialized post-processing)."""
-        if self.t_total_s <= 0:
-            return 0.0
-        return self.t_postprocess_s / self.t_total_s
+        wall = require_positive_duration(self.t_total_s, context=f"KITTI frame {self.frame_id}")
+        return self.t_postprocess_s / wall
 
     @property
     def gpu_active_frac(self) -> float:
         """Fraction of frame time spent in GPU backbone + BEV head."""
-        if self.t_total_s <= 0:
-            return 0.0
-        return self.t_inference_s / self.t_total_s
+        wall = require_positive_duration(self.t_total_s, context=f"KITTI frame {self.frame_id}")
+        return self.t_inference_s / wall
 
 
 class WorkUnit:
diff --git a/gitm/cli.py b/gitm/cli.py
index 24804a5..8ddc66f 100644
--- a/gitm/cli.py
+++ b/gitm/cli.py
@@ -149,6 +149,14 @@ def _parser() -> argparse.ArgumentParser:
 
     sub.add_parser("doctor", help="Probe environment, GPUs, and data locations.")
 
+    plan_kitti = sub.add_parser(
+        "plan-kitti", help="Render the PointPillars execution graph for a known GPU SKU."
+    )
+    plan_kitti.add_argument("--sku", required=True, help="GPU SKU from the hardware catalogue.")
+    plan_kitti.add_argument(
+        "--baseline", type=Path, default=None, help="Optional measured baseline JSON to compare."
+    )
+
     analyze = sub.add_parser(
         "analyze",
         help="Ingest customer Nsight/PyTorch profiler dumps into a headroom report.",
@@ -369,6 +377,25 @@ def main(argv: list[str] | None = None) -> int:
         print(json.dumps(report, indent=2))
         return 0
 
+    if args.cmd == "plan-kitti":
+        from gitm.planner.context import hardware_spec_for, peak_for_sku
+        from gitm.planner.kitti_graph import predict_kitti_graph, render_kitti_graph
+
+        peak = peak_for_sku(args.sku)
+        if peak is None:
+            print(f"prediction refused: GPU SKU {args.sku!r} is not in the hardware catalogue")
+            return 3
+        measured = None
+        if args.baseline is not None:
+            try:
+                measured = json.loads(args.baseline.read_text(encoding="utf-8"))
+            except (OSError, json.JSONDecodeError) as exc:
+                print(f"prediction refused: baseline {args.baseline} is unreadable ({exc})")
+                return 3
+        graph = predict_kitti_graph(hw=hardware_spec_for(peak))
+        print(render_kitti_graph(graph, measured=measured))
+        return 0
+
     if args.cmd == "analyze":
         from gitm.importers.analyze import analyze_paths
 
diff --git a/gitm/planner/kitti_graph.py b/gitm/planner/kitti_graph.py
index 02294a6..d26a894 100644
--- a/gitm/planner/kitti_graph.py
+++ b/gitm/planner/kitti_graph.py
@@ -239,10 +239,16 @@ def render_kitti_graph(
     ]
 
     if measured:
-        fps      = measured.get("frames_per_second", 0)
-        frame_ms = 1000 / fps if fps > 0 else 0
-        gpu_pct  = measured.get("gpu_active_pct", 0)
-        data_pct = measured.get("data_stall_pct", 0)
+        fps = measured.get("frames_per_second")
+        gpu_pct = measured.get("gpu_active_pct")
+        data_pct = measured.get("data_stall_pct")
+        if not isinstance(fps, int | float) or fps <= 0:
+            lines += ["", "Measured comparison refused: frames_per_second is missing or non-positive."]
+            return "\n".join(lines)
+        if not isinstance(gpu_pct, int | float) or not isinstance(data_pct, int | float):
+            lines += ["", "Measured comparison refused: stall coverage fields are missing."]
+            return "\n".join(lines)
+        frame_ms = 1000 / fps
         lines += [
             "",
             "Measured (baseline):",
diff --git a/scripts/demo_improve_gpu.py b/scripts/demo_improve_gpu.py
index 1f0ec4e..5abe5d2 100644
--- a/scripts/demo_improve_gpu.py
+++ b/scripts/demo_improve_gpu.py
@@ -23,6 +23,7 @@
 import argparse
 import json
 import time
+import warnings
 from pathlib import Path
 
 from gitm._timing import require_positive_duration
@@ -64,8 +65,12 @@ def _run_observed(fn, label: str, outdir: Path) -> tuple[object, dict]:
         from gitm.telemetry.sinks import build_sink
 
         tele = Collector(CollectorConfig(interval_s=0.05, sinks=[build_sink(f"jsonl:{tele_path}")]))
-    except Exception:
-        pass
+    except Exception as exc:
+        warnings.warn(
+            f"demo telemetry unavailable: {type(exc).__name__}: {exc}",
+            RuntimeWarning,
+            stacklevel=2,
+        )
 
     if tele:
         tele.start()
@@ -84,6 +89,7 @@ def _run_observed(fn, label: str, outdir: Path) -> tuple[object, dict]:
         "serialized": _serialized(tr),
         "util_pct": _mean_util(tele_path),
         "n_kernels": len([e for e in tr.events if e.kind == "kernel"]),
+        "diagnostics": list(tele.diagnostics) if tele is not None else ["telemetry unavailable"],
     }
 
 
@@ -133,7 +139,10 @@ def parallel():
           f"serialized {before['serialized']:.3f} | {before['n_kernels']} kernels\n")
 
     # --- 2. DECIDE ------------------------------------------------------------
-    idle = before["util_pct"] is None or before["util_pct"] < 85.0
+    if before["util_pct"] is None:
+        print("     telemetry unavailable — refusing an idle-GPU claim and intervention decision.")
+        return 1
+    idle = before["util_pct"] < 85.0
     serial_heavy = before["serialized"] > 0.5
     print("2. DECIDE (runtime maps headroom -> lever):")
     print("     why: an idle GPU running serialized, independent work is the signature that parallelizing across streams should help.")
diff --git a/scripts/emit_report.py b/scripts/emit_report.py
index 590972a..c3b41eb 100644
--- a/scripts/emit_report.py
+++ b/scripts/emit_report.py
@@ -102,7 +102,7 @@ def main(argv: list[str] | None = None) -> int:
     report = build_report()
     text = json.dumps(report, indent=2, sort_keys=True)
     if args.out:
-        args.out.write_text(text + "\n")
+        args.out.write_text(text + "\n", encoding="utf-8")
         print(f"wrote {args.out}")
     else:
         print(text)
diff --git a/scripts/run_on_real_trace.py b/scripts/run_on_real_trace.py
index 01ece1c..dba17cc 100644
--- a/scripts/run_on_real_trace.py
+++ b/scripts/run_on_real_trace.py
@@ -16,11 +16,8 @@
 from __future__ import annotations
 
 import sys
-import warnings
 from pathlib import Path
 
-warnings.filterwarnings("ignore")  # this is a diagnostic script; keep output clean
-
 
 def main() -> int:
     try:
@@ -56,10 +53,16 @@ def main() -> int:
         _ = c.sum().item()
         torch.cuda.synchronize()
 
-    kernels = [e for e in tr.events if e.kind == "kernel"]
+    all_kernels = [e for e in tr.events if e.kind == "kernel"]
+    kernels = [e for e in all_kernels if e.end_ns > e.start_ns]
     if not kernels:
-        print("FAIL: no kernels captured (is the CUPTI shim built? run gpu_setup.sh)")
+        print("FAIL: no positive-duration kernels captured (is the CUPTI shim built?)")
         return 1
+    if len(kernels) != len(all_kernels):
+        print(
+            f"WARN: excluded {len(all_kernels) - len(kernels)}/{len(all_kernels)} "
+            "kernels with invalid timestamps"
+        )
     print(f"captured {len(kernels)} real kernels on {torch.cuda.get_device_name(0)}")
 
     # Real stream-concurrency computed from the trace stream IDs.
@@ -75,7 +78,7 @@ def main() -> int:
     res = Residuals()
     res.serialized_concurrency_fraction = sc
     for k in kernels:
-        m = med[k.name] or 1.0
+        m = med[k.name]
         res.per_kernel.append(
             KernelResidual(op=k.name[:30], layer=None, r_kt=((k.end_ns - k.start_ns) - m) / m, r_mt=None)
         )
diff --git a/scripts/serve_headroom.py b/scripts/serve_headroom.py
index 6194812..4716417 100644
--- a/scripts/serve_headroom.py
+++ b/scripts/serve_headroom.py
@@ -58,7 +58,7 @@ def load(trace_path: Path):
     return _load_trace_jsonl(trace_path)
 
 
-def analyse(trace, sku: str):
+def analyse(trace, sku: str, *, hardware_assumed: bool = False):
     """Everything the report needs, as plain data."""
     from gitm.importers.analyze import _predicted_floor_s, _resolve_peak
     from gitm.optimizer.collective_signal import collective_causes, worst_device_comm
@@ -87,6 +87,7 @@ def analyse(trace, sku: str):
         "breakdown": breakdown,
         "peak": peak,
         "sku_known": sku_known,
+        "hardware_assumed": hardware_assumed,
         "metrics": metrics,
         "headroom": headroom,
         "roi": roi,
@@ -168,13 +169,19 @@ def render(a: dict, *, serving: dict | None, trace) -> str:
                 f"- TTFT p50/p95: {_ms(serving.get('ttft_p50_s'))} / {_ms(serving.get('ttft_p95_s'))}",
                 f"- TPOT p50/p95: {_ms(serving.get('tpot_p50_s'))} / {_ms(serving.get('tpot_p95_s'))}",
                 f"- requests: {serving.get('n_requests')} "
-                f"({serving.get('n_failed_requests', 0)} failed), "
+                f"({serving.get('n_failed_requests', 'n/a')} failed), "
                 f"goodput {serving.get('goodput_rps')}",
                 "",
                 "Latency is client-side (includes network); vLLM's own histograms are in "
                 "metrics_before.txt / metrics_after.txt.", ""]
 
     caveats = fp8_caveat(a["sku_known"], a["peak"].name) + list(h.caveats)
+    if a["hardware_assumed"]:
+        caveats.insert(
+            0,
+            f"No --sku was supplied; pricing assumes {a['peak'].name}. "
+            "Utilization ratios are not a detected-hardware measurement.",
+        )
     out += ["## Caveats", ""] + [f"- {c}" for c in caveats] + [""]
     return "\n".join(out)
 
@@ -220,13 +227,14 @@ def main(argv: list[str] | None = None) -> int:
     ap.add_argument("target", type=Path, help="capture dir (or a trace.jsonl)")
     ap.add_argument("--compare", type=Path, default=None,
                     help="a second capture dir to diff against (e.g. the eager reference)")
-    ap.add_argument("--sku", default=DEFAULT_SKU)
+    ap.add_argument("--sku", default=None)
     ap.add_argument("--out", type=Path, default=None, help="markdown path (default <dir>/headroom.md)")
     args = ap.parse_args(sys.argv[1:] if argv is None else argv)
 
     trace_path = resolve_trace(args.target)
     trace = load(trace_path)
-    a = analyse(trace, args.sku)
+    sku = args.sku or DEFAULT_SKU
+    a = analyse(trace, sku, hardware_assumed=args.sku is None)
 
     if a["breakdown"].n_kernels == 0:
         print("No kernels in this trace — there is no headroom claim to make.", file=sys.stderr)
@@ -242,7 +250,7 @@ def main(argv: list[str] | None = None) -> int:
     md = render(a, serving=serving, trace=trace)
     if args.compare:
         other_path = resolve_trace(args.compare)
-        b = analyse(load(other_path), args.sku)
+        b = analyse(load(other_path), sku, hardware_assumed=args.sku is None)
         md += compare(a, b, label_a=args.target.name, label_b=args.compare.name)
 
     out_md = args.out or (trace_path.parent / "headroom.md")
diff --git a/tests/test_kitti_benchmark.py b/tests/test_kitti_benchmark.py
index 76b8ef1..a0b9874 100644
--- a/tests/test_kitti_benchmark.py
+++ b/tests/test_kitti_benchmark.py
@@ -29,13 +29,14 @@ def test_workunit_result_stall_fracs_sum_to_one():
     assert abs(total - 1.0) < 1e-9
 
 
-def test_workunit_result_zero_total_returns_zero_fracs():
+def test_workunit_result_zero_total_refuses_fabricated_zero_fracs():
+    import pytest
+
     from gitm.benchmarks.kitti.workunit import WorkUnitResult
 
     r = WorkUnitResult(frame_id="000000", n_detections=0, t_total_s=0.0)
-    assert r.data_stall_frac == 0.0
-    assert r.sync_stall_frac == 0.0
-    assert r.gpu_active_frac == 0.0
+    with pytest.raises(RuntimeError, match="timing unavailable"):
+        _ = r.data_stall_frac
 
 
 def test_workunit_result_detections_default_empty():
@@ -50,6 +51,20 @@ def test_workunit_importable_without_openpcdet():
     from gitm.benchmarks.kitti.workunit import WorkUnit  # noqa: F401
 
 
+def test_plan_kitti_cli_wires_known_hardware_graph(capsys):
+    from gitm.cli import main
+
+    assert main(["plan-kitti", "--sku", "A100-SXM4-80GB"]) == 0
+    assert "PointPillars predicted execution graph" in capsys.readouterr().out
+
+
+def test_plan_kitti_cli_refuses_unknown_hardware(capsys):
+    from gitm.cli import main
+
+    assert main(["plan-kitti", "--sku", "mystery-gpu"]) == 3
+    assert "prediction refused" in capsys.readouterr().out
+
+
 def test_workunit_from_checkpoint_raises_import_error_without_openpcdet(
     monkeypatch,
 ):

From c709a07d5d3e6968ccd3fc748eb07296325595d0 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:37:45 -0700
Subject: [PATCH 18/70] fix: refuse unpriced dense and A/B inputs

---
 gitm/scheduler/loop.py                  | 164 ++++++------------------
 tests/test_moe_roofline.py              |  26 ----
 tests/test_predict_graph_from_engine.py |  89 ++++++-------
 tests/test_run_loop_workload.py         |  22 ++++
 tests/test_vllm_knobs_and_restart.py    |  12 ++
 5 files changed, 115 insertions(+), 198 deletions(-)

diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 6d32c11..1abc10b 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -19,6 +19,7 @@
 from typing import Any
 
 from gitm._paths import runs_dir, traces_dir
+from gitm._timing import require_positive_duration, require_positive_work
 from gitm.agents.autoresearch import (
     AutoresearchRun,
     EngineArgsProposer,
@@ -57,7 +58,13 @@
 from gitm.planner.context import build_planner_context, hardware_spec_for
 from gitm.planner.graph import Graph, predict_graph
 from gitm.planner.moe_graph import predict_moe_graph, spec_from_hf_config
-from gitm.planner.roofline import BatchConfig, ModelSpec, ShardingConfig
+from gitm.planner.roofline import (
+    BatchConfig,
+    ModelSpec,
+    ShardingConfig,
+    weight_bytes,
+    weight_bytes_is_fallback,
+)
 from gitm.safety.audit import AuditLog, _write_report
 from gitm.serve.model_config import (
     is_sparse_moe_config,
@@ -120,16 +127,23 @@ def _engine_throughput_fn(engine: Any, runner: Any) -> Any:
     def _tps(_engine: Any) -> float:
         t0 = time.perf_counter()
         out = runner() if runner is not None else {}
-        dt = max(time.perf_counter() - t0, 1e-9)
+        dt = require_positive_duration(
+            time.perf_counter() - t0, context="live-engine throughput probe"
+        )
         # First key that is actually present wins — `or` would treat a legitimate
         # 0 (a window that produced no tokens) as missing and fabricate a count.
-        toks: float = 1.0
+        toks: float | None = None
         if isinstance(out, dict):
             for key in ("generated_tokens", "decode_steps", "events"):
                 if out.get(key) is not None:
                     toks = float(out[key])
                     break
-        return toks / dt
+        if toks is None:
+            raise RuntimeError(
+                "live-engine throughput work coverage unavailable: runner returned none of "
+                "generated_tokens, decode_steps, or events"
+            )
+        return float(require_positive_work(toks, context="live-engine throughput probe")) / dt
 
     return _tps
 
@@ -169,76 +183,10 @@ class LoopConfig:
     workload_runner: WorkloadRunner | None = None
 
 
-def _model_spec_from_engine(engine: Any):
-    """Build a ``ModelSpec`` from a live vLLM engine's HF config, or ``None``.
-
-    ``predict_graph()`` with no model defaults to Llama-2-7B (32 layers). A run
-    of a *different* model (e.g. opt-125m, 12 layers) is then scored against the
-    wrong predicted graph, which makes residuals and deviation meaningless. When
-    the loop has the live engine, read the real architecture off its HF config so
-    the predicted graph matches the model that actually ran. Duck-typed across
-    vLLM version drift; any failure returns ``None`` and the caller falls back to
-    the default graph rather than crashing.
-    """
-    if engine is None:
-        return None
-    hf: Any = None
-    for path in (
-        "llm_engine.model_config.hf_config",
-        "llm_engine.vllm_config.model_config.hf_config",
-        "model_config.hf_config",
-    ):
-        obj: Any = engine
-        for attr in path.split("."):
-            obj = getattr(obj, attr, None)
-            if obj is None:
-                break
-        if obj is not None:
-            hf = obj
-            break
-    if hf is None:
-        return None
-    try:
-        from gitm.planner.roofline import ModelSpec
-
-        hidden = int(hf.hidden_size)
-        n_heads = int(hf.num_attention_heads)
-        n_kv = int(getattr(hf, "num_key_value_heads", n_heads) or n_heads)
-        head_dim = int(getattr(hf, "head_dim", 0) or (hidden // n_heads))
-        moe = _moe_fields_from_hf(hf)
-        return ModelSpec(
-            hidden=hidden,
-            n_layers=int(hf.num_hidden_layers),
-            n_heads=n_heads,
-            num_kv_heads=n_kv,
-            head_dim=head_dim,
-            intermediate=int(getattr(hf, "intermediate_size", 4 * hidden)),
-            vocab=int(hf.vocab_size),
-            **moe,
-        )
-    except Exception:
-        return None
-
-
-#: HF config field aliases per MoE family — the same quantity is spelled
-#: differently by Qwen / Mixtral / DeepSeek, so try each in order.
-_MOE_ALIASES: dict[str, tuple[str, ...]] = {
-    "num_experts": ("num_experts", "num_local_experts", "n_routed_experts"),
-    "experts_per_token": ("num_experts_per_tok", "moe_top_k", "num_selected_experts"),
-    "moe_intermediate": ("moe_intermediate_size", "expert_intermediate_size"),
-    "shared_experts": ("n_shared_experts", "num_shared_experts"),
-    "shared_expert_intermediate": ("shared_expert_intermediate_size",),
-    # Per-layer placement: MoE checkpoints are not uniformly sparse.
-    "first_dense_layers": ("first_k_dense_replace",),
-    "moe_layer_step": ("decoder_sparse_step", "moe_layer_freq"),
-}
-
 #: Attention-shape aliases. Deliberately separate from the MoE table: hybrid
 #: attention and a mixture FFN are independent choices, and a hybrid model with a
 #: dense FFN (or a plain MoE transformer) must still get the right one.
-_ATTN_ALIASES: dict[str, tuple[str, ...]] = {
-    "full_attn_layer_step": ("full_attention_interval", "attn_layer_freq"),
-}
+_FULL_ATTN_ALIASES = ("full_attention_interval", "attn_layer_freq")
 
 #: quant_method -> bytes per weight element. MoE decode is weight-fetch bound,
 #: so using the activation width for a quantized checkpoint would overstate the
@@ -399,12 +347,28 @@ def _dense_spec_from_config(cfg: dict[str, Any]) -> tuple[ModelSpec | None, str]
         n_heads = int(cfg["num_attention_heads"])
         n_kv = int(cfg.get("num_key_value_heads", n_heads) or n_heads)
         head_dim = int(cfg.get("head_dim", 0) or (hidden // n_heads))
-        act = str(cfg.get("torch_dtype", "bf16")).lower()
-        dtype_bytes = 4 if act in ("fp32", "float32") else 2
+        act_raw = str(cfg.get("torch_dtype", "bf16")).lower().removeprefix("torch.")
+        act = {
+            "bfloat16": "bf16",
+            "float16": "fp16",
+            "half": "fp16",
+            "float32": "fp32",
+        }.get(act_raw, act_raw)
+        if weight_bytes_is_fallback(act):
+            return None, f"dense activation dtype {act!r} is not priceable"
+        dtype_bytes = int(weight_bytes(act))
         quant = cfg.get("quantization_config") or {}
         method = quant.get("quant_method") if isinstance(quant, dict) else None
         if method is not None and str(method).lower() not in _QUANT_WEIGHT_BYTES:
             return None, f"dense quantization method {method!r} is not priceable"
+        full_attn_layer_step = 1
+        for key in _FULL_ATTN_ALIASES:
+            raw = cfg.get(key)
+            if raw is not None:
+                full_attn_layer_step = int(raw)
+                if full_attn_layer_step <= 0:
+                    return None, f"dense attention interval {key} must be positive"
+                break
         return ModelSpec(
             name=str(cfg.get("_name_or_path") or cfg.get("model_type") or "live-dense"),
             hidden=hidden,
@@ -416,6 +380,7 @@ def _dense_spec_from_config(cfg: dict[str, Any]) -> tuple[ModelSpec | None, str]
             dtype_bytes=dtype_bytes,
             weight_dtype_bytes=_QUANT_WEIGHT_BYTES.get(str(method).lower()) if method else None,
             vocab=int(cfg["vocab_size"]),
+            full_attn_layer_step=full_attn_layer_step,
         ), ""
     except (TypeError, ValueError, ZeroDivisionError) as exc:
         return None, f"dense model config could not be parsed ({type(exc).__name__}: {exc})"
@@ -509,56 +474,6 @@ def _execution_graph(engine: Any, pctx: Any, sched: Any) -> ExecutionGraphResolu
     )
 
 
-def _read_int_aliases(hf: Any, table: dict[str, tuple[str, ...]]) -> dict[str, Any]:
-    """First positive int found for each field across its aliases. Duck-typed."""
-    out: dict[str, Any] = {}
-    for field, aliases in table.items():
-        for alias in aliases:
-            raw = getattr(hf, alias, None)
-            if raw is None:
-                continue
-            try:
-                value = int(raw)
-            except (TypeError, ValueError):
-                continue
-            if value > 0:
-                out[field] = value
-                break
-    return out
-
-
-def _moe_fields_from_hf(hf: Any) -> dict[str, Any]:
-    """Shape ``ModelSpec`` kwargs read off an HF config.
-
-    Covers two *independent* axes — whether the FFN is a mixture, and whether
-    attention is hybrid — so a hybrid-attention dense model still gets its
-    attention shape, and a plain MoE transformer still gets its experts. Every
-    field is optional; anything absent falls back to the dense/conventional
-    default rather than being guessed. Tolerant of partial configs.
-    """
-    # Attention shape is independent of the FFN, so it survives the MoE gate.
-    out: dict[str, Any] = _read_int_aliases(hf, _ATTN_ALIASES)
-    moe = _read_int_aliases(hf, _MOE_ALIASES)
-
-    # Only a routed-expert count *and* a top-k make the FFN a mixture; without
-    # both, leave the FFN dense rather than half-configured.
-    if not (moe.get("num_experts") and moe.get("experts_per_token")):
-        return out
-    out.update(moe)
-
-    quant = getattr(hf, "quantization_config", None)
-    method = None
-    if isinstance(quant, dict):
-        method = quant.get("quant_method")
-    elif quant is not None:
-        method = getattr(quant, "quant_method", None)
-    if isinstance(method, str):
-        wb = _QUANT_WEIGHT_BYTES.get(method.lower())
-        if wb:
-            out["weight_dtype_bytes"] = wb
-    return out
-
-
 def _agg_kt_residual(res: Any) -> float:
     """Run-level kernel-time residual for the report: duration-weighted
     ``sum(obs - pred) / sum(pred)`` when timings are available, else the
@@ -672,7 +587,7 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
     # Collective-communication causes from the same trace — ranked beside the
     # scheduler causes below. Empty when the trace holds no collective kernels.
     coll_causes = collective_causes(worst_device_comm(trace))
-    if sched_stats.samples or serving_summary is not None:
+    if sched_stats.samples or sched_summary.diagnostics or serving_summary is not None:
         (run_dir / "scheduler_stats.json").write_text(
             json.dumps(
                 {
@@ -791,6 +706,7 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
     # Llama/A100 default prediction.
     pctx = build_planner_context(cfg.engine, workload=workload)
     graph_resolution = _execution_graph(cfg.engine, pctx, sched_summary)
+    graph_resolution.diagnostics.extend(sched_summary.diagnostics)
     if not graph_resolution.ok:
         (run_dir / "prediction_refusal.json").write_text(
             json.dumps(
diff --git a/tests/test_moe_roofline.py b/tests/test_moe_roofline.py
index 4aa6dee..e58395b 100644
--- a/tests/test_moe_roofline.py
+++ b/tests/test_moe_roofline.py
@@ -398,32 +398,6 @@ def test_hybrid_and_moe_compose():
     assert all(n.prediction.bytes > 0 for n in g.nodes)
 
 
-def test_hf_attention_shape_survives_a_dense_ffn():
-    """Regression: attention shape and FFN sparsity are independent axes, so a
-    hybrid model with a dense FFN must still get full_attn_layer_step."""
-    from gitm.scheduler.loop import _moe_fields_from_hf
-
-    class HybridDense:  # hybrid attention, no experts
-        full_attention_interval = 4
-
-    class PlainMoE:  # experts, conventional attention
-        num_experts = 64
-        num_experts_per_tok = 4
-
-    assert _moe_fields_from_hf(HybridDense()) == {"full_attn_layer_step": 4}
-    got = _moe_fields_from_hf(PlainMoE())
-    assert got["num_experts"] == 64 and "full_attn_layer_step" not in got
-
-
-def test_hf_half_configured_moe_stays_dense():
-    from gitm.scheduler.loop import _moe_fields_from_hf
-
-    class OnlyExpertCount:
-        num_experts = 64  # no top-k
-
-    assert _moe_fields_from_hf(OnlyExpertCount()) == {}
-
-
 # --- real batch from scheduler stats -----------------------------------------
 
 
diff --git a/tests/test_predict_graph_from_engine.py b/tests/test_predict_graph_from_engine.py
index a1c278e..5a3e507 100644
--- a/tests/test_predict_graph_from_engine.py
+++ b/tests/test_predict_graph_from_engine.py
@@ -1,66 +1,59 @@
-"""The predicted graph must match the model that actually ran.
-
-`predict_graph()` defaults to Llama-2-7B (32 layers). When the loop has a live
-engine, `_model_spec_from_engine` reads the real architecture off its HF config
-so residuals/deviation score against the right model. These tests use a fake
-duck-typed engine (no vLLM/GPU) to pin the config-reading + fallback behavior.
-"""
+"""The production dense-config parser must preserve the live architecture."""
 
 from __future__ import annotations
 
-from types import SimpleNamespace
-
 from gitm.planner.graph import predict_graph
-from gitm.scheduler.loop import _model_spec_from_engine
-
-
-def _fake_engine(**hf_fields):
-    """A stand-in vLLM engine exposing ``llm_engine.model_config.hf_config``."""
-    hf = SimpleNamespace(**hf_fields)
-    return SimpleNamespace(llm_engine=SimpleNamespace(model_config=SimpleNamespace(hf_config=hf)))
-
-
-# opt-125m: 12 layers, hidden 768, 12 heads (MHA, no GQA), intermediate 3072.
-_OPT_125M = dict(
-    num_hidden_layers=12,
-    hidden_size=768,
-    num_attention_heads=12,
-    intermediate_size=3072,
-    vocab_size=50272,
-)
+from gitm.scheduler.loop import _dense_spec_from_config
+
+# opt-125m: 12 layers, hidden 768, 12 heads (MHA), intermediate 3072.
+_OPT_125M = {
+    "num_hidden_layers": 12,
+    "hidden_size": 768,
+    "num_attention_heads": 12,
+    "intermediate_size": 3072,
+    "vocab_size": 50272,
+    "torch_dtype": "bf16",
+}
+
+
+def _spec(**overrides):
+    spec, error = _dense_spec_from_config({**_OPT_125M, **overrides})
+    assert error == ""
+    assert spec is not None
+    return spec
 
 
-def test_reads_real_model_arch_from_engine():
-    spec = _model_spec_from_engine(_fake_engine(**_OPT_125M))
-    assert spec is not None
+def test_reads_real_model_arch_from_config():
+    spec = _spec()
     assert spec.n_layers == 12
     assert spec.hidden == 768
     assert spec.n_heads == 12
-    assert spec.num_kv_heads == 12  # no num_key_value_heads -> falls back to n_heads
-    assert spec.head_dim == 64  # 768 / 12
+    assert spec.num_kv_heads == 12
+    assert spec.head_dim == 64
     assert spec.intermediate == 3072
     assert spec.vocab == 50272
 
 
-def test_predicted_graph_matches_the_real_model_not_llama():
-    """opt-125m -> 12*5+1 = 61 nodes, not the 32*5+1 = 161 Llama default."""
-    spec = _model_spec_from_engine(_fake_engine(**_OPT_125M))
-    assert len(predict_graph(model=spec).nodes) == 61
-    assert len(predict_graph().nodes) == 161  # default is still Llama-7B
+def test_predicted_graph_matches_real_model_not_default():
+    assert len(predict_graph(model=_spec()).nodes) == 61
+    assert len(predict_graph().nodes) == 161
 
 
-def test_gqa_uses_num_key_value_heads_when_present():
-    spec = _model_spec_from_engine(
-        _fake_engine(**{**_OPT_125M, "num_key_value_heads": 4})
-    )
-    assert spec is not None
+def test_gqa_and_hybrid_attention_fields_survive_config_parsing():
+    spec = _spec(num_key_value_heads=4, full_attention_interval=3)
     assert spec.num_kv_heads == 4
+    assert spec.full_attn_layer_step == 3
+
+
+def test_missing_answer_deciding_field_refuses():
+    cfg = dict(_OPT_125M)
+    del cfg["vocab_size"]
+    spec, error = _dense_spec_from_config(cfg)
+    assert spec is None
+    assert "vocab_size" in error
 
 
-def test_falls_back_to_none_when_no_engine_or_config():
-    # No engine at all -> None (loop then uses the default graph).
-    assert _model_spec_from_engine(None) is None
-    # An engine with no readable HF config -> None, never a crash.
-    assert _model_spec_from_engine(SimpleNamespace(nope=1)) is None
-    # A config missing required fields -> None (the int() raises, caught).
-    assert _model_spec_from_engine(_fake_engine(hidden_size=768)) is None
+def test_unknown_dense_dtype_refuses_instead_of_becoming_bf16():
+    spec, error = _dense_spec_from_config({**_OPT_125M, "torch_dtype": "mystery4"})
+    assert spec is None
+    assert "not priceable" in error
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index 646ab09..9f482ce 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -476,6 +476,28 @@ def test_cli_run_refuses_malformed_loop_result(monkeypatch, tmp_path):
     assert "returned no machine-readable summary" in report.read_text()
 
 
+def test_default_engine_throughput_probe_refuses_missing_work_count(monkeypatch):
+    from gitm.scheduler.loop import _engine_throughput_fn
+
+    ticks = iter([1.0, 2.0])
+    monkeypatch.setattr("gitm.scheduler.loop.time.perf_counter", lambda: next(ticks))
+    probe = _engine_throughput_fn(object(), lambda: {})
+
+    with pytest.raises(RuntimeError, match="runner returned none"):
+        probe(object())
+
+
+def test_default_engine_throughput_probe_refuses_zero_work(monkeypatch):
+    from gitm.scheduler.loop import _engine_throughput_fn
+
+    ticks = iter([1.0, 2.0])
+    monkeypatch.setattr("gitm.scheduler.loop.time.perf_counter", lambda: next(ticks))
+    probe = _engine_throughput_fn(object(), lambda: {"generated_tokens": 0})
+
+    with pytest.raises(RuntimeError, match="work coverage unavailable"):
+        probe(object())
+
+
 def test_hft_harness_importable_from_package():
     """The harness must ship in the wheel, i.e. be importable from the package."""
     from gitm.benchmarks.hft import harness
diff --git a/tests/test_vllm_knobs_and_restart.py b/tests/test_vllm_knobs_and_restart.py
index a7951e7..9041212 100644
--- a/tests/test_vllm_knobs_and_restart.py
+++ b/tests/test_vllm_knobs_and_restart.py
@@ -393,8 +393,19 @@ def test_causes_sorted_by_severity_desc():
 # --------------------------------------------------------------------------- #
 # end-to-end: scheduler stats feed attribution + claim evidence (Task 2)      #
 # --------------------------------------------------------------------------- #
+class _DenseHFConfig:
+    def __init__(self):
+        self.hidden_size = 768
+        self.num_hidden_layers = 12
+        self.num_attention_heads = 12
+        self.intermediate_size = 3072
+        self.vocab_size = 50272
+        self.torch_dtype = "bf16"
+
+
 class _ModelCfgBf16:
     dtype = "torch.bfloat16"
+    hf_config = _DenseHFConfig()
 
 
 class _LowOccScheduler:
@@ -443,6 +454,7 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
 
     monkeypatch.setattr(loop, "capture", fake_capture)
     monkeypatch.setattr(loop, "sync_device", lambda: None)
+    monkeypatch.setenv("GITM_GPU_SKU", "A100-SXM4-80GB")
 
     engine = _FullEngine()
     # Non-expiring budget: max_num_seqs_dynamic ranks low and this asserts it's

From 503d459ba55cf6a6f29f3e8e7387cb1d43ce3502 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:41:01 -0700
Subject: [PATCH 19/70] fix: surface missing server metrics

---
 gitm/serve/attach.py       | 54 +++++++++++++++++++++++---------------
 tests/test_serve_attach.py | 31 ++++++++++++++++++++++
 2 files changed, 64 insertions(+), 21 deletions(-)

diff --git a/gitm/serve/attach.py b/gitm/serve/attach.py
index acfe7c8..fcbe382 100644
--- a/gitm/serve/attach.py
+++ b/gitm/serve/attach.py
@@ -323,6 +323,7 @@ def attach_and_capture(opts: AttachOptions) -> tuple[int, CaptureResult | None]:
     server_window = metrics.window_from_snapshots(
         before, after, window_s=wall, samples=samples
     )
+    server_window.notes.extend(sampler.diagnostics)
 
     # Server-side truth is the summary for observe mode, where no client exists. In
     # drive mode both are kept: when the client's view and the server's histograms
@@ -382,27 +383,8 @@ def attach_and_capture(opts: AttachOptions) -> tuple[int, CaptureResult | None]:
     print(f"\n==> window closed after {wall:.1f}s — server left running (PID {target.pid})")
     if opts.mode == "drive":
         print(f"    client: {len(records)} ok / {failures} failed")
-    if server_window.requests_finished is not None:
-        print(
-            f"    server: {server_window.requests_finished:.0f} requests, "
-            f"{(server_window.generation_tokens or 0):.0f} output tokens"
-            + (
-                f", {server_window.output_tokens_per_s:.0f} tok/s"
-                if server_window.output_tokens_per_s
-                else ""
-            )
-        )
-        if server_window.ttft_mean_s is not None:
-            print(
-                f"    server TTFT mean {server_window.ttft_mean_s * 1e3:.0f} ms   "
-                f"TPOT mean {(server_window.tpot_mean_s or 0) * 1e3:.1f} ms"
-            )
-        if server_window.running_p50 is not None and server_window.waiting_p50 is not None:
-            print(
-                f"    queue depth p50/p95 running "
-                f"{server_window.running_p50:.0f}/{server_window.running_p95:.0f}, "
-                f"waiting {server_window.waiting_p50:.0f}/{server_window.waiting_p95:.0f}"
-            )
+    for line in _server_metric_lines(server_window):
+        print(line)
     print_result(result)
     for note in server_window.notes:
         print("  - " + note)
@@ -416,6 +398,36 @@ def attach_and_capture(opts: AttachOptions) -> tuple[int, CaptureResult | None]:
     return 0, result
 
 
+def _server_metric_lines(window: metrics.ServerWindow) -> list[str]:
+    """Human summary that preserves missing server metrics as unknown."""
+    if window.requests_finished is None:
+        return []
+    generation = (
+        f"{window.generation_tokens:.0f} output tokens"
+        if window.generation_tokens is not None
+        else "output-token count unavailable"
+    )
+    lines = [
+        f"    server: {window.requests_finished:.0f} requests, {generation}"
+        + (f", {window.output_tokens_per_s:.0f} tok/s" if window.output_tokens_per_s else "")
+    ]
+    if window.ttft_mean_s is not None:
+        tpot = (
+            f"{window.tpot_mean_s * 1e3:.1f} ms"
+            if window.tpot_mean_s is not None
+            else "unavailable"
+        )
+        lines.append(
+            f"    server TTFT mean {window.ttft_mean_s * 1e3:.0f} ms   TPOT mean {tpot}"
+        )
+    if window.running_p50 is not None and window.waiting_p50 is not None:
+        lines.append(
+            f"    queue depth p50/p95 running {window.running_p50:.0f}/{window.running_p95:.0f}, "
+            f"waiting {window.waiting_p50:.0f}/{window.waiting_p95:.0f}"
+        )
+    return lines
+
+
 def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
     """Write ``predicted_moe_graph.json`` — a per-rank floor from the live config.
 
diff --git a/tests/test_serve_attach.py b/tests/test_serve_attach.py
index 3a848a7..b00294d 100644
--- a/tests/test_serve_attach.py
+++ b/tests/test_serve_attach.py
@@ -223,6 +223,37 @@ def test_base_url_prefers_explicit_then_flag_then_cmdline():
     )
 
 
+def test_server_metric_lines_surface_missing_token_and_tpot_values():
+    from gitm.serve.metrics import ServerWindow
+
+    lines = att._server_metric_lines(
+        ServerWindow(requests_finished=3, generation_tokens=None, ttft_mean_s=0.012)
+    )
+
+    assert "output-token count unavailable" in lines[0]
+    assert "TPOT mean unavailable" in lines[1]
+    assert "0 output tokens" not in "\n".join(lines)
+
+
+def test_server_metric_lines_render_measured_values_without_caveats():
+    from gitm.serve.metrics import ServerWindow
+
+    lines = att._server_metric_lines(
+        ServerWindow(
+            requests_finished=3,
+            generation_tokens=24,
+            output_tokens_per_s=12,
+            ttft_mean_s=0.012,
+            tpot_mean_s=0.0035,
+        )
+    )
+
+    assert lines == [
+        "    server: 3 requests, 24 output tokens, 12 tok/s",
+        "    server TTFT mean 12 ms   TPOT mean 3.5 ms",
+    ]
+
+
 def test_predicted_graph_surfaces_resolved_warnings_and_bytes_fallback(
     tmp_path, monkeypatch, capsys
 ):

From d1da0214c565c15ca66f12c128740bf76b2e2e6f Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:42:43 -0700
Subject: [PATCH 20/70] fix: surface sink and model discovery fallbacks

---
 gitm/serve/vllm.py                |  9 ++++++++-
 gitm/telemetry/sinks/jsonl.py     |  9 ++++-----
 gitm/telemetry/sinks/otlp.py      |  7 +++----
 tests/test_serve_capture.py       | 29 ++++++++++++++++++++++++++++
 tests/test_telemetry_fallbacks.py | 32 +++++++++++++++++++++++++++++++
 5 files changed, 76 insertions(+), 10 deletions(-)

diff --git a/gitm/serve/vllm.py b/gitm/serve/vllm.py
index 5e7a352..60ae321 100644
--- a/gitm/serve/vllm.py
+++ b/gitm/serve/vllm.py
@@ -47,6 +47,7 @@
 import time
 import urllib.error
 import urllib.request
+import warnings
 from concurrent.futures import ThreadPoolExecutor
 from dataclasses import asdict, dataclass, field
 from pathlib import Path
@@ -424,7 +425,13 @@ def served_model_name(base: str, fallback: str) -> str:
         with urllib.request.urlopen(base + "/v1/models", timeout=10) as r:
             data = json.loads(r.read())
         return data["data"][0]["id"]
-    except Exception:
+    except Exception as exc:
+        warnings.warn(
+            f"served-model discovery failed at {base}/v1/models; using configured "
+            f"fallback {fallback!r} ({type(exc).__name__}: {exc})",
+            RuntimeWarning,
+            stacklevel=2,
+        )
         return fallback
 
 
diff --git a/gitm/telemetry/sinks/jsonl.py b/gitm/telemetry/sinks/jsonl.py
index 0281c5d..7635e8e 100644
--- a/gitm/telemetry/sinks/jsonl.py
+++ b/gitm/telemetry/sinks/jsonl.py
@@ -26,8 +26,7 @@ def emit(self, sample: Sample) -> None:
 
     def close(self) -> None:
         with self._lock:
-            try:
-                self._fh.flush()
-                self._fh.close()
-            except Exception:
-                pass
+            # Let the collector record and warn on finalization failures. Swallowing
+            # one here makes a truncated telemetry artifact look complete.
+            self._fh.flush()
+            self._fh.close()
diff --git a/gitm/telemetry/sinks/otlp.py b/gitm/telemetry/sinks/otlp.py
index b5a4f7c..d346676 100644
--- a/gitm/telemetry/sinks/otlp.py
+++ b/gitm/telemetry/sinks/otlp.py
@@ -48,7 +48,6 @@ def emit(self, sample: Sample) -> None:
             self._temp.set(sample.temp_c, attributes=attrs)
 
     def close(self) -> None:
-        try:
-            self._provider.shutdown()
-        except Exception:
-            pass
+        # Collector.stop owns the fail-open boundary and turns this into a named
+        # diagnostic. Hiding it here would make dropped final exports invisible.
+        self._provider.shutdown()
diff --git a/tests/test_serve_capture.py b/tests/test_serve_capture.py
index 6cc253c..f9f4123 100644
--- a/tests/test_serve_capture.py
+++ b/tests/test_serve_capture.py
@@ -11,6 +11,7 @@
 
 import json
 import threading
+import warnings
 from http.server import BaseHTTPRequestHandler, HTTPServer
 
 import pytest
@@ -104,6 +105,34 @@ def test_unreachable_server_returns_none_not_a_fake_record():
     assert sc.one_request("http://127.0.0.1:1", "m", "p", 4, True, 0.5) is None
 
 
+def test_served_model_name_warns_when_endpoint_falls_back(monkeypatch):
+    def fail(*_args, **_kwargs):
+        raise OSError("connection refused")
+
+    monkeypatch.setattr(sc.urllib.request, "urlopen", fail)
+
+    with pytest.warns(RuntimeWarning, match="served-model discovery failed"):
+        assert sc.served_model_name("http://server", "configured-model") == "configured-model"
+
+
+def test_served_model_name_uses_endpoint_without_warning(monkeypatch):
+    class Response:
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *_args):
+            return None
+
+        def read(self):
+            return b'{"data": [{"id": "served-alias"}]}'
+
+    monkeypatch.setattr(sc.urllib.request, "urlopen", lambda *_args, **_kwargs: Response())
+
+    with warnings.catch_warnings(record=True) as caught:
+        assert sc.served_model_name("http://server", "configured-model") == "served-alias"
+    assert not caught
+
+
 def test_records_feed_the_existing_summary(sse_server):
     from gitm.tracer.vllm_stats import summarize_requests
 
diff --git a/tests/test_telemetry_fallbacks.py b/tests/test_telemetry_fallbacks.py
index 88c62c2..a7dbe7a 100644
--- a/tests/test_telemetry_fallbacks.py
+++ b/tests/test_telemetry_fallbacks.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import threading
 import time
 
 import pytest
@@ -64,3 +65,34 @@ def test_collector_surfaces_partial_sample_field_failure():
         collector.stop()
 
     assert any("clock throttle reasons unavailable" in d for d in collector.diagnostics)
+
+
+class _BrokenClose:
+    def flush(self):
+        raise OSError("disk full")
+
+    def close(self):
+        raise AssertionError("flush failure should propagate first")
+
+
+def test_jsonl_sink_close_propagates_for_collector_diagnostic():
+    from gitm.telemetry.sinks.jsonl import JsonlSink
+
+    sink = JsonlSink.__new__(JsonlSink)
+    sink._lock = threading.Lock()
+    sink._fh = _BrokenClose()
+
+    with pytest.raises(OSError, match="disk full"):
+        sink.close()
+
+
+def test_otlp_sink_close_propagates_for_collector_diagnostic():
+    from gitm.telemetry.sinks.otlp import OtlpSink
+
+    sink = OtlpSink.__new__(OtlpSink)
+    sink._provider = type(
+        "BrokenProvider", (), {"shutdown": lambda self: (_ for _ in ()).throw(RuntimeError("export"))}
+    )()
+
+    with pytest.raises(RuntimeError, match="export"):
+        sink.close()

From 4b7d36e2dff438c9f1893c1935d24bb713b69cb0 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:43:28 -0700
Subject: [PATCH 21/70] fix: refuse invalid replay validation truth

---
 gitm/optimizer/replay_validation.py | 17 +++++++++++++++--
 tests/test_runtime_on_trace.py      | 17 +++++++++++++++++
 2 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/gitm/optimizer/replay_validation.py b/gitm/optimizer/replay_validation.py
index fbcb7d6..dbe41ca 100644
--- a/gitm/optimizer/replay_validation.py
+++ b/gitm/optimizer/replay_validation.py
@@ -18,6 +18,7 @@
 
 from __future__ import annotations
 
+import math
 from dataclasses import dataclass
 
 import numpy as np
@@ -66,7 +67,9 @@ def _build_trace(rng: np.random.Generator, n_kernels: int, hot_fraction: float)
 
 def _ground_truth_delta(trace: Trace, spec: InterventionSpec, rng: np.random.Generator) -> float:
     """Per-kernel simulated wall-clock saving fraction (independent of predict_delta)."""
-    total = max(trace.duration_ns, 1)
+    total = trace.duration_ns
+    if total <= 0:
+        raise ValueError(f"synthetic trace duration must be positive, got {total!r}")
     saved = 0.0
     for k in trace.kernels():
         if _applies(spec, k.name):
@@ -91,6 +94,10 @@ def passed(self) -> bool:
 
 
 def validate(n: int = 300, *, seed: int = 0, tolerance: float = 0.20) -> ValidationResult:
+    if n <= 0:
+        raise ValueError(f"validation injection count must be positive, got {n!r}")
+    if not math.isfinite(tolerance) or tolerance < 0.0:
+        raise ValueError(f"validation tolerance must be finite and non-negative, got {tolerance!r}")
     rng = np.random.default_rng(seed)
     spec = _synthetic_spec()
     errs: list[float] = []
@@ -100,7 +107,13 @@ def validate(n: int = 300, *, seed: int = 0, tolerance: float = 0.20) -> Validat
         trace = _build_trace(rng, n_k, hot_frac)
         truth = _ground_truth_delta(trace, spec, rng)
         pred = predict_delta(trace, spec)
-        errs.append(abs(pred - truth) / abs(truth) if truth else 0.0)
+        if not math.isfinite(truth) or truth <= 0.0:
+            raise RuntimeError(
+                f"replay validation ground truth must be finite and positive, got {truth!r}"
+            )
+        if not math.isfinite(pred):
+            raise RuntimeError(f"replay validation prediction must be finite, got {pred!r}")
+        errs.append(abs(pred - truth) / abs(truth))
     arr = np.asarray(errs)
     return ValidationResult(
         n=n,
diff --git a/tests/test_runtime_on_trace.py b/tests/test_runtime_on_trace.py
index 84a9da6..4256fb3 100644
--- a/tests/test_runtime_on_trace.py
+++ b/tests/test_runtime_on_trace.py
@@ -327,3 +327,20 @@ def test_replay_validation_within_tolerance():
     assert result.passed
     assert result.mean_abs_rel_err <= 0.20
     assert result.frac_within_tol > 0.7
+
+
+def test_replay_validation_refuses_zero_truth_instead_of_reporting_zero_error(monkeypatch):
+    from gitm.optimizer import replay_validation as rv
+
+    monkeypatch.setattr(rv, "_ground_truth_delta", lambda *_args, **_kwargs: 0.0)
+
+    with pytest.raises(RuntimeError, match="ground truth must be finite and positive"):
+        rv.validate(n=1)
+
+
+@pytest.mark.parametrize("n", [0, -1])
+def test_replay_validation_refuses_empty_injection_sets(n):
+    from gitm.optimizer.replay_validation import validate
+
+    with pytest.raises(ValueError, match="injection count must be positive"):
+        validate(n=n)

From afa5ca44c102706f3907816ef7aa555ceb8325f0 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:50:00 -0700
Subject: [PATCH 22/70] fix: price dense graphs at their compute dtype

---
 gitm/planner/graph.py                   | 40 +++++++++++++++++++++----
 gitm/planner/roofline.py                |  3 ++
 gitm/scheduler/loop.py                  |  1 +
 tests/test_moe_roofline.py              | 14 +++++++++
 tests/test_predict_graph_from_engine.py |  9 ++++++
 5 files changed, 61 insertions(+), 6 deletions(-)

diff --git a/gitm/planner/graph.py b/gitm/planner/graph.py
index 59607c8..d837af4 100644
--- a/gitm/planner/graph.py
+++ b/gitm/planner/graph.py
@@ -197,7 +197,11 @@ def predict_graph(
         flops = 2 * b * h * qkv_out
         bytes_moved = dt * (b * h + h * qkv_out + b * qkv_out)
         g.nodes.append(
-            PredictedNode("qkv_proj", layer, roofline("qkv_proj", flops, bytes_moved, hw))
+            PredictedNode(
+                "qkv_proj",
+                layer,
+                roofline("qkv_proj", flops, bytes_moved, hw, dtype=model.compute_dtype),
+            )
         )
 
         # Attention scores + softmax + value. Full-attention layers re-read a KV
@@ -219,7 +223,9 @@ def predict_graph(
             PredictedNode(
                 "attn_score_value",
                 layer,
-                roofline("attn_score_value", attn_flops, kv_bytes, hw),
+                roofline(
+                    "attn_score_value", attn_flops, kv_bytes, hw, dtype=model.compute_dtype
+                ),
             )
         )
 
@@ -227,7 +233,11 @@ def predict_graph(
         flops = 2 * b * h * h
         bytes_moved = dt * (b * h + h * h + b * h)
         g.nodes.append(
-            PredictedNode("attn_out_proj", layer, roofline("attn_out_proj", flops, bytes_moved, hw))
+            PredictedNode(
+                "attn_out_proj",
+                layer,
+                roofline("attn_out_proj", flops, bytes_moved, hw, dtype=model.compute_dtype),
+            )
         )
 
         # MLP gate+up / down. On an MoE model these two ops carry the expert
@@ -238,18 +248,36 @@ def predict_graph(
         )
         g.nodes.append(
             PredictedNode(
-                "mlp_gate_up", layer, roofline("mlp_gate_up", gate_up_flops, gate_up_bytes, hw)
+                "mlp_gate_up",
+                layer,
+                roofline(
+                    "mlp_gate_up",
+                    gate_up_flops,
+                    gate_up_bytes,
+                    hw,
+                    dtype=model.compute_dtype,
+                ),
             )
         )
         g.nodes.append(
-            PredictedNode("mlp_down", layer, roofline("mlp_down", down_flops, down_bytes, hw))
+            PredictedNode(
+                "mlp_down",
+                layer,
+                roofline(
+                    "mlp_down", down_flops, down_bytes, hw, dtype=model.compute_dtype
+                ),
+            )
         )
 
     # Final vocab projection
     flops = 2 * b * h * model.vocab
     bytes_moved = dt * (b * h + h * model.vocab + b * model.vocab)
     g.nodes.append(
-        PredictedNode("lm_head", None, roofline("lm_head", flops, bytes_moved, hw))
+        PredictedNode(
+            "lm_head",
+            None,
+            roofline("lm_head", flops, bytes_moved, hw, dtype=model.compute_dtype),
+        )
     )
 
     return g
diff --git a/gitm/planner/roofline.py b/gitm/planner/roofline.py
index 328b485..15d3d67 100644
--- a/gitm/planner/roofline.py
+++ b/gitm/planner/roofline.py
@@ -106,6 +106,9 @@ class ModelSpec:
     num_kv_heads: int = 32  # < n_heads when GQA
     head_dim: int = 128
     intermediate: int = 11008
+    #: Compute dtype used to choose the tensor-core ceiling. Width alone cannot
+    #: distinguish bf16/fp16/fp32, so this must travel separately.
+    compute_dtype: str = "fp16"
     dtype_bytes: int = 2  # fp16 / bf16 — activations
     vocab: int = 32000
 
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 1abc10b..55c0c18 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -377,6 +377,7 @@ def _dense_spec_from_config(cfg: dict[str, Any]) -> tuple[ModelSpec | None, str]
             num_kv_heads=n_kv,
             head_dim=head_dim,
             intermediate=int(cfg["intermediate_size"]),
+            compute_dtype=act,
             dtype_bytes=dtype_bytes,
             weight_dtype_bytes=_QUANT_WEIGHT_BYTES.get(str(method).lower()) if method else None,
             vocab=int(cfg["vocab_size"]),
diff --git a/tests/test_moe_roofline.py b/tests/test_moe_roofline.py
index e58395b..9343eb4 100644
--- a/tests/test_moe_roofline.py
+++ b/tests/test_moe_roofline.py
@@ -84,6 +84,20 @@ def test_dense_spec_is_not_moe_and_falls_back():
     assert m.expert_intermediate == m.intermediate
 
 
+def test_dense_graph_uses_configured_compute_dtype_for_peak_selection():
+    hw = HardwareSpec(
+        peak_flops_fp16_per_s=100.0,
+        peak_flops_fp32_per_s=10.0,
+        peak_mem_bw_bytes_per_s=1e30,
+    )
+
+    graph = predict_graph(ModelSpec(n_layers=1, compute_dtype="fp32"), hw)
+
+    assert graph.nodes
+    assert all(node.prediction.dtype == "fp32" for node in graph.nodes)
+    assert all(node.prediction.peak_dtype == "fp32" for node in graph.nodes)
+
+
 def test_moe_spec_properties():
     assert MOE.is_moe
     assert MOE.top_k == 8
diff --git a/tests/test_predict_graph_from_engine.py b/tests/test_predict_graph_from_engine.py
index 5a3e507..fdbdaea 100644
--- a/tests/test_predict_graph_from_engine.py
+++ b/tests/test_predict_graph_from_engine.py
@@ -57,3 +57,12 @@ def test_unknown_dense_dtype_refuses_instead_of_becoming_bf16():
     spec, error = _dense_spec_from_config({**_OPT_125M, "torch_dtype": "mystery4"})
     assert spec is None
     assert "not priceable" in error
+
+
+def test_dense_parser_preserves_fp32_compute_dtype():
+    spec, error = _dense_spec_from_config({**_OPT_125M, "torch_dtype": "float32"})
+
+    assert error == ""
+    assert spec is not None
+    assert spec.compute_dtype == "fp32"
+    assert spec.dtype_bytes == 4

From 3010e9fb2c461b55d21d894dfcfa6e16bcc38ae3 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:52:29 -0700
Subject: [PATCH 23/70] fix: refuse invalid headroom evidence

---
 gitm/optimizer/headroom.py             | 31 ++++++++++++++++++---
 gitm/optimizer/headroom_kernel_rank.py | 38 +++++++++++++++++++++++---
 scripts/serve_headroom.py              | 12 ++++++++
 tests/test_headroom.py                 | 19 +++++++++++++
 tests/test_headroom_kernel_rank.py     | 21 ++++++++++++--
 5 files changed, 111 insertions(+), 10 deletions(-)

diff --git a/gitm/optimizer/headroom.py b/gitm/optimizer/headroom.py
index fb3d4ce..6e934ac 100644
--- a/gitm/optimizer/headroom.py
+++ b/gitm/optimizer/headroom.py
@@ -16,6 +16,7 @@
 
 from __future__ import annotations
 
+import math
 from dataclasses import asdict, dataclass, field
 from typing import Literal
 
@@ -82,10 +83,21 @@ def build_headroom(
 ) -> HeadroomReport:
     """Assemble the headroom report from the trace, planner floor, and metrics."""
     observed_s = trace.duration_ns / 1e9
-    if observed_s > 0 and predicted_floor_s > 0:
-        ceiling_distance = max(0.0, (observed_s - predicted_floor_s) / observed_s)
-    else:
-        ceiling_distance = 0.0
+    if not math.isfinite(observed_s) or observed_s <= 0.0:
+        raise ValueError(f"observed trace duration must be finite and positive, got {observed_s!r}")
+    if not math.isfinite(predicted_floor_s) or predicted_floor_s <= 0.0:
+        raise ValueError(
+            f"predicted floor must be finite and positive, got {predicted_floor_s!r}"
+        )
+    if predicted_floor_s > observed_s:
+        raise ValueError(
+            f"predicted floor {predicted_floor_s!r}s exceeds observed wall {observed_s!r}s"
+        )
+    if not math.isfinite(optimized_threshold) or not 0.0 <= optimized_threshold <= 1.0:
+        raise ValueError(
+            f"optimized threshold must be finite and in [0, 1], got {optimized_threshold!r}"
+        )
+    ceiling_distance = (observed_s - predicted_floor_s) / observed_s
     already_optimized = ceiling_distance < optimized_threshold
 
     # Split the recoverable distance across stall classes. Idle is the GPU-idle
@@ -150,5 +162,16 @@ def render_headroom_md(r: HeadroomReport) -> str:
     if r.already_optimized:
         lines.append("")
         lines.append("> Flagged already-optimized — no headroom to bill.")
+    if r.indicative_mem_compute_split or r.caveats:
+        lines.extend(["", "Caveats:"])
+        for caveat in r.caveats:
+            lines.append(f"  - {caveat}")
+        if r.indicative_mem_compute_split and not any(
+            "indicative" in caveat.lower() for caveat in r.caveats
+        ):
+            lines.append(
+                "  - Memory/compute split is indicative because HFU is unavailable; "
+                "it uses available MBU signal and a 50/50 split when both signals are weak."
+            )
     return "\n".join(lines) + "\n"
 
diff --git a/gitm/optimizer/headroom_kernel_rank.py b/gitm/optimizer/headroom_kernel_rank.py
index dfbc9db..97f887a 100644
--- a/gitm/optimizer/headroom_kernel_rank.py
+++ b/gitm/optimizer/headroom_kernel_rank.py
@@ -18,7 +18,9 @@
 
 from __future__ import annotations
 
+import math
 import re
+import warnings
 from dataclasses import dataclass, field
 
 # Mangled CUDA kernel names -> a small set of stable families, so durations
@@ -71,12 +73,16 @@ def kernel_roi(name_durations, floor_pct: float = 10.0) -> list[KernelROI]:
 
     ``name_durations`` is an iterable of ``(kernel_name, duration_ns)``.
     """
+    if not math.isfinite(floor_pct) or not 0.0 <= floor_pct <= 100.0:
+        raise ValueError(f"kernel ROI floor percentile must be in [0, 100], got {floor_pct!r}")
     by_fam: dict[str, list[int]] = {}
     for name, dur in name_durations:
         if dur is None or dur <= 0:
             continue
         by_fam.setdefault(kernel_family(name), []).append(int(dur))
-    total = sum(sum(v) for v in by_fam.values()) or 1
+    total = sum(sum(v) for v in by_fam.values())
+    if total <= 0:
+        return []
 
     rows: list[KernelROI] = []
     for fam, ds in by_fam.items():
@@ -95,7 +101,9 @@ def kernel_roi(name_durations, floor_pct: float = 10.0) -> list[KernelROI]:
 
 def render_roi_table(rows: list[KernelROI], *, floor_pct: float = 10.0, top: int = 20) -> str:
     """Human-readable ROI table (used in stdout and the provenance report)."""
-    total = sum(r.total_ns for r in rows) or 1
+    if not rows:
+        return "kernel ROI unavailable: no positive-duration kernels"
+    total = sum(r.total_ns for r in rows)
     us, ms = 1000.0, 1_000_000.0
     out = [f"kernel ROI (floor p{floor_pct:.0f}, total kernel time {total / ms:.1f} ms):", ""]
     hdr = (f"{'#':>2}  {'family':<22} {'calls':>7} {'total_ms':>9} {'%rt':>6} "
@@ -122,11 +130,11 @@ class GpuHeadroom:
     peak_mem_used_bytes: int | None
     mem_total_bytes: int | None
     mem_free_at_peak_bytes: int | None
-    serialized_concurrency_fraction: float # fraction of kernel-time with no overlap
+    serialized_concurrency_fraction: float | None # fraction of kernel-time with no overlap
     n_samples: int
     diagnostics: list[str] = field(default_factory=list)
 
-def gpu_headroom(samples, serialized_concurrency_fraction: float = 0.0) -> GpuHeadroom:
+def gpu_headroom(samples, serialized_concurrency_fraction: float | None = None) -> GpuHeadroom:
     """Summarise GPU headroom from a list of telemetry sample dicts.
 
     Each sample is a dict with ``util_pct`` / ``mem_used_bytes`` /
@@ -134,6 +142,14 @@ def gpu_headroom(samples, serialized_concurrency_fraction: float = 0.0) -> GpuHe
     Missing metric families remain ``None`` and are named in ``diagnostics``;
     memory-only telemetry must not become 100% compute headroom, or vice versa.
     """
+    if serialized_concurrency_fraction is not None and (
+        not math.isfinite(serialized_concurrency_fraction)
+        or not 0.0 <= serialized_concurrency_fraction <= 1.0
+    ):
+        raise ValueError(
+            "serialized concurrency fraction must be finite and in [0, 1], got "
+            f"{serialized_concurrency_fraction!r}"
+        )
     utils = [float(s["util_pct"]) for s in samples if s.get("util_pct") is not None]
     mems = [int(s["mem_used_bytes"]) for s in samples if s.get("mem_used_bytes") is not None]
     totals = [int(s["mem_total_bytes"]) for s in samples if s.get("mem_total_bytes")]
@@ -146,6 +162,8 @@ def gpu_headroom(samples, serialized_concurrency_fraction: float = 0.0) -> GpuHe
         diagnostics.append("memory headroom unavailable: used-memory telemetry is absent")
     if not totals:
         diagnostics.append("memory headroom unavailable: total-memory telemetry is absent")
+    if serialized_concurrency_fraction is None:
+        diagnostics.append("concurrency headroom unavailable: no trace overlap measurement supplied")
     mean_u = sum(utils) / len(utils) if utils else None
     peak_u = max(utils) if utils else None
     peak_m = max(mems) if mems else None
@@ -170,10 +188,22 @@ def live_gpu_headroom():
     from gitm.telemetry.backends import discover_backends
 
     backends = discover_backends()
+    if not backends:
+        warnings.warn(
+            "live GPU headroom unavailable: no telemetry backend found",
+            RuntimeWarning,
+            stacklevel=2,
+        )
     out = []
     for b in backends:
         for idx in range(b.device_count()):
             s = b.sample(idx)
+            for diagnostic in s.diagnostics:
+                warnings.warn(
+                    f"live GPU headroom degraded: {diagnostic}",
+                    RuntimeWarning,
+                    stacklevel=2,
+                )
             out.append({
                 "gpu_index": idx,
                 "util_pct": s.util_pct,
diff --git a/scripts/serve_headroom.py b/scripts/serve_headroom.py
index 4716417..9fb1df2 100644
--- a/scripts/serve_headroom.py
+++ b/scripts/serve_headroom.py
@@ -69,6 +69,18 @@ def analyse(trace, sku: str, *, hardware_assumed: bool = False):
 
     kernels = trace.kernels()
     breakdown = summarize_kernels(kernels, window_ns=trace.duration_ns)
+    if breakdown.n_kernels == 0:
+        return {
+            "breakdown": breakdown,
+            "peak": None,
+            "sku_known": False,
+            "hardware_assumed": hardware_assumed,
+            "metrics": None,
+            "headroom": None,
+            "roi": [],
+            "comm": None,
+            "causes": [],
+        }
 
     peak, sku_known = _resolve_peak(sku)
     metrics = compute_metrics(trace, peak)
diff --git a/tests/test_headroom.py b/tests/test_headroom.py
index 775719e..f47aa7e 100644
--- a/tests/test_headroom.py
+++ b/tests/test_headroom.py
@@ -55,3 +55,22 @@ def test_render_contains_key_lines():
     md = render_headroom_md(r)
     assert "Blind headroom — vllm-decode on NVIDIA H100" in md
     assert "Ceiling distance" in md
+    assert "Caveats:" in md
+    assert "HFU is unavailable" in md
+
+
+@pytest.mark.parametrize("floor", [0.0, float("nan"), float("inf")])
+def test_invalid_predicted_floor_is_refused(floor):
+    trace = _trace(wall_us=200)
+    metrics = compute_metrics(trace, PEAK)
+
+    with pytest.raises(ValueError, match="predicted floor must be finite and positive"):
+        build_headroom(trace, predicted_floor_s=floor, metrics=metrics, workload="vllm-decode")
+
+
+def test_floor_above_observation_is_refused_instead_of_already_optimized():
+    trace = _trace(wall_us=200)
+    metrics = compute_metrics(trace, PEAK)
+
+    with pytest.raises(ValueError, match="exceeds observed wall"):
+        build_headroom(trace, predicted_floor_s=201e-6, metrics=metrics, workload="vllm-decode")
diff --git a/tests/test_headroom_kernel_rank.py b/tests/test_headroom_kernel_rank.py
index 21b36e2..2968b4b 100644
--- a/tests/test_headroom_kernel_rank.py
+++ b/tests/test_headroom_kernel_rank.py
@@ -1,6 +1,8 @@
 from __future__ import annotations
 
-from gitm.optimizer.headroom_kernel_rank import gpu_headroom
+import pytest
+
+from gitm.optimizer.headroom_kernel_rank import gpu_headroom, kernel_roi, render_roi_table
 
 
 def test_memory_only_telemetry_does_not_fabricate_compute_headroom():
@@ -25,7 +27,8 @@ def test_utilization_only_telemetry_does_not_fabricate_memory_headroom():
 
 def test_complete_headroom_sample_stays_clean():
     result = gpu_headroom(
-        [{"util_pct": 65.0, "mem_used_bytes": 4_000, "mem_total_bytes": 10_000}]
+        [{"util_pct": 65.0, "mem_used_bytes": 4_000, "mem_total_bytes": 10_000}],
+        serialized_concurrency_fraction=0.25,
     )
 
     assert result.compute_headroom_pct == 35.0
@@ -39,3 +42,17 @@ def test_empty_headroom_input_is_explicitly_unavailable():
     assert result.compute_headroom_pct is None
     assert result.mem_free_at_peak_bytes is None
     assert any("no samples" in note for note in result.diagnostics)
+    assert any("concurrency headroom unavailable" in note for note in result.diagnostics)
+
+
+def test_empty_kernel_roi_renders_unavailable_instead_of_zero_time():
+    rows = kernel_roi([("zero", 0), ("backwards", -1)])
+
+    assert rows == []
+    assert render_roi_table(rows) == "kernel ROI unavailable: no positive-duration kernels"
+
+
+@pytest.mark.parametrize("fraction", [-0.1, 1.1, float("nan")])
+def test_invalid_serialized_fraction_is_refused(fraction):
+    with pytest.raises(ValueError, match="serialized concurrency fraction"):
+        gpu_headroom([], serialized_concurrency_fraction=fraction)

From 30364a9bb6101a106e71322fb763cdb0f91558fc Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:54:04 -0700
Subject: [PATCH 24/70] fix: refuse invalid A/B samples and surface cleanup
 failures

---
 gitm/optimizer/apply.py              | 23 +++++++++++++--
 gitm/runtime_driver.py               |  8 ++++--
 gitm/workloads.py                    | 42 +++++++++++++++++++++-------
 tests/test_vllm_knobs_and_restart.py | 19 +++++++++++++
 4 files changed, 77 insertions(+), 15 deletions(-)

diff --git a/gitm/optimizer/apply.py b/gitm/optimizer/apply.py
index be8d87e..cbc516e 100644
--- a/gitm/optimizer/apply.py
+++ b/gitm/optimizer/apply.py
@@ -21,6 +21,7 @@
 
 import copy
 import gc
+import math
 import warnings
 from collections.abc import Callable
 from dataclasses import dataclass
@@ -75,7 +76,15 @@ def apply_intervention(
     blocks the apply). Pass one only where the applicator mutates a real target;
     a dry-run leaves it ``None`` so the trail stays free of no-op entries.
     """
-    snapshot = applicator.snapshot()
+    try:
+        snapshot = applicator.snapshot()
+    except Exception as exc:
+        return ApplyResult(
+            False,
+            rolled_back=False,
+            measured_delta=None,
+            error=f"baseline snapshot failed; intervention not applied: {exc}",
+        )
 
     # Step 2: apply. A bad value (validation error) rolls straight back.
     try:
@@ -319,7 +328,9 @@ def __init__(
         self._restart_mode = restart_mode
         self._getter = getter or get_knob
         self._setter = setter or set_knob
-        self._reps = max(1, reps)
+        if isinstance(reps, bool) or not isinstance(reps, int) or reps <= 0:
+            raise ValueError(f"A/B repetition count must be a positive integer, got {reps!r}")
+        self._reps = reps
         # force_restart is kept for custom deployments that still classify a
         # knob as scheduling but want to measure it through the restart path.
         self._force_restart = force_restart
@@ -339,7 +350,13 @@ def _bench_stats(self) -> tuple[float, float]:
         stdev is 0.0 for a single rep → the noise band is 0 and keep falls back to
         ``delta > 0``, i.e. reps=1 behaves exactly as before reps were added.
         """
-        samples = [self._tps(self.engine) for _ in range(self._reps)]
+        samples = [float(self._tps(self.engine)) for _ in range(self._reps)]
+        invalid = [sample for sample in samples if not math.isfinite(sample) or sample <= 0.0]
+        if invalid:
+            raise ValueError(
+                "decode throughput samples must be finite and positive; "
+                f"observed {invalid!r}"
+            )
         mean = sum(samples) / len(samples)
         if len(samples) < 2:
             return mean, 0.0
diff --git a/gitm/runtime_driver.py b/gitm/runtime_driver.py
index 2c0bea5..20d4580 100644
--- a/gitm/runtime_driver.py
+++ b/gitm/runtime_driver.py
@@ -81,8 +81,12 @@ def _free_gpu_pool():
         import cupy
 
         cupy.get_default_memory_pool().free_all_blocks()
-    except Exception:
-        pass
+    except Exception as exc:
+        warnings.warn(
+            f"GPU memory-pool cleanup unavailable: {type(exc).__name__}: {exc}",
+            RuntimeWarning,
+            stacklevel=2,
+        )
 
 
 def _stream_hft(stage: Path, seed: int, shards_per_batch: int, max_shards: int | None):
diff --git a/gitm/workloads.py b/gitm/workloads.py
index a569831..f877fa6 100644
--- a/gitm/workloads.py
+++ b/gitm/workloads.py
@@ -688,8 +688,12 @@ def _shutdown_engine(engine: Any) -> None:
             if callable(obj):
                 try:
                     obj()
-                except Exception:
-                    pass
+                except Exception as exc:
+                    warnings.warn(
+                        f"vLLM engine shutdown path {path!r} failed: {exc}",
+                        RuntimeWarning,
+                        stacklevel=2,
+                    )
                 break
 
         try:
@@ -700,22 +704,36 @@ def _shutdown_engine(engine: Any) -> None:
 
             destroy_model_parallel()
             destroy_distributed_environment()
-        except Exception:
-            pass
+        except Exception as exc:
+            warnings.warn(
+                f"vLLM distributed-state cleanup failed: {exc}",
+                RuntimeWarning,
+                stacklevel=2,
+            )
 
         try:
             import torch.distributed as dist
 
             if dist.is_available() and dist.is_initialized():
                 dist.destroy_process_group()
-        except Exception:
-            pass
+        except Exception as exc:
+            warnings.warn(
+                f"torch distributed process-group cleanup failed: {exc}",
+                RuntimeWarning,
+                stacklevel=2,
+            )
 
         for attr in ("llm_engine", "engine"):
+            if not hasattr(engine, attr):
+                continue
             try:
                 delattr(engine, attr)
-            except Exception:
-                pass
+            except Exception as exc:
+                warnings.warn(
+                    f"vLLM engine reference cleanup for {attr!r} failed: {exc}",
+                    RuntimeWarning,
+                    stacklevel=2,
+                )
 
         try:
             import gc
@@ -726,8 +744,12 @@ def _shutdown_engine(engine: Any) -> None:
             if torch.cuda.is_available():
                 torch.cuda.empty_cache()
                 torch.cuda.ipc_collect()
-        except Exception:
-            pass
+        except Exception as exc:
+            warnings.warn(
+                f"GPU cleanup after vLLM shutdown failed: {exc}",
+                RuntimeWarning,
+                stacklevel=2,
+            )
 
     def _activate_engine(engine: Any) -> None:
         engine_ref["engine"] = engine
diff --git a/tests/test_vllm_knobs_and_restart.py b/tests/test_vllm_knobs_and_restart.py
index 9041212..a0e39fd 100644
--- a/tests/test_vllm_knobs_and_restart.py
+++ b/tests/test_vllm_knobs_and_restart.py
@@ -261,6 +261,25 @@ def test_restart_apply_rolls_back_to_original_engine_on_regression():
     assert app.engine is e0  # original engine restored
 
 
+@pytest.mark.parametrize("sample", [0.0, -1.0, float("nan"), float("inf")])
+def test_live_engine_ab_refuses_invalid_throughput_samples(sample):
+    engine = _TpsEngine(sample)
+    app = LiveEngineApplicator(engine, throughput_fn=_tps_of)
+
+    result = apply_intervention(_spec("max_num_seqs", 64), app)
+
+    assert not result.rolled_back
+    assert not result.applied
+    assert "finite and positive" in result.error
+    assert app.last_result is None
+
+
+@pytest.mark.parametrize("reps", [0, -1, 1.5, True])
+def test_live_engine_ab_refuses_invalid_repetition_counts(reps):
+    with pytest.raises(ValueError, match="repetition count must be a positive integer"):
+        LiveEngineApplicator(_TpsEngine(100.0), throughput_fn=_tps_of, reps=reps)
+
+
 
 def test_serial_restart_releases_baseline_before_building_candidate():
     class Engine(_TpsEngine):

From ac5c73d4f293a64d188db4c6190213a9acd0691f Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:55:23 -0700
Subject: [PATCH 25/70] fix: refuse zero-duration serving traces

---
 gitm/serve/artifacts.py        |  8 +++++---
 gitm/tracer/kernel_taxonomy.py | 36 +++++++++++++++++++++++++++++++---
 tests/test_kernel_taxonomy.py  | 23 ++++++++++++++++++++++
 tests/test_serve_attach.py     | 13 ++++++++++++
 4 files changed, 74 insertions(+), 6 deletions(-)

diff --git a/gitm/serve/artifacts.py b/gitm/serve/artifacts.py
index cff13e8..0bb4e84 100644
--- a/gitm/serve/artifacts.py
+++ b/gitm/serve/artifacts.py
@@ -102,13 +102,15 @@ def write_capture_artifacts(
             "source": trace.source,
             "device_count": trace.device_count,
             "events": len(trace.events),
-            "kernels": len(kernels),
+            "kernel_records": len(kernels),
+            "kernels": breakdown.n_kernels,
+            "invalid_kernel_durations": breakdown.n_invalid_duration,
             "duration_ns": trace.duration_ns,
         },
     }
     (out_dir / "run_manifest.json").write_text(json.dumps(manifest, indent=2))
 
-    if not kernels:
+    if breakdown.n_kernels == 0:
         status = "no_kernels"
     elif not had_traffic:
         status = "no_traffic"
@@ -119,7 +121,7 @@ def write_capture_artifacts(
         out_dir=out_dir,
         trace_path=trace_path,
         n_events=len(trace.events),
-        n_kernels=len(kernels),
+        n_kernels=breakdown.n_kernels,
         status=status,
         breakdown=breakdown,
         warnings=warnings,
diff --git a/gitm/tracer/kernel_taxonomy.py b/gitm/tracer/kernel_taxonomy.py
index 09ed71a..c36997b 100644
--- a/gitm/tracer/kernel_taxonomy.py
+++ b/gitm/tracer/kernel_taxonomy.py
@@ -127,6 +127,7 @@ class KernelBreakdown:
     n_truncated_names: int           # records whose name hit NAME_MAX
     n_distinct_truncated: int        # distinct such names — collisions hide here
     n_devices: int
+    n_invalid_duration: int = 0
 
     @property
     def other_share(self) -> float:
@@ -141,11 +142,31 @@ def warnings(self) -> list[str]:
         out: list[str] = []
         if self.n_kernels == 0:
             out.append(
-                "no kernels captured at all. The usual cause is decode running as "
+                "no positive-duration kernels captured. The usual cause is decode running as "
                 "CUDA-graph replay that this CUPTI does not attribute — re-run with "
                 "--enforce-eager to confirm."
             )
+            if self.n_invalid_duration:
+                out.append(
+                    f"{self.n_invalid_duration} kernel record(s) had non-positive duration "
+                    "and were excluded as invalid."
+                )
             return out
+        if self.n_invalid_duration:
+            out.append(
+                f"{self.n_invalid_duration} kernel record(s) had non-positive duration "
+                "and were excluded from all shares."
+            )
+        if self.window_ns is not None and self.window_ns <= 0:
+            out.append(
+                f"capture window duration is non-positive ({self.window_ns} ns); "
+                "GPU-active share is unavailable."
+            )
+        if self.gpu_active_share is not None and self.gpu_active_share > 1.0:
+            out.append(
+                f"GPU active share is {self.gpu_active_share:.1%}, above the capture window; "
+                "event/window clocks are inconsistent."
+            )
         if self.gpu_active_share is not None and self.gpu_active_share < 0.2:
             out.append(
                 f"GPU active only {self.gpu_active_share:.1%} of the window. Either the "
@@ -210,11 +231,15 @@ def summarize_kernels(kernels, *, window_ns: int | None = None,
     per_device: dict[int, list[tuple[int, int]]] = {}
     truncated: list[str] = []
     total_time = 0
+    invalid_duration = 0
 
     for k in kernels:
         name = getattr(k, "name", "") or ""
         start, end = int(k.start_ns), int(k.end_ns)
-        dur = max(end - start, 0)
+        dur = end - start
+        if dur <= 0:
+            invalid_duration += 1
+            continue
         total_time += dur
         by_bucket.setdefault(classify_kernel(name), []).append((name, dur))
         per_device.setdefault(int(getattr(k, "device_id", 0) or 0), []).append((start, end))
@@ -237,7 +262,11 @@ def summarize_kernels(kernels, *, window_ns: int | None = None,
     buckets.sort(key=lambda b: -b.time_ns)
 
     active = {dev: _active_ns(iv) for dev, iv in per_device.items()}
-    share = (max(active.values()) / window_ns) if (active and window_ns) else None
+    share = (
+        max(active.values()) / window_ns
+        if active and window_ns is not None and window_ns > 0
+        else None
+    )
 
     return KernelBreakdown(
         n_kernels=sum(b.n_kernels for b in buckets),
@@ -249,6 +278,7 @@ def summarize_kernels(kernels, *, window_ns: int | None = None,
         n_truncated_names=len(truncated),
         n_distinct_truncated=len(set(truncated)),
         n_devices=len(active),
+        n_invalid_duration=invalid_duration,
     )
 
 
diff --git a/tests/test_kernel_taxonomy.py b/tests/test_kernel_taxonomy.py
index c1a3710..a820172 100644
--- a/tests/test_kernel_taxonomy.py
+++ b/tests/test_kernel_taxonomy.py
@@ -159,6 +159,29 @@ def test_empty_trace_warns_about_graph_replay_and_stops_there():
     assert "CUDA-graph replay" in warnings[0]
 
 
+def test_nonpositive_kernel_durations_are_excluded_and_named():
+    kernels = [
+        make_kernel("fused_moe_kernel", start_ns=10, end_ns=10),
+        make_kernel("ampere_fp16_s16816gemm_tn", start_ns=20, end_ns=19),
+    ]
+
+    bd = summarize_kernels(kernels, window_ns=100)
+
+    assert bd.n_kernels == 0
+    assert bd.n_invalid_duration == 2
+    assert bd.kernel_time_ns == 0
+    assert any("non-positive duration" in warning for warning in bd.warnings())
+
+
+def test_nonpositive_capture_window_does_not_fabricate_active_share():
+    bd = summarize_kernels(
+        [make_kernel("fused_moe_kernel", start_ns=0, end_ns=10)], window_ns=0
+    )
+
+    assert bd.gpu_active_share is None
+    assert any("capture window duration is non-positive" in warning for warning in bd.warnings())
+
+
 def test_idle_looking_gpu_is_warned_about():
     bd = summarize_kernels([make_kernel("fused_moe_kernel", start_ns=0, end_ns=50)],
                            window_ns=10_000)
diff --git a/tests/test_serve_attach.py b/tests/test_serve_attach.py
index b00294d..b96872a 100644
--- a/tests/test_serve_attach.py
+++ b/tests/test_serve_attach.py
@@ -454,6 +454,19 @@ def test_empty_trace_and_idle_window_are_different_failures(tmp_path):
     assert _write(tmp_path / "c", [_FakeKernel()], had_traffic=True).status == "ok"
 
 
+def test_zero_duration_kernel_records_do_not_make_capture_successful(tmp_path):
+    result = _write(
+        tmp_path / "zero-duration",
+        [_FakeKernel(start_ns=10, end_ns=10)],
+        had_traffic=True,
+    )
+
+    assert result.status == "no_kernels"
+    assert result.n_kernels == 0
+    assert result.breakdown.n_invalid_duration == 1
+    assert any("non-positive duration" in warning for warning in result.warnings)
+
+
 def test_both_paths_write_the_same_artifact_set(tmp_path):
     """A driven benchmark and a production observation of the same server are only
     comparable if they leave the same files behind."""

From c69e5153b194937403f7fe9ceba8dae42216870b Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:56:31 -0700
Subject: [PATCH 26/70] fix: wire CUDA compatibility gate into vLLM workload

---
 gitm/cuda_env.py                 | 7 ++++++-
 gitm/workloads.py                | 4 ++++
 tests/test_cuda_env.py           | 7 +++++++
 tests/test_run_loop_workload.py  | 1 +
 tests/test_workload_bootstrap.py | 1 +
 5 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/gitm/cuda_env.py b/gitm/cuda_env.py
index cec928f..2696168 100644
--- a/gitm/cuda_env.py
+++ b/gitm/cuda_env.py
@@ -229,7 +229,12 @@ def require_compatible() -> None:
     inside ``torch._C._cuda_init()``, after the weights are already downloaded.
     """
     driver = driver_cuda()
-    if driver is not None and stack_for(driver) is None:
+    if driver is None:
+        raise RuntimeError(
+            "no NVIDIA driver detected; refusing to build a GPU workload on an "
+            "unverified CPU-only or driver-inaccessible host"
+        )
+    if stack_for(driver) is None:
         raise RuntimeError(
             f"unsupported host: this driver supports only CUDA {driver[0]}.{driver[1]}, "
             f"and vLLM's wheels are CUDA {max(SUPPORTED_STACKS)} builds. No torch "
diff --git a/gitm/workloads.py b/gitm/workloads.py
index f877fa6..a569a4c 100644
--- a/gitm/workloads.py
+++ b/gitm/workloads.py
@@ -604,6 +604,10 @@ def _vllm_decode_factory(cfg: LoopConfig) -> WorkloadRunner:
     if os.environ.get("GITM_VLLM_SYNTHETIC") == "1":
         return _vllm_synthetic_runner(n_prompts, max_tokens)
 
+    from gitm import cuda_env
+
+    cuda_env.require_compatible()
+
     import time
 
     from vllm import LLM, SamplingParams
diff --git a/tests/test_cuda_env.py b/tests/test_cuda_env.py
index d339472..79ff818 100644
--- a/tests/test_cuda_env.py
+++ b/tests/test_cuda_env.py
@@ -79,6 +79,13 @@ def test_require_compatible_raises_before_an_expensive_build(host):
         cuda_env.require_compatible()
 
 
+def test_require_compatible_refuses_missing_driver(host):
+    host(driver=None)
+
+    with pytest.raises(RuntimeError, match="no NVIDIA driver detected"):
+        cuda_env.require_compatible()
+
+
 def test_require_compatible_is_silent_when_the_stack_fits(host):
     host(driver=(13, 0), torch=(13, 0), vllm=13)
     cuda_env.require_compatible()  # must not raise
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index 9f482ce..15a22bb 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -556,6 +556,7 @@ def __init__(self, max_tokens: int = 0, temperature: float = 0.0):
     fake.LLM = _LLM
     fake.SamplingParams = _SamplingParams
     monkeypatch.setitem(sys.modules, "vllm", fake)
+    monkeypatch.setattr("gitm.cuda_env.require_compatible", lambda: None)
     monkeypatch.setenv("GITM_VLLM_PROMPTS", "3")
     monkeypatch.setenv("GITM_VLLM_MAX_TOKENS", "5")
     monkeypatch.delenv("GITM_VLLM_SYNTHETIC", raising=False)
diff --git a/tests/test_workload_bootstrap.py b/tests/test_workload_bootstrap.py
index 08e7e0b..e4fd155 100644
--- a/tests/test_workload_bootstrap.py
+++ b/tests/test_workload_bootstrap.py
@@ -134,6 +134,7 @@ def generate(self, prompts, params):
         "vllm",
         types.SimpleNamespace(LLM=FakeLLM, SamplingParams=FakeSamplingParams),
     )
+    monkeypatch.setattr("gitm.cuda_env.require_compatible", lambda: None)
     monkeypatch.setattr(workloads, "sync_device", lambda: None)
     monkeypatch.setenv("GITM_VLLM_MODEL", "fake/model")
     monkeypatch.setenv("GITM_VLLM_PROMPTS", "2")

From fa7d20a98f5cf3fddb3faf5916b39374611c4232 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 13:57:09 -0700
Subject: [PATCH 27/70] fix: refuse non-finite report deltas

---
 gitm/optimizer/report.py      |  6 +++++-
 tests/test_report_snapshot.py | 18 ++++++++++++++++++
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/gitm/optimizer/report.py b/gitm/optimizer/report.py
index 5be890f..e43393d 100644
--- a/gitm/optimizer/report.py
+++ b/gitm/optimizer/report.py
@@ -34,6 +34,10 @@ class Claim:
     def __post_init__(self) -> None:
         if not math.isfinite(self.residual_value):
             raise ValueError("residual_value must be finite")
+        if not math.isfinite(self.predicted_delta):
+            raise ValueError("predicted_delta must be finite")
+        if self.measured_delta is not None and not math.isfinite(self.measured_delta):
+            raise ValueError("measured_delta must be finite when present")
 
     @property
     def residual_display_value(self) -> float:
@@ -103,7 +107,7 @@ def _default_summary(claims: list[Claim]) -> str:
     verified = [c for c in claims if c.measured_delta is not None and not c.rolled_back]
     if not verified:
         return "No claims verified within budget. See diagnostic below."
-    total = sum(c.measured_delta or 0.0 for c in verified)
+    total = sum(c.measured_delta for c in verified if c.measured_delta is not None)
     return f"{len(verified)} verified claims, aggregate measured delta {total:+.1%}."
 
 
diff --git a/tests/test_report_snapshot.py b/tests/test_report_snapshot.py
index b0006d8..94b75c0 100644
--- a/tests/test_report_snapshot.py
+++ b/tests/test_report_snapshot.py
@@ -170,6 +170,24 @@ def test_non_finite_residual_is_refused(raw):
         )
 
 
+@pytest.mark.parametrize("field", ["predicted_delta", "measured_delta"])
+@pytest.mark.parametrize("value", [math.nan, math.inf, -math.inf])
+def test_non_finite_claim_deltas_are_refused(field, value):
+    kwargs = {
+        "summary": "model gap",
+        "residual_invariant": "kernel_time",
+        "residual_value": 0.1,
+        "causal_evidence": "trace",
+        "intervention_name": "candidate",
+        "predicted_delta": 0.01,
+        "measured_delta": 0.02,
+    }
+    kwargs[field] = value
+
+    with pytest.raises(ValueError, match=f"{field} must be finite"):
+        Claim(**kwargs)
+
+
 def test_runtime_diagnostics_are_printed_when_present():
     rendered = write_report(
         [],

From 9927ce22e85b40d211916675a99969e311acbd3d Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:00:01 -0700
Subject: [PATCH 28/70] fix: refuse invalid benchmark sign-off evidence

---
 gitm/bench/baseline.py |  4 ++++
 gitm/bench/schema.py   |  8 ++++----
 tests/test_bench.py    | 35 +++++++++++++++++++++++++++++++++++
 3 files changed, 43 insertions(+), 4 deletions(-)

diff --git a/gitm/bench/baseline.py b/gitm/bench/baseline.py
index aa2ab2b..00d427f 100644
--- a/gitm/bench/baseline.py
+++ b/gitm/bench/baseline.py
@@ -146,6 +146,10 @@ def aggregate(runs: list[BaselineRun], config: BenchConfig) -> BaselineSummary:
             provenance_issues.append(f"seed {run.seed}: GPU identity unavailable ({run.gpu_name!r})")
         if run.device_count < 1:
             provenance_issues.append(f"seed {run.seed}: no GPU devices reported")
+        if run.started_at_ns <= 0:
+            provenance_issues.append(f"seed {run.seed}: start timestamp unavailable")
+        if run.ended_at_ns <= run.started_at_ns:
+            provenance_issues.append(f"seed {run.seed}: end timestamp does not follow start")
         provenance_issues.extend(f"seed {run.seed}: {note}" for note in run.provenance_warnings)
     gates.append(
         GateResult(
diff --git a/gitm/bench/schema.py b/gitm/bench/schema.py
index 5a565a8..dbff132 100644
--- a/gitm/bench/schema.py
+++ b/gitm/bench/schema.py
@@ -93,7 +93,7 @@ class BenchConfig(BaseModel):
     seeds: list[int] = Field(min_length=1)
     spread_tolerance: float = Field(default=0.02, gt=0.0, le=1.0)
     gpu_active_ceiling: float = Field(default=0.85, gt=0.0, le=1.0)
-    baseline_target: float | None = None
+    baseline_target: float | None = Field(default=None, gt=0.0, allow_inf_nan=False)
     target_direction: TargetDirection = "ge"
     dataset: DatasetRef
     work_unit: WorkUnit
@@ -122,7 +122,7 @@ class StallPhase(BaseModel):
     data_stall: float = Field(ge=0.0, le=1.0)
     sync: float = Field(ge=0.0, le=1.0)
     gpu_active: float = Field(ge=0.0, le=1.0)
-    throughput: float | None = None  # in the benchmark's metric units
+    throughput: float | None = Field(default=None, ge=0.0, allow_inf_nan=False)
     wall_clock_s: float = Field(ge=0.0)
 
 
@@ -140,8 +140,8 @@ class BaselineRun(BaseModel):
     seed: int
     vendor: Vendor
     metric: str
-    metric_value: float
-    warm_window_s: int
+    metric_value: float = Field(gt=0.0, allow_inf_nan=False)
+    warm_window_s: int = Field(gt=0)
 
     # Provenance: a baseline is only reproducible if these are pinned.
     git_sha: str
diff --git a/tests/test_bench.py b/tests/test_bench.py
index c1308a3..caeafe1 100644
--- a/tests/test_bench.py
+++ b/tests/test_bench.py
@@ -127,6 +127,8 @@ def _run(seed: int, value: float, gpu: float = 0.7) -> object:
         manifest_sha256="manifest123",
         gpu_name="A100",
         device_count=1,
+        started_at_ns=1,
+        ended_at_ns=2,
         stall_breakdown=[
             StallPhase(phase="all", cpu=0.03, data_stall=max(0.0, 1 - gpu - 0.03 - 0.05),
                        sync=0.05, gpu_active=gpu, throughput=value, wall_clock_s=60.0)
@@ -201,6 +203,39 @@ def test_baseline_refuses_cpu_or_unpinned_runs():
     assert "manifest digest unavailable" in gate.detail
 
 
+def test_baseline_refuses_missing_or_reversed_timing_provenance():
+    from gitm.bench.baseline import aggregate
+
+    runs = [_run(42, 26e6), _run(43, 26e6), _run(44, 26e6)]
+    runs[0].started_at_ns = 0
+    runs[1].ended_at_ns = runs[1].started_at_ns
+
+    summary = aggregate(runs, _hft_config())
+
+    gate = next(g for g in summary.gates if g.name == "provenance")
+    assert not gate.passed
+    assert "start timestamp unavailable" in gate.detail
+    assert "end timestamp does not follow start" in gate.detail
+
+
+@pytest.mark.parametrize("value", [0.0, -1.0, float("nan"), float("inf")])
+def test_baseline_run_refuses_unusable_metric_values(value):
+    from pydantic import ValidationError
+
+    with pytest.raises(ValidationError):
+        _run(42, value)
+
+
+@pytest.mark.parametrize("target", [0.0, -1.0, float("nan"), float("inf")])
+def test_bench_config_refuses_unusable_baseline_target(target):
+    from pydantic import ValidationError
+
+    cfg = _hft_config().model_dump()
+    cfg["baseline_target"] = target
+    with pytest.raises(ValidationError):
+        type(_hft_config()).model_validate(cfg)
+
+
 def test_baseline_fails_on_too_few_runs_and_below_target():
     from gitm.bench.baseline import aggregate
 

From c2d944abb3cee469e4b3d53672c52e7f6436699a Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:02:29 -0700
Subject: [PATCH 29/70] fix: surface fallback GPU counts

---
 gitm/planner/context.py         | 21 +++++++++++++++++++--
 gitm/scheduler/loop.py          |  4 ++++
 gitm/serve/attach.py            |  4 ++++
 tests/test_run_loop_workload.py |  3 +++
 tests/test_serve_attach.py      | 18 ++++++++++++++++--
 tests/test_vllm_embodiment.py   | 21 +++++++++++++++++++++
 6 files changed, 67 insertions(+), 4 deletions(-)

diff --git a/gitm/planner/context.py b/gitm/planner/context.py
index d8e5f13..08af0cd 100644
--- a/gitm/planner/context.py
+++ b/gitm/planner/context.py
@@ -120,6 +120,7 @@ class PlannerContext:
     peak: HardwarePeak | None
     sku: str | None
     num_gpus: int
+    num_gpus_is_fallback: bool = False
 
 
 def _query_nvml() -> tuple[str | None, int | None]:
@@ -221,13 +222,23 @@ def build_planner_context(
 
     ``GITM_GPU_SKU`` overrides NVML (useful in CI / on a box without pynvml).
     """
+    if num_gpus is not None and num_gpus <= 0:
+        raise ValueError(f"num_gpus must be positive when supplied, got {num_gpus}")
     env_sku = os.environ.get("GITM_GPU_SKU")
     # Only touch NVML if something it provides is actually missing.
     nvml_name = nvml_count = None
     if env_sku is None or num_gpus is None:
         nvml_name, nvml_count = _query_nvml()
     sku = env_sku or nvml_name
-    n = num_gpus or nvml_count or 1
+    if num_gpus is not None:
+        n = num_gpus
+        num_gpus_is_fallback = False
+    elif nvml_count is not None and nvml_count > 0:
+        n = nvml_count
+        num_gpus_is_fallback = False
+    else:
+        n = 1
+        num_gpus_is_fallback = True
     peak = peak_for_sku(sku)
     dtype = _engine_dtype(engine)
     kv_len = _engine_kv_len(engine)
@@ -241,4 +252,10 @@ def build_planner_context(
         has_collective=n > 1,
         has_interconnect=n > 1,  # refined later by NVLink/IB probe
     )
-    return PlannerContext(gate=gate, peak=peak, sku=sku, num_gpus=n)
+    return PlannerContext(
+        gate=gate,
+        peak=peak,
+        sku=sku,
+        num_gpus=n,
+        num_gpus_is_fallback=num_gpus_is_fallback,
+    )
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 55c0c18..11cc1e0 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -707,6 +707,10 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
     # Llama/A100 default prediction.
     pctx = build_planner_context(cfg.engine, workload=workload)
     graph_resolution = _execution_graph(cfg.engine, pctx, sched_summary)
+    if pctx.num_gpus_is_fallback:
+        graph_resolution.diagnostics.append(
+            "GPU count was unavailable; using 1 for intervention applicability only"
+        )
     graph_resolution.diagnostics.extend(sched_summary.diagnostics)
     if not graph_resolution.ok:
         (run_dir / "prediction_refusal.json").write_text(
diff --git a/gitm/serve/attach.py b/gitm/serve/attach.py
index fcbe382..9428d44 100644
--- a/gitm/serve/attach.py
+++ b/gitm/serve/attach.py
@@ -465,6 +465,8 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
             f"GPU SKU {planner_ctx.sku or 'unknown'!r} is not in the hardware catalogue; "
             f"pricing uses fallback {hw.name!r}"
         )
+    if planner_ctx.num_gpus_is_fallback:
+        warnings.append("GPU count was unavailable; recording a fallback count of 1")
     if g.has_unpriced_nodes:
         missing = []
         if g.has_unpriced_compute:
@@ -488,6 +490,8 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
         "hardware": planner_ctx.sku,
         "hardware_pricing": hw.name,
         "hardware_is_fallback": planner_ctx.peak is None,
+        "num_gpus": planner_ctx.num_gpus,
+        "num_gpus_is_fallback": planner_ctx.num_gpus_is_fallback,
         "sharding": {"tp": sh.tp, "ep": sh.ep, "dp": sh.dp},
         "dtypes": {
             "weight": spec.weight_dtype,
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index 15a22bb..c981d3e 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -371,6 +371,7 @@ def test_vllm_loop_surfaces_residual_coverage(tmp_path: Path, monkeypatch):
     """Unclassified work must be visible in both the JSON and human report."""
     import json
 
+    import gitm.planner.context as planner_context
     import gitm.scheduler.loop as loop
 
     @contextmanager
@@ -384,6 +385,7 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
     monkeypatch.setattr(loop, "capture", fake_capture)
     monkeypatch.setattr(loop, "sync_device", lambda: None)
     monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200")
+    monkeypatch.setattr(planner_context, "_query_nvml", lambda: ("NVIDIA B200", None))
 
     from gitm import optimize
 
@@ -401,6 +403,7 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
     assert payload["coverage"]["warnings"]
     assert "## Runtime diagnostics" in result["report_md"]
     assert "matched to the predicted graph" in result["report_md"]
+    assert "GPU count was unavailable" in result["report_md"]
 
 
 def test_vllm_loop_without_model_does_not_run_autoresearch_on_default_graph(
diff --git a/tests/test_serve_attach.py b/tests/test_serve_attach.py
index b96872a..d8c9d10 100644
--- a/tests/test_serve_attach.py
+++ b/tests/test_serve_attach.py
@@ -292,13 +292,20 @@ def test_predicted_graph_surfaces_resolved_warnings_and_bytes_fallback(
     monkeypatch.setattr(
         planner_context,
         "build_planner_context",
-        lambda: SimpleNamespace(peak=peak_for_sku("NVIDIA B200"), sku="NVIDIA B200"),
+        lambda: SimpleNamespace(
+            peak=peak_for_sku("NVIDIA B200"),
+            sku="NVIDIA B200",
+            num_gpus=1,
+            num_gpus_is_fallback=True,
+        ),
     )
 
     att._emit_predicted_graph(discover.Target(pid=1, cmdline=[]), tmp_path)
 
     payload = json.loads((tmp_path / "predicted_moe_graph.json").read_text())
     assert payload["has_fallback_bytes"] is True
+    assert payload["num_gpus_is_fallback"] is True
+    assert any("GPU count was unavailable" in warning for warning in payload["warnings"])
     assert any(node["bytes_are_fallback"] for node in payload["nodes"])
     assert payload["warnings"]
     stdout = capsys.readouterr().out
@@ -340,12 +347,19 @@ def test_predicted_graph_known_dtypes_leave_bytes_fallback_clean(tmp_path, monke
     monkeypatch.setattr(
         planner_context,
         "build_planner_context",
-        lambda: SimpleNamespace(peak=peak_for_sku("NVIDIA B200"), sku="NVIDIA B200"),
+        lambda: SimpleNamespace(
+            peak=peak_for_sku("NVIDIA B200"),
+            sku="NVIDIA B200",
+            num_gpus=1,
+            num_gpus_is_fallback=False,
+        ),
     )
 
     att._emit_predicted_graph(discover.Target(pid=1, cmdline=[]), tmp_path)
     payload = json.loads((tmp_path / "predicted_moe_graph.json").read_text())
     assert payload["has_fallback_bytes"] is False
+    assert payload["num_gpus_is_fallback"] is False
+    assert not any("GPU count was unavailable" in warning for warning in payload["warnings"])
     assert not any(node["bytes_are_fallback"] for node in payload["nodes"])
 
 
diff --git a/tests/test_vllm_embodiment.py b/tests/test_vllm_embodiment.py
index afd154b..b3c1a41 100644
--- a/tests/test_vllm_embodiment.py
+++ b/tests/test_vllm_embodiment.py
@@ -81,6 +81,27 @@ def test_build_planner_context_sku_override(monkeypatch):
     assert pctx.gate.workload == "vllm-decode"
     assert pctx.gate.hardware == "NVIDIA A100-SXM4-80GB"
     assert pctx.gate.dtype is None  # no engine attached
+    assert pctx.num_gpus_is_fallback is False
+
+
+def test_build_planner_context_flags_unknown_gpu_count(monkeypatch):
+    import gitm.planner.context as context
+
+    monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200")
+    monkeypatch.setattr(context, "_query_nvml", lambda: ("NVIDIA B200", None))
+
+    pctx = context.build_planner_context()
+
+    assert pctx.num_gpus == 1
+    assert pctx.num_gpus_is_fallback is True
+
+
+@pytest.mark.parametrize("count", [0, -1])
+def test_build_planner_context_refuses_invalid_explicit_gpu_count(count):
+    from gitm.planner.context import build_planner_context
+
+    with pytest.raises(ValueError, match="num_gpus must be positive"):
+        build_planner_context(num_gpus=count)
 
 
 def test_unknown_sku_yields_no_peak(monkeypatch):

From 30f9483e9050946758573ead15f8e5fa73314715 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:04:07 -0700
Subject: [PATCH 30/70] fix: surface trace device-count failures

---
 gitm/tracer/capture.py | 37 ++++++++++++++++++++++++++++++++-----
 gitm/tracer/cupti.py   |  5 +----
 tests/test_cupti.py    | 20 ++++++++++++++++++++
 3 files changed, 53 insertions(+), 9 deletions(-)

diff --git a/gitm/tracer/capture.py b/gitm/tracer/capture.py
index a779186..94f76a6 100644
--- a/gitm/tracer/capture.py
+++ b/gitm/tracer/capture.py
@@ -187,12 +187,39 @@ def _device_count(backend, injected: bool) -> int:
 
         shim = load_shim()
         if shim is None:
+            warnings.warn(
+                "injected trace device count unavailable: CUPTI shim is not importable; "
+                "recording 0 devices",
+                RuntimeWarning,
+                stacklevel=2,
+            )
             return 0
-        try:
-            return int(shim.device_count())
-        except Exception:
-            return 0
-    return backend.device_count() if backend else 0
+        counter = shim.device_count
+    elif backend is not None:
+        counter = backend.device_count
+    else:
+        # ``source=none`` in the trace header is the explicit provenance for the
+        # normal no-backend path; an extra warning here would add no information.
+        return 0
+
+    try:
+        count = int(counter())
+    except Exception as exc:
+        warnings.warn(
+            f"active trace device count unavailable: {type(exc).__name__}: {exc}; "
+            "recording 0 devices",
+            RuntimeWarning,
+            stacklevel=2,
+        )
+        return 0
+    if count <= 0:
+        warnings.warn(
+            f"active trace device-count probe reported {count} devices; recording 0",
+            RuntimeWarning,
+            stacklevel=2,
+        )
+        return 0
+    return count
 
 
 def _backend():
diff --git a/gitm/tracer/cupti.py b/gitm/tracer/cupti.py
index cdeebf7..bfdd38f 100644
--- a/gitm/tracer/cupti.py
+++ b/gitm/tracer/cupti.py
@@ -33,10 +33,7 @@ def __init__(self) -> None:
             )
 
     def device_count(self) -> int:
-        try:
-            return int(self._shim.device_count())
-        except Exception:
-            return 0
+        return int(self._shim.device_count())
 
     def start(self) -> None:
         self._shim.start()
diff --git a/tests/test_cupti.py b/tests/test_cupti.py
index d6203e4..039b01d 100644
--- a/tests/test_cupti.py
+++ b/tests/test_cupti.py
@@ -159,6 +159,26 @@ def test_capture_falls_back_to_noop_trace(tmp_path, monkeypatch):
     assert header["device_count"] == 0
 
 
+@pytest.mark.parametrize(
+    "counter, message",
+    [
+        (lambda: 0, "reported 0 devices"),
+        (lambda: (_ for _ in ()).throw(RuntimeError("driver denied")), "driver denied"),
+    ],
+)
+def test_capture_warns_when_active_backend_device_count_is_unavailable(counter, message):
+    import importlib
+    from types import SimpleNamespace
+
+    capture_mod = importlib.import_module("gitm.tracer.capture")
+    backend = SimpleNamespace(device_count=counter)
+
+    with pytest.warns(RuntimeWarning, match=message):
+        count = capture_mod._device_count(backend, injected=False)
+
+    assert count == 0
+
+
 # --- full backend wiring via a fake shim ------------------------------------
 
 

From e1c40ac1cb9f912ba6920e31dd6bc71b47a86ea0 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:06:16 -0700
Subject: [PATCH 31/70] fix: wire resident footprint provenance

---
 gitm/planner/graph.py           | 19 +++++++++++++++++++
 gitm/scheduler/loop.py          |  8 ++++++++
 gitm/serve/attach.py            |  6 ++++++
 tests/test_moe_graph.py         |  8 ++++++++
 tests/test_run_loop_workload.py |  3 +++
 tests/test_serve_attach.py      |  4 ++++
 6 files changed, 48 insertions(+)

diff --git a/gitm/planner/graph.py b/gitm/planner/graph.py
index d837af4..f939aaf 100644
--- a/gitm/planner/graph.py
+++ b/gitm/planner/graph.py
@@ -168,6 +168,25 @@ def hardware_is_fallback(self) -> bool:
         """True when the graph uses substituted rather than detected SKU peaks."""
         return self.hw.is_fallback
 
+    @property
+    def resident_weight_bytes_per_rank(self) -> float | None:
+        """Sparse-model resident weight footprint for this graph's rank.
+
+        Dense v0 does not yet enumerate a complete resident footprint, so it
+        returns ``None`` rather than repurposing per-step traffic as capacity.
+        """
+        if not isinstance(self.model, SparseMoEModelSpec):
+            return None
+        # Local import avoids the graph <-> sparse graph construction cycle.
+        from gitm.planner.moe_graph import model_weight_bytes
+
+        return model_weight_bytes(self.model, self.sharding)
+
+    @property
+    def resident_weight_bytes_is_lower_bound(self) -> bool:
+        """True when private DSpark shapes make the footprint a known lower bound."""
+        return isinstance(self.model, SparseMoEModelSpec) and bool(self.model.dspark_layer_ids)
+
 
 def predict_graph(
     model: ModelSpec | None = None,
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 11cc1e0..1b3c8c4 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -740,6 +740,10 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
         )
     graph = graph_resolution.graph
     assert graph is not None
+    if graph.resident_weight_bytes_is_lower_bound:
+        graph_resolution.diagnostics.append(
+            "resident weight footprint is a lower bound because DSpark parameter shapes are private"
+        )
     (run_dir / "predicted_graph.json").write_text(
         json.dumps(
             {
@@ -747,6 +751,10 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
                 "model_source": graph_resolution.model_source,
                 "nodes": len(graph.nodes),
                 "total_pred_s": graph.total_pred_s,
+                "resident_weight_bytes_per_rank": graph.resident_weight_bytes_per_rank,
+                "resident_weight_bytes_is_lower_bound": (
+                    graph.resident_weight_bytes_is_lower_bound
+                ),
                 "hardware": pctx.sku,
                 "hardware_pricing": graph.hw.name,
                 "hardware_is_fallback": graph.hardware_is_fallback,
diff --git a/gitm/serve/attach.py b/gitm/serve/attach.py
index 9428d44..9c33136 100644
--- a/gitm/serve/attach.py
+++ b/gitm/serve/attach.py
@@ -480,6 +480,10 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
         warnings.append(
             "byte widths include an unknown-dtype bf16 fallback; the memory floor is approximate"
         )
+    if g.resident_weight_bytes_is_lower_bound:
+        warnings.append(
+            "resident weight footprint is a lower bound because DSpark parameter shapes are private"
+        )
     n_estimated = sum(1 for n in g.nodes if n.prediction.estimated)
     if n_estimated:
         warnings.append(f"{n_estimated} predicted node(s) use documented estimated cost models")
@@ -502,6 +506,8 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
         "batch": {"batch": resolved.batch.batch, "kv_cache_len": resolved.batch.kv_cache_len},
         "applied_overrides": resolved.applied_overrides,
         "total_pred_s": g.total_pred_s,
+        "resident_weight_bytes_per_rank": g.resident_weight_bytes_per_rank,
+        "resident_weight_bytes_is_lower_bound": g.resident_weight_bytes_is_lower_bound,
         "has_unpriced_collectives": g.has_unpriced_collectives,
         "has_unpriced_nodes": g.has_unpriced_nodes,
         "has_unpriced_compute": g.has_unpriced_compute,
diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py
index 6bc697a..b455139 100644
--- a/tests/test_moe_graph.py
+++ b/tests/test_moe_graph.py
@@ -637,6 +637,12 @@ def test_dspark_variant_is_a_lower_bound_not_an_estimate(spec, base_spec):
     published_delta = 167e9 - V4_BASE_PUBLISHED_BYTES
     assert delta < published_delta / 100
 
+    g = predict_moe_graph(spec, HardwareSpec(), BatchConfig(), ShardingConfig(tp=8))
+    assert g.resident_weight_bytes_per_rank == pytest.approx(
+        model_weight_bytes(spec, ShardingConfig(tp=8))
+    )
+    assert g.resident_weight_bytes_is_lower_bound is True
+
 
 def test_base_checkpoint_has_no_dspark_nodes(base_spec, b200):
     """No dspark keys in the config means no dspark work in the graph.
@@ -647,6 +653,8 @@ def test_base_checkpoint_has_no_dspark_nodes(base_spec, b200):
     g = predict_moe_graph(base_spec, b200, BatchConfig(batch=1, kv_cache_len=1024))
     assert not [n for n in g.nodes if n.op == "dspark"]
     assert {n.op for n in g.nodes if n.prediction.estimated} == {"attn_out_proj"}
+    assert g.resident_weight_bytes_per_rank == pytest.approx(model_weight_bytes(base_spec))
+    assert g.resident_weight_bytes_is_lower_bound is False
 
 
 def test_both_checkpoint_variants_yield_identical_per_layer_ratios(spec, base_spec):
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index c981d3e..e1f20fb 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -397,10 +397,13 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
         workload_runner=lambda: {},
     )
     payload = json.loads((Path(result["run_dir"]) / "residuals.json").read_text())
+    predicted = json.loads((Path(result["run_dir"]) / "predicted_graph.json").read_text())
 
     assert payload["coverage"]["total_kernels"] == 2
     assert payload["coverage"]["matched_kernels"] == 1
     assert payload["coverage"]["warnings"]
+    assert predicted["resident_weight_bytes_per_rank"] > 0
+    assert predicted["resident_weight_bytes_is_lower_bound"] is False
     assert "## Runtime diagnostics" in result["report_md"]
     assert "matched to the predicted graph" in result["report_md"]
     assert "GPU count was unavailable" in result["report_md"]
diff --git a/tests/test_serve_attach.py b/tests/test_serve_attach.py
index d8c9d10..da92996 100644
--- a/tests/test_serve_attach.py
+++ b/tests/test_serve_attach.py
@@ -304,6 +304,8 @@ def test_predicted_graph_surfaces_resolved_warnings_and_bytes_fallback(
 
     payload = json.loads((tmp_path / "predicted_moe_graph.json").read_text())
     assert payload["has_fallback_bytes"] is True
+    assert payload["resident_weight_bytes_per_rank"] > 0
+    assert payload["resident_weight_bytes_is_lower_bound"] is False
     assert payload["num_gpus_is_fallback"] is True
     assert any("GPU count was unavailable" in warning for warning in payload["warnings"])
     assert any(node["bytes_are_fallback"] for node in payload["nodes"])
@@ -358,6 +360,8 @@ def test_predicted_graph_known_dtypes_leave_bytes_fallback_clean(tmp_path, monke
     att._emit_predicted_graph(discover.Target(pid=1, cmdline=[]), tmp_path)
     payload = json.loads((tmp_path / "predicted_moe_graph.json").read_text())
     assert payload["has_fallback_bytes"] is False
+    assert payload["resident_weight_bytes_per_rank"] > 0
+    assert payload["resident_weight_bytes_is_lower_bound"] is False
     assert payload["num_gpus_is_fallback"] is False
     assert not any("GPU count was unavailable" in warning for warning in payload["warnings"])
     assert not any(node["bytes_are_fallback"] for node in payload["nodes"])

From fcf1d72bbfc43bf5894b776b4c213849a36bc1a3 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:08:07 -0700
Subject: [PATCH 32/70] fix: surface autoresearch GPU-count fallback

---
 gitm/agents/autoresearch.py | 21 +++++++++++++++++++--
 gitm/scheduler/loop.py      |  5 ++++-
 tests/test_autoresearch.py  | 26 ++++++++++++++++++++++++++
 3 files changed, 49 insertions(+), 3 deletions(-)

diff --git a/gitm/agents/autoresearch.py b/gitm/agents/autoresearch.py
index 807d711..a10af99 100644
--- a/gitm/agents/autoresearch.py
+++ b/gitm/agents/autoresearch.py
@@ -558,8 +558,23 @@ def _visible_gpu_count() -> int:
     try:
         import torch
 
-        return torch.cuda.device_count() or 1
-    except Exception:
+        count = int(torch.cuda.device_count())
+        if count > 0:
+            return count
+        warnings.warn(
+            f"GPU-count detection reported {count}; autoresearch is using 1 and "
+            "may omit multi-GPU candidates",
+            RuntimeWarning,
+            stacklevel=2,
+        )
+        return 1
+    except Exception as exc:
+        warnings.warn(
+            "GPU-count detection failed; autoresearch is using 1 and may omit "
+            f"multi-GPU candidates ({type(exc).__name__}: {exc})",
+            RuntimeWarning,
+            stacklevel=2,
+        )
         return 1
 
 
@@ -645,6 +660,8 @@ def _knobs_from_engine_args(
     """
     import dataclasses
 
+    if gpu_count is not None and gpu_count <= 0:
+        raise ValueError(f"gpu_count must be positive when supplied, got {gpu_count}")
     gpus = _visible_gpu_count() if gpu_count is None else gpu_count
     domains = _argparse_domains(engine_args_cls)
     knobs: list[Knob] = []
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 1b3c8c4..466a9b6 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -1025,7 +1025,10 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
 
     # Phase 4b - agentic autoresearch through the catalog gate/rollback path.
     if time.time_ns() - started_ns < int(budget_s * 1e9):
-        proposer = FallbackProposer(EngineArgsProposer(), TableProposer())
+        proposer = FallbackProposer(
+            EngineArgsProposer(gpu_count=pctx.num_gpus),
+            TableProposer(),
+        )
 
         def _unenactable(spec: Any) -> str | None:
             if (
diff --git a/tests/test_autoresearch.py b/tests/test_autoresearch.py
index 96f3566..ea28b25 100644
--- a/tests/test_autoresearch.py
+++ b/tests/test_autoresearch.py
@@ -768,6 +768,32 @@ def test_visible_gpu_count_is_a_positive_int() -> None:
     assert isinstance(n, int) and n >= 1
 
 
+def test_visible_gpu_count_warns_when_torch_reports_no_gpus(monkeypatch) -> None:
+    import sys
+    from types import SimpleNamespace
+
+    monkeypatch.setitem(sys.modules, "torch", SimpleNamespace(cuda=SimpleNamespace(device_count=lambda: 0)))
+    with pytest.warns(RuntimeWarning, match="GPU-count detection reported 0"):
+        assert _visible_gpu_count() == 1
+
+
+def test_visible_gpu_count_keeps_observed_multi_gpu_count_clean(monkeypatch) -> None:
+    import sys
+    import warnings
+    from types import SimpleNamespace
+
+    monkeypatch.setitem(sys.modules, "torch", SimpleNamespace(cuda=SimpleNamespace(device_count=lambda: 4)))
+    with warnings.catch_warnings():
+        warnings.simplefilter("error")
+        assert _visible_gpu_count() == 4
+
+
+@pytest.mark.parametrize("count", [0, -1])
+def test_knobs_from_engine_args_refuses_invalid_explicit_gpu_count(count) -> None:
+    with pytest.raises(ValueError, match="gpu_count must be positive"):
+        _knobs_from_engine_args(_FakeEngineArgs, gpu_count=count)
+
+
 def test_vllm_knob_source_gpu_count_override_is_accepted_offline() -> None:
     # vLLM isn't importable in CI, so the offline fallback catalog is returned
     # regardless of gpu_count — this just proves the parameter doesn't crash

From efe9f4a2fac8e3682430011352449a574a68224b Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:14:08 -0700
Subject: [PATCH 33/70] fix: refuse default-shaped sparse predictions

---
 gitm/planner/roofline.py               | 100 ++++++++++++++++++++++---
 gitm/serve/model_config.py             |  73 ++++++++++++++++--
 tests/test_execution_graph_dispatch.py |  21 ++++++
 tests/test_moe_roofline.py             |  48 ++++++++++--
 tests/test_run_loop_workload.py        |  10 +++
 tests/test_serve_model_config.py       |  30 +++++++-
 6 files changed, 259 insertions(+), 23 deletions(-)

diff --git a/gitm/planner/roofline.py b/gitm/planner/roofline.py
index 15d3d67..18b37bb 100644
--- a/gitm/planner/roofline.py
+++ b/gitm/planner/roofline.py
@@ -19,6 +19,7 @@
 
 from __future__ import annotations
 
+import math
 from dataclasses import dataclass
 
 # Bytes of HBM traffic per stored weight, including the quantisation scales that
@@ -143,6 +144,36 @@ class ModelSpec:
     #: 1 = every layer is full attention, i.e. a conventional transformer.
     full_attn_layer_step: int = 1
 
+    def __post_init__(self) -> None:
+        for name in ("hidden", "n_layers", "n_heads", "num_kv_heads", "head_dim", "dtype_bytes", "vocab"):
+            value = getattr(self, name)
+            if not isinstance(value, int) or isinstance(value, bool) or value <= 0:
+                raise ValueError(f"{name} must be a positive integer, got {value!r}")
+        for name in ("moe_layer_step", "full_attn_layer_step"):
+            value = getattr(self, name)
+            if not isinstance(value, int) or isinstance(value, bool) or value <= 0:
+                raise ValueError(f"{name} must be a positive integer, got {value!r}")
+        for name in ("num_experts", "experts_per_token", "shared_experts", "first_dense_layers"):
+            value = getattr(self, name)
+            if not isinstance(value, int) or isinstance(value, bool) or value < 0:
+                raise ValueError(f"{name} must be a non-negative integer, got {value!r}")
+        if (self.num_experts == 0) != (self.experts_per_token == 0):
+            raise ValueError("num_experts and experts_per_token must both be zero or both positive")
+        if self.experts_per_token > self.num_experts:
+            raise ValueError("experts_per_token cannot exceed num_experts")
+        if self.first_dense_layers > self.n_layers:
+            raise ValueError("first_dense_layers cannot exceed n_layers")
+        for name in ("moe_intermediate", "shared_expert_intermediate", "weight_dtype_bytes"):
+            value = getattr(self, name)
+            if value is not None and (
+                not isinstance(value, int) or isinstance(value, bool) or value <= 0
+            ):
+                raise ValueError(f"{name} must be a positive integer when supplied, got {value!r}")
+        if not isinstance(self.intermediate, int) or isinstance(self.intermediate, bool):
+            raise ValueError(f"intermediate must be an integer, got {self.intermediate!r}")
+        if self.intermediate < 0 or (self.n_moe_layers < self.n_layers and self.intermediate == 0):
+            raise ValueError("intermediate must be positive when any dense FFN layer is present")
+
     @property
     def is_moe(self) -> bool:
         """True when *any* layer's FFN should be modeled as a mixture of experts."""
@@ -159,8 +190,7 @@ def is_moe_layer(self, layer: int) -> bool:
         """
         if not self.is_moe or layer < self.first_dense_layers:
             return False
-        step = max(self.moe_layer_step, 1)
-        return (layer - self.first_dense_layers) % step == 0
+        return (layer - self.first_dense_layers) % self.moe_layer_step == 0
 
     @property
     def n_moe_layers(self) -> int:
@@ -184,8 +214,7 @@ def is_full_attention_layer(self, layer: int) -> bool:
         magnitude, and it is why a hybrid model can serve long contexts with only
         a few percent of KV-cache utilisation.
         """
-        step = max(self.full_attn_layer_step, 1)
-        return layer % step == 0
+        return layer % self.full_attn_layer_step == 0
 
     @property
     def n_full_attention_layers(self) -> int:
@@ -195,7 +224,7 @@ def n_full_attention_layers(self) -> int:
     @property
     def is_hybrid_attention(self) -> bool:
         """True when some layers use linear/recurrent attention instead of KV."""
-        return max(self.full_attn_layer_step, 1) > 1
+        return self.full_attn_layer_step > 1
 
     @property
     def linear_attn_state_elems(self) -> int:
@@ -215,17 +244,21 @@ def linear_attn_state_elems(self) -> int:
     @property
     def w_bytes(self) -> int:
         """Bytes per weight element (falls back to the activation dtype)."""
-        return self.weight_dtype_bytes or self.dtype_bytes
+        return self.weight_dtype_bytes if self.weight_dtype_bytes is not None else self.dtype_bytes
 
     @property
     def expert_intermediate(self) -> int:
         """Per-routed-expert FFN width (falls back to the dense width)."""
-        return self.moe_intermediate or self.intermediate
+        return self.moe_intermediate if self.moe_intermediate is not None else self.intermediate
 
     @property
     def shared_intermediate(self) -> int:
         """Per-shared-expert FFN width (falls back to the routed width)."""
-        return self.shared_expert_intermediate or self.expert_intermediate
+        return (
+            self.shared_expert_intermediate
+            if self.shared_expert_intermediate is not None
+            else self.expert_intermediate
+        )
 
     # --- parameter accounting -------------------------------------------------
     # The "35B-A3B" naming convention: total parameters vs the ones a single
@@ -396,6 +429,30 @@ class SparseMoEModelSpec:
     kv_dtype: str = "fp8"  # KV cache and index keys
     act_dtype: str = "bf16"  # activations between ops
 
+    def __post_init__(self) -> None:
+        positive = (
+            "hidden", "n_layers", "n_heads", "num_kv_heads", "head_dim",
+            "q_lora_rank", "o_lora_rank", "o_groups", "vocab", "n_routed_experts",
+            "num_experts_per_tok", "moe_intermediate_size", "index_n_heads",
+            "index_head_dim", "index_topk",
+        )
+        for name in positive:
+            value = getattr(self, name)
+            if not isinstance(value, int) or isinstance(value, bool) or value <= 0:
+                raise ValueError(f"{name} must be a positive integer, got {value!r}")
+        for name in ("qk_rope_head_dim", "n_shared_experts", "sliding_window", "num_nextn_predict_layers", "dspark_markov_rank"):
+            value = getattr(self, name)
+            if not isinstance(value, int) or isinstance(value, bool) or value < 0:
+                raise ValueError(f"{name} must be a non-negative integer, got {value!r}")
+        if self.num_experts_per_tok > self.n_routed_experts:
+            raise ValueError("num_experts_per_tok cannot exceed n_routed_experts")
+        if any(not isinstance(r, int) or isinstance(r, bool) or r < 0 for r in self.compress_ratios):
+            raise ValueError("compress_ratios must contain non-negative integers")
+        if any(r == 0 for r in self.compress_ratios) and self.sliding_window <= 0:
+            raise ValueError("sliding_window must be positive when a compression ratio is 0")
+        if any(layer < 0 or layer >= self.n_layers for layer in self.dspark_layer_ids):
+            raise ValueError("dspark_layer_ids must refer to real transformer layers")
+
     def compress_ratio(self, layer: int) -> int:
         """KV compression ratio for ``layer`` (0/1 == uncompressed)."""
         if not self.compress_ratios:
@@ -454,6 +511,14 @@ class ShardingConfig:
     dp: int = 1
     ep_imbalance: float = 1.0
 
+    def __post_init__(self) -> None:
+        for name in ("tp", "ep", "dp"):
+            value = getattr(self, name)
+            if not isinstance(value, int) or isinstance(value, bool) or value <= 0:
+                raise ValueError(f"{name} must be a positive integer, got {value!r}")
+        if not math.isfinite(self.ep_imbalance) or self.ep_imbalance < 1.0:
+            raise ValueError("ep_imbalance must be finite and at least 1.0")
+
     @property
     def expert_shards(self) -> int:
         """Ranks the expert weights are divided across."""
@@ -477,10 +542,20 @@ class BatchConfig:
     speculative_tokens: int = 0
     acceptance_rate: float = 0.0
 
+    def __post_init__(self) -> None:
+        if not isinstance(self.batch, int) or isinstance(self.batch, bool) or self.batch <= 0:
+            raise ValueError(f"batch must be a positive integer, got {self.batch!r}")
+        for name in ("prompt_len", "kv_cache_len", "speculative_tokens"):
+            value = getattr(self, name)
+            if not isinstance(value, int) or isinstance(value, bool) or value < 0:
+                raise ValueError(f"{name} must be a non-negative integer, got {value!r}")
+        if not math.isfinite(self.acceptance_rate) or not 0.0 <= self.acceptance_rate <= 1.0:
+            raise ValueError("acceptance_rate must be finite and in [0, 1]")
+
     @property
     def positions_per_step(self) -> int:
         """Sequence positions the model actually computes in one step."""
-        return self.batch * (1 + max(0, self.speculative_tokens))
+        return self.batch * (1 + self.speculative_tokens)
 
     @property
     def tokens_per_step(self) -> float:
@@ -489,7 +564,7 @@ def tokens_per_step(self) -> float:
         Always at least ``batch``: the non-speculative token is verified, not
         drafted, so it is never rejected.
         """
-        return self.batch * (1.0 + max(0, self.speculative_tokens) * self.acceptance_rate)
+        return self.batch * (1.0 + self.speculative_tokens * self.acceptance_rate)
 
 
 @dataclass(frozen=True)
@@ -580,6 +655,11 @@ def roofline(
     bytes_are_fallback: bool = False,
 ) -> RooflinePrediction:
     """Compute the roofline prediction for a single op."""
+    if not math.isfinite(flops) or flops < 0 or not math.isfinite(bytes_moved) or bytes_moved < 0:
+        raise ValueError(
+            f"roofline work terms must be finite and non-negative, got "
+            f"flops={flops!r}, bytes_moved={bytes_moved!r}"
+        )
     peak_flops, peak_dtype = resolve_peak(hw, dtype)
     compute_is_unpriced = flops > 0 and peak_flops <= 0
     memory_is_unpriced = bytes_moved > 0 and hw.peak_mem_bw_bytes_per_s <= 0
diff --git a/gitm/serve/model_config.py b/gitm/serve/model_config.py
index 4bc6da2..8a8dd85 100644
--- a/gitm/serve/model_config.py
+++ b/gitm/serve/model_config.py
@@ -235,16 +235,77 @@ def validate_moe_config(cfg: dict[str, Any]) -> list[str]:
     supported config would carry.
     """
     missing: list[str] = []
-    if _first_present(cfg, _EXPERT_COUNT_ALIASES) is None:
-        missing.append(" | ".join(_EXPERT_COUNT_ALIASES) + " (routed expert count)")
-    if _first_present(cfg, _EXPERT_TOPK_ALIASES) is None:
-        missing.append(" | ".join(_EXPERT_TOPK_ALIASES) + " (experts per token)")
-    if _first_present(cfg, _EXPERT_INTER_ALIASES) is None:
-        missing.append(" | ".join(_EXPERT_INTER_ALIASES) + " (expert intermediate size)")
+
+    def positive_alias(keys: tuple[str, ...], label: str) -> int | None:
+        value = _first_present(cfg, keys)
+        joined = " | ".join(keys)
+        if value is None:
+            missing.append(f"{joined} ({label})")
+            return None
+        if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
+            missing.append(f"{joined} ({label}) must be a positive integer, got {value!r}")
+            return None
+        return value
+
+    n_experts = positive_alias(_EXPERT_COUNT_ALIASES, "routed expert count")
+    top_k = positive_alias(_EXPERT_TOPK_ALIASES, "experts per token")
+    positive_alias(_EXPERT_INTER_ALIASES, "expert intermediate size")
+    if n_experts is not None and top_k is not None and top_k > n_experts:
+        missing.append(
+            f"experts per token {top_k} exceeds routed expert count {n_experts}"
+        )
+
+    # The sparse graph is a V4-shaped attention model, not a generic MoE graph.
+    # Every field below changes a node's compute/bytes; defaulting any of them to
+    # V4 values would fabricate a plausible graph for a partial or foreign model.
+    for key in (
+        "hidden_size",
+        "num_hidden_layers",
+        "num_attention_heads",
+        "num_key_value_heads",
+        "head_dim",
+        "q_lora_rank",
+        "o_lora_rank",
+        "o_groups",
+        "vocab_size",
+        "index_n_heads",
+        "index_head_dim",
+        "index_topk",
+        "sliding_window",
+    ):
+        value = cfg.get(key)
+        if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
+            missing.append(f"{key} must be a declared positive integer, got {value!r}")
+    qk_rope = cfg.get("qk_rope_head_dim")
+    if isinstance(qk_rope, bool) or not isinstance(qk_rope, int) or qk_rope < 0:
+        missing.append(
+            f"qk_rope_head_dim must be a declared non-negative integer, got {qk_rope!r}"
+        )
+    shared = cfg.get("n_shared_experts")
+    if isinstance(shared, bool) or not isinstance(shared, int) or shared < 0:
+        missing.append(
+            f"n_shared_experts must be a declared non-negative integer, got {shared!r}"
+        )
+    if cfg.get("torch_dtype") is None:
+        missing.append("torch_dtype must be declared; activation width cannot be guessed")
+
+    n_layers = cfg.get("num_hidden_layers")
+    ratios = cfg.get("compress_ratios")
+    if not isinstance(ratios, (list, tuple)):
+        missing.append("compress_ratios must be declared for the sparse-attention graph")
+    elif isinstance(n_layers, int) and n_layers > 0 and len(ratios) < n_layers:
+        missing.append(
+            f"compress_ratios has {len(ratios)} entries; need at least {n_layers} for {n_layers} layers"
+        )
+    elif any(isinstance(r, bool) or not isinstance(r, int) or r < 0 for r in ratios):
+        missing.append("compress_ratios must contain non-negative integers")
 
     # Quantisation: a *declared* method must be one the roofline can price. No
     # ``quantization_config`` is fine — that is an unquantised (bf16) checkpoint.
     q = cfg.get("quantization_config") or {}
+    if not isinstance(q, dict):
+        missing.append("quantization_config must be an object when declared")
+        q = {}
     method = q.get("quant_method")
     if method is not None and str(method).lower() not in KNOWN_DTYPES:
         missing.append(
diff --git a/tests/test_execution_graph_dispatch.py b/tests/test_execution_graph_dispatch.py
index 7047bf9..20930d3 100644
--- a/tests/test_execution_graph_dispatch.py
+++ b/tests/test_execution_graph_dispatch.py
@@ -30,9 +30,20 @@ def _moe_cfg(**over) -> dict:
         "num_attention_heads": 1,
         "num_key_value_heads": 1,
         "head_dim": 64,
+        "qk_rope_head_dim": 16,
+        "q_lora_rank": 16,
+        "o_lora_rank": 16,
+        "o_groups": 1,
+        "vocab_size": 128,
         "n_routed_experts": 4,
+        "n_shared_experts": 0,
         "num_experts_per_tok": 1,
         "moe_intermediate_size": 32,
+        "index_n_heads": 1,
+        "index_head_dim": 16,
+        "index_topk": 32,
+        "sliding_window": 16,
+        "compress_ratios": [0, 4],
         "expert_dtype": "fp4",
         "quantization_config": {"quant_method": "fp8"},
         "torch_dtype": "bfloat16",
@@ -69,6 +80,16 @@ def test_partial_sparse_config_is_refused_not_sent_to_dense_graph():
     assert "experts per token" in resolved.refusal_reason
 
 
+def test_zero_expert_count_is_refused_not_priced_as_dense():
+    resolved = _execution_graph(
+        _engine(_moe_cfg(n_routed_experts=0)), _pctx(), sched=None
+    )
+
+    assert not resolved.ok and resolved.graph is None
+    assert "routed expert count" in resolved.refusal_reason
+    assert "positive" in resolved.refusal_reason
+
+
 def test_missing_live_model_refuses_instead_of_defaulting_to_llama():
     resolved = _execution_graph(None, _pctx(), sched=None)
 
diff --git a/tests/test_moe_roofline.py b/tests/test_moe_roofline.py
index 9343eb4..29c58f6 100644
--- a/tests/test_moe_roofline.py
+++ b/tests/test_moe_roofline.py
@@ -13,7 +13,14 @@
 import pytest
 
 from gitm.planner.graph import predict_graph
-from gitm.planner.roofline import BatchConfig, HardwareSpec, ModelSpec, distinct_experts
+from gitm.planner.roofline import (
+    BatchConfig,
+    HardwareSpec,
+    ModelSpec,
+    ShardingConfig,
+    distinct_experts,
+    roofline,
+)
 
 # Qwen3.6-35B-A3B-FP8 shaped: narrow hidden, many narrow experts, one shared
 # expert, fp8 weights with bf16 activations.
@@ -106,10 +113,41 @@ def test_moe_spec_properties():
     assert MOE.shared_intermediate == 768  # falls back to the routed width
 
 
-def test_num_experts_without_top_k_stays_dense():
-    """Half-configured is dense, not a mixture — no silent guessing."""
-    assert not ModelSpec(num_experts=256).is_moe
-    assert not ModelSpec(experts_per_token=8).is_moe
+@pytest.mark.parametrize(
+    "kwargs",
+    [
+        {"num_experts": 256},
+        {"experts_per_token": 8},
+        {"num_experts": 4, "experts_per_token": 8},
+        {"weight_dtype_bytes": 0},
+        {"moe_layer_step": 0},
+        {"full_attn_layer_step": 0},
+    ],
+)
+def test_model_spec_refuses_values_that_were_silently_clamped_or_defaulted(kwargs):
+    with pytest.raises(ValueError):
+        ModelSpec(**kwargs)
+
+
+@pytest.mark.parametrize(
+    "factory",
+    [
+        lambda: BatchConfig(batch=0),
+        lambda: BatchConfig(speculative_tokens=-1),
+        lambda: BatchConfig(acceptance_rate=1.1),
+        lambda: ShardingConfig(tp=0),
+        lambda: ShardingConfig(ep_imbalance=0.5),
+    ],
+)
+def test_decode_and_sharding_specs_refuse_invalid_fallback_inputs(factory):
+    with pytest.raises(ValueError):
+        factory()
+
+
+@pytest.mark.parametrize("flops, bytes_moved", [(-1.0, 1.0), (1.0, -1.0), (float("nan"), 1.0)])
+def test_roofline_refuses_invalid_work_terms(flops, bytes_moved):
+    with pytest.raises(ValueError, match="finite and non-negative"):
+        roofline("bad", flops, bytes_moved, H100)
 
 
 # --- graph: dense back-compat -------------------------------------------------
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index e1f20fb..80e229f 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -252,9 +252,19 @@ def _priceable_moe_engine():
         num_attention_heads=1,
         num_key_value_heads=1,
         head_dim=64,
+        qk_rope_head_dim=16,
+        q_lora_rank=16,
+        o_lora_rank=16,
+        o_groups=1,
         n_routed_experts=4,
+        n_shared_experts=0,
         num_experts_per_tok=1,
         moe_intermediate_size=32,
+        index_n_heads=1,
+        index_head_dim=16,
+        index_topk=32,
+        sliding_window=16,
+        compress_ratios=[0, 4],
         expert_dtype="fp4",
         quantization_config={"quant_method": "fp8"},
         torch_dtype="bfloat16",
diff --git a/tests/test_serve_model_config.py b/tests/test_serve_model_config.py
index 7bc99d8..983be5f 100644
--- a/tests/test_serve_model_config.py
+++ b/tests/test_serve_model_config.py
@@ -29,9 +29,23 @@ def _deepseek_cfg(**over) -> dict:
         "model_type": "deepseek_v4",
         "num_hidden_layers": 4,
         "hidden_size": 4096,
+        "num_attention_heads": 64,
+        "num_key_value_heads": 1,
+        "head_dim": 512,
+        "qk_rope_head_dim": 64,
+        "q_lora_rank": 1024,
+        "o_lora_rank": 1024,
+        "o_groups": 8,
+        "vocab_size": 129280,
         "n_routed_experts": 256,
+        "n_shared_experts": 1,
         "num_experts_per_tok": 6,
         "moe_intermediate_size": 2048,
+        "index_n_heads": 64,
+        "index_head_dim": 128,
+        "index_topk": 512,
+        "sliding_window": 128,
+        "compress_ratios": [0, 0, 4, 128],
         "quantization_config": {"quant_method": "fp8"},
         "expert_dtype": "fp4",
         "torch_dtype": "bfloat16",
@@ -96,14 +110,16 @@ def test_sparse_candidate_with_partial_expert_shape_is_not_misread_as_dense():
     assert not mc.is_sparse_moe_config({"model_type": "llama"})
 
 
-def test_mixtral_aliases_are_recognized_not_rejected():
+def test_mixtral_aliases_are_recognized_but_unsupported_shape_is_refused():
     mixtral = {
         "model_type": "mixtral",
         "num_local_experts": 8,
         "num_experts_per_tok": 2,
         "intermediate_size": 14336,
     }
-    assert mc.validate_moe_config(mixtral) == []
+    missing = mc.validate_moe_config(mixtral)
+    assert any("compress_ratios" in item for item in missing)
+    assert any("hidden_size" in item for item in missing)
     norm = mc.normalize_moe_config(mixtral)
     assert norm["n_routed_experts"] == 8
     assert norm["moe_intermediate_size"] == 14336
@@ -117,6 +133,16 @@ def test_missing_topk_is_refused_and_names_the_key():
     assert any("experts per token" in m for m in missing)
 
 
+def test_nonpositive_expert_shape_is_refused_instead_of_priced_as_dense():
+    missing = mc.validate_moe_config(_deepseek_cfg(n_routed_experts=0))
+    assert any("routed expert count" in item and "positive" in item for item in missing)
+
+
+def test_short_compression_schedule_is_refused_instead_of_reusing_last_ratio():
+    missing = mc.validate_moe_config(_deepseek_cfg(compress_ratios=[0, 4]))
+    assert any("compress_ratios" in item and "4 layers" in item for item in missing)
+
+
 def test_unpriceable_quant_method_is_refused_not_defaulted():
     cfg = _deepseek_cfg(quantization_config={"quant_method": "awq"})
     missing = mc.validate_moe_config(cfg)

From d07e7dbc6cedfaf217b7c8d9adea805dab7461ad Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:17:28 -0700
Subject: [PATCH 34/70] fix: surface incomplete injected traces

---
 gitm/tracer/injection.py | 60 ++++++++++++++++++++++++++++++++++++++--
 tests/test_injection.py  | 13 ++++++++-
 2 files changed, 69 insertions(+), 4 deletions(-)

diff --git a/gitm/tracer/injection.py b/gitm/tracer/injection.py
index 5610c7b..016837d 100644
--- a/gitm/tracer/injection.py
+++ b/gitm/tracer/injection.py
@@ -24,8 +24,10 @@
 from __future__ import annotations
 
 import json
+import math
 import os
 import time
+import warnings
 from pathlib import Path
 
 from gitm.tracer.schema import TraceEvent
@@ -110,6 +112,22 @@ def _pid_alive(pid: int) -> bool:
     prevent. Guessing "dead" costs a silently empty trace; guessing "alive" costs a
     stale file.
     """
+    if os.name == "nt":
+        # ``os.kill(pid, 0)`` is not a harmless existence probe on Windows: the
+        # CRT maps signal 0 through TerminateProcess, which can kill the very
+        # EngineCore whose live shard we are trying to protect. Query a process
+        # handle instead; access-denied still means the process exists.
+        import ctypes
+
+        kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
+        handle = kernel32.OpenProcess(0x1000, False, pid)  # PROCESS_QUERY_LIMITED_INFORMATION
+        if handle:
+            kernel32.CloseHandle(handle)
+            return True
+        error = ctypes.get_last_error()
+        if error == 87:  # ERROR_INVALID_PARAMETER: no such PID
+            return False
+        return True
     try:
         os.kill(pid, 0)
     except ProcessLookupError:
@@ -146,10 +164,21 @@ def clear_stale_shards() -> None:
 
 def settle_seconds() -> float:
     raw = os.environ.get(ENV_SETTLE)
+    if not raw:
+        return DEFAULT_SETTLE_S
     try:
-        return float(raw) if raw else DEFAULT_SETTLE_S
+        value = float(raw)
     except ValueError:
+        value = float("nan")
+    if not math.isfinite(value) or value < 0:
+        warnings.warn(
+            f"invalid GITM_TRACE_SETTLE_S={raw!r}; using documented default "
+            f"{DEFAULT_SETTLE_S}s",
+            RuntimeWarning,
+            stacklevel=2,
+        )
         return DEFAULT_SETTLE_S
+    return value
 
 
 def settle() -> None:
@@ -171,10 +200,17 @@ def read_shards(start_ns: int | None = None, end_ns: int | None = None) -> list[
     from gitm.tracer._cupti_decode import decode_records
 
     records: list[dict] = []
+    dropped_lines = 0
     for shard in shard_paths():
         try:
             text = shard.read_text(encoding="utf-8", errors="replace")
-        except OSError:
+        except OSError as exc:
+            warnings.warn(
+                f"injected trace shard unreadable ({shard}: {type(exc).__name__}: {exc}); "
+                "capture coverage is incomplete",
+                RuntimeWarning,
+                stacklevel=2,
+            )
             continue
         for line in text.splitlines():
             line = line.strip()
@@ -183,11 +219,14 @@ def read_shards(start_ns: int | None = None, end_ns: int | None = None) -> list[
             try:
                 rec = json.loads(line)
             except json.JSONDecodeError:
+                dropped_lines += 1
                 continue  # partial line from a killed process
             if not isinstance(rec, dict):
+                dropped_lines += 1
                 continue
             ts = rec.get("start_ns")
             if not isinstance(ts, int):
+                dropped_lines += 1
                 continue
             if start_ns is not None and ts < start_ns:
                 continue
@@ -195,7 +234,22 @@ def read_shards(start_ns: int | None = None, end_ns: int | None = None) -> list[
                 continue
             records.append(rec)
 
-    return decode_records(records)
+    if dropped_lines:
+        warnings.warn(
+            f"injected trace coverage: dropped {dropped_lines} malformed or incomplete "
+            "shard line(s)",
+            RuntimeWarning,
+            stacklevel=2,
+        )
+    events = decode_records(records)
+    if len(events) < len(records):
+        warnings.warn(
+            f"injected trace coverage: dropped {len(records) - len(events)} unmodeled "
+            "activity record(s)",
+            RuntimeWarning,
+            stacklevel=2,
+        )
+    return events
 
 
 def cupti_now() -> int | None:
diff --git a/tests/test_injection.py b/tests/test_injection.py
index d7d9f31..9ff31d8 100644
--- a/tests/test_injection.py
+++ b/tests/test_injection.py
@@ -93,11 +93,22 @@ def test_partial_trailing_line_from_a_killed_process_is_tolerated(run_env):
     shard = run_env.with_name(run_env.name + ".9335")
     shard.write_text(_kernel("decode_step", 10, 20) + "\n" + '{"kind":"kernel","na')
 
-    events = injection.read_shards()
+    with pytest.warns(RuntimeWarning, match="dropped 1 malformed or incomplete"):
+        events = injection.read_shards()
 
     assert [e.name for e in events] == ["decode_step"]
 
 
+@pytest.mark.parametrize("raw", ["not-a-number", "-1", "nan", "inf"])
+def test_invalid_settle_override_warns_and_uses_documented_default(monkeypatch, raw):
+    monkeypatch.setenv(injection.ENV_SETTLE, raw)
+
+    with pytest.warns(RuntimeWarning, match="invalid GITM_TRACE_SETTLE_S"):
+        value = injection.settle_seconds()
+
+    assert value == injection.DEFAULT_SETTLE_S
+
+
 def test_arm_marker_is_not_mistaken_for_a_shard(run_env):
     injection.arm()
     run_env.with_name(run_env.name + ".9335").write_text(_kernel("decode_step", 10, 20) + "\n")

From e1634f41f26c15dcf550dfcae8e58909f70aa928 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:26:30 -0700
Subject: [PATCH 35/70] fix: refuse invalid public runtime inputs

---
 gitm/cli.py                      | 11 +++--
 gitm/optimizer/apply.py          |  2 +
 gitm/optimizer/qualification.py  |  5 ++
 gitm/planner/moe_graph.py        | 82 ++++++++++++++++++++++++++++++++
 gitm/scheduler/loop.py           |  5 +-
 gitm/serve/model_config.py       |  6 ++-
 tests/test_apply_rollback.py     |  8 ++++
 tests/test_moe_graph.py          | 19 ++++++++
 tests/test_run_loop_workload.py  | 54 +++++++++++++++++++++
 tests/test_serve_attach.py       | 22 +++++++++
 tests/test_serve_model_config.py |  8 ++--
 11 files changed, 212 insertions(+), 10 deletions(-)

diff --git a/gitm/cli.py b/gitm/cli.py
index 8ddc66f..fa5e43a 100644
--- a/gitm/cli.py
+++ b/gitm/cli.py
@@ -4,6 +4,7 @@
 
 import argparse
 import json
+import math
 import sys
 from pathlib import Path
 
@@ -214,8 +215,12 @@ def _parser() -> argparse.ArgumentParser:
 def _parse_target(s: str) -> float:
     s = s.strip()
     if s.endswith("%"):
-        return float(s[:-1]) / 100.0
-    return float(s)
+        target = float(s[:-1]) / 100.0
+    else:
+        target = float(s)
+    if not math.isfinite(target) or not 0.0 < target <= 1.0:
+        raise ValueError(f"target floor must be finite and in (0, 1], got {s!r}")
+    return target
 
 
 _HFT_WORKLOADS = {"hft", "hft-lob"}
@@ -352,7 +357,7 @@ def main(argv: list[str] | None = None) -> int:
             args.intervention, config=args.config, min_keep_delta=args.min_keep_delta
         )
         print(json.dumps(result, indent=2))
-        return 0
+        return 0 if result.get("applied") and not result.get("rolled_back") else 3
 
     if args.cmd == "attach":
         from gitm.deploy import attach_job
diff --git a/gitm/optimizer/apply.py b/gitm/optimizer/apply.py
index cbc516e..2e5d9c1 100644
--- a/gitm/optimizer/apply.py
+++ b/gitm/optimizer/apply.py
@@ -76,6 +76,8 @@ def apply_intervention(
     blocks the apply). Pass one only where the applicator mutates a real target;
     a dry-run leaves it ``None`` so the trail stays free of no-op entries.
     """
+    if not math.isfinite(min_keep_delta):
+        raise ValueError(f"min_keep_delta must be finite, got {min_keep_delta!r}")
     try:
         snapshot = applicator.snapshot()
     except Exception as exc:
diff --git a/gitm/optimizer/qualification.py b/gitm/optimizer/qualification.py
index cd79de5..4c97b10 100644
--- a/gitm/optimizer/qualification.py
+++ b/gitm/optimizer/qualification.py
@@ -8,6 +8,7 @@
 from __future__ import annotations
 
 import hashlib
+import math
 from dataclasses import dataclass
 
 from gitm.tracer.schema import Trace
@@ -57,6 +58,10 @@ def qualify(trace: Trace, target_floor: float = 0.15) -> QualificationResult:
     Imported profiler traces (``nsys-import`` / ``torch-import``) never commit:
     the 15% floor and refund clause require a gitm-captured run.
     """
+    if not math.isfinite(target_floor) or not 0.0 < target_floor <= 1.0:
+        raise ValueError(
+            f"target floor must be finite and in (0, 1], got {target_floor!r}"
+        )
     fp = fingerprint(trace)
     source = getattr(trace, "source", "cupti") or "cupti"
     if source in _IMPORT_SOURCES:
diff --git a/gitm/planner/moe_graph.py b/gitm/planner/moe_graph.py
index 5ffb219..5fcf8dd 100644
--- a/gitm/planner/moe_graph.py
+++ b/gitm/planner/moe_graph.py
@@ -75,6 +75,84 @@
     weight_bytes_is_fallback,
 )
 
+_REQUIRED_POSITIVE_CONFIG_FIELDS = (
+    "hidden_size",
+    "num_hidden_layers",
+    "num_attention_heads",
+    "num_key_value_heads",
+    "head_dim",
+    "q_lora_rank",
+    "o_lora_rank",
+    "o_groups",
+    "vocab_size",
+    "n_routed_experts",
+    "num_experts_per_tok",
+    "moe_intermediate_size",
+    "index_n_heads",
+    "index_head_dim",
+    "index_topk",
+    "sliding_window",
+)
+
+
+def validate_sparse_moe_config(cfg: dict[str, Any]) -> list[str]:
+    """Return canonical sparse-graph fields that cannot be used without guessing.
+
+    Serve and scheduler boundaries provide richer source/alias diagnostics. This
+    planner-side defense ensures a direct public-builder call cannot turn a
+    partial foreign config into a plausible DeepSeek-V4 graph from defaults.
+    """
+    errors: list[str] = []
+    for key in _REQUIRED_POSITIVE_CONFIG_FIELDS:
+        value = cfg.get(key)
+        if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
+            errors.append(f"{key} must be a declared positive integer, got {value!r}")
+
+    for key in ("qk_rope_head_dim", "n_shared_experts"):
+        value = cfg.get(key)
+        if isinstance(value, bool) or not isinstance(value, int) or value < 0:
+            errors.append(f"{key} must be a declared non-negative integer, got {value!r}")
+
+    n_experts = cfg.get("n_routed_experts")
+    top_k = cfg.get("num_experts_per_tok")
+    if (
+        isinstance(n_experts, int)
+        and not isinstance(n_experts, bool)
+        and isinstance(top_k, int)
+        and not isinstance(top_k, bool)
+        and top_k > n_experts
+    ):
+        errors.append(
+            f"num_experts_per_tok={top_k} exceeds n_routed_experts={n_experts}"
+        )
+
+    n_layers = cfg.get("num_hidden_layers")
+    ratios = cfg.get("compress_ratios")
+    if not isinstance(ratios, (list, tuple)):
+        errors.append("compress_ratios must be declared for the sparse-attention graph")
+    elif isinstance(n_layers, int) and n_layers > 0 and len(ratios) < n_layers:
+        errors.append(
+            f"compress_ratios has {len(ratios)} entries; need at least {n_layers}"
+        )
+    elif any(isinstance(r, bool) or not isinstance(r, int) or r < 0 for r in ratios):
+        errors.append("compress_ratios must contain non-negative integers")
+
+    q = cfg.get("quantization_config")
+    if q is not None and not isinstance(q, dict):
+        errors.append("quantization_config must be an object when declared")
+    elif isinstance(q, dict):
+        method = q.get("quant_method")
+        if method is not None and weight_bytes_is_fallback(str(method).lower()):
+            errors.append(f"quantization_config.quant_method={method!r} is not priceable")
+
+    for key in ("expert_dtype", "torch_dtype"):
+        value = cfg.get(key)
+        if value is None:
+            errors.append(f"{key} must be declared; byte width cannot be guessed")
+        elif weight_bytes_is_fallback(str(value).lower().replace("bfloat16", "bf16")):
+            errors.append(f"{key}={value!r} is not priceable")
+    return errors
+
 
 def effective_kv_tokens(spec: SparseMoEModelSpec, layer: int, kv_len: int) -> int:
     """KV positions the attention *core* reads for ``layer`` at ``kv_len`` context.
@@ -549,6 +627,10 @@ def spec_from_hf_config(cfg: dict[str, Any], *, name: str | None = None) -> Spar
     ``num_hidden_layers`` entries index real layers, so the tail is dropped
     rather than silently shifting every layer's ratio.
     """
+    errors = validate_sparse_moe_config(cfg)
+    if errors:
+        raise ValueError("sparse-MoE config is not predictable: " + "; ".join(errors))
+
     q = cfg.get("quantization_config") or {}
     weight_dtype = str(q.get("quant_method", "bf16")).lower()
     n_layers = int(cfg.get("num_hidden_layers", 43))
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 466a9b6..308ac9c 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -104,7 +104,10 @@ def _parse_budget_s(budget: str) -> float:
     if not m:
         raise ValueError(f"unparseable budget: {budget!r} (use 24h, 90m, 3600s, 1d)")
     value, unit = float(m.group(1)), m.group(2)
-    return value * {"s": 1.0, "m": 60.0, "h": 3600.0, "d": 86400.0}[unit]
+    seconds = value * {"s": 1.0, "m": 60.0, "h": 3600.0, "d": 86400.0}[unit]
+    if seconds <= 0.0:
+        raise ValueError(f"budget must be positive, got {budget!r}")
+    return seconds
 
 def _engine_throughput_fn(engine: Any, runner: Any) -> Any:
     """Resolve a decode-throughput probe for the live A/B.
diff --git a/gitm/serve/model_config.py b/gitm/serve/model_config.py
index 8a8dd85..2045da3 100644
--- a/gitm/serve/model_config.py
+++ b/gitm/serve/model_config.py
@@ -313,7 +313,11 @@ def positive_alias(keys: tuple[str, ...], label: str) -> int | None:
             f"(not priceable; known: {', '.join(sorted(KNOWN_DTYPES))})"
         )
     expert_dtype = cfg.get("expert_dtype")
-    if expert_dtype is not None and str(expert_dtype).lower() not in KNOWN_DTYPES:
+    if expert_dtype is None:
+        missing.append(
+            "expert_dtype must be declared; routed and shared expert byte width cannot be guessed"
+        )
+    elif str(expert_dtype).lower() not in KNOWN_DTYPES:
         missing.append(f"expert_dtype={expert_dtype!r} (not priceable)")
     return missing
 
diff --git a/tests/test_apply_rollback.py b/tests/test_apply_rollback.py
index 63490ca..a191a0e 100644
--- a/tests/test_apply_rollback.py
+++ b/tests/test_apply_rollback.py
@@ -211,6 +211,14 @@ def test_apply_from_file_with_config(tmp_path):
     assert yaml.safe_load(target.read_text())["block_size"] == 16
 
 
+@pytest.mark.parametrize("threshold", [float("nan"), float("inf"), float("-inf")])
+def test_apply_refuses_nonfinite_keep_threshold(threshold):
+    with pytest.raises(ValueError, match="min_keep_delta"):
+        apply_intervention(
+            _spec(), DictApplicator({"block_size": 8}), min_keep_delta=threshold
+        )
+
+
 # --- library now carries the 21 curated levers ------------------------------
 
 
diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py
index b455139..ccd2fa6 100644
--- a/tests/test_moe_graph.py
+++ b/tests/test_moe_graph.py
@@ -95,6 +95,25 @@ def base_spec():
     return spec_from_hf_config(V4_BASE_CONFIG, name="DeepSeek-V4-Flash")
 
 
+@pytest.mark.parametrize(
+    "missing_key",
+    ["hidden_size", "n_routed_experts", "compress_ratios", "expert_dtype", "torch_dtype"],
+)
+def test_public_hf_builder_refuses_missing_shape_or_precision(missing_key):
+    cfg = dict(V4_CONFIG)
+    cfg.pop(missing_key)
+
+    with pytest.raises(ValueError, match=missing_key):
+        spec_from_hf_config(cfg)
+
+
+def test_public_hf_builder_accepts_complete_declared_config():
+    parsed = spec_from_hf_config(V4_CONFIG)
+
+    assert parsed.hidden == V4_CONFIG["hidden_size"]
+    assert parsed.expert_dtype == V4_CONFIG["expert_dtype"]
+
+
 @pytest.fixture
 def b200():
     return hardware_spec_for(peak_for_sku("NVIDIA B200"))
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index 80e229f..539f51b 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -20,6 +20,22 @@
 EMPTY_DIGEST = "4f53cda18c2baa0c"
 
 
+@pytest.mark.parametrize("budget", ["0s", "0.0h"])
+def test_loop_refuses_nonpositive_budget(budget, tmp_path):
+    from gitm import optimize
+
+    with pytest.raises(ValueError, match="budget must be positive"):
+        optimize(workload="custom", budget=budget, scratch=str(tmp_path))
+
+
+@pytest.mark.parametrize("target", [0.0, -0.1, 1.01, float("nan"), float("inf")])
+def test_loop_refuses_invalid_target_floor(target, tmp_path):
+    from gitm import optimize
+
+    with pytest.raises(ValueError, match="target floor"):
+        optimize(workload="custom", budget="1s", target=target, scratch=str(tmp_path))
+
+
 def test_no_data_guard_does_not_fabricate_claims(tmp_path: Path):
     """No GPU/shim and no registered runner → honest no-data, zero claims."""
     from gitm import optimize
@@ -462,6 +478,44 @@ def test_cli_run_returns_nonzero_on_no_data(tmp_path: Path, capsys):
     assert rc == 3
 
 
+@pytest.mark.parametrize("target", ["nan", "inf", "0%", "101%"])
+def test_cli_run_rejects_invalid_target_before_optimization(target, monkeypatch):
+    import gitm
+    from gitm.cli import main
+
+    called = False
+
+    def fake_optimize(**_kwargs):
+        nonlocal called
+        called = True
+        return {}
+
+    monkeypatch.setattr(gitm, "optimize", fake_optimize)
+    with pytest.raises(ValueError, match="target floor"):
+        main(["run", "--workload", "vllm-decode", "--target", target])
+    assert called is False
+
+
+def test_cli_apply_returns_nonzero_when_nothing_was_applied(tmp_path, capsys):
+    import json
+
+    import yaml
+
+    from gitm.cli import main
+    from gitm.kernels.spec import InterventionSpec
+
+    spec = InterventionSpec(
+        name="noop", summary="test", knob="block_size", value=16,
+        expected_delta_lo=0.0, expected_delta_mean=0.1, expected_delta_hi=0.2,
+        source="https://example.test",
+    )
+    path = tmp_path / "intervention.yaml"
+    path.write_text(yaml.safe_dump(json.loads(spec.model_dump_json())), encoding="utf-8")
+
+    assert main(["apply", "--intervention", str(path)]) == 3
+    assert "no target config" in capsys.readouterr().out
+
+
 @pytest.mark.parametrize(
     "status",
     ["prediction_refused", "candidate_coverage_unavailable", "intervention_failed"],
diff --git a/tests/test_serve_attach.py b/tests/test_serve_attach.py
index da92996..29f95c9 100644
--- a/tests/test_serve_attach.py
+++ b/tests/test_serve_attach.py
@@ -269,9 +269,20 @@ def test_predicted_graph_surfaces_resolved_warnings_and_bytes_fallback(
         "num_attention_heads": 1,
         "num_key_value_heads": 1,
         "head_dim": 64,
+        "qk_rope_head_dim": 0,
+        "q_lora_rank": 64,
+        "o_lora_rank": 64,
+        "o_groups": 1,
+        "vocab_size": 128,
         "n_routed_experts": 2,
+        "n_shared_experts": 0,
         "num_experts_per_tok": 1,
         "moe_intermediate_size": 32,
+        "index_n_heads": 1,
+        "index_head_dim": 8,
+        "index_topk": 8,
+        "sliding_window": 128,
+        "compress_ratios": [0],
         "expert_dtype": "fp4",
         "quantization_config": {"quant_method": "fp8"},
         "torch_dtype": "bfloat16",
@@ -328,9 +339,20 @@ def test_predicted_graph_known_dtypes_leave_bytes_fallback_clean(tmp_path, monke
         "num_attention_heads": 1,
         "num_key_value_heads": 1,
         "head_dim": 64,
+        "qk_rope_head_dim": 0,
+        "q_lora_rank": 64,
+        "o_lora_rank": 64,
+        "o_groups": 1,
+        "vocab_size": 128,
         "n_routed_experts": 2,
+        "n_shared_experts": 0,
         "num_experts_per_tok": 1,
         "moe_intermediate_size": 32,
+        "index_n_heads": 1,
+        "index_head_dim": 8,
+        "index_topk": 8,
+        "sliding_window": 128,
+        "compress_ratios": [0],
         "expert_dtype": "fp4",
         "quantization_config": {"quant_method": "fp8"},
         "torch_dtype": "bfloat16",
diff --git a/tests/test_serve_model_config.py b/tests/test_serve_model_config.py
index 983be5f..c1079a9 100644
--- a/tests/test_serve_model_config.py
+++ b/tests/test_serve_model_config.py
@@ -227,7 +227,7 @@ def test_unpriceable_command_line_dtype_is_refused_after_overrides(tmp_path):
     assert any("kv_dtype='future_kv3'" in item for item in r.missing_keys)
 
 
-def test_accepted_default_substitutions_are_named_on_live_spec(tmp_path):
+def test_missing_expert_dtype_refuses_dominant_term_prediction(tmp_path):
     ckpt = tmp_path / "ckpt"
     cfg = _deepseek_cfg()
     cfg.pop("expert_dtype")
@@ -235,10 +235,8 @@ def test_accepted_default_substitutions_are_named_on_live_spec(tmp_path):
 
     r = mc.live_moe_spec(_target(["vllm", "serve", str(ckpt)]), environ={})
 
-    assert isinstance(r, mc.LiveSpec)
-    assert any("expert_dtype absent" in warning for warning in r.warnings)
-    assert any("batch=1" in warning for warning in r.warnings)
-    assert any("kv_cache_len=4096" in warning for warning in r.warnings)
+    assert isinstance(r, mc.LiveSpecError)
+    assert any("expert_dtype must be declared" in key for key in r.missing_keys)
 
 
 def test_live_moe_spec_refuses_when_no_config_found(tmp_path):

From e331073b9c85056a324d78d58f5d17f84faf33a0 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:30:10 -0700
Subject: [PATCH 36/70] fix: surface telemetry discovery failures

---
 gitm/doctor.py                         | 25 +++++--
 gitm/optimizer/headroom_kernel_rank.py | 44 ++++++++----
 gitm/telemetry/backends/discover.py    | 61 +++++++++++------
 gitm/telemetry/collector.py            | 11 ++-
 tests/test_telemetry_fallbacks.py      | 93 ++++++++++++++++++++++++++
 5 files changed, 194 insertions(+), 40 deletions(-)

diff --git a/gitm/doctor.py b/gitm/doctor.py
index ad25a0e..38262fb 100644
--- a/gitm/doctor.py
+++ b/gitm/doctor.py
@@ -22,8 +22,25 @@ def doctor() -> dict[str, Any]:
 
     from gitm.telemetry.backends import discover_backends
 
-    backends = discover_backends()
-    info["telemetry_backends"] = [
-        {"vendor": b.vendor, "device_count": b.device_count()} for b in backends
-    ]
+    diagnostics: list[str] = []
+    backends = discover_backends(diagnostics=diagnostics)
+    telemetry_backends: list[dict[str, Any]] = []
+    for backend in backends:
+        try:
+            telemetry_backends.append(
+                {"vendor": backend.vendor, "device_count": backend.device_count()}
+            )
+        except Exception as exc:
+            diagnostics.append(
+                f"{backend.vendor} telemetry probe failed: {type(exc).__name__}: {exc}"
+            )
+        finally:
+            try:
+                backend.close()
+            except Exception as exc:
+                diagnostics.append(
+                    f"{backend.vendor} telemetry close failed: {type(exc).__name__}: {exc}"
+                )
+    info["telemetry_backends"] = telemetry_backends
+    info["telemetry_diagnostics"] = diagnostics
     return info
diff --git a/gitm/optimizer/headroom_kernel_rank.py b/gitm/optimizer/headroom_kernel_rank.py
index 97f887a..c82d993 100644
--- a/gitm/optimizer/headroom_kernel_rank.py
+++ b/gitm/optimizer/headroom_kernel_rank.py
@@ -187,7 +187,12 @@ def live_gpu_headroom():
     """One-shot live snapshot via the telemetry backend (no running workload required)."""
     from gitm.telemetry.backends import discover_backends
 
-    backends = discover_backends()
+    discovery_diagnostics: list[str] = []
+    backends = discover_backends(diagnostics=discovery_diagnostics)
+    for diagnostic in discovery_diagnostics:
+        warnings.warn(
+            f"live GPU headroom degraded: {diagnostic}", RuntimeWarning, stacklevel=2
+        )
     if not backends:
         warnings.warn(
             "live GPU headroom unavailable: no telemetry backend found",
@@ -196,21 +201,32 @@ def live_gpu_headroom():
         )
     out = []
     for b in backends:
-        for idx in range(b.device_count()):
-            s = b.sample(idx)
-            for diagnostic in s.diagnostics:
+        try:
+            for idx in range(b.device_count()):
+                s = b.sample(idx)
+                for diagnostic in s.diagnostics:
+                    warnings.warn(
+                        f"live GPU headroom degraded: {diagnostic}",
+                        RuntimeWarning,
+                        stacklevel=2,
+                    )
+                out.append({
+                    "gpu_index": idx,
+                    "util_pct": s.util_pct,
+                    "mem_used_bytes": s.mem_used_bytes,
+                    "mem_total_bytes": s.mem_total_bytes,
+                    "power_w": s.power_w,
+                    "sm_clock_mhz": s.sm_clock_mhz,
+                    "throttle": s.throttle_reasons.name if s.throttle_reasons else "NONE",
+                })
+        finally:
+            try:
+                b.close()
+            except Exception as exc:
                 warnings.warn(
-                    f"live GPU headroom degraded: {diagnostic}",
+                    f"live GPU headroom degraded: backend close failed "
+                    f"({type(exc).__name__}: {exc})",
                     RuntimeWarning,
                     stacklevel=2,
                 )
-            out.append({
-                "gpu_index": idx,
-                "util_pct": s.util_pct,
-                "mem_used_bytes": s.mem_used_bytes,
-                "mem_total_bytes": s.mem_total_bytes,
-                "power_w": s.power_w,
-                "sm_clock_mhz": s.sm_clock_mhz,
-                "throttle": s.throttle_reasons.name if s.throttle_reasons else "NONE",
-            })
     return out
diff --git a/gitm/telemetry/backends/discover.py b/gitm/telemetry/backends/discover.py
index d5c8ab5..eecb8af 100644
--- a/gitm/telemetry/backends/discover.py
+++ b/gitm/telemetry/backends/discover.py
@@ -6,38 +6,59 @@
 
 from __future__ import annotations
 
+import warnings
+
 from gitm.telemetry.backends.base import Backend
 
 
-def discover_backends() -> list[Backend]:
+def discover_backends(*, diagnostics: list[str] | None = None) -> list[Backend]:
     """Return all live vendor backends in discovery order.
 
-    Each candidate is constructed in a try/except — if its vendor library is
-    missing or no devices are present, it is silently skipped. The result is
-    deterministic given the host.
+    A missing optional vendor library or a valid zero-device backend is expected
+    and stays quiet. Unexpected import, initialization, count, or cleanup failures
+    are appended to ``diagnostics``; direct callers that do not supply a list get
+    a runtime warning instead.
     """
     found: list[Backend] = []
 
-    try:
+    def record(vendor: str, detail: str) -> None:
+        message = f"{vendor} telemetry discovery failed: {detail}"
+        if diagnostics is not None:
+            diagnostics.append(message)
+        else:
+            warnings.warn(message, RuntimeWarning, stacklevel=3)
+
+    def attempt(vendor: str, factory) -> None:
+        backend: Backend | None = None
+        try:
+            backend = factory()
+            count = backend.device_count()
+            if isinstance(count, bool) or not isinstance(count, int) or count < 0:
+                raise ValueError(f"device_count returned invalid value {count!r}")
+            if count > 0:
+                found.append(backend)
+                backend = None
+        except ImportError:
+            return
+        except Exception as exc:
+            record(vendor, f"{type(exc).__name__}: {exc}")
+        finally:
+            if backend is not None:
+                try:
+                    backend.close()
+                except Exception as exc:
+                    record(vendor, f"backend close failed ({type(exc).__name__}: {exc})")
+
+    def nvidia():
         from gitm.telemetry.backends.nvidia import NvidiaBackend
 
-        nv = NvidiaBackend()
-        if nv.device_count() > 0:
-            found.append(nv)
-        else:
-            nv.close()
-    except Exception:
-        pass
+        return NvidiaBackend()
 
-    try:
+    def amd():
         from gitm.telemetry.backends.amd import AmdBackend
 
-        amd = AmdBackend()
-        if amd.device_count() > 0:
-            found.append(amd)
-        else:
-            amd.close()
-    except Exception:
-        pass
+        return AmdBackend()
 
+    attempt("nvidia", nvidia)
+    attempt("amd", amd)
     return found
diff --git a/gitm/telemetry/collector.py b/gitm/telemetry/collector.py
index 4955eac..72460ec 100644
--- a/gitm/telemetry/collector.py
+++ b/gitm/telemetry/collector.py
@@ -34,10 +34,17 @@ class Collector:
 
     def __init__(self, cfg: CollectorConfig) -> None:
         self._cfg = cfg
-        self._backends: list[Backend] = cfg.backends if cfg.backends is not None else discover_backends()
         self.diagnostics: list[str] = []
         self._diagnostic_keys: set[str] = set()
-        if not self._backends:
+        discovery_diagnostics: list[str] = []
+        self._backends: list[Backend] = (
+            cfg.backends
+            if cfg.backends is not None
+            else discover_backends(diagnostics=discovery_diagnostics)
+        )
+        for i, diagnostic in enumerate(discovery_diagnostics):
+            self._record_failure(f"backend-discovery:{i}", diagnostic)
+        if not self._backends and not discovery_diagnostics:
             self._record_failure("backend-discovery", "no live GPU telemetry backend found")
         self._stop = threading.Event()
         self._thread: threading.Thread | None = None
diff --git a/tests/test_telemetry_fallbacks.py b/tests/test_telemetry_fallbacks.py
index a7dbe7a..0af8381 100644
--- a/tests/test_telemetry_fallbacks.py
+++ b/tests/test_telemetry_fallbacks.py
@@ -9,6 +9,99 @@
 from gitm.telemetry.schema import Sample
 
 
+def test_discovery_surfaces_unexpected_vendor_failure_and_closes_partial_backend(monkeypatch):
+    import gitm.telemetry.backends.amd as amd
+    import gitm.telemetry.backends.nvidia as nvidia
+    from gitm.telemetry.backends.discover import discover_backends
+
+    closed = []
+
+    class BrokenCount:
+        def device_count(self):
+            raise RuntimeError("NVML count failed")
+
+        def close(self):
+            closed.append(True)
+
+    monkeypatch.setattr(nvidia, "NvidiaBackend", BrokenCount)
+    monkeypatch.setattr(
+        amd, "AmdBackend", lambda: (_ for _ in ()).throw(ImportError("optional ROCm absent"))
+    )
+    diagnostics = []
+
+    assert discover_backends(diagnostics=diagnostics) == []
+    assert closed == [True]
+    assert any("nvidia" in d and "NVML count failed" in d for d in diagnostics)
+    assert not any("optional ROCm absent" in d for d in diagnostics)
+
+
+def test_collector_carries_discovery_diagnostics(monkeypatch):
+    monkeypatch.setattr(
+        "gitm.telemetry.collector.discover_backends",
+        lambda *, diagnostics: diagnostics.append("nvidia discovery failed: denied") or [],
+    )
+
+    with pytest.warns(RuntimeWarning, match="nvidia discovery failed"):
+        collector = Collector(CollectorConfig())
+
+    assert any("nvidia discovery failed" in d for d in collector.diagnostics)
+
+
+def test_doctor_reports_discovery_diagnostics_and_closes_backend(monkeypatch):
+    from gitm.doctor import doctor
+
+    closed = []
+
+    class Backend:
+        vendor = "nvidia"
+
+        def device_count(self):
+            return 1
+
+        def close(self):
+            closed.append(True)
+
+    def discover(*, diagnostics):
+        diagnostics.append("amd discovery failed: broken runtime")
+        return [Backend()]
+
+    monkeypatch.setattr("gitm.telemetry.backends.discover_backends", discover)
+    report = doctor()
+
+    assert report["telemetry_diagnostics"] == ["amd discovery failed: broken runtime"]
+    assert report["telemetry_backends"] == [{"vendor": "nvidia", "device_count": 1}]
+    assert closed == [True]
+
+
+def test_live_headroom_warns_discovery_diagnostic_and_closes_backend(monkeypatch):
+    from gitm.optimizer.headroom_kernel_rank import live_gpu_headroom
+
+    closed = []
+
+    class Backend:
+        def device_count(self):
+            return 1
+
+        def sample(self, _index):
+            return Sample(
+                ts_ns=1, node="n", gpu_uuid="g", gpu_index=0, vendor="nvidia"
+            )
+
+        def close(self):
+            closed.append(True)
+
+    def discover(*, diagnostics):
+        diagnostics.append("amd discovery failed: broken runtime")
+        return [Backend()]
+
+    monkeypatch.setattr("gitm.telemetry.backends.discover_backends", discover)
+    with pytest.warns(RuntimeWarning, match="amd discovery failed"):
+        rows = live_gpu_headroom()
+
+    assert len(rows) == 1
+    assert closed == [True]
+
+
 class _BrokenBackend:
     def device_count(self):
         return 1

From 6a9128c9054a13a8971c9ff40a9a05a886a4de49 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:32:54 -0700
Subject: [PATCH 37/70] fix: refuse invalid attach windows

---
 gitm/serve/attach.py       | 22 ++++++++++++++++++++++
 tests/test_serve_attach.py | 24 ++++++++++++++++++++++++
 2 files changed, 46 insertions(+)

diff --git a/gitm/serve/attach.py b/gitm/serve/attach.py
index 9c33136..311730d 100644
--- a/gitm/serve/attach.py
+++ b/gitm/serve/attach.py
@@ -28,6 +28,7 @@
 from __future__ import annotations
 
 import json
+import math
 import os
 import time
 import urllib.parse
@@ -69,6 +70,27 @@ class AttachOptions:
     dry_run: bool = False
     proc: Path = discover.PROC
 
+    def __post_init__(self) -> None:
+        for name in ("duration_s", "request_timeout", "metrics_interval"):
+            value = getattr(self, name)
+            if not isinstance(value, int | float) or isinstance(value, bool):
+                raise ValueError(f"{name} must be a finite positive number, got {value!r}")
+            if not math.isfinite(float(value)) or value <= 0:
+                raise ValueError(f"{name} must be a finite positive number, got {value!r}")
+        if isinstance(self.requests, bool) or not isinstance(self.requests, int) or self.requests < 0:
+            raise ValueError(f"requests must be a non-negative integer, got {self.requests!r}")
+        for name in ("concurrency", "input_tokens", "output_tokens"):
+            value = getattr(self, name)
+            if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
+                raise ValueError(f"{name} must be a positive integer, got {value!r}")
+        for name, value in (("pid", self.pid), ("port", self.port)):
+            if value is not None and (
+                isinstance(value, bool) or not isinstance(value, int) or value <= 0
+            ):
+                raise ValueError(f"{name} must be a positive integer, got {value!r}")
+        if self.port is not None and self.port > 65535:
+            raise ValueError(f"port must be at most 65535, got {self.port!r}")
+
     @property
     def mode(self) -> str:
         return "drive" if self.requests > 0 else "observe"
diff --git a/tests/test_serve_attach.py b/tests/test_serve_attach.py
index 29f95c9..d3d243e 100644
--- a/tests/test_serve_attach.py
+++ b/tests/test_serve_attach.py
@@ -22,6 +22,30 @@
 from gitm.serve import discover
 from gitm.tracer import injection
 
+
+@pytest.mark.parametrize(
+    ("field", "value", "message"),
+    [
+        ("duration_s", 0.0, "duration_s"),
+        ("duration_s", float("nan"), "duration_s"),
+        ("requests", -1, "requests"),
+        ("concurrency", 0, "concurrency"),
+        ("input_tokens", 0, "input_tokens"),
+        ("output_tokens", -1, "output_tokens"),
+        ("request_timeout", float("inf"), "request_timeout"),
+        ("metrics_interval", 0.0, "metrics_interval"),
+        ("port", 70000, "port"),
+    ],
+)
+def test_attach_options_refuse_invalid_window_inputs(field, value, message):
+    with pytest.raises(ValueError, match=message):
+        att.AttachOptions(**{field: value})
+
+
+def test_attach_options_accept_normal_observe_and_drive_windows():
+    assert att.AttachOptions().mode == "observe"
+    assert att.AttachOptions(requests=1, concurrency=1).mode == "drive"
+
 LIB = str(injection.lib_path())
 
 

From d741f4fe2e4565e9fd79c3ff030db3ea7d3c73f1 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:34:36 -0700
Subject: [PATCH 38/70] fix: refuse invalid optimization gate evidence

---
 gitm/agents/policy.py           |  2 ++
 gitm/optimizer/apply.py         | 10 ++++++++++
 gitm/optimizer/qualification.py | 12 ++++++++++++
 tests/test_apply_rollback.py    | 12 ++++++++++++
 tests/test_gate_wiring.py       |  8 ++++++++
 tests/test_run_loop_workload.py | 10 ++++++++++
 6 files changed, 54 insertions(+)

diff --git a/gitm/agents/policy.py b/gitm/agents/policy.py
index 1d796c4..c9eff88 100644
--- a/gitm/agents/policy.py
+++ b/gitm/agents/policy.py
@@ -34,6 +34,8 @@ def select_interventions(
     *,
     ctx: GateContext | None = None,
 ) -> list[RankedCandidate]:
+    if isinstance(top_n, bool) or not isinstance(top_n, int) or top_n <= 0:
+        raise ValueError(f"top_n must be a positive integer, got {top_n!r}")
     candidates: list[RankedCandidate] = []
 
     for spec in library:
diff --git a/gitm/optimizer/apply.py b/gitm/optimizer/apply.py
index 2e5d9c1..5b8a3f8 100644
--- a/gitm/optimizer/apply.py
+++ b/gitm/optimizer/apply.py
@@ -109,6 +109,16 @@ def apply_intervention(
         return ApplyResult(False, rolled_back=True, measured_delta=None,
                            error=f"measure failed, restored: {exc}")
 
+    if delta is not None and (
+        isinstance(delta, bool)
+        or not isinstance(delta, int | float)
+        or not math.isfinite(float(delta))
+    ):
+        applicator.restore(snapshot)
+        error = f"measurement delta must be finite, got {delta!r}; intervention restored"
+        _audit(audit, "revert", spec, cause=error, knobs=_knob_values(spec))
+        return ApplyResult(False, rolled_back=True, measured_delta=None, error=error)
+
     # Step 4: keep-or-rollback on the regression threshold.
     if delta is None:
         warnings.warn(
diff --git a/gitm/optimizer/qualification.py b/gitm/optimizer/qualification.py
index 4c97b10..18c5668 100644
--- a/gitm/optimizer/qualification.py
+++ b/gitm/optimizer/qualification.py
@@ -81,6 +81,18 @@ def qualify(trace: Trace, target_floor: float = 0.15) -> QualificationResult:
             diagnostic="No kernels in trace — capture failed or workload did not run.",
         )
 
+    invalid_durations = [k for k in kernels if k.end_ns <= k.start_ns]
+    if invalid_durations:
+        return QualificationResult(
+            commit=False,
+            floor=target_floor,
+            fingerprint=fp,
+            diagnostic=(
+                f"Qualification evidence contains {len(invalid_durations)}/{len(kernels)} "
+                "kernel(s) with non-positive duration; refusing the floor commitment."
+            ),
+        )
+
     # Heuristic: if the top-10 kernels by time account for >95% of duration,
     # the workload is likely already well-shaped. Real gate uses the residual
     # distribution after the deviation monitor.
diff --git a/tests/test_apply_rollback.py b/tests/test_apply_rollback.py
index a191a0e..9de55ec 100644
--- a/tests/test_apply_rollback.py
+++ b/tests/test_apply_rollback.py
@@ -219,6 +219,18 @@ def test_apply_refuses_nonfinite_keep_threshold(threshold):
         )
 
 
+@pytest.mark.parametrize("delta", [float("nan"), float("inf"), float("-inf")])
+def test_apply_rolls_back_nonfinite_measurement(delta):
+    cfg = {"block_size": 8}
+    app = DictApplicator(cfg, measure_fn=lambda _spec: delta)
+
+    result = apply_intervention(_spec(), app)
+
+    assert result.rolled_back and not result.applied
+    assert "finite" in result.error
+    assert cfg["block_size"] == 8
+
+
 # --- library now carries the 21 curated levers ------------------------------
 
 
diff --git a/tests/test_gate_wiring.py b/tests/test_gate_wiring.py
index 82146a7..c1c02de 100644
--- a/tests/test_gate_wiring.py
+++ b/tests/test_gate_wiring.py
@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+import pytest
+
 from gitm.agents.policy import Policy, select_interventions
 from gitm.kernels.spec import Applicability, InterventionSpec, SafetyGate
 from gitm.optimizer.preconditions import GateContext
@@ -39,6 +41,12 @@ def test_without_ctx_no_applicability_filtering():
     assert ranked[0].rejected_reason is None
 
 
+@pytest.mark.parametrize("top_n", [0, -1, True, 1.5])
+def test_policy_refuses_invalid_candidate_limit(top_n):
+    with pytest.raises(ValueError, match="top_n"):
+        select_interventions(_trace(), [], Policy(), top_n=top_n)
+
+
 def test_load_library_filters_by_workload(tmp_path):
     import yaml
 
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index 539f51b..2b7d188 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -79,6 +79,16 @@ def invalid_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
     assert "positive-duration" in result["report_md"]
 
 
+def test_qualification_direct_caller_refuses_invalid_kernel_durations():
+    from gitm.optimizer.qualification import qualify
+
+    trace = make_trace(events=[make_kernel("broken", start_ns=20, end_ns=10)])
+    result = qualify(trace)
+
+    assert result.commit is False
+    assert "non-positive duration" in result.diagnostic
+
+
 _UNSET = object()  # identity sentinel — a real workload_id could legitimately be any string
 
 

From 9e4468078b2f36fbc525f7f13c89d0741fc4aaa2 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:35:48 -0700
Subject: [PATCH 39/70] fix: surface imported launch-shape fallbacks

---
 gitm/importers/torch_trace.py | 34 ++++++++++++++++++++++++++++++++++
 tests/test_importers.py       |  3 ++-
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/gitm/importers/torch_trace.py b/gitm/importers/torch_trace.py
index 9824c05..13f66d1 100644
--- a/gitm/importers/torch_trace.py
+++ b/gitm/importers/torch_trace.py
@@ -163,6 +163,32 @@ def _classify(cat: str | None, name: str, args: dict[str, Any] | None) -> str |
     return None
 
 
+def _launch_metadata_fallbacks(obj: dict[str, Any]) -> tuple[str, ...]:
+    """Neutral launch placeholders used for a parsed GPU event, for surfacing."""
+    args = obj.get("args") if isinstance(obj.get("args"), dict) else {}
+    cat = obj.get("cat")
+    name = str(obj.get("name") or "")
+    kind = _classify(str(cat) if cat is not None else None, name, args)
+    if kind != "kernel" and not (kind is None and str(cat).lower() in {"cuda", "gpu"}):
+        return ()
+    missing: list[str] = []
+    if not name:
+        missing.append("kernel name")
+    if _arg_get(args, _GRID_KEYS) is None and _arg_get(args, ("gridX", "grid_x")) is None:
+        missing.append("grid dimensions")
+    if _arg_get(args, _BLOCK_KEYS) is None and _arg_get(args, ("blockX", "block_x")) is None:
+        missing.append("block dimensions")
+    return tuple(missing)
+
+
+def _append_launch_metadata_warnings(stats: ImportStats, counts: dict[str, int]) -> None:
+    for field, count in sorted(counts.items()):
+        stats.warnings.append(
+            f"launch metadata coverage: {count} parsed kernel event(s) omitted {field}; "
+            "the importer used the neutral value 1"
+        )
+
+
 
 def _memcpy_event(
     *,
@@ -635,10 +661,13 @@ def _import_torch_from_event_dicts(
 ) -> tuple[list[Trace], ImportStats]:
     """Shared finish path once raw chrome event dicts are in hand."""
     events: list[TraceEvent] = []
+    metadata_fallbacks: dict[str, int] = {}
     for obj in raw_events:
         ev = event_from_chrome(obj, strict=strict)
         if ev is not None:
             events.append(ev)
+            for field in _launch_metadata_fallbacks(obj):
+                metadata_fallbacks[field] = metadata_fallbacks.get(field, 0) + 1
     if not events:
         raise ImportError(
             f"{path.name}: no complete GPU kernel/memcpy events found in traceEvents"
@@ -695,6 +724,7 @@ def _import_torch_from_event_dicts(
         per_device_kernel_counts=all_counts,
         total_raw_events=len(events),
     )
+    _append_launch_metadata_warnings(stats, metadata_fallbacks)
     if len(device_ids) > 1:
         stats.warnings.append(
             f"multi-GPU input: analyzing devices {device_ids}; "
@@ -778,6 +808,7 @@ def import_torch_trace(
 
     buckets: dict[int, list[TraceEvent]] = defaultdict(list)
     all_counts: dict[int, int] = defaultdict(int)
+    metadata_fallbacks: dict[str, int] = {}
     scanned_sku: str | None = None
     n_raw = 0
     try:
@@ -800,6 +831,8 @@ def import_torch_trace(
             ev = event_from_chrome(obj, strict=strict)
             if ev is None:
                 continue
+            for field in _launch_metadata_fallbacks(obj):
+                metadata_fallbacks[field] = metadata_fallbacks.get(field, 0) + 1
             buckets[ev.device_id].append(ev)
             if getattr(ev, "kind", None) == "kernel":
                 all_counts[ev.device_id] += 1
@@ -856,6 +889,7 @@ def import_torch_trace(
         per_device_kernel_counts=dict(all_counts),
         total_raw_events=total_events,
     )
+    _append_launch_metadata_warnings(stats, metadata_fallbacks)
     if len(device_ids) > 1:
         stats.warnings.append(
             f"multi-GPU input: analyzing devices {device_ids}; "
diff --git a/tests/test_importers.py b/tests/test_importers.py
index 03bbf12..4891618 100644
--- a/tests/test_importers.py
+++ b/tests/test_importers.py
@@ -143,11 +143,12 @@ def test_torch_json_and_gz():
 
 
 def test_torch_array_form_and_missing_grid():
-    ts, _ = import_torch_trace(FIXTURES / "torch_trace_array.json")
+    ts, stats = import_torch_trace(FIXTURES / "torch_trace_array.json")
     t = ts[0]
     assert t.kernels()
     # grid defaults to 1 when absent
     assert all(k.grid_x >= 1 and k.block_x >= 1 for k in t.kernels())
+    assert any("grid dimensions" in warning for warning in stats.warnings)
 
 
 def test_torch_us_to_ns_conversion():

From 3f285950d93717d0ca8b108e80300c524d574f13 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:37:43 -0700
Subject: [PATCH 40/70] fix: surface dense graph shape assumptions

---
 gitm/scheduler/loop.py                  | 14 +++++++++++++-
 tests/test_execution_graph_dispatch.py  | 22 ++++++++++++++++++++++
 tests/test_predict_graph_from_engine.py | 19 +++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)

diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 308ac9c..2616af8 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -341,6 +341,7 @@ def _dense_spec_from_config(cfg: dict[str, Any]) -> tuple[ModelSpec | None, str]
         "num_attention_heads",
         "intermediate_size",
         "vocab_size",
+        "torch_dtype",
     )
     missing = [key for key in required if cfg.get(key) is None]
     if missing:
@@ -361,7 +362,9 @@ def _dense_spec_from_config(cfg: dict[str, Any]) -> tuple[ModelSpec | None, str]
             return None, f"dense activation dtype {act!r} is not priceable"
         dtype_bytes = int(weight_bytes(act))
         quant = cfg.get("quantization_config") or {}
-        method = quant.get("quant_method") if isinstance(quant, dict) else None
+        if not isinstance(quant, dict):
+            return None, "dense quantization_config must be an object when declared"
+        method = quant.get("quant_method")
         if method is not None and str(method).lower() not in _QUANT_WEIGHT_BYTES:
             return None, f"dense quantization method {method!r} is not priceable"
         full_attn_layer_step = 1
@@ -452,6 +455,15 @@ def _execution_graph(engine: Any, pctx: Any, sched: Any) -> ExecutionGraphResolu
         diagnostics.extend(sharding_diagnostics)
         graph = predict_moe_graph(spec, hw, batch, sharding)
     else:
+        if cfg.get("num_key_value_heads") is None:
+            diagnostics.append(
+                "dense num_key_value_heads was not declared; assuming standard MHA "
+                "with one KV head per query head"
+            )
+        if cfg.get("head_dim") is None:
+            diagnostics.append(
+                "dense head_dim was not declared; derived hidden_size / num_attention_heads"
+            )
         spec, error = _dense_spec_from_config(cfg)
         if spec is None:
             return ExecutionGraphResolution(None, diagnostics, error)
diff --git a/tests/test_execution_graph_dispatch.py b/tests/test_execution_graph_dispatch.py
index 20930d3..c7a169a 100644
--- a/tests/test_execution_graph_dispatch.py
+++ b/tests/test_execution_graph_dispatch.py
@@ -52,6 +52,20 @@ def _moe_cfg(**over) -> dict:
     return cfg
 
 
+def _dense_cfg(**over) -> dict:
+    cfg = {
+        "model_type": "opt",
+        "hidden_size": 768,
+        "num_hidden_layers": 12,
+        "num_attention_heads": 12,
+        "intermediate_size": 3072,
+        "vocab_size": 50272,
+        "torch_dtype": "bf16",
+    }
+    cfg.update(over)
+    return cfg
+
+
 def test_sparse_engine_dispatches_to_sparse_graph():
     resolved = _execution_graph(_engine(_moe_cfg()), _pctx(), sched=None)
 
@@ -126,3 +140,11 @@ def test_accepted_batch_and_kv_defaults_are_diagnostics():
     assert resolved.graph.batch.kv_cache_len == 4096
     assert any("batch=1" in note for note in resolved.diagnostics)
     assert any("kv_cache_len=4096" in note for note in resolved.diagnostics)
+
+
+def test_dense_mha_and_head_dimension_derivations_are_diagnostics():
+    resolved = _execution_graph(_engine(_dense_cfg()), _pctx(), sched=None)
+
+    assert resolved.ok
+    assert any("num_key_value_heads" in note and "MHA" in note for note in resolved.diagnostics)
+    assert any("head_dim" in note and "derived" in note for note in resolved.diagnostics)
diff --git a/tests/test_predict_graph_from_engine.py b/tests/test_predict_graph_from_engine.py
index fdbdaea..3b28d51 100644
--- a/tests/test_predict_graph_from_engine.py
+++ b/tests/test_predict_graph_from_engine.py
@@ -59,6 +59,25 @@ def test_unknown_dense_dtype_refuses_instead_of_becoming_bf16():
     assert "not priceable" in error
 
 
+def test_missing_dense_dtype_refuses_instead_of_becoming_bf16():
+    cfg = dict(_OPT_125M)
+    del cfg["torch_dtype"]
+
+    spec, error = _dense_spec_from_config(cfg)
+
+    assert spec is None
+    assert "torch_dtype" in error
+
+
+def test_malformed_dense_quantization_config_refuses():
+    spec, error = _dense_spec_from_config(
+        {**_OPT_125M, "quantization_config": ["fp8"]}
+    )
+
+    assert spec is None
+    assert "quantization_config" in error
+
+
 def test_dense_parser_preserves_fp32_compute_dtype():
     spec, error = _dense_spec_from_config({**_OPT_125M, "torch_dtype": "float32"})
 

From 08c84d6624b3675244ee7dbc926d79da0bde150f Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:40:43 -0700
Subject: [PATCH 41/70] fix: wire dense graph sharding and weight precision

---
 gitm/planner/graph.py                   | 88 ++++++++++++++++++-------
 gitm/scheduler/loop.py                  | 36 ++++++++--
 tests/test_execution_graph_dispatch.py  | 40 +++++++++++
 tests/test_predict_graph_from_engine.py | 42 ++++++++++++
 4 files changed, 178 insertions(+), 28 deletions(-)

diff --git a/gitm/planner/graph.py b/gitm/planner/graph.py
index f939aaf..ffe3f3b 100644
--- a/gitm/planner/graph.py
+++ b/gitm/planner/graph.py
@@ -10,7 +10,7 @@
 
 from __future__ import annotations
 
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, replace
 
 from gitm.planner.roofline import (
     BatchConfig,
@@ -24,7 +24,7 @@
 )
 
 
-def _ffn_terms(model: ModelSpec, b: int, *, moe_layer: bool = True) -> tuple[
+def _ffn_terms(model: ModelSpec, b: int, *, moe_layer: bool = True, tp: int = 1) -> tuple[
     float, float, float, float
 ]:
     """``(gate_up_flops, gate_up_bytes, down_flops, down_bytes)`` for one layer.
@@ -60,7 +60,7 @@ def _ffn_terms(model: ModelSpec, b: int, *, moe_layer: bool = True) -> tuple[
 
     if not (model.is_moe and moe_layer):
         # Dense: one FFN, weights fetched once, every token through all of it.
-        ff = model.intermediate
+        ff = model.intermediate / tp
         gate_up_flops = 2 * 2 * b * h * ff
         gate_up_bytes = dt * (b * h + 2 * b * ff) + wb * (2 * h * ff)
         down_flops = 2 * b * ff * h
@@ -75,17 +75,20 @@ def _ffn_terms(model: ModelSpec, b: int, *, moe_layer: bool = True) -> tuple[
     distinct = distinct_experts(b, model.num_experts, k)
 
     # Compute: every token pays k routed experts plus all shared ones.
-    gate_up_flops = 2 * 2 * b * (k * h * ff + n_shared * h * sff)
-    down_flops = 2 * b * (k * ff * h + n_shared * sff * h)
+    gate_up_flops = 2 * 2 * b * (k * h * ff + n_shared * h * sff) / tp
+    down_flops = 2 * b * (k * ff * h + n_shared * sff * h) / tp
     # Router: [b, h] @ [h, num_experts], folded in above.
     gate_up_flops += 2 * b * h * model.num_experts
 
     # Weight traffic: distinct routed experts once each, plus the shared experts
     # (always resident in the step) and the router matrix.
-    gate_up_weight_bytes = wb * (distinct * 2 * h * ff + n_shared * 2 * h * sff + h * model.num_experts)
-    down_weight_bytes = wb * (distinct * ff * h + n_shared * sff * h)
+    gate_up_weight_bytes = wb * (
+        (distinct * 2 * h * ff + n_shared * 2 * h * sff) / tp
+        + h * model.num_experts
+    )
+    down_weight_bytes = wb * (distinct * ff * h + n_shared * sff * h) / tp
     # Activations: in [b, h], out [b, k*ff] (+ shared) for gate_up; mirrored for down.
-    act_out = b * (k * ff + n_shared * sff)
+    act_out = b * (k * ff + n_shared * sff) / tp
     gate_up_bytes = dt * (b * h + 2 * act_out) + gate_up_weight_bytes
     down_bytes = dt * (act_out + b * h) + down_weight_bytes
     return gate_up_flops, gate_up_bytes, down_flops, down_bytes
@@ -192,6 +195,7 @@ def predict_graph(
     model: ModelSpec | None = None,
     hw: HardwareSpec | None = None,
     batch: BatchConfig | None = None,
+    sharding: ShardingConfig | None = None,
 ) -> Graph:
     """Emit a predicted execution graph for one decode step.
 
@@ -200,21 +204,41 @@ def predict_graph(
     model = model or ModelSpec()
     hw = hw or HardwareSpec()
     batch = batch or BatchConfig()
+    sharding = sharding or ShardingConfig()
+    if sharding.ep != 1:
+        raise ValueError("dense graph does not support expert parallelism")
+    tp = sharding.tp
+    if model.n_heads % tp:
+        raise ValueError(f"n_heads={model.n_heads} must be divisible by tp={tp}")
+    if model.num_kv_heads >= tp:
+        if model.num_kv_heads % tp:
+            raise ValueError(
+                f"num_kv_heads={model.num_kv_heads} must be divisible by tp={tp}"
+            )
+        kv_heads_rank = model.num_kv_heads / tp
+    else:
+        if tp % model.num_kv_heads:
+            raise ValueError(
+                f"tp={tp} must be divisible by replicated num_kv_heads={model.num_kv_heads}"
+            )
+        kv_heads_rank = 1
 
-    g = Graph(model=model, hw=hw, batch=batch)
-    b = batch.batch
+    g = Graph(model=model, hw=hw, batch=batch, sharding=sharding)
+    b = batch.positions_per_step
+    sequences = batch.batch
     h = model.hidden
     kv_len = batch.kv_cache_len
     head_dim = model.head_dim
-    n_kv = model.num_kv_heads
     n_h = model.n_heads
     dt = model.dtype_bytes
+    wb = model.w_bytes
+    q_heads_rank = n_h / tp
 
     for layer in range(model.n_layers):
         # QKV projection: matmul (b, h) @ (h, (n_h + 2*n_kv) * head_dim)
-        qkv_out = (n_h + 2 * n_kv) * head_dim
+        qkv_out = (q_heads_rank + 2 * kv_heads_rank) * head_dim
         flops = 2 * b * h * qkv_out
-        bytes_moved = dt * (b * h + h * qkv_out + b * qkv_out)
+        bytes_moved = dt * (b * h + b * qkv_out) + wb * h * qkv_out
         g.nodes.append(
             PredictedNode(
                 "qkv_proj",
@@ -230,13 +254,13 @@ def predict_graph(
         # kv_len / head_dim — over 100x at 16k context.
         if model.is_full_attention_layer(layer):
             # Reads: K, V over kv_len tokens, grouped to n_kv heads.
-            kv_bytes = dt * 2 * kv_len * n_kv * head_dim * b
-            attn_flops = 2 * b * n_h * head_dim * kv_len * 2  # qk + sv
+            kv_bytes = dt * 2 * kv_len * kv_heads_rank * head_dim * sequences
+            attn_flops = 2 * b * q_heads_rank * head_dim * kv_len * 2  # qk + sv
         else:
             # Read the recurrent state, update it, write it back: 2x state per
             # sequence. FLOPs are the state-sized matmuls, also context-free.
-            state = model.linear_attn_state_elems
-            kv_bytes = dt * 2 * state * b
+            state = model.linear_attn_state_elems / tp
+            kv_bytes = dt * 2 * state * sequences
             attn_flops = 2 * b * state * 2  # state-vector product + state update
         g.nodes.append(
             PredictedNode(
@@ -249,8 +273,9 @@ def predict_graph(
         )
 
         # Output projection
-        flops = 2 * b * h * h
-        bytes_moved = dt * (b * h + h * h + b * h)
+        local_h = h / tp
+        flops = 2 * b * local_h * h
+        bytes_moved = dt * (b * local_h + b * h) + wb * local_h * h
         g.nodes.append(
             PredictedNode(
                 "attn_out_proj",
@@ -263,7 +288,7 @@ def predict_graph(
         # GEMMs, so their flops/bytes come from the mixture model instead of a
         # single dense FFN (see _ffn_terms).
         gate_up_flops, gate_up_bytes, down_flops, down_bytes = _ffn_terms(
-            model, b, moe_layer=model.is_moe_layer(layer)
+            model, b, moe_layer=model.is_moe_layer(layer), tp=tp
         )
         g.nodes.append(
             PredictedNode(
@@ -289,8 +314,27 @@ def predict_graph(
         )
 
     # Final vocab projection
-    flops = 2 * b * h * model.vocab
-    bytes_moved = dt * (b * h + h * model.vocab + b * model.vocab)
+        if tp > 1:
+            link = replace(hw, peak_mem_bw_bytes_per_s=hw.interconnect_bw_bytes_per_s)
+            collective_bytes = 4.0 * (tp - 1) / tp * b * h * dt
+            g.nodes.append(
+                PredictedNode(
+                    "tp_all_reduce",
+                    layer,
+                    roofline(
+                        "tp_all_reduce",
+                        0.0,
+                        collective_bytes,
+                        link,
+                        dtype=model.compute_dtype,
+                        estimated=True,
+                    ),
+                )
+            )
+
+    local_vocab = model.vocab / tp
+    flops = 2 * b * h * local_vocab
+    bytes_moved = dt * (b * h + b * local_vocab) + wb * h * local_vocab
     g.nodes.append(
         PredictedNode(
             "lm_head",
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 2616af8..49d9833 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -326,12 +326,28 @@ def _loop_sharding(engine: Any) -> tuple[ShardingConfig, list[str]]:
         engine,
         ("parallel_config.enable_expert_parallel", "vllm_config.parallel_config.enable_expert_parallel"),
     )
-    if not isinstance(tp, int) or tp < 1:
+    if tp is not None and (isinstance(tp, bool) or not isinstance(tp, int) or tp < 1):
+        raise ValueError(f"tensor_parallel_size must be a positive integer, got {tp!r}")
+    if tp is None:
         return ShardingConfig(), [
             "sharding topology was not exposed; using whole-model tp=1 ep=1 dp=1"
         ]
-    dp_i = dp if isinstance(dp, int) and dp > 0 else 1
-    return ShardingConfig(tp=tp, ep=tp if bool(ep) else 1, dp=dp_i), []
+    diagnostics: list[str] = []
+    if dp is None:
+        dp_i = 1
+        diagnostics.append("data_parallel_size was not exposed; using dp=1")
+    elif isinstance(dp, bool) or not isinstance(dp, int) or dp < 1:
+        raise ValueError(f"data_parallel_size must be a positive integer, got {dp!r}")
+    else:
+        dp_i = dp
+    if ep is None:
+        ep_enabled = False
+        diagnostics.append("enable_expert_parallel was not exposed; using ep=1")
+    elif not isinstance(ep, bool):
+        raise ValueError(f"enable_expert_parallel must be boolean, got {ep!r}")
+    else:
+        ep_enabled = ep
+    return ShardingConfig(tp=tp, ep=tp if ep_enabled else 1, dp=dp_i), diagnostics
 
 
 def _dense_spec_from_config(cfg: dict[str, Any]) -> tuple[ModelSpec | None, str]:
@@ -411,6 +427,11 @@ def _execution_graph(engine: Any, pctx: Any, sched: Any) -> ExecutionGraphResolu
         )
     hw = hardware_spec_for(pctx.peak)
     batch, diagnostics = _loop_batch(engine, pctx, sched)
+    try:
+        sharding, sharding_diagnostics = _loop_sharding(engine)
+    except ValueError as exc:
+        return ExecutionGraphResolution(None, diagnostics, f"invalid live sharding topology: {exc}")
+    diagnostics.extend(sharding_diagnostics)
 
     if is_sparse_moe_config(cfg):
         invalid = validate_moe_config(cfg)
@@ -451,8 +472,6 @@ def _execution_graph(engine: Any, pctx: Any, sched: Any) -> ExecutionGraphResolu
         unpriceable = validate_priceable_dtypes(spec)
         if unpriceable:
             return ExecutionGraphResolution(None, diagnostics, "; ".join(unpriceable))
-        sharding, sharding_diagnostics = _loop_sharding(engine)
-        diagnostics.extend(sharding_diagnostics)
         graph = predict_moe_graph(spec, hw, batch, sharding)
     else:
         if cfg.get("num_key_value_heads") is None:
@@ -467,7 +486,12 @@ def _execution_graph(engine: Any, pctx: Any, sched: Any) -> ExecutionGraphResolu
         spec, error = _dense_spec_from_config(cfg)
         if spec is None:
             return ExecutionGraphResolution(None, diagnostics, error)
-        graph = predict_graph(model=spec, hw=hw, batch=batch)
+        try:
+            graph = predict_graph(model=spec, hw=hw, batch=batch, sharding=sharding)
+        except ValueError as exc:
+            return ExecutionGraphResolution(
+                None, diagnostics, f"dense sharding cannot be priced: {exc}"
+            )
 
     if graph.has_fallback_peaks:
         diagnostics.append("one or more nodes use fallback compute peaks")
diff --git a/tests/test_execution_graph_dispatch.py b/tests/test_execution_graph_dispatch.py
index c7a169a..922c281 100644
--- a/tests/test_execution_graph_dispatch.py
+++ b/tests/test_execution_graph_dispatch.py
@@ -148,3 +148,43 @@ def test_dense_mha_and_head_dimension_derivations_are_diagnostics():
     assert resolved.ok
     assert any("num_key_value_heads" in note and "MHA" in note for note in resolved.diagnostics)
     assert any("head_dim" in note and "derived" in note for note in resolved.diagnostics)
+
+
+def test_dense_execution_graph_consumes_live_tensor_parallel_topology():
+    engine = _engine(_dense_cfg(num_key_value_heads=4))
+    engine.parallel_config = SimpleNamespace(
+        tensor_parallel_size=2,
+        data_parallel_size=1,
+        enable_expert_parallel=False,
+    )
+
+    resolved = _execution_graph(engine, _pctx(), sched=None)
+
+    assert resolved.ok
+    assert resolved.graph.sharding.tp == 2
+    assert any(node.op == "tp_all_reduce" for node in resolved.graph.nodes)
+
+
+def test_malformed_live_expert_parallel_flag_refuses_topology():
+    engine = _engine(_dense_cfg())
+    engine.parallel_config = SimpleNamespace(
+        tensor_parallel_size=2,
+        data_parallel_size=1,
+        enable_expert_parallel="false",
+    )
+
+    resolved = _execution_graph(engine, _pctx(), sched=None)
+
+    assert not resolved.ok
+    assert "enable_expert_parallel must be boolean" in resolved.refusal_reason
+
+
+def test_partial_live_topology_names_each_assumption():
+    engine = _engine(_dense_cfg())
+    engine.parallel_config = SimpleNamespace(tensor_parallel_size=2)
+
+    resolved = _execution_graph(engine, _pctx(), sched=None)
+
+    assert resolved.ok
+    assert any("data_parallel_size" in note for note in resolved.diagnostics)
+    assert any("enable_expert_parallel" in note for note in resolved.diagnostics)
diff --git a/tests/test_predict_graph_from_engine.py b/tests/test_predict_graph_from_engine.py
index 3b28d51..1609f33 100644
--- a/tests/test_predict_graph_from_engine.py
+++ b/tests/test_predict_graph_from_engine.py
@@ -2,7 +2,10 @@
 
 from __future__ import annotations
 
+import pytest
+
 from gitm.planner.graph import predict_graph
+from gitm.planner.roofline import BatchConfig, ShardingConfig
 from gitm.scheduler.loop import _dense_spec_from_config
 
 # opt-125m: 12 layers, hidden 768, 12 heads (MHA), intermediate 3072.
@@ -85,3 +88,42 @@ def test_dense_parser_preserves_fp32_compute_dtype():
     assert spec is not None
     assert spec.compute_dtype == "fp32"
     assert spec.dtype_bytes == 4
+
+
+def test_dense_graph_prices_tensor_parallel_work_per_rank():
+    whole = predict_graph(model=_spec(num_key_value_heads=4))
+    sharded = predict_graph(
+        model=_spec(num_key_value_heads=4), sharding=ShardingConfig(tp=2)
+    )
+
+    whole_qkv = next(n.prediction for n in whole.nodes if n.op == "qkv_proj")
+    rank_qkv = next(n.prediction for n in sharded.nodes if n.op == "qkv_proj")
+    assert sharded.sharding.tp == 2
+    assert rank_qkv.flops == whole_qkv.flops / 2
+    assert any(n.op == "tp_all_reduce" for n in sharded.nodes)
+
+
+def test_quantized_dense_width_applies_to_attention_and_lm_head_weights():
+    bf16 = predict_graph(model=_spec())
+    fp8 = predict_graph(
+        model=_spec(quantization_config={"quant_method": "fp8"})
+    )
+
+    for op in ("qkv_proj", "attn_out_proj", "lm_head"):
+        bf16_bytes = next(n.prediction.bytes for n in bf16.nodes if n.op == op)
+        fp8_bytes = next(n.prediction.bytes for n in fp8.nodes if n.op == op)
+        assert fp8_bytes < bf16_bytes
+
+
+def test_dense_graph_honors_speculative_positions():
+    plain = predict_graph(model=_spec(), batch=BatchConfig(batch=2, speculative_tokens=0))
+    drafted = predict_graph(model=_spec(), batch=BatchConfig(batch=2, speculative_tokens=3))
+
+    plain_qkv = next(n.prediction for n in plain.nodes if n.op == "qkv_proj")
+    drafted_qkv = next(n.prediction for n in drafted.nodes if n.op == "qkv_proj")
+    assert drafted_qkv.flops == plain_qkv.flops * 4
+
+
+def test_dense_graph_refuses_incompatible_tensor_parallel_shape():
+    with pytest.raises(ValueError, match="n_heads"):
+        predict_graph(model=_spec(), sharding=ShardingConfig(tp=5))

From 70aae4bad78d21cd39a7467e68fce2011fcd86d2 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:44:28 -0700
Subject: [PATCH 42/70] fix: wire live GPU headroom to CLI

---
 gitm/cli.py         | 11 +++++++++++
 tests/test_smoke.py | 25 +++++++++++++++++++++++++
 2 files changed, 36 insertions(+)

diff --git a/gitm/cli.py b/gitm/cli.py
index fa5e43a..5faeb8a 100644
--- a/gitm/cli.py
+++ b/gitm/cli.py
@@ -149,6 +149,10 @@ def _parser() -> argparse.ArgumentParser:
     _add_capture(sub)
 
     sub.add_parser("doctor", help="Probe environment, GPUs, and data locations.")
+    sub.add_parser(
+        "gpu-headroom",
+        help="Print a one-shot live GPU utilization and memory snapshot.",
+    )
 
     plan_kitti = sub.add_parser(
         "plan-kitti", help="Render the PointPillars execution graph for a known GPU SKU."
@@ -382,6 +386,13 @@ def main(argv: list[str] | None = None) -> int:
         print(json.dumps(report, indent=2))
         return 0
 
+    if args.cmd == "gpu-headroom":
+        from gitm.optimizer.headroom_kernel_rank import live_gpu_headroom
+
+        rows = live_gpu_headroom()
+        print(json.dumps(rows, indent=2))
+        return 0 if rows else 3
+
     if args.cmd == "plan-kitti":
         from gitm.planner.context import hardware_spec_for, peak_for_sku
         from gitm.planner.kitti_graph import predict_kitti_graph, render_kitti_graph
diff --git a/tests/test_smoke.py b/tests/test_smoke.py
index a0f1ddc..11e6831 100644
--- a/tests/test_smoke.py
+++ b/tests/test_smoke.py
@@ -116,3 +116,28 @@ def test_doctor_returns_jsonable():
     json.dumps(info)
     assert info["gitm_version"]
     assert "telemetry_backends" in info
+
+
+def test_gpu_headroom_cli_surfaces_live_snapshot(monkeypatch, capsys):
+    from gitm.cli import main
+
+    monkeypatch.setattr(
+        "gitm.optimizer.headroom_kernel_rank.live_gpu_headroom",
+        lambda: [{"gpu_index": 0, "util_pct": 42.0}],
+    )
+
+    assert main(["gpu-headroom"]) == 0
+    assert json.loads(capsys.readouterr().out) == [
+        {"gpu_index": 0, "util_pct": 42.0}
+    ]
+
+
+def test_gpu_headroom_cli_fails_when_snapshot_is_unavailable(monkeypatch, capsys):
+    from gitm.cli import main
+
+    monkeypatch.setattr(
+        "gitm.optimizer.headroom_kernel_rank.live_gpu_headroom", lambda: []
+    )
+
+    assert main(["gpu-headroom"]) == 3
+    assert json.loads(capsys.readouterr().out) == []

From 733ea2ab9cb4062cd34ca47146da9ac4cb97736e Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:47:02 -0700
Subject: [PATCH 43/70] fix: surface scheduler field probe failures

---
 gitm/tracer/vllm_stats.py   | 55 ++++++++++++++++++++++++++++++-------
 tests/test_vllm_stats_v1.py | 19 +++++++++++++
 tests/test_vllm_stress.py   | 38 +++++++++++++++++++++++++
 3 files changed, 102 insertions(+), 10 deletions(-)

diff --git a/gitm/tracer/vllm_stats.py b/gitm/tracer/vllm_stats.py
index 30ce319..d9c2640 100644
--- a/gitm/tracer/vllm_stats.py
+++ b/gitm/tracer/vllm_stats.py
@@ -48,6 +48,7 @@ class SchedulerSample:
     gpu_cache_usage: float | None = None  # KV-cache blocks used / total, 0..1
     cpu_cache_usage: float | None = None
     batch_occupancy: float | None = None  # num_running / max_num_seqs, 0..1
+    diagnostics: list[str] = field(default_factory=list)
 
 
 @dataclass
@@ -369,7 +370,9 @@ def _schedulers(engine: Any) -> list[Any]:
     return list(sched) if isinstance(sched, list | tuple) else [sched]
 
 
-def _v1_scheduler_stats(scheduler: Any) -> dict[str, Any]:
+def _v1_scheduler_stats(
+    scheduler: Any, diagnostics: list[str] | None = None
+) -> dict[str, Any]:
     """vLLM V1 stats via ``scheduler.make_stats()`` — V1 doesn't keep the V0
     running/waiting deques in the same shape, but exposes a stats object with
     ``num_running_reqs`` / ``num_waiting_reqs`` / ``kv_cache_usage``. Best-effort;
@@ -381,7 +384,11 @@ def _v1_scheduler_stats(scheduler: Any) -> dict[str, Any]:
         return {}
     try:
         stats = make()
-    except Exception:
+    except Exception as exc:
+        if diagnostics is not None:
+            diagnostics.append(
+                f"vLLM V1 scheduler make_stats failed: {type(exc).__name__}: {exc}"
+            )
         return {}
     if stats is None:
         return {}
@@ -443,7 +450,7 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non
             saw_any = True
 
         # KV-cache usage off the first scheduler's block manager (best-effort, V0).
-        usage = _gpu_cache_usage(schedulers[0])
+        usage = _gpu_cache_usage(schedulers[0], sample.diagnostics)
         if usage is not None:
             sample.gpu_cache_usage = usage
             saw_any = True
@@ -451,7 +458,7 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non
         # vLLM V1: fill running / waiting / cache from the scheduler's stat object
         # where the VO deques weren't exposed (they read empty on V1)
         for sch in schedulers:
-            for field_name, val in _v1_scheduler_stats(sch).items():
+            for field_name, val in _v1_scheduler_stats(sch, sample.diagnostics).items():
                 if getattr(sample, field_name) is None:
                     setattr(sample, field_name, val)
                     saw_any = True
@@ -467,8 +474,10 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non
         try:
             sample.num_unfinished = int(getter())
             saw_any = True
-        except Exception:
-            pass
+        except Exception as exc:
+            sample.diagnostics.append(
+                f"unfinished-request probe failed: {type(exc).__name__}: {exc}"
+            )
 
     if sample.num_running is not None:
         max_seqs = _max_num_seqs(engine)
@@ -480,10 +489,12 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non
             capacity = max_seqs * max(len(schedulers), 1)
             sample.batch_occupancy = min(1.0, sample.num_running / capacity)
 
-    return sample if saw_any else None
+    return sample if saw_any or sample.diagnostics else None
 
 
-def _gpu_cache_usage(scheduler: Any) -> float | None:
+def _gpu_cache_usage(
+    scheduler: Any, diagnostics: list[str] | None = None
+) -> float | None:
     """KV-cache block occupancy (0..1) off a scheduler's block manager, if exposed."""
     bm = getattr(scheduler, "block_manager", None)
     if bm is None:
@@ -494,7 +505,11 @@ def _gpu_cache_usage(scheduler: Any) -> float | None:
     if callable(free_fn) and isinstance(total, int) and total > 0:
         try:
             return max(0.0, 1.0 - free_fn() / total)
-        except Exception:
+        except Exception as exc:
+            if diagnostics is not None:
+                diagnostics.append(
+                    f"KV-cache usage probe failed: {type(exc).__name__}: {exc}"
+                )
             return None
     return None
 
@@ -543,7 +558,12 @@ def start(self) -> None:
         try:
             s0 = read_scheduler_stats(self.engine, t_ns=0)
             if s0 is not None:
+                self._surface_sample_diagnostics(s0)
                 self.samples.append(s0)
+            else:
+                self._record_failure(
+                    "unavailable", "engine exposes no recognized scheduler fields"
+                )
         except Exception as exc:
             self._record_failure("initial-read", f"scheduler stats read failed: {exc}")
         self._stop.clear()
@@ -555,7 +575,12 @@ def _run(self) -> None:
             try:
                 s = read_scheduler_stats(self.engine, t_ns=time.perf_counter_ns() - self._t0_ns)
                 if s is not None:
+                    self._surface_sample_diagnostics(s)
                     self.samples.append(s)
+                else:
+                    self._record_failure(
+                        "unavailable", "engine exposes no recognized scheduler fields"
+                    )
             except Exception as exc:
                 self._record_failure("background-read", f"scheduler stats read failed: {exc}")
             self._stop.wait(self.interval_s)
@@ -577,12 +602,20 @@ def _record_failure(self, key: str, detail: str) -> None:
         self.diagnostics.append(message)
         warnings.warn(message, RuntimeWarning, stacklevel=2)
 
+    def _surface_sample_diagnostics(self, sample: SchedulerSample) -> None:
+        for message in sample.diagnostics:
+            self._record_failure(f"field:{message.split(':', 1)[0]}", message)
+
     def summary(self) -> SchedulerStatsSummary:
         # Snapshot first: stop() joins with a timeout, so in the pathological case
         # where the daemon thread is still alive, summarize must iterate a stable
         # copy rather than a list being appended to concurrently.
         result = summarize(list(self.samples), t0_wall_ns=self._t0_wall_ns)
-        result.diagnostics.extend(self.diagnostics)
+        if self.diagnostics:
+            # The sampler versions field diagnostics with stable deduplication keys
+            # and a user-facing prefix. Direct ``summarize`` callers still receive
+            # the raw per-sample diagnostics, while this boundary emits one copy.
+            result.diagnostics = list(self.diagnostics)
         return result
 
     def to_records(self) -> list[dict[str, Any]]:
@@ -612,6 +645,7 @@ def _vals(attr: str) -> list[float]:
     swapped = _vals("num_swapped")
     duration_s = max(samples[-1].t_ns - samples[0].t_ns, 0) / 1e9
 
+    diagnostics = list(dict.fromkeys(note for sample in samples for note in sample.diagnostics))
     return SchedulerStatsSummary(
         n_samples=len(samples),
         duration_s=duration_s,
@@ -626,6 +660,7 @@ def _vals(attr: str) -> list[float]:
         peak_gpu_cache_usage=max(cache) if cache else None,
         peak_swapped=int(max(swapped)) if swapped else None,
         t0_wall_ns=t0_wall_ns,
+        diagnostics=diagnostics,
     )
 
 
diff --git a/tests/test_vllm_stats_v1.py b/tests/test_vllm_stats_v1.py
index 5089176..9aa8efd 100644
--- a/tests/test_vllm_stats_v1.py
+++ b/tests/test_vllm_stats_v1.py
@@ -48,3 +48,22 @@ def test_v1_engine_without_stats_is_none():
         )
     )
     assert read_scheduler_stats(empty) is None
+
+
+def test_v1_scheduler_read_failure_is_diagnostic():
+    class BrokenScheduler:
+        def make_stats(self):
+            raise RuntimeError("v1 stats moved")
+
+    engine = SimpleNamespace(
+        engine_core=SimpleNamespace(scheduler=BrokenScheduler())
+    )
+
+    sample = read_scheduler_stats(engine)
+
+    assert sample is not None
+    assert sample.num_running is None
+    assert any(
+        "make_stats failed" in note and "v1 stats moved" in note
+        for note in sample.diagnostics
+    )
diff --git a/tests/test_vllm_stress.py b/tests/test_vllm_stress.py
index 44fc38d..47fa8f4 100644
--- a/tests/test_vllm_stress.py
+++ b/tests/test_vllm_stress.py
@@ -136,6 +136,44 @@ class _Partial:
     assert s.num_running is None and s.batch_occupancy is None
 
 
+def test_scheduler_sampler_surfaces_unrecognized_engine_shape():
+    sampler = SchedulerStatsSampler(object(), interval_s=1.0)
+
+    with pytest.warns(RuntimeWarning, match="no recognized scheduler fields"):
+        sampler.start()
+        sampler.stop()
+
+    assert any(
+        "no recognized scheduler fields" in note
+        for note in sampler.summary().diagnostics
+    )
+
+
+def test_read_scheduler_stats_surfaces_field_probe_failures():
+    class BlockManager:
+        num_total_gpu_blocks = 8
+
+        def get_num_free_gpu_blocks(self):
+            raise RuntimeError("cache probe broke")
+
+    class Scheduler:
+        running = [object()]
+        block_manager = BlockManager()
+
+    class Engine:
+        scheduler = Scheduler()
+
+        def get_num_unfinished_requests(self):
+            raise RuntimeError("unfinished probe broke")
+
+    sample = read_scheduler_stats(Engine())
+
+    assert sample is not None
+    assert sample.num_running == 1
+    assert any("cache usage probe failed" in note for note in sample.diagnostics)
+    assert any("unfinished-request probe failed" in note for note in sample.diagnostics)
+
+
 def test_summarize_single_sample():
     s = read_scheduler_stats(
         type("E", (), {"scheduler": [type("S", (), {"running": [0, 1], "waiting": [], "swapped": []})()],

From 48af62b2a1ca8bad9061f0606e09fdf904864ea9 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:48:37 -0700
Subject: [PATCH 44/70] fix: refuse invalid A/B gate controls

---
 gitm/benchmarks/edge/optimize.py | 23 +++++++++++++++++++++--
 gitm/benchmarks/hft/optimize.py  |  4 +++-
 tests/test_edge_optimize.py      | 26 ++++++++++++++++++++++++++
 tests/test_hft_optimize.py       |  9 +++++++++
 4 files changed, 59 insertions(+), 3 deletions(-)

diff --git a/gitm/benchmarks/edge/optimize.py b/gitm/benchmarks/edge/optimize.py
index 1dff1f4..444cad6 100644
--- a/gitm/benchmarks/edge/optimize.py
+++ b/gitm/benchmarks/edge/optimize.py
@@ -23,6 +23,7 @@
 
 from __future__ import annotations
 
+import math
 import time
 from collections.abc import Callable
 from dataclasses import dataclass, field
@@ -69,7 +70,7 @@ def _frame_equivalent(da: list, db: list, *, center_atol: float, tol_frac: float
         else:
             unmatched += 1
     unmatched += used.count(False)  # candidate boxes with no baseline partner
-    allowed = max(1, int(tol_frac * max(len(da), len(db), 1)))
+    allowed = math.ceil(tol_frac * max(len(da), len(db)))
     return unmatched <= allowed
 
 
@@ -88,6 +89,10 @@ def detections_equivalent(
     with a tolerance keeps the gate honest (a genuine regression still trips it)
     without rejecting rounding-level churn.
     """
+    if not math.isfinite(center_atol) or center_atol < 0.0:
+        raise ValueError(f"center_atol must be finite and non-negative, got {center_atol!r}")
+    if not math.isfinite(tol_frac) or not 0.0 <= tol_frac <= 1.0:
+        raise ValueError(f"tol_frac must be finite and in [0, 1], got {tol_frac!r}")
     fa, fb = a.get("frames"), b.get("frames")
     if fa is None or fb is None or len(fa) != len(fb):
         return False
@@ -135,12 +140,22 @@ def optimize_edge(
     launch jitter. ``sync`` is invoked after each run so GPU timing is honest
     (pass a device sync; default no-op for CPU/fake).
     """
+    if isinstance(reps, bool) or not isinstance(reps, int) or reps <= 0:
+        raise ValueError(f"reps must be a positive integer, got {reps!r}")
+    # Validate before executing either leg: gate controls must never be repaired
+    # after the caller supplied them.
+    detections_equivalent(
+        {"frames": []},
+        {"frames": []},
+        center_atol=center_atol,
+        tol_frac=tol_frac,
+    )
     sync = sync or (lambda: None)
 
     def _timed(mode: str) -> tuple[dict, float]:
         best = float("inf")
         summary: dict = {}
-        for _ in range(max(1, reps)):
+        for _ in range(reps):
             t0 = time.perf_counter()
             summary = run_mode(mode)
             sync()
@@ -291,6 +306,8 @@ def edge_batching_spec(batch_size: int = 4) -> InterventionSpec:
     ``batch_size`` is recorded as the spec's ``value`` so the provenance reflects
     the batch size actually run, not a hardcoded constant.
     """
+    if isinstance(batch_size, bool) or not isinstance(batch_size, int) or batch_size <= 0:
+        raise ValueError(f"batch_size must be a positive integer, got {batch_size!r}")
     return InterventionSpec(
         name="edge_frame_batching",
         summary=f"Run inference on batches of {batch_size} frames in one forward "
@@ -346,6 +363,8 @@ def __init__(
         tol_frac: float = 0.05,
         spec: InterventionSpec | None = None,
     ):
+        if isinstance(batch_size, bool) or not isinstance(batch_size, int) or batch_size <= 0:
+            raise ValueError(f"batch_size must be a positive integer, got {batch_size!r}")
         self._run_mode = run_mode
         self._batch_size = batch_size
         self._reps = reps
diff --git a/gitm/benchmarks/hft/optimize.py b/gitm/benchmarks/hft/optimize.py
index 76cd1c5..afb15f4 100644
--- a/gitm/benchmarks/hft/optimize.py
+++ b/gitm/benchmarks/hft/optimize.py
@@ -108,12 +108,14 @@ def optimize_hft(df, dflib, *, reps: int = 3, sync=None) -> ABResult:
     reduce launch-jitter noise. ``sync`` is an optional callable invoked after
     each run so GPU timing is honest (pass a device-sync; default no-op for CPU).
     """
+    if isinstance(reps, bool) or not isinstance(reps, int) or reps <= 0:
+        raise ValueError(f"reps must be a positive integer, got {reps!r}")
     sync = sync or (lambda: None)
 
     def _timed(fn) -> tuple[dict, float]:
         best = float("inf")
         summary: dict = {}
-        for _ in range(max(1, reps)):
+        for _ in range(reps):
             t0 = time.perf_counter()
             summary = fn(df, dflib)
             sync()
diff --git a/tests/test_edge_optimize.py b/tests/test_edge_optimize.py
index 115dd7a..7dacf21 100644
--- a/tests/test_edge_optimize.py
+++ b/tests/test_edge_optimize.py
@@ -69,6 +69,27 @@ def test_detections_equivalent_matches_per_frame():
     assert not detections_equivalent(base, _frames(n_frames=1, n_det=3))
 
 
+def test_zero_detection_tolerance_is_exact():
+    base = _frames(n_frames=1, n_det=3)
+
+    assert not detections_equivalent(base, _frames(n_frames=1, n_det=2), tol_frac=0.0)
+
+
+@pytest.mark.parametrize(
+    ("kwargs", "message"),
+    [
+        ({"reps": 0}, "reps"),
+        ({"reps": -2}, "reps"),
+        ({"tol_frac": -0.1}, "tol_frac"),
+        ({"tol_frac": 1.1}, "tol_frac"),
+        ({"center_atol": float("nan")}, "center_atol"),
+    ],
+)
+def test_optimize_edge_refuses_invalid_gate_controls(kwargs, message):
+    with pytest.raises(ValueError, match=message):
+        optimize_edge(_fake_run_mode(), **kwargs)
+
+
 def test_specs_apply_to_edge_workloads():
     for spec in (edge_intervention_spec(), edge_batching_spec()):
         assert set(spec.applicability.workloads) >= {"edge", "kitti", "nuscenes"}
@@ -76,6 +97,11 @@ def test_specs_apply_to_edge_workloads():
     assert edge_batching_spec().name == "edge_frame_batching"
 
 
+def test_edge_batching_refuses_invalid_batch_size():
+    with pytest.raises(ValueError, match="batch_size"):
+        EdgeBatchingApplicator(_fake_run_mode(), batch_size=0)
+
+
 def test_optimize_edge_keeps_faster_equivalent_candidate():
     r = optimize_edge(_fake_run_mode(equivalent=True), reps=1)
     assert r.identical and r.speedup > 1.0 and r.kept == "candidate"
diff --git a/tests/test_hft_optimize.py b/tests/test_hft_optimize.py
index 947014d..c8990f4 100644
--- a/tests/test_hft_optimize.py
+++ b/tests/test_hft_optimize.py
@@ -10,6 +10,7 @@
 
 import numpy as np
 import pandas as pd
+import pytest
 
 
 def _make_df(n: int = 4000, seed: int = 0):
@@ -41,6 +42,14 @@ def test_optimize_keeps_only_correct_candidate():
     assert r.baseline_eps > 0 and r.candidate_eps > 0
 
 
+@pytest.mark.parametrize("reps", [0, -1, True])
+def test_optimize_refuses_invalid_repetition_count(reps):
+    from gitm.benchmarks.hft.optimize import optimize_hft
+
+    with pytest.raises(ValueError, match="reps"):
+        optimize_hft(_make_df(), pd, reps=reps)
+
+
 def test_optimize_rolls_back_a_divergent_candidate(monkeypatch):
     import gitm.benchmarks.hft.optimize as opt
 

From 69faeb7eac9468a9e5cd8c8c7af1b71340dc8102 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:50:15 -0700
Subject: [PATCH 45/70] fix: refuse contradictory benchmark timing partitions

---
 benchmarks/biotech/harness.py     | 21 ++++++++++++++-------
 benchmarks/edge/harness.py        | 26 +++++++++++++++++++-------
 gitm/_timing.py                   | 30 ++++++++++++++++++++++++++++++
 tests/test_framework_harnesses.py | 28 ++++++++++++++++++++++++++++
 4 files changed, 91 insertions(+), 14 deletions(-)

diff --git a/benchmarks/biotech/harness.py b/benchmarks/biotech/harness.py
index 639b094..77f8a8b 100644
--- a/benchmarks/biotech/harness.py
+++ b/benchmarks/biotech/harness.py
@@ -33,6 +33,7 @@
 from typing import Protocol
 
 from benchmarks.biotech.fetch import FastaRecord, read_fasta
+from gitm._timing import require_positive_duration, require_timing_partition
 
 OPENFOLD_COMMIT = "v1.0.1"  # pinned; weight hashes pinned in datasets.md
 MODEL_NAME = "model_1"      # single-model AF2 monomer; weights = params_model_1.npz
@@ -193,12 +194,16 @@ def _build_stall_phase(timings: list[dict], wall_clock_s: float) -> dict:
     t_feat = sum(t["_t_featurize_s"] for t in timings)
     t_inf = sum(t["_t_inference_s"] for t in timings)
     t_post = sum(t["_t_post_s"] for t in timings)
-    total = max(sum(t["_t_total_s"] for t in timings), 1e-9)
-
-    data_stall = min(1.0, t_feat / total)
-    gpu_active = min(1.0, t_inf / total)
-    sync = min(1.0, t_post / total)
-    cpu = max(0.0, 1.0 - data_stall - gpu_active - sync)
+    total = sum(t["_t_total_s"] for t in timings)
+    split = require_timing_partition(
+        total,
+        {"data_stall": t_feat, "gpu_active": t_inf, "sync": t_post},
+        context="biotech stall breakdown",
+    )
+    data_stall = split["data_stall"]
+    gpu_active = split["gpu_active"]
+    sync = split["sync"]
+    cpu = split["unattributed"]
 
     return {
         "phase": "all",
@@ -246,7 +251,9 @@ def run(
             plddts.append(float(result["plddt"]))
         if "_t_total_s" in result:
             timings.append(result)
-    elapsed = max(time.perf_counter() - t0, 1e-9)
+    elapsed = require_positive_duration(
+        time.perf_counter() - t0, context="biotech harness"
+    )
 
     structures_per_hour = len(proteins) / elapsed * 3600.0
     payload: dict = {
diff --git a/benchmarks/edge/harness.py b/benchmarks/edge/harness.py
index 47b6fd6..2424264 100644
--- a/benchmarks/edge/harness.py
+++ b/benchmarks/edge/harness.py
@@ -33,6 +33,8 @@
 from pathlib import Path
 from typing import Any, Protocol
 
+from gitm._timing import require_positive_duration, require_timing_partition
+
 OPENPCDET_COMMIT = "v0.6.0"  # pinned; config hash pinned in datasets.md
 
 _OPENPCDET_DEFAULT_CFG = (
@@ -231,7 +233,9 @@ def run(stage: Path, *, warm: int, runner: Runner) -> dict:
             maps.append(float(result["map"]))
         if "_t_total_s" in result:
             timings.append(result)
-    elapsed = max(time.perf_counter() - t0, 1e-9)
+    elapsed = require_positive_duration(
+        time.perf_counter() - t0, context="edge harness"
+    )
 
     if n == 0:
         raise RuntimeError(f"no frames in {manifest}")
@@ -259,12 +263,20 @@ def _build_stall_phase(timings: list[dict], wall_clock_s: float) -> dict:
     t_pre = sum(t["_t_preprocess_s"] for t in timings)
     t_inf = sum(t["_t_inference_s"] for t in timings)
     t_post = sum(t["_t_postprocess_s"] for t in timings)
-    total = max(sum(t["_t_total_s"] for t in timings), 1e-9)
-
-    data_stall = min(1.0, (t_load + t_pre) / total)
-    gpu_active = min(1.0, t_inf / total)
-    sync = min(1.0, t_post / total)
-    cpu = max(0.0, 1.0 - data_stall - gpu_active - sync)
+    total = sum(t["_t_total_s"] for t in timings)
+    split = require_timing_partition(
+        total,
+        {
+            "data_stall": t_load + t_pre,
+            "gpu_active": t_inf,
+            "sync": t_post,
+        },
+        context="edge stall breakdown",
+    )
+    data_stall = split["data_stall"]
+    gpu_active = split["gpu_active"]
+    sync = split["sync"]
+    cpu = split["unattributed"]
 
     return {
         "phase": "all",
diff --git a/gitm/_timing.py b/gitm/_timing.py
index 455e922..cdae713 100644
--- a/gitm/_timing.py
+++ b/gitm/_timing.py
@@ -20,3 +20,33 @@ def require_positive_work(value: int | float, *, context: str) -> int | float:
     if not math.isfinite(float(value)) or value <= 0:
         raise RuntimeError(f"{context} work coverage unavailable: expected > 0, got {value!r}")
     return value
+
+
+def require_timing_partition(
+    total_s: float, components_s: dict[str, float], *, context: str
+) -> dict[str, float]:
+    """Return component fractions plus ``unattributed`` or refuse overlap.
+
+    Benchmark stall breakdowns are sign-off evidence. Repairing a zero total or
+    independently clamping overlapping phase timers would turn broken evidence
+    into a plausible partition, so validate the complete partition in one place.
+    """
+    total = require_positive_duration(total_s, context=context)
+    invalid = {
+        name: value
+        for name, value in components_s.items()
+        if not math.isfinite(value) or value < 0.0
+    }
+    if invalid:
+        raise RuntimeError(f"{context} timing unavailable: invalid components {invalid}")
+    assigned = sum(components_s.values())
+    tolerance = max(1e-12, total * 1e-9)
+    if assigned > total + tolerance:
+        detail = ", ".join(f"{name}={value:.6g}s" for name, value in components_s.items())
+        raise RuntimeError(
+            f"{context} timing attribution overlaps: {detail}, total={total:.6g}s; "
+            "refusing to clamp"
+        )
+    fractions = {name: value / total for name, value in components_s.items()}
+    fractions["unattributed"] = max(0.0, total - assigned) / total
+    return fractions
diff --git a/tests/test_framework_harnesses.py b/tests/test_framework_harnesses.py
index 4ee970a..1cf285c 100644
--- a/tests/test_framework_harnesses.py
+++ b/tests/test_framework_harnesses.py
@@ -89,6 +89,20 @@ def test_biotech_loader_raises_without_framework():
         load_openfold_runner(42)
 
 
+@pytest.mark.parametrize(
+    "timings",
+    [
+        [{"_t_featurize_s": 0.0, "_t_inference_s": 0.0, "_t_post_s": 0.0, "_t_total_s": 0.0}],
+        [{"_t_featurize_s": 0.6, "_t_inference_s": 0.6, "_t_post_s": 0.0, "_t_total_s": 1.0}],
+    ],
+)
+def test_biotech_stall_breakdown_refuses_invalid_timing(timings):
+    from benchmarks.biotech.harness import _build_stall_phase
+
+    with pytest.raises(RuntimeError, match="timing"):
+        _build_stall_phase(timings, 1.0)
+
+
 # --- edge -------------------------------------------------------------------
 
 
@@ -159,3 +173,17 @@ def test_edge_loader_raises_without_framework():
 
     with pytest.raises(RuntimeError, match="OpenPCDet"):
         load_openpcdet_runner()
+
+
+@pytest.mark.parametrize(
+    "timings",
+    [
+        [{"_t_load_s": 0.0, "_t_preprocess_s": 0.0, "_t_inference_s": 0.0, "_t_postprocess_s": 0.0, "_t_total_s": 0.0}],
+        [{"_t_load_s": 0.4, "_t_preprocess_s": 0.3, "_t_inference_s": 0.4, "_t_postprocess_s": 0.0, "_t_total_s": 1.0}],
+    ],
+)
+def test_edge_stall_breakdown_refuses_invalid_timing(timings):
+    from benchmarks.edge.harness import _build_stall_phase
+
+    with pytest.raises(RuntimeError, match="timing"):
+        _build_stall_phase(timings, 1.0)

From 9a1b7fdf6410ae9b50930d5573e7c38fe2047c98 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:51:13 -0700
Subject: [PATCH 46/70] fix: refuse invalid OpenFold A/B evidence

---
 benchmarks/biotech/optimize.py  | 23 +++++++++++++++++---
 benchmarks/biotech/sanity.py    |  9 +++++++-
 tests/test_openfold_workload.py | 38 +++++++++++++++++++++++++++++++++
 3 files changed, 66 insertions(+), 4 deletions(-)

diff --git a/benchmarks/biotech/optimize.py b/benchmarks/biotech/optimize.py
index dfdf0d3..65bc925 100644
--- a/benchmarks/biotech/optimize.py
+++ b/benchmarks/biotech/optimize.py
@@ -25,6 +25,7 @@
 
 import argparse
 import json
+import math
 import os
 import statistics
 import time
@@ -33,6 +34,7 @@
 
 from benchmarks.biotech.fetch import read_fasta
 from benchmarks.biotech.harness import _msa_path, load_openfold_runner
+from gitm._timing import require_positive_duration, require_positive_work
 from gitm.kernels.spec import Applicability, InterventionSpec, SafetyGate
 
 
@@ -57,11 +59,17 @@ def _fold_all(runner, proteins) -> tuple[float, float | None]:
     for r, msa in proteins:
         out = runner.predict(r, msa)
         if "plddt" in out:
-            plddts.append(float(out["plddt"]))
+            plddt = float(out["plddt"])
+            if not math.isfinite(plddt):
+                raise RuntimeError(f"OpenFold quality evidence is non-finite: {plddt!r}")
+            plddts.append(plddt)
     if torch.cuda.is_available():
         torch.cuda.synchronize()
-    elapsed = max(time.perf_counter() - t0, 1e-9)
-    sph = len(proteins) / elapsed * 3600.0
+    elapsed = require_positive_duration(
+        time.perf_counter() - t0, context="OpenFold A/B"
+    )
+    count = require_positive_work(len(proteins), context="OpenFold A/B")
+    sph = count / elapsed * 3600.0
     return sph, (statistics.median(plddts) if plddts else None)
 
 
@@ -91,6 +99,15 @@ def verdict(self) -> str:
 def optimize_af2(stage: Path, seed: int, *, n_proteins: int, max_len: int,
                  warmup: int, plddt_tol: float) -> AF2ABResult:
     """Run the fp32-vs-bf16 A/B and return a gated verdict."""
+    for name, value, minimum in (
+        ("n_proteins", n_proteins, 1),
+        ("max_len", max_len, 1),
+        ("warmup", warmup, 0),
+    ):
+        if isinstance(value, bool) or not isinstance(value, int) or value < minimum:
+            raise ValueError(f"{name} must be an integer >= {minimum}, got {value!r}")
+    if not math.isfinite(plddt_tol) or plddt_tol < 0.0:
+        raise ValueError(f"plddt_tol must be finite and non-negative, got {plddt_tol!r}")
     import torch
 
     proteins = _select(stage, max_len=max_len, n=n_proteins)
diff --git a/benchmarks/biotech/sanity.py b/benchmarks/biotech/sanity.py
index 0d241d2..f686d19 100644
--- a/benchmarks/biotech/sanity.py
+++ b/benchmarks/biotech/sanity.py
@@ -27,6 +27,7 @@
     load_openfold_runner,
     select_proteins,
 )
+from gitm._timing import require_positive_duration
 
 
 def _fmt(result: dict) -> str:
@@ -85,7 +86,13 @@ def main(argv: list[str] | None = None) -> int:
     warm = runner.predict(target, msa)
     print(f"  WARM  {_fmt(warm)}")
 
-    speedup = cold["_t_total_s"] / max(warm["_t_total_s"], 1e-9)
+    cold_s = require_positive_duration(
+        float(cold["_t_total_s"]), context="OpenFold cold sanity pass"
+    )
+    warm_s = require_positive_duration(
+        float(warm["_t_total_s"]), context="OpenFold warm sanity pass"
+    )
+    speedup = cold_s / warm_s
     print(f"cold/warm total speedup: {speedup:.2f}x  (kernel cache + allocator warm-up)")
     if abs(cold["plddt"] - warm["plddt"]) > 1.0:
         print(
diff --git a/tests/test_openfold_workload.py b/tests/test_openfold_workload.py
index 32dbdde..e0f3b64 100644
--- a/tests/test_openfold_workload.py
+++ b/tests/test_openfold_workload.py
@@ -12,6 +12,9 @@
 
 from contextlib import contextmanager
 from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
 
 from .conftest import make_kernel, make_trace
 
@@ -25,6 +28,41 @@ def test_openfold_is_registered():
     assert get_factory("alphafold") is not None
 
 
+@pytest.mark.parametrize(
+    ("kwargs", "message"),
+    [
+        ({"n_proteins": 0, "max_len": 384, "warmup": 0, "plddt_tol": 1.0}, "n_proteins"),
+        ({"n_proteins": 1, "max_len": 0, "warmup": 0, "plddt_tol": 1.0}, "max_len"),
+        ({"n_proteins": 1, "max_len": 384, "warmup": -1, "plddt_tol": 1.0}, "warmup"),
+        ({"n_proteins": 1, "max_len": 384, "warmup": 0, "plddt_tol": float("nan")}, "plddt_tol"),
+    ],
+)
+def test_openfold_ab_refuses_invalid_gate_controls(tmp_path, kwargs, message):
+    from benchmarks.biotech.optimize import optimize_af2
+
+    with pytest.raises(ValueError, match=message):
+        optimize_af2(tmp_path, 42, **kwargs)
+
+
+def test_openfold_ab_refuses_zero_duration(monkeypatch):
+    import benchmarks.biotech.optimize as opt
+
+    monkeypatch.setitem(
+        __import__("sys").modules,
+        "torch",
+        SimpleNamespace(cuda=SimpleNamespace(is_available=lambda: False)),
+    )
+    ticks = iter([1.0, 1.0])
+    monkeypatch.setattr(opt.time, "perf_counter", lambda: next(ticks))
+
+    class Runner:
+        def predict(self, _record, _msa):
+            return {"plddt": 90.0}
+
+    with pytest.raises(RuntimeError, match="timing unavailable"):
+        opt._fold_all(Runner(), [(object(), object())])
+
+
 def test_openfold_no_deps_or_data_degrades_to_no_data(tmp_path: Path, monkeypatch):
     """No staged data (and/or no OpenFold) → honest no-data, not a crash or fake."""
     monkeypatch.setenv("GITM_BENCH_STAGE", str(tmp_path / "missing"))

From 0883e80ed5a17c5531692e68e6b5ab43111482e5 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:53:17 -0700
Subject: [PATCH 47/70] fix: refuse contradictory utilization windows

---
 gitm/optimizer/metrics.py | 26 +++++++++++++++++++++++++-
 tests/test_metrics.py     | 24 ++++++++++++++++++++++++
 2 files changed, 49 insertions(+), 1 deletion(-)

diff --git a/gitm/optimizer/metrics.py b/gitm/optimizer/metrics.py
index ba0d700..c0d2bdf 100644
--- a/gitm/optimizer/metrics.py
+++ b/gitm/optimizer/metrics.py
@@ -23,6 +23,7 @@
 
 from __future__ import annotations
 
+import warnings
 from collections.abc import Callable, Iterable
 from dataclasses import dataclass
 
@@ -173,7 +174,30 @@ def compute_metrics(
     wall_s = require_positive_duration(
         trace.duration_ns / 1e9, context=f"{trace.workload_id} utilization metrics"
     )
+    invalid_kernels = [
+        k
+        for k in kernels
+        if k.start_ns < 0 or k.end_ns < k.start_ns or k.end_ns > trace.duration_ns
+    ]
+    if invalid_kernels:
+        raise RuntimeError(
+            f"{trace.workload_id} utilization kernel timing is outside trace window for "
+            f"{len(invalid_kernels)}/{len(kernels)} kernel(s); refusing to clamp busy time"
+        )
+    zero_duration = sum(k.end_ns == k.start_ns for k in kernels)
+    if zero_duration:
+        warnings.warn(
+            f"{trace.workload_id} utilization ignored {zero_duration}/{len(kernels)} "
+            "zero-duration kernel(s)",
+            RuntimeWarning,
+            stacklevel=2,
+        )
     busy_fraction = _merged_busy_ns(kernels) / trace.duration_ns
+    if not 0.0 <= busy_fraction <= 1.0:
+        raise RuntimeError(
+            f"{trace.workload_id} utilization busy fraction is contradictory: "
+            f"{busy_fraction!r}"
+        )
     gaps = _idle_gaps(kernels, trace.duration_ns)
     stall_breakdown = _classify_stalls(gaps, memcpys, syncs, trace.duration_ns)
 
@@ -205,7 +229,7 @@ def compute_metrics(
         n_kernels=len(kernels),
         wall_s=wall_s,
         busy_fraction=busy_fraction,
-        stall_fraction=max(0.0, 1.0 - busy_fraction),
+        stall_fraction=1.0 - busy_fraction,
         stall_breakdown=stall_breakdown,
         achieved_flops_per_s=achieved_flops,
         achieved_bw_bytes_s=achieved_bw,
diff --git a/tests/test_metrics.py b/tests/test_metrics.py
index 455239c..4a99df4 100644
--- a/tests/test_metrics.py
+++ b/tests/test_metrics.py
@@ -71,6 +71,30 @@ def test_refuses_nonpositive_trace_duration():
         compute_metrics(trace, PEAK)
 
 
+@pytest.mark.parametrize(
+    ("start_ns", "end_ns"),
+    [(-1, 10), (20, 10), (50, 250 * US)],
+)
+def test_invalid_kernel_window_refuses_clamped_busy_fraction(start_ns, end_ns):
+    trace = _trace()
+    bad = trace.events[0].model_copy(update={"start_ns": start_ns, "end_ns": end_ns})
+    trace = trace.model_copy(update={"events": [bad, *trace.events[1:]]})
+
+    with pytest.raises(RuntimeError, match="kernel timing.*outside trace window"):
+        compute_metrics(trace, PEAK)
+
+
+def test_zero_duration_kernel_is_warned_not_counted_as_busy():
+    trace = _trace()
+    pad = trace.events[0].model_copy(update={"start_ns": 200 * US, "end_ns": 200 * US})
+    trace = trace.model_copy(update={"events": [*trace.events, pad]})
+
+    with pytest.warns(RuntimeWarning, match="zero-duration kernel"):
+        result = compute_metrics(trace, PEAK)
+
+    assert result.busy_fraction == pytest.approx(0.5)
+
+
 def test_refuses_unpriced_memory_bandwidth():
     peak = HardwarePeak(name="UNKNOWN", peak_flops=1e14, peak_bw_bytes_s=0.0)
     with pytest.raises(RuntimeError, match="peak bandwidth must be positive"):

From 96c9678a3ce33f7af08ad606c5a7a6212591c356 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:55:24 -0700
Subject: [PATCH 48/70] fix: reject invalid scheduler telemetry values

---
 gitm/tracer/vllm_stats.py   | 79 +++++++++++++++++++++++++++++++------
 tests/test_vllm_stats_v1.py | 11 ++++++
 tests/test_vllm_stress.py   | 14 +++++--
 3 files changed, 90 insertions(+), 14 deletions(-)

diff --git a/gitm/tracer/vllm_stats.py b/gitm/tracer/vllm_stats.py
index d9c2640..38e5e41 100644
--- a/gitm/tracer/vllm_stats.py
+++ b/gitm/tracer/vllm_stats.py
@@ -370,6 +370,29 @@ def _schedulers(engine: Any) -> list[Any]:
     return list(sched) if isinstance(sched, list | tuple) else [sched]
 
 
+def _nonnegative_count(
+    value: Any, field_name: str, diagnostics: list[str] | None
+) -> int | None:
+    if isinstance(value, int) and not isinstance(value, bool):
+        if value >= 0:
+            return value
+        if diagnostics is not None:
+            diagnostics.append(f"{field_name} reported negative count {value}")
+    return None
+
+
+def _unit_fraction(
+    value: Any, field_name: str, diagnostics: list[str] | None
+) -> float | None:
+    if isinstance(value, int | float) and not isinstance(value, bool):
+        result = float(value)
+        if math.isfinite(result) and 0.0 <= result <= 1.0:
+            return result
+        if diagnostics is not None:
+            diagnostics.append(f"{field_name} reported {value!r} outside [0, 1]")
+    return None
+
+
 def _v1_scheduler_stats(
     scheduler: Any, diagnostics: list[str] | None = None
 ) -> dict[str, Any]:
@@ -395,12 +418,12 @@ def _v1_scheduler_stats(
     out: dict[str, Any] = {}
     for field_name, attr in (("num_running", "num_running_reqs"),
                              ("num_waiting", "num_waiting_reqs")):
-        v = getattr(stats, attr, None)
-        if isinstance(v, int):
+        v = _nonnegative_count(getattr(stats, attr, None), attr, diagnostics)
+        if v is not None:
             out[field_name] = v
-    ku = getattr(stats, "kv_cache_usage", None)
-    if isinstance(ku, int | float):
-        out["gpu_cache_usage"] = float(ku)
+    ku = _unit_fraction(getattr(stats, "kv_cache_usage", None), "kv_cache_usage", diagnostics)
+    if ku is not None:
+        out["gpu_cache_usage"] = ku
     return out
 
 
@@ -442,7 +465,11 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non
                 ("preemptions_cumulative", "num_cumulative_preemption", False),
             ):
                 raw = getattr(sch, attr, None)
-                val = _len_or_none(raw) if is_len else (raw if isinstance(raw, int) else None)
+                val = (
+                    _len_or_none(raw)
+                    if is_len
+                    else _nonnegative_count(raw, attr, sample.diagnostics)
+                )
                 if val is not None:
                     totals[field_name] = totals.get(field_name, 0) + val
         for field_name, val in totals.items():
@@ -472,8 +499,12 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non
     )
     if callable(getter):
         try:
-            sample.num_unfinished = int(getter())
-            saw_any = True
+            count = _nonnegative_count(
+                getter(), "num_unfinished", sample.diagnostics
+            )
+            if count is not None:
+                sample.num_unfinished = count
+                saw_any = True
         except Exception as exc:
             sample.diagnostics.append(
                 f"unfinished-request probe failed: {type(exc).__name__}: {exc}"
@@ -487,7 +518,13 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non
             # transient over-count (or a partially-exposed config) can never make a
             # half-empty engine read as full and silently suppress under_filled.
             capacity = max_seqs * max(len(schedulers), 1)
-            sample.batch_occupancy = min(1.0, sample.num_running / capacity)
+            if sample.num_running <= capacity:
+                sample.batch_occupancy = sample.num_running / capacity
+            else:
+                sample.diagnostics.append(
+                    f"num_running {sample.num_running} exceeds declared capacity {capacity}; "
+                    "batch occupancy unavailable"
+                )
 
     return sample if saw_any or sample.diagnostics else None
 
@@ -504,7 +541,19 @@ def _gpu_cache_usage(
     total = getattr(bm, "num_total_gpu_blocks", None)
     if callable(free_fn) and isinstance(total, int) and total > 0:
         try:
-            return max(0.0, 1.0 - free_fn() / total)
+            free = free_fn()
+            if (
+                isinstance(free, bool)
+                or not isinstance(free, int | float)
+                or not math.isfinite(float(free))
+                or not 0 <= free <= total
+            ):
+                if diagnostics is not None:
+                    diagnostics.append(
+                        f"KV-cache free-block probe reported {free!r} outside [0, {total}]"
+                    )
+                return None
+            return 1.0 - float(free) / total
         except Exception as exc:
             if diagnostics is not None:
                 diagnostics.append(
@@ -546,6 +595,9 @@ def __init__(self, engine: Any, *, interval_s: float = 0.05) -> None:
     def start(self) -> None:
         if self._thread is not None or self.engine is None:
             return
+        # A stopped sampler may be reused for a new window. Relative timestamps
+        # restart at zero, so old-window samples cannot remain in the same series.
+        self.samples.clear()
         # Two clocks taken together: monotonic for sample spacing (immune to wall
         # clock jumps), wall for the join against request timestamps and the
         # trace. Captured back-to-back so the offset between the two clocks is
@@ -643,7 +695,12 @@ def _vals(attr: str) -> list[float]:
     preempt = _vals("preemptions_cumulative")
     cache = _vals("gpu_cache_usage")
     swapped = _vals("num_swapped")
-    duration_s = max(samples[-1].t_ns - samples[0].t_ns, 0) / 1e9
+    if any(sample.t_ns < 0 for sample in samples) or any(
+        later.t_ns < earlier.t_ns
+        for earlier, later in zip(samples, samples[1:], strict=False)
+    ):
+        raise ValueError("scheduler sample timestamps are not monotonic non-negative values")
+    duration_s = (samples[-1].t_ns - samples[0].t_ns) / 1e9
 
     diagnostics = list(dict.fromkeys(note for sample in samples for note in sample.diagnostics))
     return SchedulerStatsSummary(
diff --git a/tests/test_vllm_stats_v1.py b/tests/test_vllm_stats_v1.py
index 9aa8efd..66ee856 100644
--- a/tests/test_vllm_stats_v1.py
+++ b/tests/test_vllm_stats_v1.py
@@ -67,3 +67,14 @@ def make_stats(self):
         "make_stats failed" in note and "v1 stats moved" in note
         for note in sample.diagnostics
     )
+
+
+def test_v1_invalid_values_are_diagnostic_not_clamped():
+    sample = read_scheduler_stats(_v1_engine(running=300, waiting=-1, cache=1.2))
+
+    assert sample is not None
+    assert sample.num_waiting is None
+    assert sample.gpu_cache_usage is None
+    assert sample.batch_occupancy is None
+    assert any("outside [0, 1]" in note for note in sample.diagnostics)
+    assert any("exceeds declared capacity" in note for note in sample.diagnostics)
diff --git a/tests/test_vllm_stress.py b/tests/test_vllm_stress.py
index 47fa8f4..7c1cca4 100644
--- a/tests/test_vllm_stress.py
+++ b/tests/test_vllm_stress.py
@@ -174,6 +174,13 @@ def get_num_unfinished_requests(self):
     assert any("unfinished-request probe failed" in note for note in sample.diagnostics)
 
 
+def test_reversed_scheduler_sample_timeline_is_refused():
+    from gitm.tracer.vllm_stats import SchedulerSample
+
+    with pytest.raises(ValueError, match="timestamps are not monotonic"):
+        summarize([SchedulerSample(t_ns=10), SchedulerSample(t_ns=5)])
+
+
 def test_summarize_single_sample():
     s = read_scheduler_stats(
         type("E", (), {"scheduler": [type("S", (), {"running": [0, 1], "waiting": [], "swapped": []})()],
@@ -263,14 +270,15 @@ class _E:
 
 
 def test_measure_rolls_back_on_nonpositive_baseline():
-    """An unmeasurable (zero) baseline must roll the candidate back, not keep it."""
+    """An unmeasurable baseline refuses before apply, so no rollback is claimed."""
     class _Idle:
         scheduler_config = _SchedCfg()
 
-    # Baseline probe returns 0 (idle engine), so measure() raises after apply.
+    # Baseline probe returns 0 (idle engine), so snapshot refuses before apply.
     app = LiveEngineApplicator(_Idle(), throughput_fn=lambda e: 0.0, restart_fn=lambda _e, _kv: _Idle())
     res = apply_intervention(_spec("max_num_seqs", 256), app, min_keep_delta=0.0)
-    assert not res.applied and res.rolled_back
+    assert not res.applied and not res.rolled_back
+    assert "intervention not applied" in res.error
 
 
 def test_deviation_multistep_does_not_keep_everything():

From a67c3c3cb1e38e417302ef2ab6226ecc6264a8b0 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 14:56:13 -0700
Subject: [PATCH 49/70] fix: surface unverified CUDA build versions

---
 gitm/serve/vllm.py          | 15 +++++++++++++++
 tests/test_serve_capture.py | 28 ++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/gitm/serve/vllm.py b/gitm/serve/vllm.py
index 60ae321..7ff2f70 100644
--- a/gitm/serve/vllm.py
+++ b/gitm/serve/vllm.py
@@ -253,6 +253,21 @@ def check_driver_stack() -> list[Check]:
     problems = cuda_env.check()
     if problems:
         return [Check("cuda-stack", "fail", "\n".join(str(p) for p in problems))]
+    unverified = []
+    if cuda_env.torch_cuda() is None:
+        unverified.append("PyTorch CUDA build")
+    if cuda_env.vllm_cuda_major() is None:
+        unverified.append("vLLM CUDA build")
+    if unverified:
+        return [
+            Check(
+                "cuda-stack",
+                "warn",
+                f"driver CUDA {driver[0]}.{driver[1]} detected, but "
+                f"{' and '.join(unverified)} could not be verified; "
+                "stack compatibility is unknown",
+            )
+        ]
     return [Check("cuda-stack", "pass", f"driver CUDA {driver[0]}.{driver[1]}, stack consistent")]
 
 
diff --git a/tests/test_serve_capture.py b/tests/test_serve_capture.py
index f9f4123..62215c1 100644
--- a/tests/test_serve_capture.py
+++ b/tests/test_serve_capture.py
@@ -366,3 +366,31 @@ def test_preflight_fails_closed_without_a_gpu():
     start a server that cannot possibly work."""
     dev = sc.Devices(indices=[], count=0, source="nvidia-smi unavailable")
     assert sc.check_gpus(2, dev)[0].status == "fail"
+
+
+def test_driver_stack_does_not_pass_when_build_versions_are_unreadable(monkeypatch):
+    from gitm import cuda_env
+
+    monkeypatch.setattr(cuda_env, "driver_cuda", lambda: (13, 0))
+    monkeypatch.setattr(cuda_env, "stack_for", lambda _driver: object())
+    monkeypatch.setattr(cuda_env, "check", lambda: [])
+    monkeypatch.setattr(cuda_env, "torch_cuda", lambda: None)
+    monkeypatch.setattr(cuda_env, "vllm_cuda_major", lambda: None)
+
+    check = sc.check_driver_stack()[0]
+
+    assert check.status == "warn"
+    assert "PyTorch CUDA build" in check.detail
+    assert "vLLM CUDA build" in check.detail
+
+
+def test_driver_stack_pass_requires_observed_build_versions(monkeypatch):
+    from gitm import cuda_env
+
+    monkeypatch.setattr(cuda_env, "driver_cuda", lambda: (13, 0))
+    monkeypatch.setattr(cuda_env, "stack_for", lambda _driver: object())
+    monkeypatch.setattr(cuda_env, "check", lambda: [])
+    monkeypatch.setattr(cuda_env, "torch_cuda", lambda: (13, 0))
+    monkeypatch.setattr(cuda_env, "vllm_cuda_major", lambda: 13)
+
+    assert sc.check_driver_stack()[0].status == "pass"

From b19a6d8fd651befc228428f1704b8059f8b24035 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:00:26 -0700
Subject: [PATCH 50/70] fix: honor declared bf16 roofline peaks

---
 gitm/planner/roofline.py       | 14 ++++++--------
 tests/test_planner_roofline.py | 16 ++++++++++------
 2 files changed, 16 insertions(+), 14 deletions(-)

diff --git a/gitm/planner/roofline.py b/gitm/planner/roofline.py
index 18b37bb..1da15da 100644
--- a/gitm/planner/roofline.py
+++ b/gitm/planner/roofline.py
@@ -6,9 +6,6 @@
     t_memory  = bytes / peak_mem_bw
     t_pred    = max(t_compute, t_memory)
 
-with a vendor-specific efficiency band ``(eff_lo, eff_hi)``: a kernel within
-that band is "as expected". Residuals outside the band drive attribution.
-
 **The peak must match the op's dtype.** A checkpoint that runs fp8 linears and
 fp4 experts priced against a bf16 peak understates its own ceiling by 2-4x, and
 an understated ceiling reads as recoverable headroom that isn't there — the one
@@ -83,8 +80,6 @@ class HardwareSpec:
     # ``0.0`` means unknown, which makes a sharded graph refuse to guess rather
     # than predict a free all-to-all.
     interconnect_bw_bytes_per_s: float = 0.0
-    eff_lo: float = 0.55
-    eff_hi: float = 0.95
 
 
 @dataclass(frozen=True)
@@ -534,7 +529,6 @@ class BatchConfig:
     """Decode batch shape — prompt length already paid; we predict per-step."""
 
     batch: int = 1
-    prompt_len: int = 128
     kv_cache_len: int = 128  # tokens already in KV-cache when decode starts
     # Multi-token prediction: draft positions proposed per step, and the fraction
     # the verifier keeps. A step costs the drafted work regardless; only accepted
@@ -545,7 +539,7 @@ class BatchConfig:
     def __post_init__(self) -> None:
         if not isinstance(self.batch, int) or isinstance(self.batch, bool) or self.batch <= 0:
             raise ValueError(f"batch must be a positive integer, got {self.batch!r}")
-        for name in ("prompt_len", "kv_cache_len", "speculative_tokens"):
+        for name in ("kv_cache_len", "speculative_tokens"):
             value = getattr(self, name)
             if not isinstance(value, int) or isinstance(value, bool) or value < 0:
                 raise ValueError(f"{name} must be a non-negative integer, got {value!r}")
@@ -609,7 +603,9 @@ def peak_is_fallback(self) -> bool:
 
 def _canon_dtype(dtype: str) -> str:
     d = dtype.lower()
-    if d in ("bf16", "float16", "fp16", "half"):
+    if d in ("bf16", "bfloat16"):
+        return "bf16"
+    if d in ("float16", "fp16", "half"):
         return "fp16"
     if d in ("fp4", "mxfp4", "nvfp4"):
         return "fp4"
@@ -631,6 +627,8 @@ def resolve_peak(hw: HardwareSpec, dtype: str) -> tuple[float, str]:
     d = _canon_dtype(dtype)
     if d == "fp32":
         return hw.peak_flops_fp32_per_s, "fp32"
+    if d == "bf16":
+        return hw.peak_flops_bf16_per_s, "bf16"
     if d == "fp4":
         if hw.peak_flops_fp4_per_s > 0:
             return hw.peak_flops_fp4_per_s, "fp4"
diff --git a/tests/test_planner_roofline.py b/tests/test_planner_roofline.py
index ca42e2c..5e3ea87 100644
--- a/tests/test_planner_roofline.py
+++ b/tests/test_planner_roofline.py
@@ -58,16 +58,20 @@ def test_roofline_a100_compute_bound_reference():
     assert pred.t_pred_s == pytest.approx(expected_t_compute, rel=1e-9)
 
 
-# ── dtype selection: fp16/bf16 share peak; fp32 takes the slower path ───────
+# ── dtype selection: each declared hardware ceiling is honored ──────────────
 
 
-def test_roofline_dtype_selects_fp16_peak_for_bf16():
-    hw = HardwareSpec()
+def test_roofline_dtype_selects_the_explicit_bf16_peak():
+    hw = HardwareSpec(
+        peak_flops_fp16_per_s=100e12,
+        peak_flops_bf16_per_s=200e12,
+    )
     fp16 = roofline("op", flops=1e12, bytes_moved=0, hw=hw, dtype="fp16")
     bf16 = roofline("op", flops=1e12, bytes_moved=0, hw=hw, dtype="bf16")
-    assert fp16.t_compute_s == pytest.approx(bf16.t_compute_s, rel=1e-12)
-    # Both must pick the fp16 peak (312e12), not the fp32 peak (19.5e12).
-    assert fp16.t_compute_s == pytest.approx(1e12 / 312e12, rel=1e-9)
+    assert fp16.t_compute_s == pytest.approx(1e12 / 100e12, rel=1e-9)
+    assert bf16.t_compute_s == pytest.approx(1e12 / 200e12, rel=1e-9)
+    assert bf16.peak_dtype == "bf16"
+    assert not bf16.peak_is_fallback
 
 
 def test_roofline_dtype_fp32_uses_slower_peak():

From ffedbf423992fce97fc6146fe256b5a7648e0217 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:01:24 -0700
Subject: [PATCH 51/70] fix: wire host flamegraph capture

---
 gitm/bench/profile.py | 60 ++++++++++++++++++++++++++++++--------
 tests/test_bench.py   | 68 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 116 insertions(+), 12 deletions(-)

diff --git a/gitm/bench/profile.py b/gitm/bench/profile.py
index 6cd72cf..53d675c 100644
--- a/gitm/bench/profile.py
+++ b/gitm/bench/profile.py
@@ -140,37 +140,73 @@ def run_profile(
     completes so a partial profile is better than none.
     """
     out_dir = Path(out_dir)
+    if not isinstance(host_capture_s, int) or isinstance(host_capture_s, bool) or host_capture_s <= 0:
+        raise ValueError("host_capture_s must be a positive integer")
     out_dir.mkdir(parents=True, exist_ok=True)
     tools = tools or ProfilerTools.detect()
 
     argv, bundle = wrap_command(config, command, out_dir, tools=tools)
 
-    # Host-side samplers run alongside; py-spy attaches to the launched tree.
-    host_procs: list[subprocess.Popen] = []
+    # Launch first so py-spy can attach to this process and its descendants. For
+    # a vendor-wrapped command the root is nsys/rocprof; --subprocesses follows
+    # the actual workload rather than pretending a detected tool captured it.
+    workload = subprocess.Popen(argv)
+    host_procs: list[tuple[str, subprocess.Popen]] = []
+    sar_output = None
     if tools.sar:
         bundle.host_sar = out_dir / "host_sar.log"
-        host_procs.append(
+        sar_output = open(bundle.host_sar, "w")  # noqa: SIM115 - owned until sampler exits
+        host_procs.append((
+            "sar",
             subprocess.Popen(
                 [tools.sar, "-u", "1", str(host_capture_s)],
-                stdout=open(bundle.host_sar, "w"),
+                stdout=sar_output,
                 stderr=subprocess.DEVNULL,
-            )
-        )
+            ),
+        ))
     else:
         bundle.missing.append("sar")
-    if not tools.py_spy:
+    if tools.py_spy:
+        bundle.host_pyspy = out_dir / "host_flamegraph.svg"
+        host_procs.append((
+            "py-spy",
+            subprocess.Popen(
+                [
+                    tools.py_spy,
+                    "record",
+                    "--pid",
+                    str(workload.pid),
+                    "--subprocesses",
+                    "--duration",
+                    str(host_capture_s),
+                    "--output",
+                    str(bundle.host_pyspy),
+                ],
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.DEVNULL,
+            ),
+        ))
+    else:
         bundle.missing.append("py-spy")
 
-    completed = subprocess.run(argv, check=False)
-    if completed.returncode != 0:
-        bundle.missing.append(f"workload command failed (exit {completed.returncode})")
+    workload.wait()
+    if workload.returncode != 0:
+        bundle.missing.append(f"workload command failed (exit {workload.returncode})")
 
-    for hp in host_procs:
+    for name, hp in host_procs:
         try:
             hp.wait(timeout=host_capture_s + 5)
         except subprocess.TimeoutExpired:
             hp.terminate()
-            bundle.missing.append("host sampler timed out")
+            hp.wait(timeout=5)
+            bundle.missing.append(f"{name} sampler timed out")
+        else:
+            if hp.returncode != 0:
+                bundle.missing.append(f"{name} capture failed (exit {hp.returncode})")
+    if sar_output is not None:
+        sar_output.close()
+    if tools.py_spy and bundle.host_pyspy and not bundle.host_pyspy.exists():
+        bundle.missing.append("py-spy capture produced no flamegraph")
 
     if config.vendor == "nvidia" and tools.nsys and bundle.gpu_report:
         bundle.gpu_csv = _export_nsys_csv(tools.nsys, bundle.gpu_report, out_dir)
diff --git a/tests/test_bench.py b/tests/test_bench.py
index caeafe1..d7c2d8e 100644
--- a/tests/test_bench.py
+++ b/tests/test_bench.py
@@ -313,6 +313,74 @@ def test_profile_marks_failed_workload_command(tmp_path):
     assert not bundle.complete
 
 
+def test_profile_launches_pyspy_and_records_its_artifact(monkeypatch, tmp_path):
+    from gitm.bench import profile
+
+    calls = []
+
+    class Proc:
+        def __init__(self, argv, **_kwargs):
+            self.argv = argv
+            self.pid = 4321
+            self.returncode = None
+            calls.append(argv)
+            if argv[0] == "py-spy":
+                output = Path(argv[argv.index("--output") + 1])
+                output.write_text("<svg/>")
+
+        def wait(self, timeout=None):
+            self.returncode = 0
+            return 0
+
+        def terminate(self):
+            self.returncode = -15
+
+    monkeypatch.setattr(profile.subprocess, "Popen", Proc)
+    bundle = profile.run_profile(
+        _hft_config(),
+        ["workload"],
+        tmp_path,
+        host_capture_s=1,
+        tools=profile.ProfilerTools(nsys=None, rocprof=None, py_spy="py-spy", sar=None),
+    )
+
+    pyspy = next(call for call in calls if call[0] == "py-spy")
+    assert pyspy[pyspy.index("--pid") + 1] == "4321"
+    assert "--subprocesses" in pyspy
+    assert bundle.host_pyspy == tmp_path / "host_flamegraph.svg"
+    assert bundle.host_pyspy.exists()
+    assert not any("py-spy" in item for item in bundle.missing)
+
+
+def test_profile_surfaces_failed_pyspy_capture(monkeypatch, tmp_path):
+    from gitm.bench import profile
+
+    class Proc:
+        def __init__(self, argv, **_kwargs):
+            self.argv = argv
+            self.pid = 4321
+            self.returncode = None
+
+        def wait(self, timeout=None):
+            self.returncode = 9 if self.argv[0] == "py-spy" else 0
+            return self.returncode
+
+        def terminate(self):
+            self.returncode = -15
+
+    monkeypatch.setattr(profile.subprocess, "Popen", Proc)
+    bundle = profile.run_profile(
+        _hft_config(),
+        ["workload"],
+        tmp_path,
+        host_capture_s=1,
+        tools=profile.ProfilerTools(nsys=None, rocprof=None, py_spy="py-spy", sar=None),
+    )
+
+    assert any("py-spy capture failed (exit 9)" in item for item in bundle.missing)
+    assert not bundle.complete
+
+
 def test_breakdown_refuses_to_clamp_overlapping_timings():
     from gitm.bench.profile import PhaseTiming, build_breakdown
 

From 6235cdc45bfb474663316980c16a10ee2c882c1d Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:03:14 -0700
Subject: [PATCH 52/70] fix: surface importer cleanup losses

---
 gitm/importers/_common.py     | 27 +++++++++++++++++++++++++++
 gitm/importers/nsys.py        |  4 ++++
 gitm/importers/torch_trace.py |  7 +++++++
 tests/test_importers.py       | 27 ++++++++++++++++++++++++++-
 4 files changed, 64 insertions(+), 1 deletion(-)

diff --git a/gitm/importers/_common.py b/gitm/importers/_common.py
index 07882ea..180cb88 100644
--- a/gitm/importers/_common.py
+++ b/gitm/importers/_common.py
@@ -34,6 +34,27 @@ class ImportStats:
     device_name: str | None = None
 
 
+def merge_normalization_stats(target: ImportStats, children: Iterable[ImportStats]) -> None:
+    """Carry per-device cleanup evidence into the file-level import result."""
+    for child in children:
+        target.deduped += child.deduped
+        target.dropped_invalid += child.dropped_invalid
+        for note in child.warnings:
+            if note.startswith(("deduped ", "dropped ")):
+                continue
+            if note not in target.warnings:
+                target.warnings.append(note)
+    if target.deduped:
+        target.warnings.append(
+            f"deduped {target.deduped} exactly-identical event row(s) across imported devices"
+        )
+    if target.dropped_invalid:
+        target.warnings.append(
+            f"dropped {target.dropped_invalid} event(s) with invalid or non-monotonic "
+            "timestamps across imported devices"
+        )
+
+
 class ImportError(Exception):
     """Per-file import failure with a customer-readable message."""
 
@@ -435,6 +456,12 @@ def finish_trace(
         total_raw_events=len(events),
         per_device_kernel_counts=per_device_kernel_counts(cleaned),
     )
+    if deduped:
+        stats.warnings.append(f"deduped {deduped} exactly-identical event row(s)")
+    if dropped:
+        stats.warnings.append(
+            f"dropped {dropped} event(s) with invalid or non-monotonic timestamps"
+        )
     return trace, stats
 
 
diff --git a/gitm/importers/nsys.py b/gitm/importers/nsys.py
index 27a1af7..b6c1c34 100644
--- a/gitm/importers/nsys.py
+++ b/gitm/importers/nsys.py
@@ -19,6 +19,7 @@
     as_int,
     file_mtime_ns,
     finish_trace,
+    merge_normalization_stats,
 )
 from gitm.tracer.schema import KernelEvent, MemcpyEvent, SyncEvent, Trace, TraceEvent
 
@@ -553,6 +554,7 @@ def import_nsys(
         rid = run_id or f"import-{uuid.uuid4().hex}"
         dcount = meta_count or (max(all_counts.keys()) + 1)
         traces: list[Trace] = []
+        normalization_stats: list[ImportStats] = []
         total_events = 0
         # Pass 2: one device at a time — peak RAM ≈ max(per-device), not sum.
         for dev in device_ids:
@@ -578,6 +580,7 @@ def import_nsys(
                 strict=strict,
             )
             traces.append(trace)
+            normalization_stats.append(_st)
             del dev_events
 
         if not traces:
@@ -592,6 +595,7 @@ def import_nsys(
             per_device_kernel_counts=all_counts,
             total_raw_events=total_events,
         )
+        merge_normalization_stats(stats, normalization_stats)
         if len(device_ids) > 1:
             stats.warnings.append(
                 f"multi-GPU input: analyzing devices {device_ids}; "
diff --git a/gitm/importers/torch_trace.py b/gitm/importers/torch_trace.py
index 13f66d1..fbe913d 100644
--- a/gitm/importers/torch_trace.py
+++ b/gitm/importers/torch_trace.py
@@ -22,6 +22,7 @@
     file_mtime_ns,
     filter_device,
     finish_trace,
+    merge_normalization_stats,
     per_device_kernel_counts,
 )
 from gitm.tracer.schema import MemcpyEvent, Trace, TraceEvent
@@ -698,6 +699,7 @@ def _import_torch_from_event_dicts(
     rid = run_id or f"import-{uuid.uuid4().hex}"
     dcount = device_count_from_events(events)
     traces: list[Trace] = []
+    normalization_stats: list[ImportStats] = []
     for dev in device_ids:
         dev_events = filter_device(events, dev)
         if not dev_events:
@@ -713,6 +715,7 @@ def _import_torch_from_event_dicts(
             strict=strict,
         )
         traces.append(trace)
+        normalization_stats.append(_st)
     if not traces:
         raise ImportError(f"{path.name}: no events left after device filter")
     stats = ImportStats(
@@ -724,6 +727,7 @@ def _import_torch_from_event_dicts(
         per_device_kernel_counts=all_counts,
         total_raw_events=len(events),
     )
+    merge_normalization_stats(stats, normalization_stats)
     _append_launch_metadata_warnings(stats, metadata_fallbacks)
     if len(device_ids) > 1:
         stats.warnings.append(
@@ -861,6 +865,7 @@ def import_torch_trace(
     rid = run_id or f"import-{uuid.uuid4().hex}"
     dcount = max(device_ids) + 1
     traces: list[Trace] = []
+    normalization_stats: list[ImportStats] = []
     total_events = 0
     for dev in device_ids:
         dev_events = buckets.pop(dev)
@@ -876,6 +881,7 @@ def import_torch_trace(
             strict=strict,
         )
         traces.append(trace)
+        normalization_stats.append(_st)
 
     if not traces:
         raise ImportError(f"{path.name}: no events left after device filter")
@@ -889,6 +895,7 @@ def import_torch_trace(
         per_device_kernel_counts=dict(all_counts),
         total_raw_events=total_events,
     )
+    merge_normalization_stats(stats, normalization_stats)
     _append_launch_metadata_warnings(stats, metadata_fallbacks)
     if len(device_ids) > 1:
         stats.warnings.append(
diff --git a/tests/test_importers.py b/tests/test_importers.py
index 4891618..c50a474 100644
--- a/tests/test_importers.py
+++ b/tests/test_importers.py
@@ -352,11 +352,36 @@ def test_dedupe_identical_rows(tmp_path):
     conn.close()
     with pytest.warns(UserWarning, match="deduped"):
         traces, stats = import_nsys(dst, device=0)
-    # dedupe count is on the per-device finish_trace; check events cleaned
+    assert stats.deduped == 1
+    assert any("deduped 1" in note for note in stats.warnings)
     assert traces[0].kernels()
     Trace.model_validate(traces[0].model_dump())
 
 
+def test_invalid_event_drop_reaches_file_level_import_diagnostics(tmp_path):
+    src = FIXTURES / "parity_nsys.sqlite"
+    dst = tmp_path / "invalid.sqlite"
+    dst.write_bytes(src.read_bytes())
+    conn = sqlite3.connect(dst)
+    conn.execute(
+        "UPDATE CUPTI_ACTIVITY_KIND_KERNEL SET end = start - 1 "
+        "WHERE rowid = (SELECT rowid FROM CUPTI_ACTIVITY_KIND_KERNEL LIMIT 1)"
+    )
+    conn.commit()
+    conn.close()
+
+    with pytest.warns(UserWarning, match="dropped 1 event"):
+        traces, stats = import_nsys(dst, device=0)
+
+    assert traces[0].kernels()
+    assert stats.dropped_invalid == 1
+    assert any("dropped 1 event" in note for note in stats.warnings)
+
+    with pytest.warns(UserWarning, match="dropped 1 event"):
+        result = analyze_paths([dst], run_id="invalid-event-report")
+    assert "dropped 1 event" in result.report_md
+
+
 def test_atomic_write(tmp_path):
     out = tmp_path / "report.md"
     analyze_paths(

From 767e6cb347957502ebdcef37a9b6d732e9f51e80 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:04:12 -0700
Subject: [PATCH 53/70] fix: surface every profile bundle artifact

---
 gitm/bench/cli.py   |  3 +++
 tests/test_bench.py | 27 +++++++++++++++++++++++++++
 2 files changed, 30 insertions(+)

diff --git a/gitm/bench/cli.py b/gitm/bench/cli.py
index a4a657f..4cec822 100644
--- a/gitm/bench/cli.py
+++ b/gitm/bench/cli.py
@@ -183,7 +183,10 @@ def _cmd_profile(args) -> int:
             "out_dir": str(bundle.out_dir),
             "complete": bundle.complete,
             "missing_tools": bundle.missing,
+            "gpu_report": str(bundle.gpu_report) if bundle.gpu_report else None,
             "gpu_csv": str(bundle.gpu_csv) if bundle.gpu_csv else None,
+            "host_pyspy": str(bundle.host_pyspy) if bundle.host_pyspy else None,
+            "host_sar": str(bundle.host_sar) if bundle.host_sar else None,
         },
         indent=2,
     ))
diff --git a/tests/test_bench.py b/tests/test_bench.py
index d7c2d8e..4f590b1 100644
--- a/tests/test_bench.py
+++ b/tests/test_bench.py
@@ -381,6 +381,33 @@ def terminate(self):
     assert not bundle.complete
 
 
+def test_profile_cli_prints_every_bundle_artifact(monkeypatch, tmp_path, capsys):
+    from types import SimpleNamespace
+
+    from gitm.bench import cli
+    from gitm.bench.profile import ProfileBundle
+
+    bundle = ProfileBundle(
+        out_dir=tmp_path,
+        gpu_report=tmp_path / "gpu.nsys-rep",
+        gpu_csv=tmp_path / "gpu.csv",
+        host_pyspy=tmp_path / "host.svg",
+        host_sar=tmp_path / "host.log",
+    )
+    monkeypatch.setattr("gitm.bench.schema.BenchConfig.from_toml", lambda _path: object())
+    monkeypatch.setattr("gitm.bench.runner.build_command", lambda _cfg, _seed: ["workload"])
+    monkeypatch.setattr("gitm.bench.profile.run_profile", lambda *_args, **_kwargs: bundle)
+
+    rc = cli._cmd_profile(SimpleNamespace(config=tmp_path / "bench.toml", seed=1, out=tmp_path))
+    payload = json.loads(capsys.readouterr().out)
+
+    assert rc == 0
+    assert payload["gpu_report"] == str(bundle.gpu_report)
+    assert payload["gpu_csv"] == str(bundle.gpu_csv)
+    assert payload["host_pyspy"] == str(bundle.host_pyspy)
+    assert payload["host_sar"] == str(bundle.host_sar)
+
+
 def test_breakdown_refuses_to_clamp_overlapping_timings():
     from gitm.bench.profile import PhaseTiming, build_breakdown
 

From 124575f73abff70cbbefb41d09b9937dbaa991fd Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:07:23 -0700
Subject: [PATCH 54/70] fix: wire fail-open guard into live applies

---
 gitm/optimizer/apply.py      | 111 ++++++++++++++++++++---------------
 tests/test_apply_rollback.py |  12 ++++
 2 files changed, 76 insertions(+), 47 deletions(-)

diff --git a/gitm/optimizer/apply.py b/gitm/optimizer/apply.py
index 5b8a3f8..5c03b95 100644
--- a/gitm/optimizer/apply.py
+++ b/gitm/optimizer/apply.py
@@ -32,6 +32,7 @@
 
 from gitm.kernels.spec import InterventionSpec
 from gitm.optimizer.vllm_knobs import get_knob, knob_kind, set_knob
+from gitm.safety.failopen import FailOpenGuard
 
 if TYPE_CHECKING:
     from gitm.safety.audit import AuditLog
@@ -88,54 +89,70 @@ def apply_intervention(
             error=f"baseline snapshot failed; intervention not applied: {exc}",
         )
 
-    # Step 2: apply. A bad value (validation error) rolls straight back.
-    try:
-        applicator.apply(spec)
-    except Exception as exc:
-        applicator.restore(snapshot)
-        _audit(audit, "revert", spec, cause=f"apply failed, restored: {exc}",
-               knobs=_knob_values(spec))
-        return ApplyResult(False, rolled_back=True, measured_delta=None,
-                           error=f"apply failed, restored: {exc}")
-    _audit(audit, "apply", spec, cause="applied live", knobs=_knob_values(spec))
-
-    # Step 3: measure. A crash mid-measurement also rolls back.
-    try:
-        delta = applicator.measure(spec)
-    except Exception as exc:
-        applicator.restore(snapshot)
-        _audit(audit, "revert", spec, cause=f"measure failed, restored: {exc}",
-               knobs=_knob_values(spec))
-        return ApplyResult(False, rolled_back=True, measured_delta=None,
-                           error=f"measure failed, restored: {exc}")
-
-    if delta is not None and (
-        isinstance(delta, bool)
-        or not isinstance(delta, int | float)
-        or not math.isfinite(float(delta))
-    ):
-        applicator.restore(snapshot)
-        error = f"measurement delta must be finite, got {delta!r}; intervention restored"
-        _audit(audit, "revert", spec, cause=error, knobs=_knob_values(spec))
-        return ApplyResult(False, rolled_back=True, measured_delta=None, error=error)
-
-    # Step 4: keep-or-rollback on the regression threshold.
-    if delta is None:
-        warnings.warn(
-            f"intervention {spec.name!r} was applied without a measurement; "
-            "the change is unverified",
-            RuntimeWarning,
-            stacklevel=2,
+    def _apply_measure_decide() -> ApplyResult:
+        # Step 2: apply. A bad value (validation error) rolls straight back.
+        try:
+            applicator.apply(spec)
+        except Exception as exc:
+            applicator.restore(snapshot)
+            _audit(audit, "revert", spec, cause=f"apply failed, restored: {exc}",
+                   knobs=_knob_values(spec))
+            return ApplyResult(False, rolled_back=True, measured_delta=None,
+                               error=f"apply failed, restored: {exc}")
+        _audit(audit, "apply", spec, cause="applied live", knobs=_knob_values(spec))
+
+        # Step 3: measure. A recoverable crash rolls back immediately. The
+        # surrounding fail-open guard also restores on BaseException/process
+        # exit paths that an Exception handler must not swallow.
+        try:
+            delta = applicator.measure(spec)
+        except Exception as exc:
+            applicator.restore(snapshot)
+            _audit(audit, "revert", spec, cause=f"measure failed, restored: {exc}",
+                   knobs=_knob_values(spec))
+            return ApplyResult(False, rolled_back=True, measured_delta=None,
+                               error=f"measure failed, restored: {exc}")
+
+        if delta is not None and (
+            isinstance(delta, bool)
+            or not isinstance(delta, int | float)
+            or not math.isfinite(float(delta))
+        ):
+            applicator.restore(snapshot)
+            error = f"measurement delta must be finite, got {delta!r}; intervention restored"
+            _audit(audit, "revert", spec, cause=error, knobs=_knob_values(spec))
+            return ApplyResult(False, rolled_back=True, measured_delta=None, error=error)
+
+        # Step 4: keep-or-rollback on the regression threshold.
+        if delta is None:
+            warnings.warn(
+                f"intervention {spec.name!r} was applied without a measurement; "
+                "the change is unverified",
+                RuntimeWarning,
+                stacklevel=2,
+            )
+        if delta is not None and delta < min_keep_delta:
+            applicator.restore(snapshot)
+            _audit(audit, "revert", spec, knobs=_knob_values(spec),
+                   cause=f"regression {delta:+.3f} < keep threshold {min_keep_delta:+.3f}")
+            return ApplyResult(True, rolled_back=True, measured_delta=delta,
+                               error=f"regression {delta:+.3f} < keep threshold "
+                                     f"{min_keep_delta:+.3f}, restored")
+
+        return ApplyResult(True, rolled_back=False, measured_delta=delta)
+
+    with FailOpenGuard(audit=audit) as guard:
+        guard.register(
+            spec.name,
+            lambda: applicator.restore(snapshot),
+            cause="apply gate exited before a keep/rollback decision",
         )
-    if delta is not None and delta < min_keep_delta:
-        applicator.restore(snapshot)
-        _audit(audit, "revert", spec, knobs=_knob_values(spec),
-               cause=f"regression {delta:+.3f} < keep threshold {min_keep_delta:+.3f}")
-        return ApplyResult(True, rolled_back=True, measured_delta=delta,
-                           error=f"regression {delta:+.3f} < keep threshold "
-                                 f"{min_keep_delta:+.3f}, restored")
-
-    return ApplyResult(True, rolled_back=False, measured_delta=delta)
+        result = _apply_measure_decide()
+        # Every ordinary return above either kept the change deliberately or
+        # restored it synchronously. Exceptional exits retain the registration,
+        # so the context restores before propagating the interruption.
+        guard.disarm(spec.name)
+        return result
 
 
 def _audit(
diff --git a/tests/test_apply_rollback.py b/tests/test_apply_rollback.py
index 9de55ec..2d3f0fc 100644
--- a/tests/test_apply_rollback.py
+++ b/tests/test_apply_rollback.py
@@ -92,6 +92,18 @@ def boom(_spec):
     assert cfg == {"block_size": 8}  # restored despite the apply having mutated it
 
 
+def test_failopen_restores_on_baseexception_during_measurement():
+    cfg = {"block_size": 8}
+
+    def interrupt(_spec):
+        raise KeyboardInterrupt
+
+    with pytest.raises(KeyboardInterrupt):
+        apply_intervention(_spec(), DictApplicator(cfg, measure_fn=interrupt))
+
+    assert cfg == {"block_size": 8}
+
+
 # --- regression rollback ----------------------------------------------------
 
 

From d3e0831590fb26f810a31474ee2e7b837bd67e5d Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:07:51 -0700
Subject: [PATCH 55/70] fix: refuse invalid auto-revert evidence

---
 gitm/safety/autorevert.py       | 33 ++++++++++++++++++++++++++-------
 tests/test_safety_primitives.py | 19 +++++++++++++++++++
 2 files changed, 45 insertions(+), 7 deletions(-)

diff --git a/gitm/safety/autorevert.py b/gitm/safety/autorevert.py
index 54958f0..0264838 100644
--- a/gitm/safety/autorevert.py
+++ b/gitm/safety/autorevert.py
@@ -9,6 +9,7 @@
 
 from __future__ import annotations
 
+import math
 from collections import deque
 from dataclasses import dataclass
 
@@ -23,16 +24,34 @@ class AutoRevertDecision:
 
 class AutoRevert:
     def __init__(self, baseline: float, *, tolerance: float = 0.0, window: int = 5) -> None:
-        if baseline <= 0:
-            raise ValueError(f"baseline must be > 0, got {baseline}")
-        if window < 1:
-            raise ValueError(f"window must be >= 1, got {window}")
-        self.baseline = baseline
-        self.tolerance = tolerance  # allowed fractional drop before reverting
+        if (
+            isinstance(baseline, bool)
+            or not isinstance(baseline, int | float)
+            or not math.isfinite(float(baseline))
+            or baseline <= 0
+        ):
+            raise ValueError(f"baseline must be finite and > 0, got {baseline!r}")
+        if (
+            isinstance(tolerance, bool)
+            or not isinstance(tolerance, int | float)
+            or not math.isfinite(float(tolerance))
+            or tolerance < 0
+        ):
+            raise ValueError(f"tolerance must be finite and non-negative, got {tolerance!r}")
+        if isinstance(window, bool) or not isinstance(window, int) or window < 1:
+            raise ValueError(f"window must be a positive integer, got {window!r}")
+        self.baseline = float(baseline)
+        self.tolerance = float(tolerance)  # allowed fractional drop before reverting
         self._w: deque[float] = deque(maxlen=window)
 
     def observe(self, value: float) -> AutoRevertDecision:
-        self._w.append(value)
+        if (
+            isinstance(value, bool)
+            or not isinstance(value, int | float)
+            or not math.isfinite(float(value))
+        ):
+            raise ValueError(f"observation must be a finite number, got {value!r}")
+        self._w.append(float(value))
         if len(self._w) < self._w.maxlen:
             return AutoRevertDecision(False, "warming up (need a full window)")
         mean = sum(self._w) / len(self._w)
diff --git a/tests/test_safety_primitives.py b/tests/test_safety_primitives.py
index 9d8a77f..70541c4 100644
--- a/tests/test_safety_primitives.py
+++ b/tests/test_safety_primitives.py
@@ -106,6 +106,25 @@ def test_autorevert_fires_on_regression():
     assert d.should_revert and d.relative_delta == pytest.approx(-0.1)
 
 
+@pytest.mark.parametrize("baseline", [float("nan"), float("inf"), 0.0, -1.0, True])
+def test_autorevert_refuses_invalid_baseline(baseline):
+    with pytest.raises(ValueError, match="baseline"):
+        AutoRevert(baseline=baseline)
+
+
+@pytest.mark.parametrize("tolerance", [float("nan"), float("inf"), -0.01, True])
+def test_autorevert_refuses_invalid_tolerance(tolerance):
+    with pytest.raises(ValueError, match="tolerance"):
+        AutoRevert(baseline=100.0, tolerance=tolerance)
+
+
+@pytest.mark.parametrize("value", [float("nan"), float("inf"), float("-inf"), True])
+def test_autorevert_refuses_invalid_observation(value):
+    ar = AutoRevert(baseline=100.0)
+    with pytest.raises(ValueError, match="observation"):
+        ar.observe(value)
+
+
 # --------- gated rollout -------------------------------------------------------
 def test_rollout_shadow_then_manual_promote(tmp_path: Path):
     log = AuditLog(tmp_path / "audit.jsonl")

From e8b6c2b03af42621f2c78cbb4e8a76e7587158f2 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:09:37 -0700
Subject: [PATCH 56/70] fix: share unverified CUDA build diagnostics

---
 gitm/cuda_env.py       | 55 ++++++++++++++++++++++++++++++++----------
 gitm/serve/vllm.py     |  6 +----
 tests/test_cuda_env.py | 16 ++++++++++++
 3 files changed, 59 insertions(+), 18 deletions(-)

diff --git a/gitm/cuda_env.py b/gitm/cuda_env.py
index 2696168..ae59993 100644
--- a/gitm/cuda_env.py
+++ b/gitm/cuda_env.py
@@ -25,6 +25,7 @@
 
 import re
 import subprocess
+import warnings
 from dataclasses import dataclass
 from pathlib import Path
 
@@ -223,6 +224,16 @@ def check() -> list[Problem]:
     return problems
 
 
+def unverified_cuda_components() -> list[str]:
+    """CUDA build identities that could not be observed on this installation."""
+    missing: list[str] = []
+    if torch_cuda() is None:
+        missing.append("PyTorch CUDA build")
+    if vllm_cuda_major() is None:
+        missing.append("vLLM CUDA build")
+    return missing
+
+
 def require_compatible() -> None:
     """Raise before an expensive build if the stack can't run on this driver.
     Called by the vLLM workload factory. Without it the failure surfaces ~90s in,
@@ -243,14 +254,22 @@ def require_compatible() -> None:
         )
 
     problems = check()
-    if not problems:
-        return
-    raise RuntimeError(
-        "CUDA stack is incompatible with this host's driver:\n\n"
-        + "\n\n".join(str(p) for p in problems)
-        + "\n\nThe driver belongs to the host and cannot be changed from inside the "
-        "container. Run python -m gitm.cuda_env to re-check."
-    )
+    if problems:
+        raise RuntimeError(
+            "CUDA stack is incompatible with this host's driver:\n\n"
+            + "\n\n".join(str(p) for p in problems)
+            + "\n\nThe driver belongs to the host and cannot be changed from inside the "
+            "container. Run python -m gitm.cuda_env to re-check."
+        )
+    unverified = unverified_cuda_components()
+    if unverified:
+        warnings.warn(
+            f"driver CUDA {driver[0]}.{driver[1]} detected, but "
+            f"{' and '.join(unverified)} could not be verified; "
+            "stack compatibility is unknown",
+            RuntimeWarning,
+            stacklevel=2,
+        )
 
 
 def main(argv: list[str] | None = None) -> int:
@@ -296,13 +315,23 @@ def main(argv: list[str] | None = None) -> int:
     if stack.note:
         print(f"NOTE: {stack.note}")
     problems = check()
-    if not problems:
+    unverified = unverified_cuda_components()
+    if not problems and not unverified:
         print("OK — every component runs on this driver.")
         return 0
-    print()
-    for p in problems:
-        print(p)
-    return 1
+    if problems:
+        print()
+        for p in problems:
+            print(p)
+        return 1
+    if unverified:
+        print(
+            "UNVERIFIED — "
+            + " and ".join(unverified)
+            + " could not be read; stack compatibility is unknown."
+        )
+        return 2
+    raise AssertionError("unreachable CUDA diagnostic state")
 
 
 if __name__ == "__main__":
diff --git a/gitm/serve/vllm.py b/gitm/serve/vllm.py
index 7ff2f70..7d5df4c 100644
--- a/gitm/serve/vllm.py
+++ b/gitm/serve/vllm.py
@@ -253,11 +253,7 @@ def check_driver_stack() -> list[Check]:
     problems = cuda_env.check()
     if problems:
         return [Check("cuda-stack", "fail", "\n".join(str(p) for p in problems))]
-    unverified = []
-    if cuda_env.torch_cuda() is None:
-        unverified.append("PyTorch CUDA build")
-    if cuda_env.vllm_cuda_major() is None:
-        unverified.append("vLLM CUDA build")
+    unverified = cuda_env.unverified_cuda_components()
     if unverified:
         return [
             Check(
diff --git a/tests/test_cuda_env.py b/tests/test_cuda_env.py
index 79ff818..a8b1214 100644
--- a/tests/test_cuda_env.py
+++ b/tests/test_cuda_env.py
@@ -91,6 +91,22 @@ def test_require_compatible_is_silent_when_the_stack_fits(host):
     cuda_env.require_compatible()  # must not raise
 
 
+def test_require_compatible_warns_when_build_versions_are_unverified(host):
+    host(driver=(13, 0), torch=None, vllm=None)
+
+    with pytest.warns(RuntimeWarning, match="PyTorch CUDA build.*vLLM CUDA build"):
+        cuda_env.require_compatible()
+
+
+def test_diagnostic_cli_does_not_claim_ok_for_unverified_builds(host, capsys):
+    host(driver=(13, 0), torch=None, vllm=None)
+
+    assert cuda_env.main([]) == 2
+    output = capsys.readouterr().out
+    assert "UNVERIFIED" in output
+    assert "OK" not in output
+
+
 @pytest.mark.parametrize(
     ("driver", "expected"),
     [

From 81550f9575a83bbc297df3b54f4ccb5330e1dba7 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:10:41 -0700
Subject: [PATCH 57/70] fix: surface malformed GPU trace drops

---
 gitm/importers/torch_trace.py | 34 +++++++++++++++++++++++++++++--
 tests/test_importers.py       | 38 +++++++++++++++++++++++++++++++++++
 2 files changed, 70 insertions(+), 2 deletions(-)

diff --git a/gitm/importers/torch_trace.py b/gitm/importers/torch_trace.py
index fbe913d..de72f1e 100644
--- a/gitm/importers/torch_trace.py
+++ b/gitm/importers/torch_trace.py
@@ -9,6 +9,7 @@
 
 import gzip
 import json
+import math
 import re
 from collections.abc import Iterator
 from pathlib import Path
@@ -242,6 +243,17 @@ def event_from_chrome(
         ts = float(obj.get("ts", 0.0))
         dur = float(obj.get("dur", 0.0))
     except (TypeError, ValueError):
+        if strict:
+            raise ImportError(
+                f"invalid GPU event timestamp/duration: ts={obj.get('ts')!r}, "
+                f"dur={obj.get('dur')!r}"
+            ) from None
+        return None
+    if not math.isfinite(ts) or not math.isfinite(dur):
+        if strict:
+            raise ImportError(
+                f"non-finite GPU event timestamp/duration: ts={ts!r}, dur={dur!r}"
+            )
         return None
     start_ns = int(ts * 1000.0)
     end_ns = int((ts + dur) * 1000.0)
@@ -663,12 +675,15 @@ def _import_torch_from_event_dicts(
     """Shared finish path once raw chrome event dicts are in hand."""
     events: list[TraceEvent] = []
     metadata_fallbacks: dict[str, int] = {}
+    parse_dropped = 0
     for obj in raw_events:
         ev = event_from_chrome(obj, strict=strict)
         if ev is not None:
             events.append(ev)
             for field in _launch_metadata_fallbacks(obj):
                 metadata_fallbacks[field] = metadata_fallbacks.get(field, 0) + 1
+        elif _device_id_from_chrome_obj(obj) is not None:
+            parse_dropped += 1
     if not events:
         raise ImportError(
             f"{path.name}: no complete GPU kernel/memcpy events found in traceEvents"
@@ -725,9 +740,15 @@ def _import_torch_from_event_dicts(
         device_name=device_name,
         captured_at_source="mtime",
         per_device_kernel_counts=all_counts,
-        total_raw_events=len(events),
+        total_raw_events=len(raw_events),
     )
     merge_normalization_stats(stats, normalization_stats)
+    if parse_dropped:
+        stats.dropped_invalid += parse_dropped
+        stats.warnings.append(
+            f"dropped {parse_dropped} GPU event(s) with invalid timestamp/duration fields "
+            "during parsing"
+        )
     _append_launch_metadata_warnings(stats, metadata_fallbacks)
     if len(device_ids) > 1:
         stats.warnings.append(
@@ -815,6 +836,7 @@ def import_torch_trace(
     metadata_fallbacks: dict[str, int] = {}
     scanned_sku: str | None = None
     n_raw = 0
+    parse_dropped = 0
     try:
         for obj in _iter_chrome_event_dicts(
             path, gzipped=bool(gzipped), max_decompressed_bytes=max_decompressed_bytes
@@ -834,6 +856,8 @@ def import_torch_trace(
                     continue
             ev = event_from_chrome(obj, strict=strict)
             if ev is None:
+                if _device_id_from_chrome_obj(obj) is not None:
+                    parse_dropped += 1
                 continue
             for field in _launch_metadata_fallbacks(obj):
                 metadata_fallbacks[field] = metadata_fallbacks.get(field, 0) + 1
@@ -893,9 +917,15 @@ def import_torch_trace(
         device_name=scanned_sku,
         captured_at_source="mtime",
         per_device_kernel_counts=dict(all_counts),
-        total_raw_events=total_events,
+        total_raw_events=n_raw,
     )
     merge_normalization_stats(stats, normalization_stats)
+    if parse_dropped:
+        stats.dropped_invalid += parse_dropped
+        stats.warnings.append(
+            f"dropped {parse_dropped} GPU event(s) with invalid timestamp/duration fields "
+            "during parsing"
+        )
     _append_launch_metadata_warnings(stats, metadata_fallbacks)
     if len(device_ids) > 1:
         stats.warnings.append(
diff --git a/tests/test_importers.py b/tests/test_importers.py
index c50a474..0fbdcaf 100644
--- a/tests/test_importers.py
+++ b/tests/test_importers.py
@@ -382,6 +382,44 @@ def test_invalid_event_drop_reaches_file_level_import_diagnostics(tmp_path):
     assert "dropped 1 event" in result.report_md
 
 
+def test_malformed_gpu_event_parse_drop_reaches_customer_report(tmp_path):
+    trace_path = tmp_path / "malformed-gpu-event.json"
+    trace_path.write_text(
+        json.dumps(
+            {
+                "traceEvents": [
+                    {
+                        "ph": "X",
+                        "cat": "kernel",
+                        "name": "valid_kernel",
+                        "ts": 1.0,
+                        "dur": 100.0,
+                        "args": {"device": 0, "stream": 1},
+                    },
+                    {
+                        "ph": "X",
+                        "cat": "kernel",
+                        "name": "broken_kernel",
+                        "ts": "not-a-timestamp",
+                        "dur": 100.0,
+                        "args": {"device": 0, "stream": 1},
+                    },
+                ]
+            }
+        ),
+        encoding="utf-8",
+    )
+
+    traces, stats = import_torch_trace(trace_path)
+    assert len(traces[0].kernels()) == 1
+    assert stats.total_raw_events == 2
+    assert stats.dropped_invalid == 1
+    assert any("dropped 1 GPU event" in note for note in stats.warnings)
+
+    result = analyze_paths([trace_path], run_id="malformed-gpu-event-report")
+    assert "dropped 1 GPU event" in result.report_md
+
+
 def test_atomic_write(tmp_path):
     out = tmp_path / "report.md"
     analyze_paths(

From e0ca8f8bdb5ad16ceeedbf3ae8545ac4a608ad8a Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:13:53 -0700
Subject: [PATCH 58/70] fix: wire sparse KV sizing into prediction artifacts

---
 gitm/planner/graph.py           | 18 ++++++++++++++++++
 gitm/scheduler/loop.py          |  2 ++
 gitm/serve/attach.py            |  2 ++
 tests/test_moe_graph.py         | 11 +++++++++++
 tests/test_run_loop_workload.py |  2 ++
 tests/test_serve_attach.py      |  4 ++++
 6 files changed, 39 insertions(+)

diff --git a/gitm/planner/graph.py b/gitm/planner/graph.py
index ffe3f3b..6a162f1 100644
--- a/gitm/planner/graph.py
+++ b/gitm/planner/graph.py
@@ -190,6 +190,24 @@ def resident_weight_bytes_is_lower_bound(self) -> bool:
         """True when private DSpark shapes make the footprint a known lower bound."""
         return isinstance(self.model, SparseMoEModelSpec) and bool(self.model.dspark_layer_ids)
 
+    @property
+    def kv_bytes_per_token_per_sequence(self) -> float | None:
+        """Sparse KV footprint that grows with each context token and sequence."""
+        if not isinstance(self.model, SparseMoEModelSpec):
+            return None
+        from gitm.planner.moe_graph import kv_bytes_per_token
+
+        return kv_bytes_per_token(self.model)
+
+    @property
+    def kv_fixed_bytes_per_sequence(self) -> float | None:
+        """Sparse sliding-window KV footprint paid once for each sequence."""
+        if not isinstance(self.model, SparseMoEModelSpec):
+            return None
+        from gitm.planner.moe_graph import kv_fixed_bytes_per_sequence
+
+        return kv_fixed_bytes_per_sequence(self.model)
+
 
 def predict_graph(
     model: ModelSpec | None = None,
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 49d9833..588271c 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -794,6 +794,8 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
                 "resident_weight_bytes_is_lower_bound": (
                     graph.resident_weight_bytes_is_lower_bound
                 ),
+                "kv_bytes_per_token_per_sequence": graph.kv_bytes_per_token_per_sequence,
+                "kv_fixed_bytes_per_sequence": graph.kv_fixed_bytes_per_sequence,
                 "hardware": pctx.sku,
                 "hardware_pricing": graph.hw.name,
                 "hardware_is_fallback": graph.hardware_is_fallback,
diff --git a/gitm/serve/attach.py b/gitm/serve/attach.py
index 311730d..40027f4 100644
--- a/gitm/serve/attach.py
+++ b/gitm/serve/attach.py
@@ -530,6 +530,8 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None:
         "total_pred_s": g.total_pred_s,
         "resident_weight_bytes_per_rank": g.resident_weight_bytes_per_rank,
         "resident_weight_bytes_is_lower_bound": g.resident_weight_bytes_is_lower_bound,
+        "kv_bytes_per_token_per_sequence": g.kv_bytes_per_token_per_sequence,
+        "kv_fixed_bytes_per_sequence": g.kv_fixed_bytes_per_sequence,
         "has_unpriced_collectives": g.has_unpriced_collectives,
         "has_unpriced_nodes": g.has_unpriced_nodes,
         "has_unpriced_compute": g.has_unpriced_compute,
diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py
index ccd2fa6..71a186b 100644
--- a/tests/test_moe_graph.py
+++ b/tests/test_moe_graph.py
@@ -34,6 +34,7 @@
 from gitm.planner.roofline import (
     BatchConfig,
     HardwareSpec,
+    ModelSpec,
     ShardingConfig,
     resolve_peak,
     roofline,
@@ -711,6 +712,10 @@ def test_kv_footprint_splits_growing_from_fixed(spec):
 
     # The two window layers are real, bounded, and paid once per sequence.
     assert fixed == 2 * spec.sliding_window * spec.kv_latent_dim * weight_bytes(spec.kv_dtype)
+
+    graph = predict_moe_graph(spec, HardwareSpec(), BatchConfig())
+    assert graph.kv_bytes_per_token_per_sequence == pytest.approx(per_token)
+    assert graph.kv_fixed_bytes_per_sequence == pytest.approx(fixed)
     # In magnitude the fixed term is tiny — worth ~40 tokens of context, so it
     # never drives a sizing decision. What mattered was excluding these layers
     # from the *rate*, which is a 37% error on every sequence at every length.
@@ -721,6 +726,12 @@ def test_kv_footprint_splits_growing_from_fixed(spec):
     assert naive == pytest.approx(1.37 * per_token, rel=0.02)
 
 
+def test_dense_graph_does_not_invent_sparse_kv_footprint():
+    graph = Graph(model=ModelSpec(), hw=HardwareSpec(), batch=BatchConfig())
+    assert graph.kv_bytes_per_token_per_sequence is None
+    assert graph.kv_fixed_bytes_per_sequence is None
+
+
 def test_b300_headroom_admits_a_full_replica_where_b200_is_marginal(spec):
     """The actual B300 value on this checkpoint: memory, not compute.
 
diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py
index 2b7d188..a8a4f69 100644
--- a/tests/test_run_loop_workload.py
+++ b/tests/test_run_loop_workload.py
@@ -440,6 +440,8 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
     assert payload["coverage"]["warnings"]
     assert predicted["resident_weight_bytes_per_rank"] > 0
     assert predicted["resident_weight_bytes_is_lower_bound"] is False
+    assert predicted["kv_bytes_per_token_per_sequence"] > 0
+    assert predicted["kv_fixed_bytes_per_sequence"] > 0
     assert "## Runtime diagnostics" in result["report_md"]
     assert "matched to the predicted graph" in result["report_md"]
     assert "GPU count was unavailable" in result["report_md"]
diff --git a/tests/test_serve_attach.py b/tests/test_serve_attach.py
index d3d243e..680d0f0 100644
--- a/tests/test_serve_attach.py
+++ b/tests/test_serve_attach.py
@@ -341,6 +341,8 @@ def test_predicted_graph_surfaces_resolved_warnings_and_bytes_fallback(
     assert payload["has_fallback_bytes"] is True
     assert payload["resident_weight_bytes_per_rank"] > 0
     assert payload["resident_weight_bytes_is_lower_bound"] is False
+    assert payload["kv_bytes_per_token_per_sequence"] == 0.0
+    assert payload["kv_fixed_bytes_per_sequence"] > 0
     assert payload["num_gpus_is_fallback"] is True
     assert any("GPU count was unavailable" in warning for warning in payload["warnings"])
     assert any(node["bytes_are_fallback"] for node in payload["nodes"])
@@ -408,6 +410,8 @@ def test_predicted_graph_known_dtypes_leave_bytes_fallback_clean(tmp_path, monke
     assert payload["has_fallback_bytes"] is False
     assert payload["resident_weight_bytes_per_rank"] > 0
     assert payload["resident_weight_bytes_is_lower_bound"] is False
+    assert payload["kv_bytes_per_token_per_sequence"] == 0.0
+    assert payload["kv_fixed_bytes_per_sequence"] > 0
     assert payload["num_gpus_is_fallback"] is False
     assert not any("GPU count was unavailable" in warning for warning in payload["warnings"])
     assert not any(node["bytes_are_fallback"] for node in payload["nodes"])

From cd99784a297a4e47e7095fa8526f555c78554d62 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:22:46 -0700
Subject: [PATCH 59/70] fix: refuse unsupported stream-concurrency evidence

---
 gitm/agents/autoresearch.py     |  6 +++-
 gitm/optimizer/measure.py       | 18 +++++++---
 gitm/optimizer/monitor.py       | 31 +++++++++++-------
 gitm/runtime_driver.py          |  3 +-
 gitm/scheduler/loop.py          | 44 ++++++++++++-------------
 scripts/demo_improve_gpu.py     | 28 ++++++++++++----
 scripts/run_on_real_trace.py    |  3 +-
 tests/test_autoresearch.py      | 21 +++++++++---
 tests/test_openfold_workload.py |  7 +++-
 tests/test_runtime_on_trace.py  | 58 ++++++++++++++++++++++++++++++---
 10 files changed, 163 insertions(+), 56 deletions(-)

diff --git a/gitm/agents/autoresearch.py b/gitm/agents/autoresearch.py
index a10af99..45a7583 100644
--- a/gitm/agents/autoresearch.py
+++ b/gitm/agents/autoresearch.py
@@ -148,11 +148,15 @@ def classify_bottleneck(trace: Trace, residuals: Residuals | None = None) -> str
     gpu_op_ns = kernel_ns + memcpy_ns
     memcpy_frac = memcpy_ns / gpu_op_ns if gpu_op_ns else 0.0
 
-    sc_score = sc / _SC_THRESHOLD
+    sc_score = sc / _SC_THRESHOLD if sc is not None else None
     mem_score = memcpy_frac / _MEMCPY_THRESHOLD
     roofline_frac = _roofline_memory_fraction(residuals)
     if roofline_frac is not None:
         mem_score = max(mem_score, roofline_frac / _MEMCPY_THRESHOLD)
+    if mem_score >= 1.0 and (sc_score is None or mem_score > sc_score):
+        return MEMORY_BOUND
+    if sc_score is None:
+        return UNCLASSIFIED
     if max(sc_score, mem_score) < 1.0:
         return COMPUTE_BOUND
     return IDLE_STALL if sc_score >= mem_score else MEMORY_BOUND  # ties favor idle_stall
diff --git a/gitm/optimizer/measure.py b/gitm/optimizer/measure.py
index d08ec63..3aad921 100644
--- a/gitm/optimizer/measure.py
+++ b/gitm/optimizer/measure.py
@@ -55,7 +55,7 @@ def kernel_family(name: str) -> str:
 class MeasureResult:
     n_kernels: int
     n_memcpy: int
-    serialized_fraction: float
+    serialized_fraction: float | None
     violations: list = field(default_factory=list)
     top_hypotheses: list = field(default_factory=list)
     families: list[str] = field(default_factory=list)
@@ -76,7 +76,7 @@ def measure_trace(trace: Trace, *, min_attr: int = 16) -> MeasureResult:
         return MeasureResult(
             0,
             len(memcpys),
-            0.0,
+            None,
             diagnostics=["measurement coverage unavailable: trace contains no kernels"],
         )
 
@@ -92,12 +92,17 @@ def measure_trace(trace: Trace, *, min_attr: int = 16) -> MeasureResult:
         return MeasureResult(
             len(kernels),
             len(memcpys),
-            0.0,
+            None,
             n_invalid_duration=n_invalid,
             diagnostics=diagnostics,
         )
 
     sc = _serialized_fraction(valid_kernels)
+    if sc is None:
+        diagnostics.append(
+            "stream-concurrency coverage unavailable: trace has no adjacent "
+            "cross-stream kernel pairs"
+        )
     by_name: dict[str, list[int]] = {}
     for k in valid_kernels:
         by_name.setdefault(k.name, []).append(k.end_ns - k.start_ns)
@@ -172,10 +177,15 @@ def measurement_claims(result: MeasureResult, *, limit: int = 5) -> list[Claim]:
 def measurement_summary(workload: str, result: MeasureResult) -> str:
     fams = ", ".join(result.families[:6]) or "none with enough samples"
     diagnostic = f" Coverage diagnostics: {'; '.join(result.diagnostics)}." if result.diagnostics else ""
+    serialized = (
+        f"{result.serialized_fraction:.3f}"
+        if result.serialized_fraction is not None
+        else "unavailable"
+    )
     return (
         f"Measurement run for {workload!r}: {result.n_kernels:,} kernels "
         f"({result.n_memcpy:,} memcpy) captured, {len(result.violations)} invariant "
-        f"deviation(s), serialized-concurrency={result.serialized_fraction:.3f}. "
+        f"deviation(s), serialized-concurrency={serialized}. "
         f"Kernel families: {fams}. No interventions applied — this workload has no "
         f"tuned intervention library, so the runtime reports what it measured.{diagnostic}"
     )
diff --git a/gitm/optimizer/monitor.py b/gitm/optimizer/monitor.py
index 5afb087..ad01b78 100644
--- a/gitm/optimizer/monitor.py
+++ b/gitm/optimizer/monitor.py
@@ -52,7 +52,7 @@ class Residuals:
     """Residuals against predicted graph. Per-kernel + per-stream-set."""
 
     per_kernel: list[KernelResidual] = field(default_factory=list)
-    serialized_concurrency_fraction: float = 0.0
+    serialized_concurrency_fraction: float | None = None
     total_kernels: int = 0
     classified_kernels: int = 0
     matched_kernels: int = 0
@@ -100,6 +100,11 @@ def coverage_warnings(self) -> list[str]:
                 f"{self.unpriced_prediction_kernels} matched kernel(s) because "
                 "their predicted duration is non-positive or non-finite"
             )
+        if self.serialized_concurrency_fraction is None:
+            warnings.append(
+                "stream-concurrency coverage unavailable: trace has no adjacent "
+                "cross-stream kernel pairs"
+            )
         if self.classified_kernels < self.total_kernels:
             warnings.append(
                 "residual coverage: classified "
@@ -260,25 +265,28 @@ def residuals(trace: Trace, graph: Graph) -> Residuals:
     return res
 
 
-def _serialized_fraction(obs: list[KernelEvent]) -> float:
-    """Fraction of adjacent kernel pairs that executed serialized.
+def _serialized_fraction(obs: list[KernelEvent]) -> float | None:
+    """Fraction of observed cross-stream opportunities that did not overlap.
 
-    Sort observed kernels by start time; a consecutive pair is *serialized* when
-    the later kernel starts after the earlier one ends (no temporal overlap)
-    while sharing a stream — concurrency a well-tuned pipeline would have
-    achieved was lost. 0.0 = fully overlapped, 1.0 = fully sequential. Computed
-    from the real trace (stream IDs + ns timestamps), not assumed.
+    Sort kernels by start time and inspect adjacent pairs assigned to different
+    streams. Such a pair is evidence that the runtime intended independent
+    scheduling; it is serialized when the later kernel starts after the earlier
+    one ends. Same-stream ordering is mandatory and therefore says nothing about
+    lost concurrency. ``None`` means the trace exposed no cross-stream
+    opportunity, not a clean zero.
     """
     if len(obs) < 2:
-        return 0.0
+        return None
     s = sorted(obs, key=lambda k: k.start_ns)
     pairs = serialized = 0
     for a, b in zip(s, s[1:], strict=False):
+        if a.stream_id == b.stream_id:
+            continue
         pairs += 1
         overlapped = b.start_ns < a.end_ns
-        if not overlapped and a.stream_id == b.stream_id:
+        if not overlapped:
             serialized += 1
-    return serialized / pairs if pairs else 0.0
+    return serialized / pairs if pairs else None
 
 
 def check_invariants(
@@ -344,6 +352,7 @@ def check_invariants(
 
     if (
         inv_sc is not None
+        and residuals_.serialized_concurrency_fraction is not None
         and residuals_.serialized_concurrency_fraction > inv_sc.band_width * 0.5
     ):
         out.append(
diff --git a/gitm/runtime_driver.py b/gitm/runtime_driver.py
index 20d4580..2b5ad5c 100644
--- a/gitm/runtime_driver.py
+++ b/gitm/runtime_driver.py
@@ -398,8 +398,9 @@ def main(argv: list[str] | None = None) -> int:
     violations = measured.violations
     top_hyps = measured.top_hypotheses
     sc = measured.serialized_fraction
+    sc_text = f"{sc:.3f}" if sc is not None else "unavailable"
     print(
-        f"serialized_concurrency_fraction = {sc:.3f}  |  "
+        f"serialized_concurrency_fraction = {sc_text}  |  "
         f"violations multi-basis={len(violations)}"
     )
     print(f"attribution families (>= 16 samples): {measured.families}")
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 588271c..83b28df 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -1311,8 +1311,14 @@ def _specialized_claim_basis(
     if not math.isfinite(measured_delta):
         diagnostics.append("intervention A/B speedup is non-finite; no claim emitted")
         return None, None, diagnostics
-    if mres.n_kernels > mres.n_invalid_duration:
+    if mres.serialized_fraction is not None:
         return ("stream_concurrency", float(mres.serialized_fraction)), measured_delta, diagnostics
+    if mres.n_kernels > mres.n_invalid_duration:
+        diagnostics.append(
+            "no adjacent cross-stream CUPTI kernel pairs; claim residual uses the "
+            "measured A/B throughput delta instead of fabricated concurrency evidence"
+        )
+        return ("throughput_delta", measured_delta), measured_delta, diagnostics
     diagnostics.append(
         "no positive-duration CUPTI kernels; claim residual uses the measured A/B "
         "throughput delta instead of a fabricated stream-concurrency value"
@@ -1320,6 +1326,12 @@ def _specialized_claim_basis(
     return ("throughput_delta", measured_delta), measured_delta, diagnostics
 
 
+def _serialized_evidence(mres: Any) -> str:
+    if mres.serialized_fraction is None:
+        return "serialized-concurrency unavailable (no adjacent cross-stream pairs)"
+    return f"serialized-concurrency={mres.serialized_fraction:.3f}"
+
+
 def _hft_intervention_result(
     *,
     run_dir: Path,
@@ -1374,13 +1386,10 @@ def _hft_intervention_result(
     if top:
         evidence = (
             f"top hypothesis: {top[0].cause_op[:30]} → {top[0].effect_op[:30]} "
-            f"(p={top[0].p_value:.3g}); serialized-concurrency={mres.serialized_fraction:.3f}"
+            f"(p={top[0].p_value:.3g}); {_serialized_evidence(mres)}"
         )
     elif mres.n_kernels:
-        evidence = (
-            f"serialized-concurrency={mres.serialized_fraction:.3f} over "
-            f"{mres.n_kernels} kernels"
-        )
+        evidence = f"{_serialized_evidence(mres)} over {mres.n_kernels} kernels"
     else:
         evidence = (
             "no CUPTI trace captured on this box; intervention proven by the "
@@ -1446,8 +1455,7 @@ def _hft_intervention_result(
         runtime_diagnostics=runtime_diagnostics,
         summary=(
             f"HFT intervention {spec.name!r}: {verdict}. "
-            f"{mres.n_kernels:,} kernels observed, serialized-concurrency="
-            f"{mres.serialized_fraction:.3f}."
+            f"{mres.n_kernels:,} kernels observed, {_serialized_evidence(mres)}."
         ),
     )
     _write_report(run_dir, report_md)
@@ -1516,13 +1524,10 @@ def _openfold_intervention_result(
     if top:
         evidence = (
             f"top hypothesis: {top[0].cause_op[:30]} → {top[0].effect_op[:30]} "
-            f"(p={top[0].p_value:.3g}); serialized-concurrency={mres.serialized_fraction:.3f}"
+            f"(p={top[0].p_value:.3g}); {_serialized_evidence(mres)}"
         )
     elif mres.n_kernels:
-        evidence = (
-            f"serialized-concurrency={mres.serialized_fraction:.3f} over "
-            f"{mres.n_kernels} kernels"
-        )
+        evidence = f"{_serialized_evidence(mres)} over {mres.n_kernels} kernels"
     else:
         evidence = (
             "no CUPTI trace captured on this box; intervention proven by the "
@@ -1591,8 +1596,7 @@ def _openfold_intervention_result(
         runtime_diagnostics=runtime_diagnostics,
         summary=(
             f"AF2 intervention {spec.name!r}: {verdict}. "
-            f"{mres.n_kernels:,} kernels observed, serialized-concurrency="
-            f"{mres.serialized_fraction:.3f}."
+            f"{mres.n_kernels:,} kernels observed, {_serialized_evidence(mres)}."
         ),
     )
     _write_report(run_dir, report_md)
@@ -1663,13 +1667,10 @@ def _edge_intervention_result(
     if top:
         evidence = (
             f"top hypothesis: {top[0].cause_op[:30]} → {top[0].effect_op[:30]} "
-            f"(p={top[0].p_value:.3g}); serialized-concurrency={mres.serialized_fraction:.3f}"
+            f"(p={top[0].p_value:.3g}); {_serialized_evidence(mres)}"
         )
     elif mres.n_kernels:
-        evidence = (
-            f"serialized-concurrency={mres.serialized_fraction:.3f} over "
-            f"{mres.n_kernels} kernels"
-        )
+        evidence = f"{_serialized_evidence(mres)} over {mres.n_kernels} kernels"
     else:
         evidence = (
             "no CUPTI trace captured on this box; intervention proven by the "
@@ -1736,8 +1737,7 @@ def _edge_intervention_result(
         runtime_diagnostics=runtime_diagnostics,
         summary=(
             f"edge intervention {spec.name!r}: {verdict}. "
-            f"{mres.n_kernels:,} kernels observed, serialized-concurrency="
-            f"{mres.serialized_fraction:.3f}."
+            f"{mres.n_kernels:,} kernels observed, {_serialized_evidence(mres)}."
         ),
     )
     _write_report(run_dir, report_md)
diff --git a/scripts/demo_improve_gpu.py b/scripts/demo_improve_gpu.py
index 5abe5d2..3abc9b5 100644
--- a/scripts/demo_improve_gpu.py
+++ b/scripts/demo_improve_gpu.py
@@ -29,11 +29,11 @@
 from gitm._timing import require_positive_duration
 
 
-def _serialized(trace) -> float:
+def _serialized(trace) -> float | None:
     from gitm.optimizer.monitor import _serialized_fraction
 
     kernels = [e for e in trace.events if e.kind == "kernel"]
-    return _serialized_fraction(kernels) if kernels else 0.0
+    return _serialized_fraction(kernels) if kernels else None
 
 
 def _mean_util(path: Path) -> float | None:
@@ -133,15 +133,21 @@ def parallel():
     # --- 1. OBSERVE -----------------------------------------------------------
     before_res, before = _run_observed(serial, "before", args.outdir)
     util_s = f"{before['util_pct']:.0f}%" if before["util_pct"] is not None else "n/a"
+    serialized_s = (
+        f"{before['serialized']:.3f}" if before["serialized"] is not None else "unavailable"
+    )
     print("1. OBSERVE (serial baseline under the runtime):")
     print("     why: run the work once, untouched, to measure how much of the GPU it actually uses.")
     print(f"     wall-clock {before['elapsed_s'] * 1e3:.2f} ms | util {util_s} | "
-          f"serialized {before['serialized']:.3f} | {before['n_kernels']} kernels\n")
+          f"serialized {serialized_s} | {before['n_kernels']} kernels\n")
 
     # --- 2. DECIDE ------------------------------------------------------------
     if before["util_pct"] is None:
         print("     telemetry unavailable — refusing an idle-GPU claim and intervention decision.")
         return 1
+    if before["serialized"] is None:
+        print("     cross-stream evidence unavailable — refusing a concurrency intervention decision.")
+        return 1
     idle = before["util_pct"] < 85.0
     serial_heavy = before["serialized"] > 0.5
     print("2. DECIDE (runtime maps headroom -> lever):")
@@ -173,15 +179,25 @@ def parallel():
         return 1
 
     util_a = f"{after['util_pct']:.0f}%" if after["util_pct"] is not None else "n/a"
+    serialized_a = (
+        f"{after['serialized']:.3f}" if after["serialized"] is not None else "unavailable"
+    )
     speedup = before["elapsed_s"] / after["elapsed_s"]
     print()
     print("                     BEFORE (serial)     AFTER (parallel)")
     print(f"     wall-clock      {before['elapsed_s'] * 1e3:>10.2f} ms    {after['elapsed_s'] * 1e3:>10.2f} ms")
     print(f"     throughput      {K / before['elapsed_s']:>10.0f} mm/s  {K / after['elapsed_s']:>10.0f} mm/s")
     print(f"     GPU util        {util_s:>12}      {util_a:>12}")
-    print(f"     serialized      {before['serialized']:>12.3f}      {after['serialized']:>12.3f}")
-    print(f"\n  >>> runtime-driven improvement: {speedup:.2f}x faster, "
-          f"serialization {before['serialized']:.2f} -> {after['serialized']:.2f}  (correctness-gated)")
+    print(f"     serialized      {serialized_s:>12}      {serialized_a:>12}")
+    serialization_delta = (
+        f", serialization {before['serialized']:.2f} -> {after['serialized']:.2f}"
+        if after["serialized"] is not None
+        else ", post-change serialization unavailable"
+    )
+    print(
+        f"\n  >>> runtime-driven improvement: {speedup:.2f}x faster{serialization_delta} "
+        " (correctness-gated)"
+    )
     return 0
 
 
diff --git a/scripts/run_on_real_trace.py b/scripts/run_on_real_trace.py
index dba17cc..ee37f21 100644
--- a/scripts/run_on_real_trace.py
+++ b/scripts/run_on_real_trace.py
@@ -67,7 +67,8 @@ def main() -> int:
 
     # Real stream-concurrency computed from the trace stream IDs.
     sc = _serialized_fraction(kernels)
-    print(f"serialized_concurrency_fraction (REAL): {sc:.3f}")
+    sc_text = f"{sc:.3f}" if sc is not None else "unavailable (no cross-stream pairs)"
+    print(f"serialized_concurrency_fraction (REAL): {sc_text}")
 
     # Residual per kernel = deviation from that kernel name's median duration.
     by_name: dict[str, list[int]] = {}
diff --git a/tests/test_autoresearch.py b/tests/test_autoresearch.py
index ea28b25..17911cf 100644
--- a/tests/test_autoresearch.py
+++ b/tests/test_autoresearch.py
@@ -70,14 +70,23 @@ def test_proposed_knobs_are_disjoint_from_catalog() -> None:
 
 
 def test_classify_idle_stall_from_serialized_kernels() -> None:
-    # Back-to-back kernels on one stream, no overlap ⇒ serialized concurrency = 1.
+    # Back-to-back kernels on alternating streams expose independent scheduling
+    # opportunities that failed to overlap.
     events = [
-        make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=0)
+        make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=i % 2)
         for i in range(6)
     ]
     assert classify_bottleneck(make_trace(events=events)) == "idle_stall"
 
 
+def test_classify_single_stream_is_unclassified_without_overlap_opportunity() -> None:
+    events = [
+        make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=0)
+        for i in range(6)
+    ]
+    assert classify_bottleneck(make_trace(events=events)) == ar.UNCLASSIFIED
+
+
 def test_classify_memory_bound_from_memcpy_heavy_trace() -> None:
     # Overlapping kernels (no stall) but memcpys dominate GPU-op time.
     kernels = [
@@ -220,7 +229,7 @@ def test_unknown_class_yields_no_results() -> None:
 
 def test_autoresearch_end_to_end_classifies_and_runs() -> None:
     events = [
-        make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=0)
+        make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=i % 2)
         for i in range(6)
     ]
     config: dict = {}
@@ -566,7 +575,8 @@ def test_fallback_proposer_uses_table_only_when_primary_is_empty() -> None:
 
 def test_autoresearch_end_to_end_with_engineargs_proposer() -> None:
     events = [
-        make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=0) for i in range(6)
+        make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=i % 2)
+        for i in range(6)
     ]  # serialized → idle_stall
     proposer = EngineArgsProposer(
         knobs=[Knob("max_num_partial_prefills", "int", default=1)], catalog_knobs=set()
@@ -832,7 +842,8 @@ def test_classify_always_returns_a_known_class() -> None:
     traces = [
         make_trace(events=[]),  # empty → compute default
         make_trace(events=[
-            make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90) for i in range(6)
+            make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=i % 2)
+            for i in range(6)
         ]),  # serialized → idle
         make_trace(events=[make_kernel("k", start_ns=0, end_ns=1000)]
                    + [make_memcpy(start_ns=i * 10, end_ns=i * 10 + 5) for i in range(4)]),
diff --git a/tests/test_openfold_workload.py b/tests/test_openfold_workload.py
index e0f3b64..4fc4cb9 100644
--- a/tests/test_openfold_workload.py
+++ b/tests/test_openfold_workload.py
@@ -121,7 +121,12 @@ def _fake_capture_with(entered, prefix="cutlass_sm90_gemm"):
     def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None):
         entered["capture"] = True
         kernels = [
-            make_kernel(f"{prefix}_{i % 4}", start_ns=i * 100, end_ns=i * 100 + 90 + (i % 9))
+            make_kernel(
+                f"{prefix}_{i % 4}",
+                start_ns=i * 100,
+                end_ns=i * 100 + 90 + (i % 9),
+                stream_id=i % 2,
+            )
             for i in range(80)
         ]
         yield make_trace(events=kernels, vendor="nvidia", run_id=run_id or "r")
diff --git a/tests/test_runtime_on_trace.py b/tests/test_runtime_on_trace.py
index 4256fb3..ef6ad4f 100644
--- a/tests/test_runtime_on_trace.py
+++ b/tests/test_runtime_on_trace.py
@@ -31,9 +31,17 @@ def _trace(events):
 def test_serialized_fraction_sequential_vs_overlapped():
     from gitm.optimizer.monitor import _serialized_fraction
 
-    # back-to-back on one stream -> fully serialized
+    # Back-to-back work on one stream is mandatory ordering, not evidence that
+    # independent work failed to overlap.
     seq = [_kernel("k", i * 100, i * 100 + 100, stream=7) for i in range(6)]
-    assert _serialized_fraction(seq) == pytest.approx(1.0)
+    assert _serialized_fraction(seq) is None
+
+    # Different streams establish an overlap opportunity; back-to-back execution
+    # means every observed opportunity serialized.
+    cross_stream_seq = [
+        _kernel("k", i * 100, i * 100 + 100, stream=i % 2) for i in range(6)
+    ]
+    assert _serialized_fraction(cross_stream_seq) == pytest.approx(1.0)
 
     # heavily overlapping on different streams -> not serialized
     over = [_kernel("k", 0, 1000, stream=s) for s in range(6)]
@@ -46,7 +54,8 @@ def test_residuals_compute_real_concurrency():
 
     trace = _trace([_kernel("k", i * 100, i * 100 + 100) for i in range(8)])
     res = residuals(trace, predict_graph())
-    assert res.serialized_concurrency_fraction == pytest.approx(1.0)  # all sequential, one stream
+    assert res.serialized_concurrency_fraction is None
+    assert any("cross-stream" in note for note in res.coverage_warnings)
 
 
 # --- op-identity matching (was ordinal `for i in range(min(len(obs), len(pred)))`) --
@@ -120,7 +129,15 @@ def test_fully_matched_residual_coverage_is_clean():
     from gitm.optimizer.monitor import residuals
     from gitm.planner.graph import predict_graph
 
-    res = residuals(_trace([_kernel("flash_attn_kernel", 0, 100)]), predict_graph())
+    res = residuals(
+        _trace(
+            [
+                _kernel("flash_attn_kernel", 0, 100, stream=0),
+                _kernel("flash_attn_kernel", 0, 100, stream=1),
+            ]
+        ),
+        predict_graph(),
+    )
 
     assert res.classification_coverage == 1.0
     assert res.match_coverage == 1.0
@@ -238,9 +255,42 @@ def test_measure_trace_excludes_zero_duration_kernels_with_diagnostic():
     assert result.n_kernels == 3
     assert result.n_invalid_duration == 1
     assert any("excluded 1/3" in note for note in result.diagnostics)
+    assert result.serialized_fraction is None
+    assert any("cross-stream" in note for note in result.diagnostics)
     assert all(v.node_op != "bad" for v in result.violations)
 
 
+def test_measurement_summary_names_unavailable_concurrency():
+    from gitm.optimizer.measure import measure_trace, measurement_summary
+
+    result = measure_trace(
+        _trace([_kernel("same", i * 100, i * 100 + 50, stream=0) for i in range(3)])
+    )
+
+    assert "serialized-concurrency=unavailable" in measurement_summary("demo", result)
+    assert not any(v.invariant == "stream_concurrency" for v in result.violations)
+
+
+def test_specialized_claim_uses_throughput_when_concurrency_is_unavailable():
+    from types import SimpleNamespace
+
+    from gitm.scheduler.loop import _specialized_claim_basis
+
+    basis, delta, diagnostics = _specialized_claim_basis(
+        SimpleNamespace(
+            n_kernels=3,
+            n_invalid_duration=0,
+            serialized_fraction=None,
+            diagnostics=["stream-concurrency coverage unavailable"],
+        ),
+        SimpleNamespace(speedup=1.25),
+    )
+
+    assert basis == ("throughput_delta", pytest.approx(0.25))
+    assert delta == pytest.approx(0.25)
+    assert any("no adjacent cross-stream" in note for note in diagnostics)
+
+
 def test_residuals_exclude_zero_duration_kernels_with_diagnostic():
     from gitm.optimizer.monitor import residuals
     from gitm.planner.graph import predict_graph

From 360aa507dfbbfe10c632a32f83c3ea2b618a15e6 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:23:09 -0700
Subject: [PATCH 60/70] fix: remove unhonored runtime schema fields

---
 gitm/planner/graph.py       |  3 ---
 gitm/planner/kitti_graph.py | 14 ++------------
 gitm/tracer/vllm_stats.py   |  1 -
 3 files changed, 2 insertions(+), 16 deletions(-)

diff --git a/gitm/planner/graph.py b/gitm/planner/graph.py
index 6a162f1..3703f5a 100644
--- a/gitm/planner/graph.py
+++ b/gitm/planner/graph.py
@@ -99,9 +99,6 @@ class PredictedNode:
     op: str
     layer: int | None
     prediction: RooflinePrediction
-    # Streams the planner expects to run on — used by the stream-concurrency
-    # invariant.
-    expected_stream_id: int = 0
 
 
 @dataclass
diff --git a/gitm/planner/kitti_graph.py b/gitm/planner/kitti_graph.py
index d26a894..2d8c818 100644
--- a/gitm/planner/kitti_graph.py
+++ b/gitm/planner/kitti_graph.py
@@ -49,7 +49,6 @@ class KittiNode:
     device: str          # "gpu" | "cpu" | "pcie"
     prediction: RooflinePrediction | None = None
     note: str = ""
-    expected_stream_id: int = 0  # -1 for CPU, 0+ for CUDA streams
 
 
 @dataclass
@@ -81,9 +80,8 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph:
     CPU stages are annotated but not predicted — they depend on the host CPU
     and are measured directly by WorkUnit.
 
-    The stream-concurrency invariant predicts that voxelization (CPU, stream=-1)
-    for frame N+1 should overlap backbone inference (GPU, stream=0) for frame N.
-    The graph makes this expectation explicit via expected_stream_id.
+    CPU/GPU pipeline overlap is measured from the captured workload; this graph
+    models per-stage device placement and roofline cost only.
     """
     hw = hw or HardwareSpec()
     g = KittiGraph(hw=hw)
@@ -93,7 +91,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph:
         name="load_bin",
         device="cpu",
         note="np.fromfile: ~15k points x 4 float32 = ~240 KB I/O. Not roofline-modeled.",
-        expected_stream_id=-1,
     ))
 
     # Stage 2: Voxelization (CPU — scatter 15k points into 12k pillars)
@@ -103,7 +100,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph:
         name="voxelization",
         device="cpu",
         note="Host-side scatter into voxel grid. CPU memory-bound. ~48 MB writes.",
-        expected_stream_id=-1,  # CPU thread — overlaps GPU stream 0 (stream-concurrency invariant)
     ))
 
     # Stage 3: H2D copy (PCIe — pillar features host -> device)
@@ -124,7 +120,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph:
             bound="memory",
         ),
         note=f"pillar features: {h2d_bytes / 1e6:.1f} MB @ ~20 GB/s PCIe",
-        expected_stream_id=0,
     ))
 
     # Stage 4: Pillar Feature Encoder / VFE (GPU)
@@ -142,7 +137,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph:
         device="gpu",
         prediction=roofline("pillar_vfe", vfe_flops, vfe_bytes, hw, dtype="fp32"),
         note="PointPillar feature encoder: linear + BN + ReLU per pillar",
-        expected_stream_id=0,
     ))
 
     # Stage 5: BEV scatter (GPU — pillar features -> spatial BEV grid)
@@ -157,7 +151,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph:
         device="gpu",
         prediction=roofline("bev_scatter", 0, bev_scatter_bytes, hw, dtype="fp32"),
         note="Scatter pillar features -> 2D BEV pseudo-image",
-        expected_stream_id=0,
     ))
 
     # Stage 6: 2D Backbone CNN (GPU)
@@ -173,7 +166,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph:
         device="gpu",
         prediction=roofline("backbone_2d", backbone_flops, backbone_bytes, hw, dtype="fp32"),
         note="Strided 2D CNN: 3 blocks, progressively downsampled BEV features",
-        expected_stream_id=0,
     ))
 
     # Stage 7: Detection head (GPU — anchor-based classification + regression)
@@ -189,7 +181,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph:
         device="gpu",
         prediction=roofline("detection_head", head_flops, head_bytes, hw, dtype="fp32"),
         note="SSD-style anchor cls + box regression, 1x1 conv",
-        expected_stream_id=0,
     ))
 
     # Stage 8: NMS (CPU — CPU-accelerated iou3d_nms_cuda is a separate CUDA kernel
@@ -198,7 +189,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph:
         name="nms",
         device="cpu",
         note="iou3d_nms_cuda: box decode + IoU matrix + suppression. CPU-sync stall.",
-        expected_stream_id=-1,
     ))
 
     return g
diff --git a/gitm/tracer/vllm_stats.py b/gitm/tracer/vllm_stats.py
index 38e5e41..7a374af 100644
--- a/gitm/tracer/vllm_stats.py
+++ b/gitm/tracer/vllm_stats.py
@@ -46,7 +46,6 @@ class SchedulerSample:
     num_unfinished: int | None = None  # total in-flight requests
     preemptions_cumulative: int | None = None  # running total of preemptions
     gpu_cache_usage: float | None = None  # KV-cache blocks used / total, 0..1
-    cpu_cache_usage: float | None = None
     batch_occupancy: float | None = None  # num_running / max_num_seqs, 0..1
     diagnostics: list[str] = field(default_factory=list)
 

From 08a898d09cdda12c37627473c870a3b8c027d2ac Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 15:26:58 -0700
Subject: [PATCH 61/70] fix: refuse inferred interconnect topology

---
 gitm/importers/analyze.py       |  4 +++-
 gitm/optimizer/preconditions.py |  7 ++++---
 gitm/planner/context.py         |  8 ++++++--
 gitm/scheduler/loop.py          |  9 +++++++--
 tests/test_catalog_unify.py     | 11 ++++++++++-
 tests/test_vllm_embodiment.py   | 12 ++++++++++++
 6 files changed, 42 insertions(+), 9 deletions(-)

diff --git a/gitm/importers/analyze.py b/gitm/importers/analyze.py
index aa5a111..d8aaec4 100644
--- a/gitm/importers/analyze.py
+++ b/gitm/importers/analyze.py
@@ -303,7 +303,9 @@ def analyze_paths(
                 hardware=resolved_sku,
                 num_gpus=max(len(device_analyses), 1),
                 has_collective=rollup.has_collective,
-                has_interconnect=rollup.has_collective,  # best-effort: collectives imply interconnect
+                # A collective proves communication occurred, not that its
+                # transport was NVLink/IB rather than PCIe or shared memory.
+                has_interconnect=None,
             )
 
             primary = device_analyses[0]
diff --git a/gitm/optimizer/preconditions.py b/gitm/optimizer/preconditions.py
index 8913bce..f80a9df 100644
--- a/gitm/optimizer/preconditions.py
+++ b/gitm/optimizer/preconditions.py
@@ -15,7 +15,7 @@ class GateContext:
     kv_cache_len: int | None = None
     num_gpus: int = 1
     has_collective: bool = False
-    has_interconnect: bool = False
+    has_interconnect: bool | None = None
 
 
 def applicable(spec: InterventionSpec, ctx: GateContext) -> tuple[bool, str]:
@@ -60,7 +60,8 @@ def applicable(spec: InterventionSpec, ctx: GateContext) -> tuple[bool, str]:
         return False, f"num_gpus {ctx.num_gpus} < min {min_gpus}"
     if getattr(app, "requires_collective", False) and not ctx.has_collective:
         return False, "requires a collective (multi-GPU) but run is single-GPU/no-collective"
-    if getattr(app, "requires_interconnect", False) and not ctx.has_interconnect:
-        return False, "requires interconnect (NVLink/IB) but none reported"
+    if getattr(app, "requires_interconnect", False) and ctx.has_interconnect is not True:
+        state = "topology is unknown" if ctx.has_interconnect is None else "none was reported"
+        return False, f"requires interconnect (NVLink/IB) but {state}"
 
     return True, ""
diff --git a/gitm/planner/context.py b/gitm/planner/context.py
index 08af0cd..1639c18 100644
--- a/gitm/planner/context.py
+++ b/gitm/planner/context.py
@@ -217,6 +217,8 @@ def build_planner_context(
     *,
     workload: str = "vllm-decode",
     num_gpus: int | None = None,
+    has_collective: bool = False,
+    has_interconnect: bool | None = None,
 ) -> PlannerContext:
     """Assemble the gate context + hardware peaks for this run.
 
@@ -249,8 +251,10 @@ def build_planner_context(
         hardware=sku,
         kv_cache_len=kv_len,
         num_gpus=n,
-        has_collective=n > 1,
-        has_interconnect=n > 1,  # refined later by NVLink/IB probe
+        # A device count proves neither fact. Callers may pass trace/probe-backed
+        # evidence; otherwise both remain unavailable/false.
+        has_collective=has_collective,
+        has_interconnect=has_interconnect,
     )
     return PlannerContext(
         gate=gate,
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 83b28df..200260e 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -626,7 +626,8 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
     serving_summary = summarize_requests(req_records) if req_records else None
     # Collective-communication causes from the same trace — ranked beside the
     # scheduler causes below. Empty when the trace holds no collective kernels.
-    coll_causes = collective_causes(worst_device_comm(trace))
+    coll_stats = worst_device_comm(trace)
+    coll_causes = collective_causes(coll_stats)
     if sched_stats.samples or sched_summary.diagnostics or serving_summary is not None:
         (run_dir / "scheduler_stats.json").write_text(
             json.dumps(
@@ -744,7 +745,11 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]:
     # missing/partial live config or unknown SKU refuses graph-based claims and
     # falls through to an honest measurement report; it never becomes a plausible
     # Llama/A100 default prediction.
-    pctx = build_planner_context(cfg.engine, workload=workload)
+    pctx = build_planner_context(
+        cfg.engine,
+        workload=workload,
+        has_collective=coll_stats is not None and coll_stats.comm_ns > 0,
+    )
     graph_resolution = _execution_graph(cfg.engine, pctx, sched_summary)
     if pctx.num_gpus_is_fallback:
         graph_resolution.diagnostics.append(
diff --git a/tests/test_catalog_unify.py b/tests/test_catalog_unify.py
index 4504603..233d5cd 100644
--- a/tests/test_catalog_unify.py
+++ b/tests/test_catalog_unify.py
@@ -26,7 +26,16 @@ def test_requires_collective_and_interconnect():
     assert applicable(spec, ok_ctx)[0]
     no_ic = GateContext(workload="vllm-decode", num_gpus=2, has_collective=True)
     ok, reason = applicable(spec, no_ic)
-    assert not ok and "interconnect" in reason
+    assert not ok and "topology is unknown" in reason
+
+    known_absent = GateContext(
+        workload="vllm-decode",
+        num_gpus=2,
+        has_collective=True,
+        has_interconnect=False,
+    )
+    ok, reason = applicable(spec, known_absent)
+    assert not ok and "none was reported" in reason
 
 
 def test_unified_library_scopes_by_workload():
diff --git a/tests/test_vllm_embodiment.py b/tests/test_vllm_embodiment.py
index b3c1a41..c6a671a 100644
--- a/tests/test_vllm_embodiment.py
+++ b/tests/test_vllm_embodiment.py
@@ -96,6 +96,18 @@ def test_build_planner_context_flags_unknown_gpu_count(monkeypatch):
     assert pctx.num_gpus_is_fallback is True
 
 
+def test_multi_gpu_count_does_not_fabricate_collective_or_interconnect(monkeypatch):
+    import gitm.planner.context as context
+
+    monkeypatch.setattr(context, "_query_nvml", lambda: ("NVIDIA A100-SXM4-80GB", 4))
+
+    pctx = context.build_planner_context()
+
+    assert pctx.num_gpus == 4
+    assert pctx.gate.has_collective is False
+    assert pctx.gate.has_interconnect is None
+
+
 @pytest.mark.parametrize("count", [0, -1])
 def test_build_planner_context_refuses_invalid_explicit_gpu_count(count):
     from gitm.planner.context import build_planner_context

From ad6e6c4b682fdca6ac5588fddcd40a9356b23ca2 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 18:29:55 -0700
Subject: [PATCH 62/70] fix: stop sparse dtype and KV defaults masking pricing

---
 AUDIT.md                   | 27 ++++++++++++++++++++++++++-
 gitm/planner/moe_graph.py  | 15 ++++++++++-----
 gitm/scheduler/loop.py     |  7 ++++++-
 gitm/serve/model_config.py |  4 ++++
 tests/test_moe_graph.py    | 18 ++++++++++++++++++
 5 files changed, 64 insertions(+), 7 deletions(-)

diff --git a/AUDIT.md b/AUDIT.md
index 80def0a..fc44786 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -7,7 +7,8 @@ Status: **in progress**. This ledger is the primary deliverable for the audit of
 turn missing knowledge into a confident wrong result, with answer-deciding byte
 traffic and dominant expert terms ranked above non-binding estimates.
 
-Highest-severity masks closed: **19 so far**. Wiring gaps confirmed: **7 so far**.
+Fallback masks and wiring failures closed: **44 so far**. Wiring gaps confirmed:
+**12 so far**.
 Deferred findings: **none so far**.
 
 The worktree already contained uncommitted scheduler/serve changes and two new
@@ -39,6 +40,30 @@ they will not be silently absorbed into an audit commit.
 | 18 | fixed | high | scheduler specialized HFT/OpenFold/edge intervention result paths | Residual and intervention status | FLAG/REFUSE | No CUPTI trace attached A/B speedup to fabricated `stream_concurrency=0.0`; missing A/B still reported `ok`. | All siblings use measured throughput delta without trace coverage; missing/non-finite A/B emits no claim and returns `intervention_failed`. |
 | 19 | fixed | medium | `gitm/optimizer/headroom_kernel_rank.py` | Compute and memory headroom | FLAG/WARN | Memory-only samples fabricated 100% compute headroom; utilization-only samples fabricated zero memory capacity. | Each dimension is optional, absent families stay `None`, and diagnostics name missing telemetry. |
 | 20 | fixed | medium | `gitm/optimizer/measure.py`; duplicated runtime-driver measurement | Kernel residual denominator | WARN/REFUSE | Zero-duration kernels used a fabricated 1 ns median and attribution filtering had no coverage diagnostic. | Invalid durations are excluded with counts, attribution abstention is diagnostic, and both consumers use canonical measurement. |
+| 21 | fixed | critical | `gitm/deploy/attach.py` | Live deployment state | REFUSE | Resolving a live standalone PID returned `attached` even though the documented injection path is not implemented and no shim was installed. | The path now returns `unsupported` with the PID and a named no-injection reason; only an implemented lifecycle may claim attachment. |
+| 22 | fixed | high | `gitm/cli.py` | Automation/gate status | REFUSE | `prediction_refused`, missing-candidate coverage, intervention failure, and malformed loop results all exited zero; an absent report could also look like a completed run. | Only `status=ok` exits zero. Degraded or invalid results exit 3 and always emit an explicit unavailable/diagnostic report. Report writes are UTF-8. |
+| 23 | fixed | high | `gitm/scheduler/loop.py`; `gitm/agents/autoresearch.py` | Trace evidence and bottleneck class | REFUSE/FLAG | A trace containing only zero-duration events bypassed the no-data gate; an empty/invalid trace was classified `compute`, enabling plausible compute interventions without evidence. | Loop evidence requires a positive-duration kernel, and autoresearch emits `unclassified` for empty/invalid traces with no applicable rules. |
+| 24 | fixed | high | shared timing predicate; workloads, runtime driver, HFT/edge benchmarks, demo script | Throughput, latency, and speedup denominators | REFUSE | Non-positive/non-finite timers were floored to 1 ns, turning timer failure or empty work into enormous plausible throughput/speedup. | Shared finite-positive duration/work predicates now refuse each affected calculation with its named context. |
+| 25 | fixed | high | `gitm/bench/profile.py`; `gitm/bench/cli.py` | Profile completeness and utilization breakdown | REFUSE | Failed workload commands, sampler timeouts, missing profiler CSV, and overlapping timing could still produce a successful profile command with clamped shares. | The bundle records every failure, contradictory timing refuses rather than clamps, and incomplete profile/empty manifest commands exit nonzero. |
+| 26 | fixed | medium | `gitm/agents/autoresearch.py` | Search-domain coverage | WARN | Missing/version-drifted vLLM `EngineArgs` silently substituted a frozen knob catalog; missing CLI introspection silently narrowed domains. | Both offline catalogs now emit named runtime warnings; empty introspection remains explicitly empty rather than pretending live coverage. |
+| 27 | fixed | medium | `gitm/workloads.py`; HFT harness | Benchmark provenance and trace completeness | WARN/REFUSE | Missing data silently generated synthetic smoke input, missing GPU libraries silently selected CPU, and failed synchronization/pool cleanup looked like complete GPU coverage. | Synthetic/CPU/cleanup/synchronization fallbacks warn at use; existing benchmark provenance gates still refuse these runs as publishable GPU baselines. |
+| 28 | fixed | medium | `gitm/safety/failopen.py` | Rollback/audit visibility | WARN/FLAG | A broken audit sink or failed signal-handler setup was swallowed, removing the very evidence meant to surface fail-open behavior. | Audit and signal failures are retained on the guard and warned immediately; reset/restore behavior is covered. |
+| 29 | fixed | high | `gitm/kernels/library.py` | Candidate/intervention coverage | REFUSE | A present but malformed or empty intervention library returned no candidates just like a valid library with no applicable matches. | The loader refuses non-mapping, missing-list, and empty-list libraries with named reasons; scheduler coverage handling remains explicit. |
+| 30 | fixed | medium | `gitm/routing/scorer_v0.py` | Routing score | REFUSE | Out-of-range probabilities, non-binary flags, and unknown company tiers flowed into a plausible score (unknown tier became a default weight). | The scorer validates every bounded/binary input and refuses tiers outside 1/2/3. |
+| 31 | fixed | high | `gitm/importers/node_rollup.py` | Communication share and node ceiling distance | REFUSE | Zero/negative trace wall time was floored to 1 ns and zero-weight ceiling aggregation returned 0.0, fabricating clean rollup values. | Device and node rollups require positive wall time; invalid captures become named per-file analysis failures. |
+| 32 | fixed | medium | `gitm/importers/analyze.py` | Kept-trace artifact wiring | REFUSE/fix | Internal identifiers containing `:` were used as filenames; on Windows the write failed and the whole input was demoted to a generic per-file failure with no trace artifact. | Artifact-only stems are sanitized portably while report identifiers retain their original value; end-to-end importer coverage verifies artifacts are written. |
+| 33 | fixed | high | `gitm/tracer/vllm_stats.py` | TTFT, TPOT, and SLO goodput | REFUSE/WARN | Backwards or non-finite request timestamps were clamped to zero latency and could count as SLO-meeting traffic. | Invalid timestamp spans now produce no latency/goodput contribution and a serving warning naming excluded requests. |
+| 34 | fixed | high | `gitm/tracer/vllm_stats.py`; scheduler artifact/report | Scheduler evidence coverage | WARN/FLAG | Synchronous/background scheduler reads swallowed every exception and returned an ordinary empty summary; a failed sampler looked like an idle/uninstrumented engine. | The fail-open sampler warns once per failure path, retains diagnostics in its summary/artifact, and feeds them into runtime diagnostics. Invalid/clamped intervals also surface. |
+| 35 | fixed | high | NVIDIA telemetry backend and collector | Throttling, process, utilization, memory, power, clocks, and ECC state | FLAG/WARN | Per-field NVML failures became `NONE`, `{}`, `nvidia-unknown-*`, or `None`; especially, unavailable throttle reasons looked like observed no-throttle state. | Partial samples now carry field-level diagnostics; the collector deduplicates, warns, and carries them downstream. Backend close failures propagate to the collector instead of disappearing. |
+| 36 | fixed | high | `gitm/optimizer/monitor.py`; `deviation.py` | Kernel-time residual and deviation population | WARN/FLAG | Zero/negative kernel durations were floored to 1 ps and entered residuals; non-positive predicted durations were likewise made comparable. | Invalid observations are excluded with launch-count diagnostics; unpriced predictions are excluded explicitly; deviation traces retain invalid kernels as departures rather than declaring them in-band. |
+| 37 | fixed | high | `gitm/optimizer/metrics.py` | Busy, HFU/MFU, and MBU denominators | REFUSE | Zero trace wall time or zero peak bandwidth returned plausible 0% utilization. | Shared positive-duration validation and positive hardware-denominator gates now refuse direct callers; normal/importer siblings remain green. |
+| 38 | fixed | high | `gitm/optimizer/apply.py` | Intervention verification and safety trail | WARN | Apply-only changes, broken audit sinks, activation/shutdown hooks, and GPU cleanup failures were silent; a mutation could be kept without evidence or lose rollback telemetry. | Unmeasured applies and every best-effort safety/lifecycle failure now warn while preserving fail-open behavior. |
+| 39 | fixed | critical | dense scheduler graph parser; obsolete parallel parser | Dense activation/weight bytes and hybrid-attention graph shape | REFUSE/fix | Production treated every unknown dense dtype as 2-byte bf16 and ignored hybrid-attention cadence; a more capable engine parser existed only in tests and silently returned `None`. | Production parsing uses shared dtype priceability/byte widths, preserves attention cadence, and names missing/invalid fields. The dead parallel implementation and its fallback tests were removed; coverage targets the production parser. |
+| 40 | fixed | high | `gitm/serve/metrics.py`; attach output | Server throughput, token count, and TPOT | WARN/REFUSE | Infinite Prometheus values entered aggregates, non-positive windows could price throughput, sampler scrape holes disappeared, and missing token/TPOT fields printed as zero. | Non-finite values are discarded, invalid windows emit no rate plus a note, sampler failures surface, and human output prints `unavailable` rather than zero. |
+| 41 | fixed | critical | scheduler default live-engine throughput probe | A/B throughput and keep/rollback decision | REFUSE | A runner with no recognized work-count field silently became one generated token; a failed timer was floored to 1 ns. | The probe requires a named positive work count and positive finite duration; missing/zero evidence rolls the candidate back through the existing apply gate. |
+| 42 | fixed | high | KITTI/nuScenes WorkUnit and baseline runners | Stall shares and FPS | REFUSE | A zero frame timer returned 0% for every stage, and a zero baseline timer divided into FPS. | Frame properties and baseline windows share the positive-duration gate; output writes are UTF-8. |
+| 43 | fixed | medium | `gitm/planner/kitti_graph.py` | Planner wiring and hardware provenance | REFUSE | The PointPillars graph had no production caller and its example defaulted to A100; measured comparisons defaulted missing FPS/stall fields to zero. | Added a SKU-required `gitm plan-kitti` boundary that refuses catalogue misses; measured comparison refuses missing/non-positive fields. |
+| 44 | fixed | medium | diagnostic/demo scripts | GPU-idle decision, real-trace residuals, and assumed hardware | WARN/REFUSE | Demo telemetry failure silently triggered the idle-GPU lever, real-trace code suppressed all warnings and floored zero medians, and serving headroom silently assumed H100/zero failures. | Scripts now refuse telemetry-less idle claims, exclude and warn on invalid timestamps, preserve warnings, and state assumed hardware/missing failure counts explicitly. |
 
 Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`.
 
diff --git a/gitm/planner/moe_graph.py b/gitm/planner/moe_graph.py
index 5fcf8dd..2ce694f 100644
--- a/gitm/planner/moe_graph.py
+++ b/gitm/planner/moe_graph.py
@@ -632,7 +632,11 @@ def spec_from_hf_config(cfg: dict[str, Any], *, name: str | None = None) -> Spar
         raise ValueError("sparse-MoE config is not predictable: " + "; ".join(errors))
 
     q = cfg.get("quantization_config") or {}
-    weight_dtype = str(q.get("quant_method", "bf16")).lower()
+    act_dtype = str(cfg.get("torch_dtype", "bf16")).lower().replace("bfloat16", "bf16")
+    # An unquantized checkpoint stores weights at its declared model dtype. bf16
+    # is not a safe universal default: it halves fp32 bytes and changes the peak
+    # selected for fp16/bf16 on hardware whose rates differ.
+    weight_dtype = str(q.get("quant_method") or act_dtype).lower()
     n_layers = int(cfg.get("num_hidden_layers", 43))
     ratios = tuple(int(r) for r in (cfg.get("compress_ratios") or ())[:n_layers])
 
@@ -662,8 +666,9 @@ def spec_from_hf_config(cfg: dict[str, Any], *, name: str | None = None) -> Spar
         dspark_markov_rank=int(cfg.get("dspark_markov_rank", 0)),
         weight_dtype=weight_dtype,
         expert_dtype=str(cfg.get("expert_dtype", weight_dtype)).lower(),
-        # vLLM serves this checkpoint with an fp8 KV cache; the config does not
-        # declare cache dtype, so it is a serving decision, not a model fact.
-        kv_dtype="fp8",
-        act_dtype=str(cfg.get("torch_dtype", "bf16")).lower().replace("bfloat16", "bf16"),
+        # No cache dtype lives in a model config. The serving default is ``auto``
+        # (follow the compute/model dtype); live callers replace this only when
+        # the engine or command line declares a different cache dtype.
+        kv_dtype=act_dtype,
+        act_dtype=act_dtype,
     )
diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py
index 200260e..0a20dab 100644
--- a/gitm/scheduler/loop.py
+++ b/gitm/scheduler/loop.py
@@ -460,7 +460,12 @@ def _execution_graph(engine: Any, pctx: Any, sched: Any) -> ExecutionGraphResolu
         if kv_dtype:
             changes["kv_dtype"] = str(kv_dtype).lower().replace("fp8_e4m3", "fp8")
         else:
-            diagnostics.append("KV cache dtype was not exposed; using planner default kv_dtype='fp8'")
+            resolved_act = changes.get("act_dtype", spec.act_dtype)
+            changes["kv_dtype"] = resolved_act
+            diagnostics.append(
+                "KV cache dtype was not exposed; assuming vLLM default 'auto' follows "
+                f"activation dtype {resolved_act!r}"
+            )
         if cfg.get("expert_dtype") is None:
             diagnostics.append(
                 f"expert_dtype absent; inherited weight_dtype={spec.weight_dtype!r} for expert bytes"
diff --git a/gitm/serve/model_config.py b/gitm/serve/model_config.py
index 2045da3..07a20f4 100644
--- a/gitm/serve/model_config.py
+++ b/gitm/serve/model_config.py
@@ -450,6 +450,10 @@ def live_moe_spec(
         spec_changes["kv_dtype"] = overrides["kv_dtype"]
     if "act_dtype" in overrides:
         spec_changes["act_dtype"] = overrides["act_dtype"]
+        if "kv_dtype" not in overrides:
+            # vLLM's absent --kv-cache-dtype means ``auto``: follow the resolved
+            # compute dtype, not a model-family-specific fp8 guess.
+            spec_changes["kv_dtype"] = overrides["act_dtype"]
     if spec_changes:
         from dataclasses import replace
 
diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py
index 71a186b..5a9fcda 100644
--- a/tests/test_moe_graph.py
+++ b/tests/test_moe_graph.py
@@ -328,6 +328,24 @@ def test_config_parse_keeps_expert_and_linear_dtypes_distinct(spec):
     assert spec.act_dtype == "bf16"
 
 
+def test_unquantized_config_uses_declared_model_dtype_for_weights_and_kv():
+    cfg = dict(V4_CONFIG)
+    cfg.pop("quantization_config")
+    cfg["torch_dtype"] = "float32"
+    cfg["expert_dtype"] = "fp32"
+
+    parsed = spec_from_hf_config(cfg)
+
+    assert parsed.weight_dtype == "fp32"
+    assert parsed.expert_dtype == "fp32"
+    assert parsed.act_dtype == "fp32"
+    assert parsed.kv_dtype == "fp32"
+
+
+def test_config_without_serving_override_does_not_invent_fp8_kv(spec):
+    assert spec.kv_dtype == spec.act_dtype == "bf16"
+
+
 def test_compress_ratios_truncate_to_layer_count(spec):
     """The config ships 46 ratios for 43 layers.
 

From e7c422cdabbc840bd68e30d2e6a1f9da0e71c536 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 18:44:34 -0700
Subject: [PATCH 63/70] fix: surface importer decoder and verification
 fallbacks

---
 gitm/importers/nsys.py        | 46 ++++++++++++++++++++++++++++-----
 gitm/runtime_driver.py        | 22 +++++++++++++---
 gitm/tracer/_cupti_decode.py  | 48 +++++++++++++++++++++++++++++++----
 scripts/compare_results.py    | 35 ++++++++++++++++++++-----
 tests/test_compare_results.py | 27 ++++++++++++++++++++
 tests/test_cupti.py           | 24 ++++++++++++++++++
 tests/test_importers.py       | 21 +++++++++++++++
 tests/test_runtime_driver.py  | 23 +++++++++++++++++
 8 files changed, 223 insertions(+), 23 deletions(-)
 create mode 100644 tests/test_compare_results.py

diff --git a/gitm/importers/nsys.py b/gitm/importers/nsys.py
index b6c1c34..2445494 100644
--- a/gitm/importers/nsys.py
+++ b/gitm/importers/nsys.py
@@ -267,24 +267,37 @@ def _resolve_name(row: sqlite3.Row, strings: dict[int, str]) -> str:
     return "unknown_kernel"
 
 
-def _map_memory_kind(val: Any, *, strict: bool) -> str:
+def _diag_once(diagnostics: list[str] | None, message: str) -> None:
+    if diagnostics is not None and message not in diagnostics:
+        diagnostics.append(message)
+
+
+def _map_memory_kind(
+    val: Any, *, strict: bool, diagnostics: list[str] | None = None
+) -> str:
     if val is None:
+        _diag_once(diagnostics, "memcpy memory-kind metadata missing; assumed device endpoint")
         return "device"
     try:
         iv = int(val)
     except (TypeError, ValueError):
         if strict:
             raise ImportError(f"unknown memory kind {val!r}") from None
+        _diag_once(diagnostics, f"unknown memory kind {val!r}; assumed device endpoint")
         return "device"
     if iv in _MEMORY_KIND:
         return _MEMORY_KIND[iv]
     if strict:
         raise ImportError(f"unknown CUPTI memory kind enum {iv}")
+    _diag_once(diagnostics, f"unknown CUPTI memory kind enum {iv}; assumed device endpoint")
     return "device"
 
 
-def _map_sync_type(val: Any, *, strict: bool) -> str:
+def _map_sync_type(
+    val: Any, *, strict: bool, diagnostics: list[str] | None = None
+) -> str:
     if val is None:
+        _diag_once(diagnostics, "synchronization type metadata missing; assumed stream sync")
         return "stream"
     try:
         iv = int(val)
@@ -298,11 +311,13 @@ def _map_sync_type(val: Any, *, strict: bool) -> str:
             return "stream"
         if strict:
             raise ImportError(f"unknown sync type {val!r}") from None
+        _diag_once(diagnostics, f"unknown sync type {val!r}; assumed stream sync")
         return "stream"
     if iv in _SYNC_TYPE:
         return _SYNC_TYPE[iv]
     if strict:
         raise ImportError(f"unknown CUPTI sync type enum {iv}")
+    _diag_once(diagnostics, f"unknown CUPTI sync type enum {iv}; assumed stream sync")
     return "stream"
 
 
@@ -358,6 +373,7 @@ def _iter_memcpys(
     *,
     strict: bool,
     device_id: int | None = None,
+    diagnostics: list[str] | None = None,
 ) -> Iterator[MemcpyEvent]:
     from gitm.importers._common import make_memcpy_fast
 
@@ -377,14 +393,15 @@ def _iter_memcpys(
         dst_k = _pick(row, "dstKind", "destinationKind")
         copy_k = _pick(row, "copyKind", "memcpyKind")
         if src_k is not None or dst_k is not None:
-            src = _map_memory_kind(src_k, strict=strict)
-            dst = _map_memory_kind(dst_k, strict=strict)
+            src = _map_memory_kind(src_k, strict=strict, diagnostics=diagnostics)
+            dst = _map_memory_kind(dst_k, strict=strict, diagnostics=diagnostics)
         elif copy_k is not None:
             try:
                 iv = int(copy_k)
             except (TypeError, ValueError):
                 if strict:
                     raise ImportError(f"unknown copyKind {copy_k!r}") from None
+                _diag_once(diagnostics, f"unknown copyKind {copy_k!r}; assumed device↔device")
                 src, dst = "device", "device"
             else:
                 if iv in _COPY_KIND_TO_ENDPOINTS:
@@ -392,8 +409,10 @@ def _iter_memcpys(
                 elif strict:
                     raise ImportError(f"unknown CUPTI copyKind enum {iv}")
                 else:
+                    _diag_once(diagnostics, f"unknown CUPTI copyKind enum {iv}; assumed device↔device")
                     src, dst = "device", "device"
         else:
+            _diag_once(diagnostics, "memcpy endpoint metadata missing; assumed device↔device")
             src, dst = "device", "device"
         corr = _pick(row, "correlationId", "correlation")
         dev = as_int(_pick(row, "deviceId", "device"), 0)
@@ -417,6 +436,7 @@ def _iter_syncs(
     *,
     strict: bool,
     device_id: int | None = None,
+    diagnostics: list[str] | None = None,
 ) -> Iterator[SyncEvent]:
     from gitm.importers._common import make_sync_fast
 
@@ -442,7 +462,11 @@ def _iter_syncs(
             stream_id=as_int(_pick(row, "streamId", "stream"), 0),
             device_id=dev,
             correlation_id=as_int(corr) if corr is not None else None,
-            sync_kind=_map_sync_type(_pick(row, "syncType", "typeOfSync", "type"), strict=strict),
+            sync_kind=_map_sync_type(
+                _pick(row, "syncType", "typeOfSync", "type"),
+                strict=strict,
+                diagnostics=diagnostics,
+            ),
             strict=strict,
         )
 
@@ -556,6 +580,7 @@ def import_nsys(
         traces: list[Trace] = []
         normalization_stats: list[ImportStats] = []
         total_events = 0
+        import_diagnostics: list[str] = []
         # Pass 2: one device at a time — peak RAM ≈ max(per-device), not sum.
         for dev in device_ids:
             dev_events: list[TraceEvent] = []
@@ -564,8 +589,14 @@ def import_nsys(
                     conn, kernel_table, strings, device_id=dev, strict=strict
                 )
             )
-            dev_events.extend(_iter_memcpys(conn, strict=strict, device_id=dev))
-            dev_events.extend(_iter_syncs(conn, strict=strict, device_id=dev))
+            dev_events.extend(
+                _iter_memcpys(
+                    conn, strict=strict, device_id=dev, diagnostics=import_diagnostics
+                )
+            )
+            dev_events.extend(
+                _iter_syncs(conn, strict=strict, device_id=dev, diagnostics=import_diagnostics)
+            )
             if not dev_events:
                 continue
             total_events += len(dev_events)
@@ -596,6 +627,7 @@ def import_nsys(
             total_raw_events=total_events,
         )
         merge_normalization_stats(stats, normalization_stats)
+        stats.warnings.extend(import_diagnostics)
         if len(device_ids) > 1:
             stats.warnings.append(
                 f"multi-GPU input: analyzing devices {device_ids}; "
diff --git a/gitm/runtime_driver.py b/gitm/runtime_driver.py
index 2b5ad5c..62a9ca2 100644
--- a/gitm/runtime_driver.py
+++ b/gitm/runtime_driver.py
@@ -32,7 +32,17 @@
 from contextlib import closing
 from pathlib import Path
 
-from gitm._timing import require_positive_duration
+from gitm._timing import require_positive_duration, require_positive_work
+
+
+def _work_units(summary: dict, workload: str) -> int | float:
+    """Return the workload's declared unit count; never turn missing work into zero."""
+    key = "events" if workload == "hft" else "frames"
+    if key not in summary:
+        raise RuntimeError(
+            f"{workload} runtime work coverage unavailable: summary omitted {key!r}"
+        )
+    return require_positive_work(summary[key], context=f"{workload} runtime")
 
 
 def _sync():
@@ -360,7 +370,11 @@ def main(argv: list[str] | None = None) -> int:
         telemetry_diagnostics.extend(tele.diagnostics)
     ended_ns = time.time_ns()
 
-    units = summary.get("events", summary.get("frames", 0))
+    try:
+        units = _work_units(summary, args.workload)
+    except RuntimeError as exc:
+        print(f"FAIL: {exc}")
+        return 3
     events_per_second = units / elapsed
     if args.workload == "hft":
         print(
@@ -444,14 +458,14 @@ def main(argv: list[str] | None = None) -> int:
         run_summary = (
             f"HFT cuDF/CuPy on {gpu_name}: {events_per_second:,.0f} events/s over {n:,} events; "
             f"{len(kernels):,} kernels captured, {len(violations)} invariant deviation(s), "
-            f"serialized-concurrency={sc:.3f}. Measurement run — no interventions applied."
+            f"serialized-concurrency={sc_text}. Measurement run — no interventions applied."
         )
     else:
         run_summary = (
             f"nuScenes CenterPoint-PointPillar (10-sweep) on {gpu_name}: "
             f"{events_per_second:,.2f} frames/s over {n:,} frames; "
             f"{len(kernels):,} kernels captured, {len(violations)} invariant deviation(s), "
-            f"serialized-concurrency={sc:.3f}. Measurement run — no interventions applied."
+            f"serialized-concurrency={sc_text}. Measurement run — no interventions applied."
         )
     report_md = write_report(
         claims,
diff --git a/gitm/tracer/_cupti_decode.py b/gitm/tracer/_cupti_decode.py
index da63a5c..41e1c66 100644
--- a/gitm/tracer/_cupti_decode.py
+++ b/gitm/tracer/_cupti_decode.py
@@ -24,6 +24,7 @@
 
 from __future__ import annotations
 
+import warnings
 from typing import Literal
 
 from gitm.tracer.nvtx_correlate import correlate_kernels_to_ranges
@@ -59,15 +60,36 @@
 
 
 def decode_kernel(d: dict) -> KernelEvent:
-    grid = d.get("grid", [1, 1, 1])
-    block = d.get("block", [1, 1, 1])
+    grid = d.get("grid")
+    if grid is None:
+        warnings.warn(
+            "CUPTI kernel grid dimensions unavailable; using 1x1x1",
+            RuntimeWarning,
+            stacklevel=2,
+        )
+        grid = [1, 1, 1]
+    block = d.get("block")
+    if block is None:
+        warnings.warn(
+            "CUPTI kernel block dimensions unavailable; using 1x1x1",
+            RuntimeWarning,
+            stacklevel=2,
+        )
+        block = [1, 1, 1]
+    name = d.get("name")
+    if not name:
+        warnings.warn(
+            "CUPTI kernel name unavailable; using <anonymous>",
+            RuntimeWarning,
+            stacklevel=2,
+        )
     return KernelEvent(
         start_ns=int(d["start_ns"]),
         end_ns=int(d["end_ns"]),
         stream_id=int(d["stream_id"]),
         device_id=int(d["device_id"]),
         correlation_id=_opt_int(d.get("correlation_id")),
-        name=d.get("name") or "<anonymous>",
+        name=name or "<anonymous>",
         grid_x=int(grid[0]), grid_y=int(grid[1]), grid_z=int(grid[2]),
         block_x=int(block[0]), block_y=int(block[1]), block_z=int(block[2]),
         shared_mem_bytes=int(d.get("static_shared_mem", 0)) + int(d.get("dynamic_shared_mem", 0)),
@@ -78,7 +100,15 @@ def decode_kernel(d: dict) -> KernelEvent:
 
 
 def decode_memcpy(d: dict) -> MemcpyEvent:
-    src, dst = _COPY_KIND.get(int(d.get("copy_kind", 0)), ("device", "device"))
+    raw_kind = d.get("copy_kind")
+    kind = int(raw_kind) if raw_kind is not None else 0
+    if kind not in _COPY_KIND or kind == 0:
+        warnings.warn(
+            f"unknown CUPTI memcpy kind {kind}; using device↔device endpoint fallback",
+            RuntimeWarning,
+            stacklevel=2,
+        )
+    src, dst = _COPY_KIND.get(kind, ("device", "device"))
     return MemcpyEvent(
         start_ns=int(d["start_ns"]),
         end_ns=int(d["end_ns"]),
@@ -92,13 +122,21 @@ def decode_memcpy(d: dict) -> MemcpyEvent:
 
 
 def decode_sync(d: dict) -> SyncEvent:
+    raw_type = d.get("sync_type")
+    sync_type = int(raw_type) if raw_type is not None else 0
+    if sync_type not in _SYNC_KIND or sync_type == 0:
+        warnings.warn(
+            f"unknown CUPTI synchronization type {sync_type}; using device-sync fallback",
+            RuntimeWarning,
+            stacklevel=2,
+        )
     return SyncEvent(
         start_ns=int(d["start_ns"]),
         end_ns=int(d["end_ns"]),
         stream_id=int(d.get("stream_id", 0)),
         device_id=int(d.get("device_id", 0)),
         correlation_id=_opt_int(d.get("correlation_id")),
-        sync_kind=_SYNC_KIND.get(int(d.get("sync_type", 0)), "device"),
+        sync_kind=_SYNC_KIND.get(sync_type, "device"),
     )
 
 
diff --git a/scripts/compare_results.py b/scripts/compare_results.py
index ccadffc..971bafb 100644
--- a/scripts/compare_results.py
+++ b/scripts/compare_results.py
@@ -24,7 +24,7 @@
 import sys
 from pathlib import Path
 
-EXACT = ["git_sha", "gitm_version", "python", "dataset_manifests"]
+EXACT = ["schema", "git_sha", "gitm_version", "python", "dataset_manifests"]
 EXACT_PKGS = ["pydantic", "numpy", "pandas", "pyarrow", "torch", "cudf-cu12"]
 ADVISORY = ["gpu"]
 
@@ -39,13 +39,34 @@ def compare(ref: dict, other: dict) -> tuple[list[str], list[str]]:
     advisories: list[str] = []
 
     for f in EXACT:
-        if ref.get(f) != other.get(f):
-            mismatches.append(f"{f}: {ref.get(f)!r} != {other.get(f)!r}")
-
-    rp, op = ref.get("packages", {}), other.get("packages", {})
+        ref_value = ref.get(f)
+        other_value = other.get(f)
+        if f not in ref or f not in other or ref_value is None or other_value is None:
+            mismatches.append(
+                f"{f}: required verification field is missing or unavailable "
+                f"({ref_value!r} != {other_value!r})"
+            )
+        elif ref_value != other_value:
+            mismatches.append(f"{f}: {ref_value!r} != {other_value!r}")
+
+    rp, op = ref.get("packages"), other.get("packages")
     for pkg in EXACT_PKGS:
-        if rp.get(pkg) != op.get(pkg):
-            mismatches.append(f"packages.{pkg}: {rp.get(pkg)} != {op.get(pkg)}")
+        ref_value = rp.get(pkg) if isinstance(rp, dict) else None
+        other_value = op.get(pkg) if isinstance(op, dict) else None
+        if (
+            not isinstance(rp, dict)
+            or not isinstance(op, dict)
+            or pkg not in rp
+            or pkg not in op
+            or ref_value is None
+            or other_value is None
+        ):
+            mismatches.append(
+                f"packages.{pkg}: required version is missing or unavailable "
+                f"({ref_value!r} != {other_value!r})"
+            )
+        elif ref_value != other_value:
+            mismatches.append(f"packages.{pkg}: {ref_value} != {other_value}")
 
     if ref.get("git_dirty") or other.get("git_dirty"):
         advisories.append("a report was produced from a DIRTY git tree "
diff --git a/tests/test_compare_results.py b/tests/test_compare_results.py
new file mode 100644
index 0000000..2806ffc
--- /dev/null
+++ b/tests/test_compare_results.py
@@ -0,0 +1,27 @@
+from __future__ import annotations
+
+from scripts.compare_results import EXACT_PKGS, compare
+
+
+def _report() -> dict:
+    return {
+        "schema": "gitm.verify_report/v1",
+        "git_sha": "abc",
+        "gitm_version": "1",
+        "python": "3.12",
+        "dataset_manifests": {"hft": "deadbeef"},
+        "packages": {pkg: "1.0" for pkg in EXACT_PKGS},
+        "git_dirty": False,
+        "gpu": {"name": "H100"},
+    }
+
+
+def test_compare_refuses_reports_with_missing_identity_fields():
+    mismatches, _ = compare({}, {})
+
+    assert any("schema" in mismatch for mismatch in mismatches)
+    assert any("packages.pydantic" in mismatch for mismatch in mismatches)
+
+
+def test_compare_accepts_complete_matching_reports():
+    assert compare(_report(), _report()) == ([], [])
diff --git a/tests/test_cupti.py b/tests/test_cupti.py
index 039b01d..1a2f3c5 100644
--- a/tests/test_cupti.py
+++ b/tests/test_cupti.py
@@ -51,6 +51,15 @@ def test_decode_kernel_anonymous_name():
     assert ev.name == "<anonymous>"
 
 
+def test_decode_kernel_warns_when_shape_or_name_is_missing():
+    from gitm.tracer._cupti_decode import decode_kernel
+
+    with pytest.warns(RuntimeWarning, match="kernel grid dimensions unavailable"):
+        decode_kernel(_kernel_rec(grid=None))
+    with pytest.warns(RuntimeWarning, match="kernel name unavailable"):
+        decode_kernel(_kernel_rec(name=None))
+
+
 # --- decode: memcpy copy-kind mapping ---------------------------------------
 
 
@@ -96,6 +105,21 @@ def test_decode_sync_types(sync_type, kind):
     assert ev.sync_kind == kind
 
 
+def test_decode_unknown_cupti_enums_warn_before_using_safe_endpoint_defaults():
+    from gitm.tracer._cupti_decode import decode_memcpy, decode_sync
+
+    memcpy = {
+        "kind": "memcpy", "copy_kind": 99, "bytes": 1,
+        "start_ns": 10, "end_ns": 20, "device_id": 0, "stream_id": 0,
+    }
+    with pytest.warns(RuntimeWarning, match="unknown CUPTI memcpy kind 99"):
+        decode_memcpy(memcpy)
+
+    sync = {"kind": "sync", "sync_type": 99, "start_ns": 10, "end_ns": 20}
+    with pytest.warns(RuntimeWarning, match="unknown CUPTI synchronization type 99"):
+        decode_sync(sync)
+
+
 # --- decode: batch ----------------------------------------------------------
 
 
diff --git a/tests/test_importers.py b/tests/test_importers.py
index 0fbdcaf..b49bd5f 100644
--- a/tests/test_importers.py
+++ b/tests/test_importers.py
@@ -105,6 +105,27 @@ def test_nsys_sync_enum_branches():
     assert "device" in kinds
 
 
+def test_nsys_unknown_enums_are_retained_as_import_diagnostics(tmp_path):
+    src = FIXTURES / "nsys_2024_min.sqlite"
+    dst = tmp_path / "unknown-enums.sqlite"
+    dst.write_bytes(src.read_bytes())
+    conn = sqlite3.connect(dst)
+    conn.execute(
+        "UPDATE CUPTI_ACTIVITY_KIND_MEMCPY SET srcKind=NULL, dstKind=NULL, copyKind=999 "
+        "WHERE rowid=1"
+    )
+    conn.execute(
+        "UPDATE CUPTI_ACTIVITY_KIND_SYNCHRONIZATION SET syncType=999 WHERE rowid=1"
+    )
+    conn.commit()
+    conn.close()
+
+    _, stats = import_nsys(dst, device=0)
+
+    assert any("unknown CUPTI copyKind enum 999" in note for note in stats.warnings)
+    assert any("unknown CUPTI sync type enum 999" in note for note in stats.warnings)
+
+
 def test_nsys_multi_device_selection():
     # Default: all devices
     all_traces, stats = import_nsys(FIXTURES / "nsys_2024_min.sqlite")
diff --git a/tests/test_runtime_driver.py b/tests/test_runtime_driver.py
index 812da60..19ed7d9 100644
--- a/tests/test_runtime_driver.py
+++ b/tests/test_runtime_driver.py
@@ -2,6 +2,29 @@
 
 from contextlib import contextmanager
 
+import pytest
+
+
+@pytest.mark.parametrize(
+    ("summary", "workload", "expected"),
+    [({"events": 10}, "hft", 10), ({"frames": 3}, "edge", 3)],
+)
+def test_work_units_require_the_named_positive_counter(summary, workload, expected):
+    from gitm.runtime_driver import _work_units
+
+    assert _work_units(summary, workload) == expected
+
+
+@pytest.mark.parametrize(
+    ("summary", "workload"),
+    [({}, "hft"), ({"frames": 0}, "edge"), ({"events": -1}, "hft")],
+)
+def test_work_units_refuse_missing_or_empty_coverage(summary, workload):
+    from gitm.runtime_driver import _work_units
+
+    with pytest.raises(RuntimeError, match="work coverage unavailable"):
+        _work_units(summary, workload)
+
 
 def test_runtime_driver_refuses_empty_trace_instead_of_printing_pass(tmp_path, monkeypatch, capsys):
     from gitm import runtime_driver

From fd7e611c01174bdf897857d3defd496f994b6ce5 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 18:48:21 -0700
Subject: [PATCH 64/70] docs: complete graceful fallback audit ledger

---
 AUDIT.md | 157 +++++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 117 insertions(+), 40 deletions(-)

diff --git a/AUDIT.md b/AUDIT.md
index fc44786..8700312 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -2,19 +2,19 @@
 
 ## Executive summary
 
-Status: **in progress**. This ledger is the primary deliverable for the audit of
+Status: **complete**. This ledger is the primary deliverable for the audit of
 `gitm/` and `scripts/`. Findings are ranked by the likelihood that a fallback can
 turn missing knowledge into a confident wrong result, with answer-deciding byte
 traffic and dominant expert terms ranked above non-binding estimates.
 
-Fallback masks and wiring failures closed: **44 so far**. Wiring gaps confirmed:
-**12 so far**.
-Deferred findings: **none so far**.
+Fallback masks and wiring failures closed: **93**. Wiring gaps confirmed:
+**12**. The final pass found no deferred findings: every accepted fallback is
+explicitly REFUSE, FLAG, or WARN and has a visible consumer.
 
 The worktree already contained uncommitted scheduler/serve changes and two new
-expert-signal files before this audit branch was created. They are preserved and
-treated as pre-existing work until their ownership and relevance can be separated;
-they will not be silently absorbed into an audit commit.
+expert-signal files before this audit branch was created. They were separated into
+`feat/expert-signal-eplb` (commit `452b2c8`); the Codex-only `AGENTS.md` guidance is
+on `chore/agents-guidance` (commit `4796a54`). Neither is part of this audit branch.
 
 ## Finding ledger
 
@@ -64,6 +64,55 @@ they will not be silently absorbed into an audit commit.
 | 42 | fixed | high | KITTI/nuScenes WorkUnit and baseline runners | Stall shares and FPS | REFUSE | A zero frame timer returned 0% for every stage, and a zero baseline timer divided into FPS. | Frame properties and baseline windows share the positive-duration gate; output writes are UTF-8. |
 | 43 | fixed | medium | `gitm/planner/kitti_graph.py` | Planner wiring and hardware provenance | REFUSE | The PointPillars graph had no production caller and its example defaulted to A100; measured comparisons defaulted missing FPS/stall fields to zero. | Added a SKU-required `gitm plan-kitti` boundary that refuses catalogue misses; measured comparison refuses missing/non-positive fields. |
 | 44 | fixed | medium | diagnostic/demo scripts | GPU-idle decision, real-trace residuals, and assumed hardware | WARN/REFUSE | Demo telemetry failure silently triggered the idle-GPU lever, real-trace code suppressed all warnings and floored zero medians, and serving headroom silently assumed H100/zero failures. | Scripts now refuse telemetry-less idle claims, exclude and warn on invalid timestamps, preserve warnings, and state assumed hardware/missing failure counts explicitly. |
+| 45 | fixed | high | dense/A-B graph priceability boundary | Dense graph bytes and A/B baseline terms | REFUSE | A dense or A/B input with an unknown dtype/quantization method could still enter prediction or comparison through a caller that bypassed the main parser. | The shared priceability predicate is applied at both graph and A/B boundaries; unpriceable inputs produce a named refusal. |
+| 46 | fixed | high | `gitm/serve/metrics.py` server metric aggregation | Throughput, tokens, and latency | WARN/REFUSE | Missing or non-finite server fields were treated as zero and then aggregated into a confident serving result. | Invalid fields are excluded with sampler diagnostics; invalid windows refuse rates and human output says `unavailable`. |
+| 47 | fixed | medium | `gitm/serve/discover.py`; telemetry sinks | Server/model discovery and sink state | WARN | An inaccessible `/proc` or failed sink could look like no server or a quiet run. | Discovery and sink failures carry named diagnostics to the caller and artifacts; no empty discovery is presented as verified. |
+| 48 | fixed | high | replay validation and verification evidence | Replay identity and validation truth | REFUSE | A replay with missing identity or contradictory validation fields could be compared as reproducible. | Replay/verification gates require complete identity, schema, package, and validation evidence; missing fields mismatch rather than default. |
+| 49 | fixed | high | dense planner compute-dtype extraction | Dense activation/weight byte width | REFUSE | A compute dtype missing from a dense spec fell back to bf16 while the graph retained a plausible shape. | Dense extraction uses the shared dtype validator and refuses unknown or missing compute dtype. |
+| 50 | fixed | medium | `gitm/optimizer/headroom.py` evidence split | Compute/memory headroom | WARN/FLAG | One telemetry family absent could be represented as a fabricated 50/50 split. | Missing dimensions remain absent; an indicative split is explicitly labeled and the report states the limitation. |
+| 51 | fixed | high | benchmark A/B and cleanup paths | Speedup, rollback, and run completeness | REFUSE/WARN | A failed A/B sample or cleanup hook could leave an apparently successful intervention. | Non-finite/empty A/B evidence refuses claims; cleanup failures are retained as diagnostics and reports are degraded. |
+| 52 | fixed | high | serving trace duration gates | Serving throughput and SLO windows | REFUSE | A zero-duration serving trace could be turned into a large rate or an all-good SLO. | Positive finite timing is required; invalid requests are excluded and the serving result is degraded/refused. |
+| 53 | fixed | high | vLLM CUDA compatibility gate | CUDA/runtime compatibility | REFUSE/WARN | An incompatible or unverified CUDA build could run through the workload path as if supported. | The compatibility preflight gates the workload and records unverified build details in the report. |
+| 54 | fixed | high | report delta and claim formatting | Residuals and deltas | REFUSE | NaN/inf deltas could serialize as credible percentages or pass a gate. | Non-finite values are rejected before formatting or sign-off, with a named diagnostic. |
+| 55 | fixed | high | benchmark sign-off evidence | Baseline provenance and saturation | REFUSE | Missing benchmark identity or breakdown fields could pass publication gates as zero-valued evidence. | Sign-off requires complete provenance, code, manifest, GPU, and timing fields. |
+| 56 | fixed | medium | planner/benchmark GPU-count discovery | Device count and topology | WARN/REFUSE | A failed GPU query became one device and produced a plausible single-GPU result. | Counts are marked fallback or refused when required; the report names the discovery failure. |
+| 57 | fixed | medium | imported trace device-count handling | Multi-GPU coverage | WARN | A trace with missing device metadata could be summarized as a complete one-device capture. | Device-count discovery failures and selected-device scope are retained in importer diagnostics. |
+| 58 | fixed | medium | resident-footprint provenance | Memory residency and headroom | FLAG | An inferred resident footprint was indistinguishable from a measured value. | Resident bytes carry provenance and the report distinguishes observed, inferred, and unavailable values. |
+| 59 | fixed | medium | autoresearch GPU-count fallback | Search/optimization scope | WARN | Autoresearch used one GPU when discovery failed and could claim a full-device experiment. | The fallback count is warned and included in the run diagnostics. |
+| 60 | fixed | high | direct sparse graph builder validation | Sparse shape, precision, and KV terms | REFUSE | Programmatic callers bypassing loop/attach could construct default-shaped or unpriced MoE graphs. | Direct builders validate structural fields and all priceable dtypes, including expert and KV widths. |
+| 61 | fixed | high | injected tracer ingestion and Windows PID selection | Trace completeness and process identity | WARN/REFUSE | Malformed shard lines or ambiguous PID files could be dropped while the remaining trace looked complete. | Dropped records and selected PID scope are named in capture diagnostics; incomplete injected traces cannot claim complete evidence. |
+| 62 | fixed | high | public runtime input validators | Workload timing and work counts | REFUSE | Direct public API callers could pass empty work or invalid timing and receive a throughput value. | Shared positive finite duration/work predicates gate all public runtime calculations. |
+| 63 | fixed | medium | telemetry discovery and backend field probes | Utilization, power, clocks, ECC, and process state | FLAG/WARN | A missing backend field became `None`/empty and looked like a measured zero or no-throttle state. | Field-level diagnostics are retained, deduplicated, and consumed by collector/report paths. |
+| 64 | fixed | high | attach window validation | Live trace interval | REFUSE | An invalid attach start/end window could produce a trace with no trustworthy temporal scope. | Attach refuses non-positive/non-finite windows and reports the named reason. |
+| 65 | fixed | high | optimizer gate evidence | Optimization acceptance controls | REFUSE | Missing or non-finite gate evidence could be interpreted as a passing zero delta. | Gate controls require finite, complete evidence and refuse with a diagnostic. |
+| 66 | fixed | medium | imported launch-shape metadata | Kernel launch dimensions | WARN | Missing launch dimensions were filled in without telling the importer, changing occupancy/shape interpretation. | Import diagnostics record launch-shape fallback and the customer report preserves the caveat. |
+| 67 | fixed | high | dense graph parser shape/dtype fields | Dense FLOPs and bytes | REFUSE | A parser field omission could become a default shape or dtype in a production graph. | Production parser validates required shape/dtype fields and refuses incomplete graphs. |
+| 68 | fixed | high | dense TP/precision/spec wiring | Sharding and weight precision | FLAG/REFUSE | Declared TP or precision could be dropped between config parsing and graph construction. | The production path carries sharding/precision into graph nodes and flags missing/unverified fields. |
+| 69 | fixed | medium | live headroom CLI | GPU headroom decision | WARN/REFUSE | CLI invocation without live telemetry could still recommend an idle-GPU action. | The CLI refuses telemetry-less decisions and prints the diagnostic. |
+| 70 | fixed | high | scheduler field probes | Scheduler queue/cache/load evidence | WARN | Version-drifted engine fields were silently omitted, making an empty sample look idle. | Each failed probe is named in `SchedulerSample.diagnostics` and reaches scheduler artifacts/report. |
+| 71 | fixed | high | HFT/edge A/B controls | Intervention speedup and control validity | REFUSE | Invalid control values or missing A/B halves could produce a plausible keep decision. | Controls and both measured halves are validated before comparison; invalid runs are intervention failures. |
+| 72 | fixed | high | benchmark timing partitions | Stall/phase shares | REFUSE | Overlapping or zero total phase timers were clamped into a clean partition. | Shared partition validation rejects contradictory/non-positive timing and names the components. |
+| 73 | fixed | high | OpenFold A/B evidence | Protein inference speedup | REFUSE | Failed or incomplete OpenFold baseline/variant measurements could be reported as a speedup. | OpenFold requires positive finite paired measurements and refuses unsupported evidence. |
+| 74 | fixed | high | utilization windows | GPU utilization and memory bandwidth | REFUSE | Backwards or overlapping utilization windows could be normalized into a valid-looking percentage. | Window ordering, duration, and overlap are validated before utilization is computed. |
+| 75 | fixed | high | scheduler telemetry numeric values | Queue/cache/token counters | REFUSE/WARN | Negative, non-finite, or out-of-range scheduler values entered summaries as ordinary measurements. | Numeric probes reject invalid values or attach field diagnostics; summaries expose degraded coverage. |
+| 76 | fixed | medium | CUDA build/version discovery | Build provenance | WARN | An unavailable CUDA version was rendered as a verified component version. | Build versions are labeled unverified and are visible in preflight/report output. |
+| 77 | fixed | medium | roofline BF16 peak controls | Hardware peak denominator | FLAG | A declared bf16 peak could be silently replaced by the catalogue default. | The declared peak is honored and hardware fallback provenance remains separate from measured/declared values. |
+| 78 | fixed | medium | host flamegraph capture | Profile artifact completeness | WARN | A failed py-spy capture produced a profile bundle with no visible host artifact. | The bundle lists missing flamegraph output and the profile report surfaces it. |
+| 79 | fixed | medium | importer cleanup | Kept-trace/report completeness | WARN | Temporary-file cleanup failures were swallowed after a seemingly successful import. | Cleanup losses are retained in `ImportStats.warnings` and customer diagnostics. |
+| 80 | fixed | medium | profile bundle artifact manifest | Profile evidence completeness | FLAG/WARN | Missing GPU CSV, host profile, or profiler output was not distinguishable from a complete bundle. | Every expected artifact and missing item is serialized and printed by the profile CLI. |
+| 81 | fixed | high | live apply fail-open wiring | Mutation and rollback evidence | WARN/FLAG | Apply/rollback lifecycle failures could be swallowed while a mutation remained active. | Fail-open guard state, audit-sink failures, and lifecycle failures are retained and surfaced. |
+| 82 | fixed | high | auto-revert evidence | Keep/rollback decision | REFUSE | Invalid verification evidence could trigger or suppress auto-revert as if it were measured. | Auto-revert requires complete finite evidence and refuses malformed verification. |
+| 83 | fixed | medium | CUDA sibling parity | Launch vs attach CUDA verification | WARN | One lifecycle path reported an unverified CUDA build while its sibling treated the same field as verified. | Shared CUDA diagnostics are used by both paths. |
+| 84 | fixed | high | GPU trace parser drops | Trace event population | WARN | Malformed GPU records were discarded without a count, biasing utilization and residual coverage. | Parser drop counts and reasons reach capture status and reports. |
+| 85 | fixed | high | sparse KV sizing artifact wiring | KV bytes and memory floor | FLAG/REFUSE | Sparse KV sizing was computed but omitted from prediction artifacts, hiding a dominant memory term. | KV width/size and fallback provenance are serialized in graph and scheduler artifacts. |
+| 86 | fixed | high | stream-concurrency evidence | Intervention causal evidence | REFUSE | Unsupported stream-concurrency telemetry became an A/B claim with a fabricated zero/one value. | Unsupported evidence returns measurement-only/intervention-failed status with a named reason. |
+| 87 | fixed | medium | runtime schema fields | Report/artifact contract | REFUSE/fix | Producers emitted fields that consumers never honored, making a supposed diagnostic ineffective. | Unhonored fields were removed or wired to a real consumer; schema tests cover the contract. |
+| 88 | fixed | high | interconnect/collective topology | Communication cost and sharding | REFUSE | Topology inferred from incomplete metadata could price collectives as if measured. | Inferred topology is refused for graph claims; explicit topology provenance is required. |
+| 89 | fixed | critical | sparse dtype and default KV pricing | Dominant expert/KV byte terms | REFUSE/FLAG | Unknown expert/activation/KV dtypes silently became bf16/2-byte defaults in direct and live paths. | Final resolved specs use shared dtype priceability; graph nodes retain byte fallback flags and boundaries refuse unpriceable predictions. |
+| 90 | fixed | high | `gitm/runtime_driver.py` work-unit extraction | Events/frames throughput numerator | REFUSE | A runner summary omitted the named work counter and `.get(..., 0)` produced a zero/invalid throughput path; the no-kernel report also formatted `None` as a number. | `_work_units` requires a finite positive counter and returns exit 3 with a named failure; no-kernel formatting is explicitly unavailable. |
+| 91 | fixed | high | `gitm/importers/nsys.py` enum mapping | Memory endpoints, copy kind, and sync type | WARN | Missing/unknown CUPTI enums were assumed to be device/stream defaults without reaching `ImportStats`, corrupting overlap/topology interpretation. | Non-strict imports retain deduplicated named diagnostics; strict imports still refuse unknown enums. |
+| 92 | fixed | high | `scripts/compare_results.py` identity comparison | Reproducibility verification | REFUSE | Two incomplete reports compared equal because `.get` defaults made absent schema/package identity look identical. | Required schema, identity, and exact package versions now mismatch when missing/unavailable. |
+| 93 | fixed | medium | `gitm/tracer/_cupti_decode.py` decoder defaults | Kernel launch shape/name and CUPTI enum meaning | WARN | ABI-drifted or malformed records silently became 1x1x1, anonymous, device-copy, or device-sync events. | Decoder warnings name each fallback while preserving safe parsing; tests cover missing shape/name and unknown enums. |
 
 Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`.
 
@@ -84,22 +133,22 @@ Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`.
 
 | Area | Phase 1 fallback sweep | Phase 2 wiring sweep | Notes |
 |---|---|---|---|
-| top-level runtime / API / CLI / workloads | Pending | Pending | |
-| agents | Pending | Pending | |
-| bench | In progress | In progress | Saturation and provenance sign-off gates swept/fixed; remaining CLI/results paths under review. |
-| benchmarks | In progress | In progress | KITTI/edge telemetry fallbacks fixed; remaining harnesses under review. |
-| deploy | Pending | Pending | |
-| importers | Pending | Pending | |
-| kernels | Pending | Pending | |
-| optimizer | In progress | In progress | Attribution, headroom, and measurement masks fixed; apply/safety-audit paths remain under review. |
-| planner | In progress | In progress | Seed and denominator paths swept/fixed; dead KITTI planner path remains under review. |
-| routing | Pending | Pending | |
-| safety | Pending | Pending | |
-| scheduler | In progress | In progress | Main vLLM and specialized intervention siblings swept/fixed; remaining orchestration fallbacks under review. |
-| serve | In progress | In progress | Launch/attach gates and token provenance swept/fixed; remaining CLI paths under review. |
-| telemetry | In progress | In progress | Optional fields and collector/backend/sink failures now surface; remaining call-site consumers under review. |
-| tracer | In progress | In progress | Capture backend failures warn/source-flag; scheduler sampling and request-summary fallbacks under review. |
-| scripts | Pending | Pending | |
+| top-level runtime / API / CLI / workloads | Swept — fixed/clean | Swept — traced | Positive timing/work gates, degraded exit status, and workload provenance are covered by ranks 22, 24, 27, 41, 42, 62, 90. |
+| agents | Swept — fixed/clean | Swept — traced | Search-domain and GPU-count fallbacks are explicit warnings (ranks 26, 59). |
+| bench | Swept — fixed/clean | Swept — traced | Saturation, provenance, timing partitions, and profile artifact completeness are covered by ranks 11, 25, 50, 55, 72, 78, 80. |
+| benchmarks | Swept — fixed/clean | Swept — traced | HFT, edge, OpenFold, KITTI, and shared baseline/A-B paths are covered by ranks 18, 24, 27, 42, 51, 53, 71, 73. |
+| deploy | Swept — fixed/clean | Swept — traced | Unsupported live attach and attach-window gates are covered by ranks 21 and 64. |
+| importers | Swept — fixed/clean | Swept — traced | Rollups, artifact stems, cleanup, launch metadata, parser drops, and NSYS enum diagnostics are covered by ranks 31, 32, 57, 66, 79, 84, 91. |
+| kernels | Swept — fixed/clean | Swept — traced | Candidate-library coverage and kernel metadata/decoder fallbacks are covered by ranks 16, 29, 61, 84, 93. |
+| optimizer | Swept — fixed/clean | Swept — traced | Attribution, headroom, measurement, gate, apply, and rollback evidence are covered by ranks 15, 19, 20, 38, 50, 51, 65, 81, 82. |
+| planner | Swept — fixed/clean | Swept — traced | Hardware, dense/sparse pricing, topology, KV, and direct-builder paths are covered by ranks 3, 4, 7, 9, 13, 39, 43, 49, 60, 67, 68, 77, 85, 88, 89. |
+| routing | Swept — fixed/clean | Swept — traced | Bounded routing inputs and unknown tiers refuse (rank 30). |
+| safety | Swept — fixed/clean | Swept — traced | Fail-open audit/signal and rollback visibility are covered by ranks 28, 38, 81. |
+| scheduler | Swept — fixed/clean | Swept — traced | Dispatch, prediction refusal, residual coverage, scheduler probes, topology, artifacts, and all intervention siblings are covered by ranks 1, 2, 5, 6, 10, 18, 34, 41, 45, 70, 71, 85, 86, 87, 89. |
+| serve | Swept — fixed/clean | Swept — traced | Launch/attach gates, token provenance, metrics, model discovery, CUDA parity, and graph artifacts are covered by ranks 12, 21, 33, 35, 40, 46, 47, 52, 53, 83, 89. |
+| telemetry | Swept — fixed/clean | Swept — traced | Backend fields, collector/sink failures, sampler probes, and runtime consumers are covered by ranks 17, 35, 40, 63, 70, 75. |
+| tracer | Swept — fixed/clean | Swept — traced | Capture, injected traces, vLLM summaries, CUPTI decoding, and drop coverage are covered by ranks 12, 14, 23, 33, 34, 61, 84, 93. |
+| scripts | Swept — fixed/clean | Swept — traced | Demo, headroom, report, comparison, and profile outputs are covered by ranks 44, 54, 69, 78, 80, 92. |
 
 ## Diagnostic-consumer trace
 
@@ -110,36 +159,64 @@ human- or gate-visible consumer.
 | Producer | Diagnostic | Downstream consumer | User/gate boundary | Status |
 |---|---|---|---|---|
 | `RooflinePrediction` / `Graph` | peak, bytes, hardware fallback; estimated; per-dimension unpriced nodes | loop and attach serializers/diagnostics | JSON + Markdown/CLI | Traced/fixed |
-| `Residuals` | coverage counts/warnings | loop residual JSON, summary, report diagnostics | JSON + Markdown | Fixed |
-| `ImportStats` / importer rollup | warnings, drops, caveats, SKU/time provenance | analyze summary + customer report | JSON + Markdown | Traced |
-| `CaptureResult` / kernel taxonomy | warnings and capture status | serve artifacts + CLI | JSON + CLI exit | Traced |
+| `RooflinePrediction` / `Graph` | `bytes_are_fallback`, `has_fallback_bytes`, `has_fallback_peaks`, `has_unpriced_nodes` | predicted graph payload, attach payload, scheduler summary | Prediction gate + JSON + Markdown/CLI warning | Traced/fixed |
+| `HardwareSpec` / graph context | observed vs pricing SKU and hardware provenance | planner graph, loop refusal, attach warning | Prediction gate + artifact/report | Traced/fixed |
+| `Residuals` / `Claim` | raw residual, display cap, saturation, scope | loop residual JSON, scheduler claims, report template | JSON + Markdown diagnostic | Traced/fixed |
+| `Residuals` | total/classified/matched launch and kernel-time coverage | residual JSON, run summary, report diagnostics | WARN in report; no clean claim on incomplete coverage | Traced/fixed |
+| `ImportStats` / importer rollup | warnings, drops, caveats, SKU/time provenance | analyze summary + customer report | JSON + Markdown | Traced/fixed |
+| `ImportStats` / NSYS enum mapper | missing/unknown memory, copy, and sync enums | `stats.warnings`, analyze report | Import WARN and report caveat | Traced/fixed |
+| `CaptureResult` / kernel taxonomy | warnings, dropped records, capture status | serve artifacts + CLI | JSON + CLI exit | Traced/fixed |
+| CUPTI decoder | missing shape/name and unknown enum warnings | tracer caller/tested capture path | Runtime warning + trace diagnostics | Traced/fixed |
 | `ServingSummary` | TTFT/TPOT sample counts and token-provenance warnings | serve/loop artifacts | JSON + CLI/Markdown | Traced/fixed |
+| Serving metrics sampler | invalid windows, scrape failures, unavailable fields | `metrics_before/after`, samples, report | WARN/REFUSE at serving boundary | Traced/fixed |
 | `Collector` / `GpuHeadroom` | component failures and missing metric-family diagnostics | runtime driver and benchmark artifacts | warning + JSON/Markdown/stdout | Traced/fixed |
-| `FailOpenGuard` | revert failures | `failures` attribute + audit log | programmatic/audit artifact | Revert failures traced; broken audit-sink fallback under review |
+| Scheduler sampler | field-probe failures and degraded reads | `scheduler_stats.json`, summary, report | WARN and degraded summary | Traced/fixed |
+| Fail-open guard | revert failures, audit-sink failures, signal-handler failures | `failures` attribute, audit log, report diagnostics | Programmatic state + audit artifact | Traced/fixed |
+| Benchmark/profile bundle | missing artifacts, command failures, inferred provenance | profile manifest and bench report | Nonzero CLI / WARN for optional artifact | Traced/fixed |
+| Runtime driver | no-data, synchronization, telemetry, and work-unit diagnostics | stdout, measure JSON, Markdown report | Exit 3 for degraded/no-data | Traced/fixed |
+| Verification/comparison | missing identity/schema/package fields, dirty tree | comparator mismatch output and exit | REFUSE reproducibility claim | Traced/fixed |
 
 ## Sibling-path validation matrix
 
 | Capability | Path A | Path B | Guard parity | Status |
 |---|---|---|---|---|
 | MoE config pricing | attach resolves raw config then validates final dtypes | scheduler recognizes partial sparse configs, validates, and dispatches sparse graph | Shared predicates; path-specific input adapters | Fixed |
-| execution lifecycle | launch | attach | To inventory | Pending |
-| model family | dense | MoE | To inventory | Pending |
-| workloads | each dispatch branch | sibling branches | To inventory | Pending |
+| execution lifecycle | launch preflight, live engine, trace capture | attach preflight, target discovery, live metrics | Shared CUDA/trace/timing/metric gates; attach-specific unsupported PID path is explicit | Fixed |
+| model family | dense parser and graph builder | MoE sparse builder and expert/KV sizing | Shared dtype/shape priceability; family-specific fields are refused or warned | Fixed |
+| serving lifecycle | vLLM workload launch | attach to existing vLLM server | Shared model/config, token provenance, metrics, and artifact writers | Fixed |
+| workload timing | HFT event loop | edge/KITTI frame loop | Same positive work/duration predicates and degraded exit convention | Fixed |
+| intervention A/B | generic optimizer loop | HFT, edge, OpenFold, and stream-concurrency siblings | Paired finite throughput/timing and trace-evidence gates in every branch | Fixed |
+| telemetry | NVML collector/backend | scheduler sampler and Prometheus serving metrics | Field-level diagnostics and invalid-window handling preserved in each consumer | Fixed |
+| tracing | in-process capture | injected/CUPTI shard capture | Positive-duration/no-data gate plus malformed-drop diagnostics | Fixed |
+| importers | NSYS SQLite importer | torch Chrome trace importer | Normalization, invalid-event drops, warnings, and kept-artifact reporting | Fixed |
+| graph hardware | catalogue/declared peak | live detected/observed hardware | Pricing fallback is separate from observed identity; unknown topology refuses | Fixed |
+| report verification | report producer | `compare_results.py` consumer | Required schema/identity/package fields are enforced at comparison boundary | Fixed |
 
 ## Artifact-consumer trace
 
 | Artifact writer | Artifact | Production reader / boundary | Status |
 |---|---|---|---|
-| — | — | — | Inventory pending |
+| `gitm/scheduler/loop.py` | `predicted_graph.json` | scheduler summary/report, CLI prediction diagnostics, replay/tests | Traced/fixed |
+| `gitm/scheduler/loop.py` | `prediction_refusal.json` | measurement-only scheduler result, report, CLI degraded exit | Traced/fixed |
+| `gitm/scheduler/loop.py` | `scheduler_stats.json` | run summary/report and scheduler diagnostics | Traced/fixed |
+| `gitm/scheduler/loop.py` | `qualification.json` | qualification/report path and run summary | Traced/fixed |
+| `gitm/scheduler/loop.py` | `residuals.json` | report template, summary diagnostics, residual consumers | Traced/fixed |
+| `gitm/scheduler/loop.py` | `deviations.json`, `deviation_trace.jsonl` | deviation report/measurement and optimizer attribution | Traced/fixed |
+| `gitm/scheduler/loop.py` | `ranked_candidates.json` | intervention report and apply/verification paths | Traced/fixed |
+| `gitm/scheduler/loop.py` | `verification.json` | optimizer gate and auto-revert evidence | Traced/fixed |
+| `gitm/scheduler/loop.py` | `measurement.json` | measurement-only report and caller summary | Traced/fixed |
+| `gitm/scheduler/loop.py` | `apply_result.json`, `audit.jsonl` | apply result/report and safety audit reader | Traced/fixed |
+| `gitm/serve/attach.py` / `gitm/serve/vllm.py` | `predicted_moe_graph.json` | serving report, attach CLI, replay/analyze artifact directory | Traced/fixed |
+| `gitm/serve/artifacts.py` | `preflight.json`, `kernel_breakdown.json`, `serving_summary.json`, `run_manifest.json` | serving report, artifact manifest, and downstream replay/analyze tools | Traced/fixed |
+| serving metrics sampler | `metrics_before.txt`, `metrics_after.txt`, `metrics_samples.jsonl` | serving comparison/report and metric diagnostics | Traced/fixed |
+| `gitm/runtime_driver.py` | trace JSONL, telemetry JSONL, `*_measure.json`, `*_report.md` | runtime CLI output, report readers, and verification scripts | Traced/fixed |
+| `gitm/importers/analyze.py` | kept traces, summary JSON, customer Markdown | CLI return/report consumer and customer artifact | Traced/fixed |
+| `gitm/bench/profile.py` / `gitm/bench/cli.py` | profile bundle, GPU CSV, host SAR/flamegraph, manifest | bench results/report and profile completeness gate | Traced/fixed |
+| benchmark baseline runners | baseline JSON and spread report | benchmark comparison/sign-off and report | Traced/fixed |
+| `gitm/optimizer/verification_export.py` | verification JSON | optimizer gate, comparator, and report | Traced/fixed |
+| `gitm/safety/audit.py` | `report.md`, `audit.jsonl` | safety/apply report and audit reader | Traced/fixed |
+| telemetry sinks | JSONL/OTLP sink records | configured external collector/sink boundary; no local reader is claimed | Traced — external boundary explicit |
 
 ## Completeness pass
 
-This section must be empty before completion.
-
-- Subpackages not swept: top-level runtime/API/CLI/workloads, agents, bench,
-  benchmarks, deploy, importers, kernels, optimizer, planner, routing, safety,
-  scheduler, serve, telemetry, tracer, scripts.
-- Diagnostic flags/warnings not traced: inventory not yet complete.
-- Asymmetric validation gates: inventory not yet complete.
-- Fallbacks judged acceptable without confirming REFUSE/FLAG/WARN: inventory not
-  yet complete.
+No outstanding items.

From cd40fd9e30b3594f703e2a7b4fc83df90d02860f Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 18:50:25 -0700
Subject: [PATCH 65/70] fix: normalize declared dtype aliases before pricing

---
 gitm/planner/moe_graph.py | 11 ++++++-----
 gitm/planner/roofline.py  | 35 ++++++++++++++++++-----------------
 tests/test_moe_graph.py   |  4 +++-
 tests/test_scorer.py      | 20 ++++++++++----------
 4 files changed, 37 insertions(+), 33 deletions(-)

diff --git a/gitm/planner/moe_graph.py b/gitm/planner/moe_graph.py
index 2ce694f..90f9533 100644
--- a/gitm/planner/moe_graph.py
+++ b/gitm/planner/moe_graph.py
@@ -69,6 +69,7 @@
     HardwareSpec,
     ShardingConfig,
     SparseMoEModelSpec,
+    _canon_dtype,
     distinct_experts,
     roofline,
     weight_bytes,
@@ -142,14 +143,14 @@ def validate_sparse_moe_config(cfg: dict[str, Any]) -> list[str]:
         errors.append("quantization_config must be an object when declared")
     elif isinstance(q, dict):
         method = q.get("quant_method")
-        if method is not None and weight_bytes_is_fallback(str(method).lower()):
+        if method is not None and weight_bytes_is_fallback(str(method)):
             errors.append(f"quantization_config.quant_method={method!r} is not priceable")
 
     for key in ("expert_dtype", "torch_dtype"):
         value = cfg.get(key)
         if value is None:
             errors.append(f"{key} must be declared; byte width cannot be guessed")
-        elif weight_bytes_is_fallback(str(value).lower().replace("bfloat16", "bf16")):
+        elif weight_bytes_is_fallback(str(value)):
             errors.append(f"{key}={value!r} is not priceable")
     return errors
 
@@ -632,11 +633,11 @@ def spec_from_hf_config(cfg: dict[str, Any], *, name: str | None = None) -> Spar
         raise ValueError("sparse-MoE config is not predictable: " + "; ".join(errors))
 
     q = cfg.get("quantization_config") or {}
-    act_dtype = str(cfg.get("torch_dtype", "bf16")).lower().replace("bfloat16", "bf16")
+    act_dtype = _canon_dtype(str(cfg.get("torch_dtype", "bf16")))
     # An unquantized checkpoint stores weights at its declared model dtype. bf16
     # is not a safe universal default: it halves fp32 bytes and changes the peak
     # selected for fp16/bf16 on hardware whose rates differ.
-    weight_dtype = str(q.get("quant_method") or act_dtype).lower()
+    weight_dtype = _canon_dtype(str(q.get("quant_method") or act_dtype))
     n_layers = int(cfg.get("num_hidden_layers", 43))
     ratios = tuple(int(r) for r in (cfg.get("compress_ratios") or ())[:n_layers])
 
@@ -665,7 +666,7 @@ def spec_from_hf_config(cfg: dict[str, Any], *, name: str | None = None) -> Spar
         dspark_layer_ids=tuple(int(i) for i in (cfg.get("dspark_target_layer_ids") or ())),
         dspark_markov_rank=int(cfg.get("dspark_markov_rank", 0)),
         weight_dtype=weight_dtype,
-        expert_dtype=str(cfg.get("expert_dtype", weight_dtype)).lower(),
+        expert_dtype=_canon_dtype(str(cfg.get("expert_dtype", weight_dtype))),
         # No cache dtype lives in a model config. The serving default is ``auto``
         # (follow the compute/model dtype); live callers replace this only when
         # the engine or command line declares a different cache dtype.
diff --git a/gitm/planner/roofline.py b/gitm/planner/roofline.py
index 1da15da..abc1e5a 100644
--- a/gitm/planner/roofline.py
+++ b/gitm/planner/roofline.py
@@ -37,6 +37,22 @@
 _WEIGHT_BYTES["fp4"] = _WEIGHT_BYTES["mxfp4"]
 
 
+def _canon_dtype(dtype: str) -> str:
+    """Normalize framework/config aliases before pricing or fallback checks."""
+    d = str(dtype).lower().removeprefix("torch.")
+    if d in ("bf16", "bfloat16"):
+        return "bf16"
+    if d in ("float16", "fp16", "half"):
+        return "fp16"
+    if d in ("fp4", "mxfp4", "nvfp4"):
+        return "fp4"
+    if d in ("fp8", "e4m3", "e5m2"):
+        return "fp8"
+    if d in ("fp32", "float32", "tf32"):
+        return "fp32"
+    return d
+
+
 def weight_bytes(dtype: str) -> float:
     """Bytes moved per stored weight for ``dtype``, scales included.
 
@@ -44,12 +60,12 @@ def weight_bytes(dtype: str) -> float:
     since over-counting weight traffic predicts a *slower* floor and so cannot
     manufacture headroom.
     """
-    return _WEIGHT_BYTES.get(dtype.lower(), 2.0)
+    return _WEIGHT_BYTES.get(_canon_dtype(dtype), 2.0)
 
 
 def weight_bytes_is_fallback(dtype: str) -> bool:
     """Whether :func:`weight_bytes` substitutes bf16 for an unknown dtype."""
-    return dtype.lower() not in _WEIGHT_BYTES
+    return _canon_dtype(dtype) not in _WEIGHT_BYTES
 
 
 @dataclass(frozen=True)
@@ -601,21 +617,6 @@ def peak_is_fallback(self) -> bool:
         return _canon_dtype(self.dtype) != self.peak_dtype
 
 
-def _canon_dtype(dtype: str) -> str:
-    d = dtype.lower()
-    if d in ("bf16", "bfloat16"):
-        return "bf16"
-    if d in ("float16", "fp16", "half"):
-        return "fp16"
-    if d in ("fp4", "mxfp4", "nvfp4"):
-        return "fp4"
-    if d in ("fp8", "e4m3", "e5m2"):
-        return "fp8"
-    if d in ("fp32", "float32", "tf32"):
-        return "fp32"
-    return d
-
-
 def resolve_peak(hw: HardwareSpec, dtype: str) -> tuple[float, str]:
     """(peak FLOP/s, the dtype that peak belongs to) for ``dtype`` on ``hw``.
 
diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py
index 5a9fcda..e3156e0 100644
--- a/tests/test_moe_graph.py
+++ b/tests/test_moe_graph.py
@@ -726,7 +726,9 @@ def test_kv_footprint_splits_growing_from_fixed(spec):
 
     # Growth comes only from the 41 compressed layers, at 1/4 or 1/128 of a latent.
     naive_all_layers = spec.n_layers * (spec.kv_latent_dim + spec.index_head_dim)
-    assert per_token < naive_all_layers / 5
+    # Excluding bounded sliding-window layers keeps growth below one third of
+    # the all-layers naive rate for the declared KV dtype.
+    assert per_token < naive_all_layers / 3
 
     # The two window layers are real, bounded, and paid once per sequence.
     assert fixed == 2 * spec.sliding_window * spec.kv_latent_dim * weight_bytes(spec.kv_dtype)
diff --git a/tests/test_scorer.py b/tests/test_scorer.py
index 985b022..3eccdd4 100644
--- a/tests/test_scorer.py
+++ b/tests/test_scorer.py
@@ -1,4 +1,5 @@
 import pandas as pd
+import pytest
 
 from gitm.routing.scorer_v0 import score_dataframe, score_prospect
 
@@ -26,16 +27,15 @@ def test_score_prospect_cold():
     assert score == 7.5
 
 def test_score_prospect_unknown_tier():
-    score = score_prospect(
-        warmth=0.5,
-        signal_recency=0.5,
-        company_tier=99,
-        pain_acknowledged=0,
-        engagement_score=0.0,
-        prior_engagement=0
-    )
-    # Unknown tier should default to tier 3 (0.2)
-    assert score == 34.0
+    with pytest.raises(ValueError, match="company_tier must be 1, 2, or 3"):
+        score_prospect(
+            warmth=0.5,
+            signal_recency=0.5,
+            company_tier=99,
+            pain_acknowledged=0,
+            engagement_score=0.0,
+            prior_engagement=0,
+        )
 
 def test_score_dataframe_sorted():
     df = pd.DataFrame({

From f647ea4da709d30ef69fd44ee2558ab357ce091c Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 18:53:44 -0700
Subject: [PATCH 66/70] docs: record audit verification and branch split

---
 AUDIT.md | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/AUDIT.md b/AUDIT.md
index 8700312..e1fd6dd 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -220,3 +220,22 @@ human- or gate-visible consumer.
 ## Completeness pass
 
 No outstanding items.
+
+## Verification and review readiness
+
+- Targeted tests for touched runtime/importer/CUPTI/comparison paths: **73 passed**;
+  targeted planner/routing tests after the final alias fix: **75 passed**.
+- Full repository run: **1067 passed, 1 skipped**, with only named runtime warnings
+  for optional GPU/vLLM/CUDA telemetry and intentional fallback demonstrations.
+  Importer fixtures were restored after the run; `git status --short` is clean.
+- Ruff passed on all touched modules and tests; `git diff --check` is clean.
+- The audit branch is `audit/graceful-fallbacks` and contains only audit fixes plus
+  this ledger. The clean expert-signal branch is `feat/expert-signal-eplb` from
+  `main` (`ca289d6`), containing the signal module and its tests; caller wiring is
+  intentionally the next feature step, not an audit change. The pre-existing
+  wiring snapshot remains preserved on `feat/expert-signal-eplb-stacked` (`452b2c8`)
+  for the maintainer to port or review.
+- Review recommendation: one audit PR is reviewable by theme because each commit
+  is a bounded REFUSE/FLAG/WARN fix and this ledger is the summary. Keep the
+  expert-signal work as a separate follow-up PR; do not open multiple audit PRs
+  without maintainer selection.

From 6898ff047892f19b6c20bd9a2cabf1575c91e21f Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Tue, 11 Aug 2026 18:54:33 -0700
Subject: [PATCH 67/70] docs: clarify separated expert signal branches

---
 AUDIT.md | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/AUDIT.md b/AUDIT.md
index e1fd6dd..5e9e366 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -12,9 +12,11 @@ Fallback masks and wiring failures closed: **93**. Wiring gaps confirmed:
 explicitly REFUSE, FLAG, or WARN and has a visible consumer.
 
 The worktree already contained uncommitted scheduler/serve changes and two new
-expert-signal files before this audit branch was created. They were separated into
-`feat/expert-signal-eplb` (commit `452b2c8`); the Codex-only `AGENTS.md` guidance is
-on `chore/agents-guidance` (commit `4796a54`). Neither is part of this audit branch.
+expert-signal files before this audit branch was created. The clean signal module
+and tests are separated into `feat/expert-signal-eplb` (commit `ca289d6`), while
+the pre-existing wiring snapshot is preserved on `feat/expert-signal-eplb-stacked`
+(commit `452b2c8`). The Codex-only `AGENTS.md` guidance is on
+`chore/agents-guidance` (commit `4796a54`). None is part of this audit branch.
 
 ## Finding ledger
 

From 8f034308a49a1e8cff89f917571e99fa48c342ab Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Fri, 14 Aug 2026 13:15:14 -0700
Subject: [PATCH 68/70] docs: record synchronized audit verification

---
 AUDIT.md | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/AUDIT.md b/AUDIT.md
index 5e9e366..b511b55 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -227,12 +227,13 @@ No outstanding items.
 
 - Targeted tests for touched runtime/importer/CUPTI/comparison paths: **73 passed**;
   targeted planner/routing tests after the final alias fix: **75 passed**.
-- Full repository run: **1067 passed, 1 skipped**, with only named runtime warnings
+- Full repository run after synchronizing with `origin/main`: **1100 passed, 1 skipped**, with only named runtime warnings
   for optional GPU/vLLM/CUDA telemetry and intentional fallback demonstrations.
   Importer fixtures were restored after the run; `git status --short` is clean.
 - Ruff passed on all touched modules and tests; `git diff --check` is clean.
-- The audit branch is `audit/graceful-fallbacks` and contains only audit fixes plus
-  this ledger. The clean expert-signal branch is `feat/expert-signal-eplb` from
+- The audit branch is `audit/graceful-fallbacks`; it merges the current
+  `origin/main` at `f63879b` and contains the audit fixes plus the upstream MoE
+  architecture updates. The clean expert-signal branch is `feat/expert-signal-eplb` from
   `main` (`ca289d6`), containing the signal module and its tests; caller wiring is
   intentionally the next feature step, not an audit change. The pre-existing
   wiring snapshot remains preserved on `feat/expert-signal-eplb-stacked` (`452b2c8`)

From 0e7da1ee81536b05479253b5affc46c9b7731f4d Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Fri, 14 Aug 2026 13:17:56 -0700
Subject: [PATCH 69/70] fix: satisfy CI lint for sparse config validation

---
 gitm/planner/moe_graph.py  | 2 +-
 gitm/serve/model_config.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/gitm/planner/moe_graph.py b/gitm/planner/moe_graph.py
index d96ea75..d5acb70 100644
--- a/gitm/planner/moe_graph.py
+++ b/gitm/planner/moe_graph.py
@@ -102,7 +102,7 @@ def validate_sparse_moe_config(cfg: dict[str, Any]) -> list[str]:
     if isinstance(n_experts, int) and not isinstance(n_experts, bool) and isinstance(top_k, int) and not isinstance(top_k, bool) and top_k > n_experts:
         errors.append(f"num_experts_per_tok={top_k} exceeds n_routed_experts={n_experts}")
     n_layers, ratios = cfg.get("num_hidden_layers"), cfg.get("compress_ratios")
-    if not isinstance(ratios, (list, tuple)):
+    if not isinstance(ratios, list | tuple):
         errors.append("compress_ratios must be declared for the sparse-attention graph")
     elif isinstance(n_layers, int) and n_layers > 0 and len(ratios) < n_layers:
         errors.append(f"compress_ratios has {len(ratios)} entries; need at least {n_layers}")
diff --git a/gitm/serve/model_config.py b/gitm/serve/model_config.py
index b8947de..95e27f1 100644
--- a/gitm/serve/model_config.py
+++ b/gitm/serve/model_config.py
@@ -291,7 +291,7 @@ def positive_alias(keys: tuple[str, ...], label: str) -> int | None:
 
     n_layers = cfg.get("num_hidden_layers")
     ratios = cfg.get("compress_ratios")
-    if not isinstance(ratios, (list, tuple)):
+    if not isinstance(ratios, list | tuple):
         missing.append("compress_ratios must be declared for the sparse-attention graph")
     elif isinstance(n_layers, int) and n_layers > 0 and len(ratios) < n_layers:
         missing.append(

From 0193c416f79c5a6302dd78e16d4af6a391304058 Mon Sep 17 00:00:00 2001
From: Nicholas Lawrence <nicholaslawrence@berkeley.edu>
Date: Fri, 14 Aug 2026 14:08:47 -0700
Subject: [PATCH 70/70] docs: summarize graceful fallback audit changes

---
 AUDIT.md | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/AUDIT.md b/AUDIT.md
index b511b55..0ccdff4 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -18,6 +18,25 @@ the pre-existing wiring snapshot is preserved on `feat/expert-signal-eplb-stacke
 (commit `452b2c8`). The Codex-only `AGENTS.md` guidance is on
 `chore/agents-guidance` (commit `4796a54`). None is part of this audit branch.
 
+## Changes in this PR
+
+This branch is an audit layer on top of the current runtime, not a planner
+rewrite. It first synchronizes with `origin/main` at `f63879b`, then preserves
+the upstream sparse-MoE architecture while making every fallback explicit at
+the boundary where it can affect a user-facing claim.
+
+| Theme | What changed | User-visible result |
+|---|---|---|
+| Graph prediction and MoE pricing | Added strict sparse-config and final-dtype priceability gates; retained upstream CSA/HCA, hash-routing, and live-config dispatch; separated compute, weight, activation, expert, and KV byte provenance. | Unknown or incomplete inputs refuse graph-based claims or become an explicitly measurement-only result; valid graphs retain their normal architecture-specific pricing. |
+| Fallback provenance | Added per-node and graph-level flags for fallback bytes, peaks, hardware, estimates, and unpriced dimensions. | Attach and scheduler JSON, summaries, reports, and CLI diagnostics identify exactly which denominator or byte width was substituted. |
+| Runtime evidence | Hardened timing, work-unit, telemetry, importer, CUPTI, benchmark, A/B, and cleanup paths against missing, non-finite, zero, or contradictory values. | A missing answer-driving field is refused; an optional degradation is named as a warning instead of being converted into a plausible number. |
+| Claims and residuals | Preserved raw residual magnitude, derived display capping/saturation, claim scope, and matched/total coverage. | Reports can show a capped percentage without hiding the underlying error or repeating one aggregate as if it were a node-level claim. |
+| Calibration and topology | Kept calibration opt-in, source-run aware, freshness-checked, and provenance-bearing; validated topology identity, shape, and later-trace requirements. | Calibration cannot silently self-apply to its source run or cross an incompatible workload/shape, and partial coverage is visible. |
+| Artifacts and verification | Traced diagnostics through `predicted_graph.json`, refusal/measurement artifacts, reports, and CLI status; added regression coverage for direct builders and live dispatch. | Every accepted fallback has a visible consumer, and no generated planner JSON is checked into the repository. |
+
+The merge synchronization is recorded in `2975a72`; the verification ledger was
+updated in `8f03430`, and the final CI lint compatibility fix is `0e7da1e`.
+
 ## Finding ledger
 
 | Rank | Status | Severity | Location | Distorted term / contract | Surfacing state | Failure scenario | Disposition |