diff --git a/AUDIT.md b/AUDIT.md new file mode 100644 index 0000000..0ccdff4 --- /dev/null +++ b/AUDIT.md @@ -0,0 +1,263 @@ +# Graceful Fallback and Runtime Wiring Audit + +## Executive summary + +Status: **complete**. This ledger is the primary deliverable for the audit of +`gitm/` and `scripts/`. Findings are ranked by the likelihood that a fallback can +turn missing knowledge into a confident wrong result, with answer-deciding byte +traffic and dominant expert terms ranked above non-binding estimates. + +Fallback masks and wiring failures closed: **93**. Wiring gaps confirmed: +**12**. The final pass found no deferred findings: every accepted fallback is +explicitly REFUSE, FLAG, or WARN and has a visible consumer. + +The worktree already contained uncommitted scheduler/serve changes and two new +expert-signal files before this audit branch was created. The clean signal module +and tests are separated into `feat/expert-signal-eplb` (commit `ca289d6`), while +the pre-existing wiring snapshot is preserved on `feat/expert-signal-eplb-stacked` +(commit `452b2c8`). The Codex-only `AGENTS.md` guidance is on +`chore/agents-guidance` (commit `4796a54`). None is part of this audit branch. + +## Changes in this PR + +This branch is an audit layer on top of the current runtime, not a planner +rewrite. It first synchronizes with `origin/main` at `f63879b`, then preserves +the upstream sparse-MoE architecture while making every fallback explicit at +the boundary where it can affect a user-facing claim. + +| Theme | What changed | User-visible result | +|---|---|---| +| Graph prediction and MoE pricing | Added strict sparse-config and final-dtype priceability gates; retained upstream CSA/HCA, hash-routing, and live-config dispatch; separated compute, weight, activation, expert, and KV byte provenance. | Unknown or incomplete inputs refuse graph-based claims or become an explicitly measurement-only result; valid graphs retain their normal architecture-specific pricing. | +| Fallback provenance | Added per-node and graph-level flags for fallback bytes, peaks, hardware, estimates, and unpriced dimensions. | Attach and scheduler JSON, summaries, reports, and CLI diagnostics identify exactly which denominator or byte width was substituted. | +| Runtime evidence | Hardened timing, work-unit, telemetry, importer, CUPTI, benchmark, A/B, and cleanup paths against missing, non-finite, zero, or contradictory values. | A missing answer-driving field is refused; an optional degradation is named as a warning instead of being converted into a plausible number. | +| Claims and residuals | Preserved raw residual magnitude, derived display capping/saturation, claim scope, and matched/total coverage. | Reports can show a capped percentage without hiding the underlying error or repeating one aggregate as if it were a node-level claim. | +| Calibration and topology | Kept calibration opt-in, source-run aware, freshness-checked, and provenance-bearing; validated topology identity, shape, and later-trace requirements. | Calibration cannot silently self-apply to its source run or cross an incompatible workload/shape, and partial coverage is visible. | +| Artifacts and verification | Traced diagnostics through `predicted_graph.json`, refusal/measurement artifacts, reports, and CLI status; added regression coverage for direct builders and live dispatch. | Every accepted fallback has a visible consumer, and no generated planner JSON is checked into the repository. | + +The merge synchronization is recorded in `2975a72`; the verification ledger was +updated in `8f03430`, and the final CI lint compatibility fix is `0e7da1e`. + +## Finding ledger + +| Rank | Status | Severity | Location | Distorted term / contract | Surfacing state | Failure scenario | Disposition | +|---:|---|---|---|---|---|---|---| +| 1 | fixed | critical | `gitm/scheduler/loop.py` execution-graph dispatcher | Entire sparse-MoE execution graph; especially dominant expert weight bytes and hybrid-attention/KV terms | REFUSE/FLAG | A live V4 engine was converted into the legacy `ModelSpec` and sent to `predict_graph`; the production loop never called `spec_from_hf_config`/`predict_moe_graph`. | Added `_execution_graph`: partial sparse configs reach the sparse gate, valid configs dispatch to `predict_moe_graph`, and unpriceable configs refuse claims into a named measurement-only result. The full graph provenance/flags are serialized. | +| 2 | fixed | critical | `gitm/scheduler/loop.py` config resolution and refusal artifact | Model identity and every residual | REFUSE | Parse/version-drift failures returned `None` and became the plausible Llama-2-7B default graph. | Typed resolution preserves exception class/message in `prediction_refusal.json` and the report; missing engine/config refuses graph-based claims instead of defaulting. | +| 3 | fixed | high | `gitm/planner/context.py`; `gitm/planner/roofline.py`; loop and attach artifacts | Compute and HBM denominators for every node | REFUSE/FLAG/WARN | Unknown/no GPU SKU became A100 and artifacts recorded A100 as if detected. | `HardwareSpec.is_fallback`/`Graph.hardware_is_fallback` distinguish substituted pricing. The loop refuses unknown hardware; attach records observed hardware separately from fallback pricing and prints a warning. | +| 4 | fixed | high | roofline, sparse graph, shared final-spec dtype gate, loop/attach boundaries | Weight/KV/activation bytes; dominant decode-binding term | FLAG/REFUSE | Unknown dtype returned 2-byte bf16; mixed-byte nodes and scalar sizing could not identify the substituted width. | Per-node `bytes_are_fallback` and `Graph.has_fallback_bytes` now cover the graph, while shared `validate_priceable_dtypes` refuses final resolved live/command-line dtypes before customer-facing predictions. JSON and CLI/report consumers surface both directions. | +| 5 | fixed | high | `gitm/scheduler/loop.py`; `gitm/optimizer/report.py`; report template | Residual magnitude shipped on every Claim | FLAG | A 10x/18x model error and a 2x error both rendered `+100%`, hiding that the model was broken and repeating one aggregate as if claim-specific. | Raw residuals now remain on `Claim`; display capping and saturation are derived, saturated rows print raw magnitude, scopes distinguish run/target-op, and non-finite residuals are refused. Report, scheduler sibling, and ruff checks pass. | +| 6 | fixed | high | `gitm/optimizer/monitor.py`; `gitm/scheduler/loop.py`; report template | Residual population / coverage | WARN | Unclassified kernels and classified ops absent from the graph were skipped, so the loop could report clean residuals over a small, biased fraction without matched/total coverage. | `Residuals` now records total/classified/matched launch and kernel-time coverage. Incomplete coverage emits warnings into `residuals.json`, the run summary, and a printed Runtime diagnostics section; fully matched traces stay clean. Monitor/report/loop sibling tests and ruff pass. | +| 7 | fixed | high | loop dispatcher dense/sparse quantization gates | Dense-path MoE expert weight bytes | REFUSE | Unknown quantization methods were ignored, leaving bf16 byte defaults. | Sparse configs no longer use the legacy dense extraction; both sparse and dense dispatch refuse unknown quantization methods with the method named. | +| 8 | fixed | medium | sparse config resolution in loop and attach | Expert weights, often the dominant term | WARN | Missing `expert_dtype` inherited linear `weight_dtype`. Official V4 Flash configs declare it (Flash=`fp4`, Base=`fp8`), but uniform foreign MoEs may omit it legitimately. | Accepted inheritance now rides in `LiveSpec.warnings` / loop diagnostics and reaches artifacts plus human output; unpriceable inherited dtypes still refuse. | +| 9 | fixed | medium | `gitm/planner/graph.py` | Zero-time byte-moving nodes | FLAG | `has_unpriced_collectives` scanned all nodes despite its narrow name. | Split `has_unpriced_nodes` (general safety net) from the genuinely collective-specific property; production trust consumers use the general flag and artifacts retain both. | +| 10 | fixed | high | loop `predicted_graph.json`, summary, and Markdown diagnostics | Prediction trust diagnostics | FLAG/WARN | Loop artifacts omitted fallback, estimate, default, model, batch, sharding, and hardware provenance. | Artifact now carries model source, observed/pricing hardware, batch/sharding, graph flags, per-node diagnostics, and warnings; the report and run summary consume them. | +| 11 | fixed | critical | `gitm/bench/schema.py`; `gitm/bench/baseline.py` | Benchmark saturation/sign-off gate | REFUSE | A missing `stall_breakdown` became 0% GPU active and could sign off a CPU/no-telemetry run as unsaturated. | Missing coverage is now `None` and fails saturation; code, manifest, and GPU identity have a separate provenance gate and safe report rendering. | +| 12 | fixed | high | `gitm/tracer/vllm_stats.py`; `gitm/serve/vllm.py` | TPOT percentiles and SLO goodput | FLAG/WARN | SSE chunk counts undercounted multi-token chunks but entered TPOT and goodput as authoritative; missing counts passed the TPOT half of the SLO. | Usage/engine counts are authoritative, chunk estimates are excluded from TPOT/goodput, and coverage warnings reach CLI and loop reports. | +| 13 | fixed | high | `gitm/planner/roofline.py`; `gitm/planner/graph.py` | Compute/HBM denominator | FLAG/WARN | Positive work with a zero catalogue rate was priced at zero and a sibling term could hide the missing denominator. | Per-dimension unpriced flags survive on each prediction, aggregate on the graph, and surface through loop/attach artifacts and diagnostics. | +| 14 | fixed | high | `gitm/runtime_driver.py` | Trace coverage and every claimed runtime detail | REFUSE/WARN | Zero captured kernels still produced `PASS: ... all details measured`. | The driver now uses canonical measurement, emits NO DATA, records diagnostics, and exits 3 without any positive-duration kernel. | +| 15 | fixed | high | `gitm/optimizer/attribution.py`; `gitm/optimizer/dr.py`; loop/driver/report consumers | Causal evidence | WARN | Import/fit failure became `no strong causal signal`; DR nuisance-model fallbacks emitted estimates silently. | Attribution carries import, sample-coverage, pair-fit, and nuisance-model diagnostics into JSON, CLI, and Markdown. | +| 16 | fixed | high | `gitm/kernels/library.py`; scheduler caller | Intervention availability / candidate coverage | REFUSE/WARN | A missing library returned `[]`, indistinguishable from no applicable levers. | The loader refuses with the path; the scheduler emits a named candidate-coverage-unavailable measurement report and no optimization claims. | +| 17 | fixed | medium | `gitm/telemetry/collector.py`; benchmark samplers and runtime-driver consumer | State-telemetry coverage | WARN | Sampling, sink, and close failures were swallowed, making empty telemetry look like a quiet GPU. | Collector failures are deduplicated warnings and report diagnostics; benchmark sampler failures ride into JSON and stdout. | +| 18 | fixed | high | scheduler specialized HFT/OpenFold/edge intervention result paths | Residual and intervention status | FLAG/REFUSE | No CUPTI trace attached A/B speedup to fabricated `stream_concurrency=0.0`; missing A/B still reported `ok`. | All siblings use measured throughput delta without trace coverage; missing/non-finite A/B emits no claim and returns `intervention_failed`. | +| 19 | fixed | medium | `gitm/optimizer/headroom_kernel_rank.py` | Compute and memory headroom | FLAG/WARN | Memory-only samples fabricated 100% compute headroom; utilization-only samples fabricated zero memory capacity. | Each dimension is optional, absent families stay `None`, and diagnostics name missing telemetry. | +| 20 | fixed | medium | `gitm/optimizer/measure.py`; duplicated runtime-driver measurement | Kernel residual denominator | WARN/REFUSE | Zero-duration kernels used a fabricated 1 ns median and attribution filtering had no coverage diagnostic. | Invalid durations are excluded with counts, attribution abstention is diagnostic, and both consumers use canonical measurement. | +| 21 | fixed | critical | `gitm/deploy/attach.py` | Live deployment state | REFUSE | Resolving a live standalone PID returned `attached` even though the documented injection path is not implemented and no shim was installed. | The path now returns `unsupported` with the PID and a named no-injection reason; only an implemented lifecycle may claim attachment. | +| 22 | fixed | high | `gitm/cli.py` | Automation/gate status | REFUSE | `prediction_refused`, missing-candidate coverage, intervention failure, and malformed loop results all exited zero; an absent report could also look like a completed run. | Only `status=ok` exits zero. Degraded or invalid results exit 3 and always emit an explicit unavailable/diagnostic report. Report writes are UTF-8. | +| 23 | fixed | high | `gitm/scheduler/loop.py`; `gitm/agents/autoresearch.py` | Trace evidence and bottleneck class | REFUSE/FLAG | A trace containing only zero-duration events bypassed the no-data gate; an empty/invalid trace was classified `compute`, enabling plausible compute interventions without evidence. | Loop evidence requires a positive-duration kernel, and autoresearch emits `unclassified` for empty/invalid traces with no applicable rules. | +| 24 | fixed | high | shared timing predicate; workloads, runtime driver, HFT/edge benchmarks, demo script | Throughput, latency, and speedup denominators | REFUSE | Non-positive/non-finite timers were floored to 1 ns, turning timer failure or empty work into enormous plausible throughput/speedup. | Shared finite-positive duration/work predicates now refuse each affected calculation with its named context. | +| 25 | fixed | high | `gitm/bench/profile.py`; `gitm/bench/cli.py` | Profile completeness and utilization breakdown | REFUSE | Failed workload commands, sampler timeouts, missing profiler CSV, and overlapping timing could still produce a successful profile command with clamped shares. | The bundle records every failure, contradictory timing refuses rather than clamps, and incomplete profile/empty manifest commands exit nonzero. | +| 26 | fixed | medium | `gitm/agents/autoresearch.py` | Search-domain coverage | WARN | Missing/version-drifted vLLM `EngineArgs` silently substituted a frozen knob catalog; missing CLI introspection silently narrowed domains. | Both offline catalogs now emit named runtime warnings; empty introspection remains explicitly empty rather than pretending live coverage. | +| 27 | fixed | medium | `gitm/workloads.py`; HFT harness | Benchmark provenance and trace completeness | WARN/REFUSE | Missing data silently generated synthetic smoke input, missing GPU libraries silently selected CPU, and failed synchronization/pool cleanup looked like complete GPU coverage. | Synthetic/CPU/cleanup/synchronization fallbacks warn at use; existing benchmark provenance gates still refuse these runs as publishable GPU baselines. | +| 28 | fixed | medium | `gitm/safety/failopen.py` | Rollback/audit visibility | WARN/FLAG | A broken audit sink or failed signal-handler setup was swallowed, removing the very evidence meant to surface fail-open behavior. | Audit and signal failures are retained on the guard and warned immediately; reset/restore behavior is covered. | +| 29 | fixed | high | `gitm/kernels/library.py` | Candidate/intervention coverage | REFUSE | A present but malformed or empty intervention library returned no candidates just like a valid library with no applicable matches. | The loader refuses non-mapping, missing-list, and empty-list libraries with named reasons; scheduler coverage handling remains explicit. | +| 30 | fixed | medium | `gitm/routing/scorer_v0.py` | Routing score | REFUSE | Out-of-range probabilities, non-binary flags, and unknown company tiers flowed into a plausible score (unknown tier became a default weight). | The scorer validates every bounded/binary input and refuses tiers outside 1/2/3. | +| 31 | fixed | high | `gitm/importers/node_rollup.py` | Communication share and node ceiling distance | REFUSE | Zero/negative trace wall time was floored to 1 ns and zero-weight ceiling aggregation returned 0.0, fabricating clean rollup values. | Device and node rollups require positive wall time; invalid captures become named per-file analysis failures. | +| 32 | fixed | medium | `gitm/importers/analyze.py` | Kept-trace artifact wiring | REFUSE/fix | Internal identifiers containing `:` were used as filenames; on Windows the write failed and the whole input was demoted to a generic per-file failure with no trace artifact. | Artifact-only stems are sanitized portably while report identifiers retain their original value; end-to-end importer coverage verifies artifacts are written. | +| 33 | fixed | high | `gitm/tracer/vllm_stats.py` | TTFT, TPOT, and SLO goodput | REFUSE/WARN | Backwards or non-finite request timestamps were clamped to zero latency and could count as SLO-meeting traffic. | Invalid timestamp spans now produce no latency/goodput contribution and a serving warning naming excluded requests. | +| 34 | fixed | high | `gitm/tracer/vllm_stats.py`; scheduler artifact/report | Scheduler evidence coverage | WARN/FLAG | Synchronous/background scheduler reads swallowed every exception and returned an ordinary empty summary; a failed sampler looked like an idle/uninstrumented engine. | The fail-open sampler warns once per failure path, retains diagnostics in its summary/artifact, and feeds them into runtime diagnostics. Invalid/clamped intervals also surface. | +| 35 | fixed | high | NVIDIA telemetry backend and collector | Throttling, process, utilization, memory, power, clocks, and ECC state | FLAG/WARN | Per-field NVML failures became `NONE`, `{}`, `nvidia-unknown-*`, or `None`; especially, unavailable throttle reasons looked like observed no-throttle state. | Partial samples now carry field-level diagnostics; the collector deduplicates, warns, and carries them downstream. Backend close failures propagate to the collector instead of disappearing. | +| 36 | fixed | high | `gitm/optimizer/monitor.py`; `deviation.py` | Kernel-time residual and deviation population | WARN/FLAG | Zero/negative kernel durations were floored to 1 ps and entered residuals; non-positive predicted durations were likewise made comparable. | Invalid observations are excluded with launch-count diagnostics; unpriced predictions are excluded explicitly; deviation traces retain invalid kernels as departures rather than declaring them in-band. | +| 37 | fixed | high | `gitm/optimizer/metrics.py` | Busy, HFU/MFU, and MBU denominators | REFUSE | Zero trace wall time or zero peak bandwidth returned plausible 0% utilization. | Shared positive-duration validation and positive hardware-denominator gates now refuse direct callers; normal/importer siblings remain green. | +| 38 | fixed | high | `gitm/optimizer/apply.py` | Intervention verification and safety trail | WARN | Apply-only changes, broken audit sinks, activation/shutdown hooks, and GPU cleanup failures were silent; a mutation could be kept without evidence or lose rollback telemetry. | Unmeasured applies and every best-effort safety/lifecycle failure now warn while preserving fail-open behavior. | +| 39 | fixed | critical | dense scheduler graph parser; obsolete parallel parser | Dense activation/weight bytes and hybrid-attention graph shape | REFUSE/fix | Production treated every unknown dense dtype as 2-byte bf16 and ignored hybrid-attention cadence; a more capable engine parser existed only in tests and silently returned `None`. | Production parsing uses shared dtype priceability/byte widths, preserves attention cadence, and names missing/invalid fields. The dead parallel implementation and its fallback tests were removed; coverage targets the production parser. | +| 40 | fixed | high | `gitm/serve/metrics.py`; attach output | Server throughput, token count, and TPOT | WARN/REFUSE | Infinite Prometheus values entered aggregates, non-positive windows could price throughput, sampler scrape holes disappeared, and missing token/TPOT fields printed as zero. | Non-finite values are discarded, invalid windows emit no rate plus a note, sampler failures surface, and human output prints `unavailable` rather than zero. | +| 41 | fixed | critical | scheduler default live-engine throughput probe | A/B throughput and keep/rollback decision | REFUSE | A runner with no recognized work-count field silently became one generated token; a failed timer was floored to 1 ns. | The probe requires a named positive work count and positive finite duration; missing/zero evidence rolls the candidate back through the existing apply gate. | +| 42 | fixed | high | KITTI/nuScenes WorkUnit and baseline runners | Stall shares and FPS | REFUSE | A zero frame timer returned 0% for every stage, and a zero baseline timer divided into FPS. | Frame properties and baseline windows share the positive-duration gate; output writes are UTF-8. | +| 43 | fixed | medium | `gitm/planner/kitti_graph.py` | Planner wiring and hardware provenance | REFUSE | The PointPillars graph had no production caller and its example defaulted to A100; measured comparisons defaulted missing FPS/stall fields to zero. | Added a SKU-required `gitm plan-kitti` boundary that refuses catalogue misses; measured comparison refuses missing/non-positive fields. | +| 44 | fixed | medium | diagnostic/demo scripts | GPU-idle decision, real-trace residuals, and assumed hardware | WARN/REFUSE | Demo telemetry failure silently triggered the idle-GPU lever, real-trace code suppressed all warnings and floored zero medians, and serving headroom silently assumed H100/zero failures. | Scripts now refuse telemetry-less idle claims, exclude and warn on invalid timestamps, preserve warnings, and state assumed hardware/missing failure counts explicitly. | +| 45 | fixed | high | dense/A-B graph priceability boundary | Dense graph bytes and A/B baseline terms | REFUSE | A dense or A/B input with an unknown dtype/quantization method could still enter prediction or comparison through a caller that bypassed the main parser. | The shared priceability predicate is applied at both graph and A/B boundaries; unpriceable inputs produce a named refusal. | +| 46 | fixed | high | `gitm/serve/metrics.py` server metric aggregation | Throughput, tokens, and latency | WARN/REFUSE | Missing or non-finite server fields were treated as zero and then aggregated into a confident serving result. | Invalid fields are excluded with sampler diagnostics; invalid windows refuse rates and human output says `unavailable`. | +| 47 | fixed | medium | `gitm/serve/discover.py`; telemetry sinks | Server/model discovery and sink state | WARN | An inaccessible `/proc` or failed sink could look like no server or a quiet run. | Discovery and sink failures carry named diagnostics to the caller and artifacts; no empty discovery is presented as verified. | +| 48 | fixed | high | replay validation and verification evidence | Replay identity and validation truth | REFUSE | A replay with missing identity or contradictory validation fields could be compared as reproducible. | Replay/verification gates require complete identity, schema, package, and validation evidence; missing fields mismatch rather than default. | +| 49 | fixed | high | dense planner compute-dtype extraction | Dense activation/weight byte width | REFUSE | A compute dtype missing from a dense spec fell back to bf16 while the graph retained a plausible shape. | Dense extraction uses the shared dtype validator and refuses unknown or missing compute dtype. | +| 50 | fixed | medium | `gitm/optimizer/headroom.py` evidence split | Compute/memory headroom | WARN/FLAG | One telemetry family absent could be represented as a fabricated 50/50 split. | Missing dimensions remain absent; an indicative split is explicitly labeled and the report states the limitation. | +| 51 | fixed | high | benchmark A/B and cleanup paths | Speedup, rollback, and run completeness | REFUSE/WARN | A failed A/B sample or cleanup hook could leave an apparently successful intervention. | Non-finite/empty A/B evidence refuses claims; cleanup failures are retained as diagnostics and reports are degraded. | +| 52 | fixed | high | serving trace duration gates | Serving throughput and SLO windows | REFUSE | A zero-duration serving trace could be turned into a large rate or an all-good SLO. | Positive finite timing is required; invalid requests are excluded and the serving result is degraded/refused. | +| 53 | fixed | high | vLLM CUDA compatibility gate | CUDA/runtime compatibility | REFUSE/WARN | An incompatible or unverified CUDA build could run through the workload path as if supported. | The compatibility preflight gates the workload and records unverified build details in the report. | +| 54 | fixed | high | report delta and claim formatting | Residuals and deltas | REFUSE | NaN/inf deltas could serialize as credible percentages or pass a gate. | Non-finite values are rejected before formatting or sign-off, with a named diagnostic. | +| 55 | fixed | high | benchmark sign-off evidence | Baseline provenance and saturation | REFUSE | Missing benchmark identity or breakdown fields could pass publication gates as zero-valued evidence. | Sign-off requires complete provenance, code, manifest, GPU, and timing fields. | +| 56 | fixed | medium | planner/benchmark GPU-count discovery | Device count and topology | WARN/REFUSE | A failed GPU query became one device and produced a plausible single-GPU result. | Counts are marked fallback or refused when required; the report names the discovery failure. | +| 57 | fixed | medium | imported trace device-count handling | Multi-GPU coverage | WARN | A trace with missing device metadata could be summarized as a complete one-device capture. | Device-count discovery failures and selected-device scope are retained in importer diagnostics. | +| 58 | fixed | medium | resident-footprint provenance | Memory residency and headroom | FLAG | An inferred resident footprint was indistinguishable from a measured value. | Resident bytes carry provenance and the report distinguishes observed, inferred, and unavailable values. | +| 59 | fixed | medium | autoresearch GPU-count fallback | Search/optimization scope | WARN | Autoresearch used one GPU when discovery failed and could claim a full-device experiment. | The fallback count is warned and included in the run diagnostics. | +| 60 | fixed | high | direct sparse graph builder validation | Sparse shape, precision, and KV terms | REFUSE | Programmatic callers bypassing loop/attach could construct default-shaped or unpriced MoE graphs. | Direct builders validate structural fields and all priceable dtypes, including expert and KV widths. | +| 61 | fixed | high | injected tracer ingestion and Windows PID selection | Trace completeness and process identity | WARN/REFUSE | Malformed shard lines or ambiguous PID files could be dropped while the remaining trace looked complete. | Dropped records and selected PID scope are named in capture diagnostics; incomplete injected traces cannot claim complete evidence. | +| 62 | fixed | high | public runtime input validators | Workload timing and work counts | REFUSE | Direct public API callers could pass empty work or invalid timing and receive a throughput value. | Shared positive finite duration/work predicates gate all public runtime calculations. | +| 63 | fixed | medium | telemetry discovery and backend field probes | Utilization, power, clocks, ECC, and process state | FLAG/WARN | A missing backend field became `None`/empty and looked like a measured zero or no-throttle state. | Field-level diagnostics are retained, deduplicated, and consumed by collector/report paths. | +| 64 | fixed | high | attach window validation | Live trace interval | REFUSE | An invalid attach start/end window could produce a trace with no trustworthy temporal scope. | Attach refuses non-positive/non-finite windows and reports the named reason. | +| 65 | fixed | high | optimizer gate evidence | Optimization acceptance controls | REFUSE | Missing or non-finite gate evidence could be interpreted as a passing zero delta. | Gate controls require finite, complete evidence and refuse with a diagnostic. | +| 66 | fixed | medium | imported launch-shape metadata | Kernel launch dimensions | WARN | Missing launch dimensions were filled in without telling the importer, changing occupancy/shape interpretation. | Import diagnostics record launch-shape fallback and the customer report preserves the caveat. | +| 67 | fixed | high | dense graph parser shape/dtype fields | Dense FLOPs and bytes | REFUSE | A parser field omission could become a default shape or dtype in a production graph. | Production parser validates required shape/dtype fields and refuses incomplete graphs. | +| 68 | fixed | high | dense TP/precision/spec wiring | Sharding and weight precision | FLAG/REFUSE | Declared TP or precision could be dropped between config parsing and graph construction. | The production path carries sharding/precision into graph nodes and flags missing/unverified fields. | +| 69 | fixed | medium | live headroom CLI | GPU headroom decision | WARN/REFUSE | CLI invocation without live telemetry could still recommend an idle-GPU action. | The CLI refuses telemetry-less decisions and prints the diagnostic. | +| 70 | fixed | high | scheduler field probes | Scheduler queue/cache/load evidence | WARN | Version-drifted engine fields were silently omitted, making an empty sample look idle. | Each failed probe is named in `SchedulerSample.diagnostics` and reaches scheduler artifacts/report. | +| 71 | fixed | high | HFT/edge A/B controls | Intervention speedup and control validity | REFUSE | Invalid control values or missing A/B halves could produce a plausible keep decision. | Controls and both measured halves are validated before comparison; invalid runs are intervention failures. | +| 72 | fixed | high | benchmark timing partitions | Stall/phase shares | REFUSE | Overlapping or zero total phase timers were clamped into a clean partition. | Shared partition validation rejects contradictory/non-positive timing and names the components. | +| 73 | fixed | high | OpenFold A/B evidence | Protein inference speedup | REFUSE | Failed or incomplete OpenFold baseline/variant measurements could be reported as a speedup. | OpenFold requires positive finite paired measurements and refuses unsupported evidence. | +| 74 | fixed | high | utilization windows | GPU utilization and memory bandwidth | REFUSE | Backwards or overlapping utilization windows could be normalized into a valid-looking percentage. | Window ordering, duration, and overlap are validated before utilization is computed. | +| 75 | fixed | high | scheduler telemetry numeric values | Queue/cache/token counters | REFUSE/WARN | Negative, non-finite, or out-of-range scheduler values entered summaries as ordinary measurements. | Numeric probes reject invalid values or attach field diagnostics; summaries expose degraded coverage. | +| 76 | fixed | medium | CUDA build/version discovery | Build provenance | WARN | An unavailable CUDA version was rendered as a verified component version. | Build versions are labeled unverified and are visible in preflight/report output. | +| 77 | fixed | medium | roofline BF16 peak controls | Hardware peak denominator | FLAG | A declared bf16 peak could be silently replaced by the catalogue default. | The declared peak is honored and hardware fallback provenance remains separate from measured/declared values. | +| 78 | fixed | medium | host flamegraph capture | Profile artifact completeness | WARN | A failed py-spy capture produced a profile bundle with no visible host artifact. | The bundle lists missing flamegraph output and the profile report surfaces it. | +| 79 | fixed | medium | importer cleanup | Kept-trace/report completeness | WARN | Temporary-file cleanup failures were swallowed after a seemingly successful import. | Cleanup losses are retained in `ImportStats.warnings` and customer diagnostics. | +| 80 | fixed | medium | profile bundle artifact manifest | Profile evidence completeness | FLAG/WARN | Missing GPU CSV, host profile, or profiler output was not distinguishable from a complete bundle. | Every expected artifact and missing item is serialized and printed by the profile CLI. | +| 81 | fixed | high | live apply fail-open wiring | Mutation and rollback evidence | WARN/FLAG | Apply/rollback lifecycle failures could be swallowed while a mutation remained active. | Fail-open guard state, audit-sink failures, and lifecycle failures are retained and surfaced. | +| 82 | fixed | high | auto-revert evidence | Keep/rollback decision | REFUSE | Invalid verification evidence could trigger or suppress auto-revert as if it were measured. | Auto-revert requires complete finite evidence and refuses malformed verification. | +| 83 | fixed | medium | CUDA sibling parity | Launch vs attach CUDA verification | WARN | One lifecycle path reported an unverified CUDA build while its sibling treated the same field as verified. | Shared CUDA diagnostics are used by both paths. | +| 84 | fixed | high | GPU trace parser drops | Trace event population | WARN | Malformed GPU records were discarded without a count, biasing utilization and residual coverage. | Parser drop counts and reasons reach capture status and reports. | +| 85 | fixed | high | sparse KV sizing artifact wiring | KV bytes and memory floor | FLAG/REFUSE | Sparse KV sizing was computed but omitted from prediction artifacts, hiding a dominant memory term. | KV width/size and fallback provenance are serialized in graph and scheduler artifacts. | +| 86 | fixed | high | stream-concurrency evidence | Intervention causal evidence | REFUSE | Unsupported stream-concurrency telemetry became an A/B claim with a fabricated zero/one value. | Unsupported evidence returns measurement-only/intervention-failed status with a named reason. | +| 87 | fixed | medium | runtime schema fields | Report/artifact contract | REFUSE/fix | Producers emitted fields that consumers never honored, making a supposed diagnostic ineffective. | Unhonored fields were removed or wired to a real consumer; schema tests cover the contract. | +| 88 | fixed | high | interconnect/collective topology | Communication cost and sharding | REFUSE | Topology inferred from incomplete metadata could price collectives as if measured. | Inferred topology is refused for graph claims; explicit topology provenance is required. | +| 89 | fixed | critical | sparse dtype and default KV pricing | Dominant expert/KV byte terms | REFUSE/FLAG | Unknown expert/activation/KV dtypes silently became bf16/2-byte defaults in direct and live paths. | Final resolved specs use shared dtype priceability; graph nodes retain byte fallback flags and boundaries refuse unpriceable predictions. | +| 90 | fixed | high | `gitm/runtime_driver.py` work-unit extraction | Events/frames throughput numerator | REFUSE | A runner summary omitted the named work counter and `.get(..., 0)` produced a zero/invalid throughput path; the no-kernel report also formatted `None` as a number. | `_work_units` requires a finite positive counter and returns exit 3 with a named failure; no-kernel formatting is explicitly unavailable. | +| 91 | fixed | high | `gitm/importers/nsys.py` enum mapping | Memory endpoints, copy kind, and sync type | WARN | Missing/unknown CUPTI enums were assumed to be device/stream defaults without reaching `ImportStats`, corrupting overlap/topology interpretation. | Non-strict imports retain deduplicated named diagnostics; strict imports still refuse unknown enums. | +| 92 | fixed | high | `scripts/compare_results.py` identity comparison | Reproducibility verification | REFUSE | Two incomplete reports compared equal because `.get` defaults made absent schema/package identity look identical. | Required schema, identity, and exact package versions now mismatch when missing/unavailable. | +| 93 | fixed | medium | `gitm/tracer/_cupti_decode.py` decoder defaults | Kernel launch shape/name and CUPTI enum meaning | WARN | ABI-drifted or malformed records silently became 1x1x1, anonymous, device-copy, or device-sync events. | Decoder warnings name each fallback while preserving safe parsing; tests cover missing shape/name and unknown enums. | + +Status values: `open`, `fixed`, `deferred (reason)`, or `won't fix (reason)`. + +## Seed-finding verification + +| Seed | Verification | Status | +|---:|---|---| +| 1 | Unknown weight dtype falls back to bf16 without a bytes-side flag. | Fixed with per-node/graph flags, shared final dtype refusal, and boundary consumers. | +| 2 | Loop MoE dispatch lacks the attach path's dtype priceability gate. | Fixed with shared predicates and typed sparse dispatcher/refusal. | +| 3 | Missing `expert_dtype` inherits `weight_dtype`, potentially mispricing the dominant expert term. | Fixed by explicit inheritance warnings; official V4 configs were confirmed to declare the field. | +| 4 | Residual percentage clamps at ±100% and loses raw error magnitude. | Fixed with raw + capped-display contract, saturation note, scope, and non-finite refusal. | +| 5 | Residual classification drops unmatched kernels without loop-path coverage. | Fixed with count/time coverage and JSON/summary/report warnings. | +| 6 | Unknown SKU becomes A100 while recording A100 as if observed. | Fixed with hardware fallback provenance, loop refusal, and attach warning/separate fields. | +| 7 | Broad model-config parse rescue becomes an unmarked default model. | Fixed in production dispatch with typed named refusal; no default graph. | +| 8 | `has_unpriced_collectives` scans all nodes despite its narrow name. | Fixed by splitting general and collective-specific predicates. | + +## Sweep coverage + +| Area | Phase 1 fallback sweep | Phase 2 wiring sweep | Notes | +|---|---|---|---| +| top-level runtime / API / CLI / workloads | Swept — fixed/clean | Swept — traced | Positive timing/work gates, degraded exit status, and workload provenance are covered by ranks 22, 24, 27, 41, 42, 62, 90. | +| agents | Swept — fixed/clean | Swept — traced | Search-domain and GPU-count fallbacks are explicit warnings (ranks 26, 59). | +| bench | Swept — fixed/clean | Swept — traced | Saturation, provenance, timing partitions, and profile artifact completeness are covered by ranks 11, 25, 50, 55, 72, 78, 80. | +| benchmarks | Swept — fixed/clean | Swept — traced | HFT, edge, OpenFold, KITTI, and shared baseline/A-B paths are covered by ranks 18, 24, 27, 42, 51, 53, 71, 73. | +| deploy | Swept — fixed/clean | Swept — traced | Unsupported live attach and attach-window gates are covered by ranks 21 and 64. | +| importers | Swept — fixed/clean | Swept — traced | Rollups, artifact stems, cleanup, launch metadata, parser drops, and NSYS enum diagnostics are covered by ranks 31, 32, 57, 66, 79, 84, 91. | +| kernels | Swept — fixed/clean | Swept — traced | Candidate-library coverage and kernel metadata/decoder fallbacks are covered by ranks 16, 29, 61, 84, 93. | +| optimizer | Swept — fixed/clean | Swept — traced | Attribution, headroom, measurement, gate, apply, and rollback evidence are covered by ranks 15, 19, 20, 38, 50, 51, 65, 81, 82. | +| planner | Swept — fixed/clean | Swept — traced | Hardware, dense/sparse pricing, topology, KV, and direct-builder paths are covered by ranks 3, 4, 7, 9, 13, 39, 43, 49, 60, 67, 68, 77, 85, 88, 89. | +| routing | Swept — fixed/clean | Swept — traced | Bounded routing inputs and unknown tiers refuse (rank 30). | +| safety | Swept — fixed/clean | Swept — traced | Fail-open audit/signal and rollback visibility are covered by ranks 28, 38, 81. | +| scheduler | Swept — fixed/clean | Swept — traced | Dispatch, prediction refusal, residual coverage, scheduler probes, topology, artifacts, and all intervention siblings are covered by ranks 1, 2, 5, 6, 10, 18, 34, 41, 45, 70, 71, 85, 86, 87, 89. | +| serve | Swept — fixed/clean | Swept — traced | Launch/attach gates, token provenance, metrics, model discovery, CUDA parity, and graph artifacts are covered by ranks 12, 21, 33, 35, 40, 46, 47, 52, 53, 83, 89. | +| telemetry | Swept — fixed/clean | Swept — traced | Backend fields, collector/sink failures, sampler probes, and runtime consumers are covered by ranks 17, 35, 40, 63, 70, 75. | +| tracer | Swept — fixed/clean | Swept — traced | Capture, injected traces, vLLM summaries, CUPTI decoding, and drop coverage are covered by ranks 12, 14, 23, 33, 34, 61, 84, 93. | +| scripts | Swept — fixed/clean | Swept — traced | Demo, headroom, report, comparison, and profile outputs are covered by ranks 44, 54, 69, 78, 80, 92. | + +## Diagnostic-consumer trace + +Every boolean flag, warning list, `estimated`, provenance field, and `has_*` or +`*_fallback` property discovered during the sweep is listed here and traced to a +human- or gate-visible consumer. + +| Producer | Diagnostic | Downstream consumer | User/gate boundary | Status | +|---|---|---|---|---| +| `RooflinePrediction` / `Graph` | peak, bytes, hardware fallback; estimated; per-dimension unpriced nodes | loop and attach serializers/diagnostics | JSON + Markdown/CLI | Traced/fixed | +| `RooflinePrediction` / `Graph` | `bytes_are_fallback`, `has_fallback_bytes`, `has_fallback_peaks`, `has_unpriced_nodes` | predicted graph payload, attach payload, scheduler summary | Prediction gate + JSON + Markdown/CLI warning | Traced/fixed | +| `HardwareSpec` / graph context | observed vs pricing SKU and hardware provenance | planner graph, loop refusal, attach warning | Prediction gate + artifact/report | Traced/fixed | +| `Residuals` / `Claim` | raw residual, display cap, saturation, scope | loop residual JSON, scheduler claims, report template | JSON + Markdown diagnostic | Traced/fixed | +| `Residuals` | total/classified/matched launch and kernel-time coverage | residual JSON, run summary, report diagnostics | WARN in report; no clean claim on incomplete coverage | Traced/fixed | +| `ImportStats` / importer rollup | warnings, drops, caveats, SKU/time provenance | analyze summary + customer report | JSON + Markdown | Traced/fixed | +| `ImportStats` / NSYS enum mapper | missing/unknown memory, copy, and sync enums | `stats.warnings`, analyze report | Import WARN and report caveat | Traced/fixed | +| `CaptureResult` / kernel taxonomy | warnings, dropped records, capture status | serve artifacts + CLI | JSON + CLI exit | Traced/fixed | +| CUPTI decoder | missing shape/name and unknown enum warnings | tracer caller/tested capture path | Runtime warning + trace diagnostics | Traced/fixed | +| `ServingSummary` | TTFT/TPOT sample counts and token-provenance warnings | serve/loop artifacts | JSON + CLI/Markdown | Traced/fixed | +| Serving metrics sampler | invalid windows, scrape failures, unavailable fields | `metrics_before/after`, samples, report | WARN/REFUSE at serving boundary | Traced/fixed | +| `Collector` / `GpuHeadroom` | component failures and missing metric-family diagnostics | runtime driver and benchmark artifacts | warning + JSON/Markdown/stdout | Traced/fixed | +| Scheduler sampler | field-probe failures and degraded reads | `scheduler_stats.json`, summary, report | WARN and degraded summary | Traced/fixed | +| Fail-open guard | revert failures, audit-sink failures, signal-handler failures | `failures` attribute, audit log, report diagnostics | Programmatic state + audit artifact | Traced/fixed | +| Benchmark/profile bundle | missing artifacts, command failures, inferred provenance | profile manifest and bench report | Nonzero CLI / WARN for optional artifact | Traced/fixed | +| Runtime driver | no-data, synchronization, telemetry, and work-unit diagnostics | stdout, measure JSON, Markdown report | Exit 3 for degraded/no-data | Traced/fixed | +| Verification/comparison | missing identity/schema/package fields, dirty tree | comparator mismatch output and exit | REFUSE reproducibility claim | Traced/fixed | + +## Sibling-path validation matrix + +| Capability | Path A | Path B | Guard parity | Status | +|---|---|---|---|---| +| MoE config pricing | attach resolves raw config then validates final dtypes | scheduler recognizes partial sparse configs, validates, and dispatches sparse graph | Shared predicates; path-specific input adapters | Fixed | +| execution lifecycle | launch preflight, live engine, trace capture | attach preflight, target discovery, live metrics | Shared CUDA/trace/timing/metric gates; attach-specific unsupported PID path is explicit | Fixed | +| model family | dense parser and graph builder | MoE sparse builder and expert/KV sizing | Shared dtype/shape priceability; family-specific fields are refused or warned | Fixed | +| serving lifecycle | vLLM workload launch | attach to existing vLLM server | Shared model/config, token provenance, metrics, and artifact writers | Fixed | +| workload timing | HFT event loop | edge/KITTI frame loop | Same positive work/duration predicates and degraded exit convention | Fixed | +| intervention A/B | generic optimizer loop | HFT, edge, OpenFold, and stream-concurrency siblings | Paired finite throughput/timing and trace-evidence gates in every branch | Fixed | +| telemetry | NVML collector/backend | scheduler sampler and Prometheus serving metrics | Field-level diagnostics and invalid-window handling preserved in each consumer | Fixed | +| tracing | in-process capture | injected/CUPTI shard capture | Positive-duration/no-data gate plus malformed-drop diagnostics | Fixed | +| importers | NSYS SQLite importer | torch Chrome trace importer | Normalization, invalid-event drops, warnings, and kept-artifact reporting | Fixed | +| graph hardware | catalogue/declared peak | live detected/observed hardware | Pricing fallback is separate from observed identity; unknown topology refuses | Fixed | +| report verification | report producer | `compare_results.py` consumer | Required schema/identity/package fields are enforced at comparison boundary | Fixed | + +## Artifact-consumer trace + +| Artifact writer | Artifact | Production reader / boundary | Status | +|---|---|---|---| +| `gitm/scheduler/loop.py` | `predicted_graph.json` | scheduler summary/report, CLI prediction diagnostics, replay/tests | Traced/fixed | +| `gitm/scheduler/loop.py` | `prediction_refusal.json` | measurement-only scheduler result, report, CLI degraded exit | Traced/fixed | +| `gitm/scheduler/loop.py` | `scheduler_stats.json` | run summary/report and scheduler diagnostics | Traced/fixed | +| `gitm/scheduler/loop.py` | `qualification.json` | qualification/report path and run summary | Traced/fixed | +| `gitm/scheduler/loop.py` | `residuals.json` | report template, summary diagnostics, residual consumers | Traced/fixed | +| `gitm/scheduler/loop.py` | `deviations.json`, `deviation_trace.jsonl` | deviation report/measurement and optimizer attribution | Traced/fixed | +| `gitm/scheduler/loop.py` | `ranked_candidates.json` | intervention report and apply/verification paths | Traced/fixed | +| `gitm/scheduler/loop.py` | `verification.json` | optimizer gate and auto-revert evidence | Traced/fixed | +| `gitm/scheduler/loop.py` | `measurement.json` | measurement-only report and caller summary | Traced/fixed | +| `gitm/scheduler/loop.py` | `apply_result.json`, `audit.jsonl` | apply result/report and safety audit reader | Traced/fixed | +| `gitm/serve/attach.py` / `gitm/serve/vllm.py` | `predicted_moe_graph.json` | serving report, attach CLI, replay/analyze artifact directory | Traced/fixed | +| `gitm/serve/artifacts.py` | `preflight.json`, `kernel_breakdown.json`, `serving_summary.json`, `run_manifest.json` | serving report, artifact manifest, and downstream replay/analyze tools | Traced/fixed | +| serving metrics sampler | `metrics_before.txt`, `metrics_after.txt`, `metrics_samples.jsonl` | serving comparison/report and metric diagnostics | Traced/fixed | +| `gitm/runtime_driver.py` | trace JSONL, telemetry JSONL, `*_measure.json`, `*_report.md` | runtime CLI output, report readers, and verification scripts | Traced/fixed | +| `gitm/importers/analyze.py` | kept traces, summary JSON, customer Markdown | CLI return/report consumer and customer artifact | Traced/fixed | +| `gitm/bench/profile.py` / `gitm/bench/cli.py` | profile bundle, GPU CSV, host SAR/flamegraph, manifest | bench results/report and profile completeness gate | Traced/fixed | +| benchmark baseline runners | baseline JSON and spread report | benchmark comparison/sign-off and report | Traced/fixed | +| `gitm/optimizer/verification_export.py` | verification JSON | optimizer gate, comparator, and report | Traced/fixed | +| `gitm/safety/audit.py` | `report.md`, `audit.jsonl` | safety/apply report and audit reader | Traced/fixed | +| telemetry sinks | JSONL/OTLP sink records | configured external collector/sink boundary; no local reader is claimed | Traced — external boundary explicit | + +## Completeness pass + +No outstanding items. + +## Verification and review readiness + +- Targeted tests for touched runtime/importer/CUPTI/comparison paths: **73 passed**; + targeted planner/routing tests after the final alias fix: **75 passed**. +- Full repository run after synchronizing with `origin/main`: **1100 passed, 1 skipped**, with only named runtime warnings + for optional GPU/vLLM/CUDA telemetry and intentional fallback demonstrations. + Importer fixtures were restored after the run; `git status --short` is clean. +- Ruff passed on all touched modules and tests; `git diff --check` is clean. +- The audit branch is `audit/graceful-fallbacks`; it merges the current + `origin/main` at `f63879b` and contains the audit fixes plus the upstream MoE + architecture updates. The clean expert-signal branch is `feat/expert-signal-eplb` from + `main` (`ca289d6`), containing the signal module and its tests; caller wiring is + intentionally the next feature step, not an audit change. The pre-existing + wiring snapshot remains preserved on `feat/expert-signal-eplb-stacked` (`452b2c8`) + for the maintainer to port or review. +- Review recommendation: one audit PR is reviewable by theme because each commit + is a bounded REFUSE/FLAG/WARN fix and this ledger is the summary. Keep the + expert-signal work as a separate follow-up PR; do not open multiple audit PRs + without maintainer selection. diff --git a/benchmarks/biotech/harness.py b/benchmarks/biotech/harness.py index 639b094..77f8a8b 100644 --- a/benchmarks/biotech/harness.py +++ b/benchmarks/biotech/harness.py @@ -33,6 +33,7 @@ from typing import Protocol from benchmarks.biotech.fetch import FastaRecord, read_fasta +from gitm._timing import require_positive_duration, require_timing_partition OPENFOLD_COMMIT = "v1.0.1" # pinned; weight hashes pinned in datasets.md MODEL_NAME = "model_1" # single-model AF2 monomer; weights = params_model_1.npz @@ -193,12 +194,16 @@ def _build_stall_phase(timings: list[dict], wall_clock_s: float) -> dict: t_feat = sum(t["_t_featurize_s"] for t in timings) t_inf = sum(t["_t_inference_s"] for t in timings) t_post = sum(t["_t_post_s"] for t in timings) - total = max(sum(t["_t_total_s"] for t in timings), 1e-9) - - data_stall = min(1.0, t_feat / total) - gpu_active = min(1.0, t_inf / total) - sync = min(1.0, t_post / total) - cpu = max(0.0, 1.0 - data_stall - gpu_active - sync) + total = sum(t["_t_total_s"] for t in timings) + split = require_timing_partition( + total, + {"data_stall": t_feat, "gpu_active": t_inf, "sync": t_post}, + context="biotech stall breakdown", + ) + data_stall = split["data_stall"] + gpu_active = split["gpu_active"] + sync = split["sync"] + cpu = split["unattributed"] return { "phase": "all", @@ -246,7 +251,9 @@ def run( plddts.append(float(result["plddt"])) if "_t_total_s" in result: timings.append(result) - elapsed = max(time.perf_counter() - t0, 1e-9) + elapsed = require_positive_duration( + time.perf_counter() - t0, context="biotech harness" + ) structures_per_hour = len(proteins) / elapsed * 3600.0 payload: dict = { diff --git a/benchmarks/biotech/optimize.py b/benchmarks/biotech/optimize.py index dfdf0d3..65bc925 100644 --- a/benchmarks/biotech/optimize.py +++ b/benchmarks/biotech/optimize.py @@ -25,6 +25,7 @@ import argparse import json +import math import os import statistics import time @@ -33,6 +34,7 @@ from benchmarks.biotech.fetch import read_fasta from benchmarks.biotech.harness import _msa_path, load_openfold_runner +from gitm._timing import require_positive_duration, require_positive_work from gitm.kernels.spec import Applicability, InterventionSpec, SafetyGate @@ -57,11 +59,17 @@ def _fold_all(runner, proteins) -> tuple[float, float | None]: for r, msa in proteins: out = runner.predict(r, msa) if "plddt" in out: - plddts.append(float(out["plddt"])) + plddt = float(out["plddt"]) + if not math.isfinite(plddt): + raise RuntimeError(f"OpenFold quality evidence is non-finite: {plddt!r}") + plddts.append(plddt) if torch.cuda.is_available(): torch.cuda.synchronize() - elapsed = max(time.perf_counter() - t0, 1e-9) - sph = len(proteins) / elapsed * 3600.0 + elapsed = require_positive_duration( + time.perf_counter() - t0, context="OpenFold A/B" + ) + count = require_positive_work(len(proteins), context="OpenFold A/B") + sph = count / elapsed * 3600.0 return sph, (statistics.median(plddts) if plddts else None) @@ -91,6 +99,15 @@ def verdict(self) -> str: def optimize_af2(stage: Path, seed: int, *, n_proteins: int, max_len: int, warmup: int, plddt_tol: float) -> AF2ABResult: """Run the fp32-vs-bf16 A/B and return a gated verdict.""" + for name, value, minimum in ( + ("n_proteins", n_proteins, 1), + ("max_len", max_len, 1), + ("warmup", warmup, 0), + ): + if isinstance(value, bool) or not isinstance(value, int) or value < minimum: + raise ValueError(f"{name} must be an integer >= {minimum}, got {value!r}") + if not math.isfinite(plddt_tol) or plddt_tol < 0.0: + raise ValueError(f"plddt_tol must be finite and non-negative, got {plddt_tol!r}") import torch proteins = _select(stage, max_len=max_len, n=n_proteins) diff --git a/benchmarks/biotech/sanity.py b/benchmarks/biotech/sanity.py index 0d241d2..f686d19 100644 --- a/benchmarks/biotech/sanity.py +++ b/benchmarks/biotech/sanity.py @@ -27,6 +27,7 @@ load_openfold_runner, select_proteins, ) +from gitm._timing import require_positive_duration def _fmt(result: dict) -> str: @@ -85,7 +86,13 @@ def main(argv: list[str] | None = None) -> int: warm = runner.predict(target, msa) print(f" WARM {_fmt(warm)}") - speedup = cold["_t_total_s"] / max(warm["_t_total_s"], 1e-9) + cold_s = require_positive_duration( + float(cold["_t_total_s"]), context="OpenFold cold sanity pass" + ) + warm_s = require_positive_duration( + float(warm["_t_total_s"]), context="OpenFold warm sanity pass" + ) + speedup = cold_s / warm_s print(f"cold/warm total speedup: {speedup:.2f}x (kernel cache + allocator warm-up)") if abs(cold["plddt"] - warm["plddt"]) > 1.0: print( diff --git a/benchmarks/edge/harness.py b/benchmarks/edge/harness.py index 47b6fd6..2424264 100644 --- a/benchmarks/edge/harness.py +++ b/benchmarks/edge/harness.py @@ -33,6 +33,8 @@ from pathlib import Path from typing import Any, Protocol +from gitm._timing import require_positive_duration, require_timing_partition + OPENPCDET_COMMIT = "v0.6.0" # pinned; config hash pinned in datasets.md _OPENPCDET_DEFAULT_CFG = ( @@ -231,7 +233,9 @@ def run(stage: Path, *, warm: int, runner: Runner) -> dict: maps.append(float(result["map"])) if "_t_total_s" in result: timings.append(result) - elapsed = max(time.perf_counter() - t0, 1e-9) + elapsed = require_positive_duration( + time.perf_counter() - t0, context="edge harness" + ) if n == 0: raise RuntimeError(f"no frames in {manifest}") @@ -259,12 +263,20 @@ def _build_stall_phase(timings: list[dict], wall_clock_s: float) -> dict: t_pre = sum(t["_t_preprocess_s"] for t in timings) t_inf = sum(t["_t_inference_s"] for t in timings) t_post = sum(t["_t_postprocess_s"] for t in timings) - total = max(sum(t["_t_total_s"] for t in timings), 1e-9) - - data_stall = min(1.0, (t_load + t_pre) / total) - gpu_active = min(1.0, t_inf / total) - sync = min(1.0, t_post / total) - cpu = max(0.0, 1.0 - data_stall - gpu_active - sync) + total = sum(t["_t_total_s"] for t in timings) + split = require_timing_partition( + total, + { + "data_stall": t_load + t_pre, + "gpu_active": t_inf, + "sync": t_post, + }, + context="edge stall breakdown", + ) + data_stall = split["data_stall"] + gpu_active = split["gpu_active"] + sync = split["sync"] + cpu = split["unattributed"] return { "phase": "all", diff --git a/gitm/_timing.py b/gitm/_timing.py new file mode 100644 index 0000000..cdae713 --- /dev/null +++ b/gitm/_timing.py @@ -0,0 +1,52 @@ +"""Shared timing trust predicates.""" + +from __future__ import annotations + +import math + + +def require_positive_duration(duration_s: float, *, context: str) -> float: + """Return a usable duration or refuse to fabricate a throughput denominator.""" + if not math.isfinite(duration_s) or duration_s <= 0.0: + raise RuntimeError( + f"{context} timing unavailable: expected a finite positive duration, " + f"got {duration_s!r}" + ) + return duration_s + + +def require_positive_work(value: int | float, *, context: str) -> int | float: + """Refuse throughput or speedup claims over an empty work unit.""" + if not math.isfinite(float(value)) or value <= 0: + raise RuntimeError(f"{context} work coverage unavailable: expected > 0, got {value!r}") + return value + + +def require_timing_partition( + total_s: float, components_s: dict[str, float], *, context: str +) -> dict[str, float]: + """Return component fractions plus ``unattributed`` or refuse overlap. + + Benchmark stall breakdowns are sign-off evidence. Repairing a zero total or + independently clamping overlapping phase timers would turn broken evidence + into a plausible partition, so validate the complete partition in one place. + """ + total = require_positive_duration(total_s, context=context) + invalid = { + name: value + for name, value in components_s.items() + if not math.isfinite(value) or value < 0.0 + } + if invalid: + raise RuntimeError(f"{context} timing unavailable: invalid components {invalid}") + assigned = sum(components_s.values()) + tolerance = max(1e-12, total * 1e-9) + if assigned > total + tolerance: + detail = ", ".join(f"{name}={value:.6g}s" for name, value in components_s.items()) + raise RuntimeError( + f"{context} timing attribution overlaps: {detail}, total={total:.6g}s; " + "refusing to clamp" + ) + fractions = {name: value / total for name, value in components_s.items()} + fractions["unattributed"] = max(0.0, total - assigned) / total + return fractions diff --git a/gitm/agents/autoresearch.py b/gitm/agents/autoresearch.py index 8ba7d3c..45a7583 100644 --- a/gitm/agents/autoresearch.py +++ b/gitm/agents/autoresearch.py @@ -38,6 +38,7 @@ from __future__ import annotations import random +import warnings from collections.abc import Callable from dataclasses import dataclass, field from typing import TYPE_CHECKING, Protocol @@ -63,6 +64,7 @@ "IDLE_STALL", "MEMORY_BOUND", "COMPUTE_BOUND", + "UNCLASSIFIED", "classify_bottleneck", "ResidualTarget", "largest_residual", @@ -97,7 +99,8 @@ IDLE_STALL = "idle_stall" MEMORY_BOUND = "memory_bound" COMPUTE_BOUND = "compute_bound" -BOTTLENECK_CLASSES = (IDLE_STALL, MEMORY_BOUND, COMPUTE_BOUND) +UNCLASSIFIED = "unclassified" +BOTTLENECK_CLASSES = (IDLE_STALL, MEMORY_BOUND, COMPUTE_BOUND, UNCLASSIFIED) #: Serialized-concurrency fraction above this ⇒ kernels ran back-to-back on one #: stream instead of overlapping: scheduling gaps / launch-bound idle time. @@ -125,18 +128,18 @@ def _roofline_memory_fraction(residuals: Residuals | None) -> float | None: def classify_bottleneck(trace: Trace, residuals: Residuals | None = None) -> str: - """Map a captured trace to one of ``idle_stall`` / ``memory_bound`` / ``compute_bound``. + """Classify a measured trace, or return ``unclassified`` without valid kernels. Two signals, scored against a threshold each; the stronger wins, neither - crossing defaults to compute bound: serialized-concurrency fraction (poor + crossing selects compute bound: serialized-concurrency fraction (poor kernel overlap ⇒ idle/scheduling gaps), and memory pressure (memcpy share of GPU-op time, widened by the roofline-predicted memory-bound fraction of matched kernel time when ``residuals`` is passed). Without ``residuals`` this is the memcpy-only heuristic. """ - kernels = trace.kernels() + kernels = [k for k in trace.kernels() if k.end_ns > k.start_ns] if not kernels: - return COMPUTE_BOUND + return UNCLASSIFIED memcpys = [e for e in trace.events if e.kind == "memcpy"] sc = _serialized_fraction(kernels) @@ -145,11 +148,15 @@ def classify_bottleneck(trace: Trace, residuals: Residuals | None = None) -> str gpu_op_ns = kernel_ns + memcpy_ns memcpy_frac = memcpy_ns / gpu_op_ns if gpu_op_ns else 0.0 - sc_score = sc / _SC_THRESHOLD + sc_score = sc / _SC_THRESHOLD if sc is not None else None mem_score = memcpy_frac / _MEMCPY_THRESHOLD roofline_frac = _roofline_memory_fraction(residuals) if roofline_frac is not None: mem_score = max(mem_score, roofline_frac / _MEMCPY_THRESHOLD) + if mem_score >= 1.0 and (sc_score is None or mem_score > sc_score): + return MEMORY_BOUND + if sc_score is None: + return UNCLASSIFIED if max(sc_score, mem_score) < 1.0: return COMPUTE_BOUND return IDLE_STALL if sc_score >= mem_score else MEMORY_BOUND # ties favor idle_stall @@ -213,6 +220,7 @@ def _op_present(trace: Trace, op: str) -> bool: ("compilation_config", 3, "raise torch.compile to level 3 for kernel fusion + piecewise CUDA graphs"), ], + "unclassified": [], } @@ -393,6 +401,7 @@ class Knob: "idle_stall": ("prefill", "partial", "schedul", "chunk", "overlap"), "memory_bound": ("cache", "swap", "offload", "block", "gpu_memory", "kv", "preempt", "cpu"), "compute_bound": ("compil", "cudagraph", "cuda_graph", "graph", "quant", "fus", "eager"), + "unclassified": (), } @@ -553,8 +562,23 @@ def _visible_gpu_count() -> int: try: import torch - return torch.cuda.device_count() or 1 - except Exception: + count = int(torch.cuda.device_count()) + if count > 0: + return count + warnings.warn( + f"GPU-count detection reported {count}; autoresearch is using 1 and " + "may omit multi-GPU candidates", + RuntimeWarning, + stacklevel=2, + ) + return 1 + except Exception as exc: + warnings.warn( + "GPU-count detection failed; autoresearch is using 1 and may omit " + f"multi-GPU candidates ({type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) return 1 @@ -585,7 +609,13 @@ def _argparse_domains(engine_args_cls: object) -> dict[str, _ArgDomain]: import argparse parser = engine_args_cls.add_cli_args(argparse.ArgumentParser()) # type: ignore[attr-defined] - except Exception: + except Exception as exc: + warnings.warn( + "vLLM CLI-domain introspection unavailable; candidate grids will use " + f"coarser dataclass annotations ({type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) return {} out: dict[str, _ArgDomain] = {} for action in getattr(parser, "_actions", []): @@ -634,6 +664,8 @@ def _knobs_from_engine_args( """ import dataclasses + if gpu_count is not None and gpu_count <= 0: + raise ValueError(f"gpu_count must be positive when supplied, got {gpu_count}") gpus = _visible_gpu_count() if gpu_count is None else gpu_count domains = _argparse_domains(engine_args_cls) knobs: list[Knob] = [] @@ -668,11 +700,32 @@ def _engine_arg_knobs(*, gpu_count: int | None = None) -> list[Knob]: """ try: from vllm import EngineArgs # type: ignore - except Exception: + except Exception as exc: + warnings.warn( + "vLLM EngineArgs unavailable; autoresearch is using the frozen fallback " + f"knob catalog ({type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) return list(_FALLBACK_KNOBS) try: - return _knobs_from_engine_args(EngineArgs, gpu_count=gpu_count) or list(_FALLBACK_KNOBS) - except Exception: + knobs = _knobs_from_engine_args(EngineArgs, gpu_count=gpu_count) + if knobs: + return knobs + warnings.warn( + "vLLM EngineArgs introspection produced no searchable knobs; autoresearch " + "is using the frozen fallback knob catalog", + RuntimeWarning, + stacklevel=2, + ) + return list(_FALLBACK_KNOBS) + except Exception as exc: + warnings.warn( + "vLLM EngineArgs introspection failed; autoresearch is using the frozen " + f"fallback knob catalog ({type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) return list(_FALLBACK_KNOBS) diff --git a/gitm/agents/policy.py b/gitm/agents/policy.py index 1d796c4..c9eff88 100644 --- a/gitm/agents/policy.py +++ b/gitm/agents/policy.py @@ -34,6 +34,8 @@ def select_interventions( *, ctx: GateContext | None = None, ) -> list[RankedCandidate]: + if isinstance(top_n, bool) or not isinstance(top_n, int) or top_n <= 0: + raise ValueError(f"top_n must be a positive integer, got {top_n!r}") candidates: list[RankedCandidate] = [] for spec in library: diff --git a/gitm/bench/baseline.py b/gitm/bench/baseline.py index 2a16f2d..00d427f 100644 --- a/gitm/bench/baseline.py +++ b/gitm/bench/baseline.py @@ -47,7 +47,7 @@ class BaselineSummary: mean: float stddev: float spread: float # (max - min) / mean - gpu_active_overall: float # worst (max) across runs + gpu_active_overall: float | None # worst (max) across runs; None = unmeasured recorded: float # the number we publish = mean gates: list[GateResult] = field(default_factory=list) seeds: list[int] = field(default_factory=list) @@ -96,7 +96,9 @@ def aggregate(runs: list[BaselineRun], config: BenchConfig) -> BaselineSummary: mean = statistics.fmean(values) stddev = statistics.pstdev(values) if len(values) > 1 else 0.0 spread = (max(values) - min(values)) / mean if mean else float("inf") - gpu_overall = max(r.gpu_active_overall() for r in runs) + gpu_values = [r.gpu_active_overall() for r in runs] + gpu_coverage_ok = all(value is not None for value in gpu_values) + gpu_overall = max(value for value in gpu_values if value is not None) if gpu_coverage_ok else None gates: list[GateResult] = [] @@ -119,13 +121,43 @@ def aggregate(runs: list[BaselineRun], config: BenchConfig) -> BaselineSummary: ) # Gate 2: saturation / swap rule. - sat_ok = gpu_overall < config.gpu_active_ceiling + sat_ok = gpu_overall is not None and gpu_overall < config.gpu_active_ceiling gates.append( GateResult( "saturation", sat_ok, - f"GPU active {gpu_overall:.1%} vs ceiling {config.gpu_active_ceiling:.0%}" - + ("" if sat_ok else " — trips swap rule, shard same day"), + ( + f"GPU active {gpu_overall:.1%} vs ceiling {config.gpu_active_ceiling:.0%}" + + ("" if sat_ok else " — trips swap rule, shard same day") + if gpu_overall is not None + else "GPU-active coverage unavailable: at least one run has no positive-wall-time " + "stall breakdown; saturation sign-off refused" + ), + ) + ) + + provenance_issues: list[str] = [] + for run in runs: + if not run.git_sha or run.git_sha == "unknown": + provenance_issues.append(f"seed {run.seed}: git SHA unavailable") + if not run.manifest_sha256: + provenance_issues.append(f"seed {run.seed}: dataset manifest digest unavailable") + if not run.gpu_name or run.gpu_name.lower() in {"cpu", "gpu-unknown"}: + provenance_issues.append(f"seed {run.seed}: GPU identity unavailable ({run.gpu_name!r})") + if run.device_count < 1: + provenance_issues.append(f"seed {run.seed}: no GPU devices reported") + if run.started_at_ns <= 0: + provenance_issues.append(f"seed {run.seed}: start timestamp unavailable") + if run.ended_at_ns <= run.started_at_ns: + provenance_issues.append(f"seed {run.seed}: end timestamp does not follow start") + provenance_issues.extend(f"seed {run.seed}: {note}" for note in run.provenance_warnings) + gates.append( + GateResult( + "provenance", + not provenance_issues, + "all runs pin code, dataset, and GPU identity" + if not provenance_issues + else "; ".join(provenance_issues), ) ) diff --git a/gitm/bench/cli.py b/gitm/bench/cli.py index dd567ef..4cec822 100644 --- a/gitm/bench/cli.py +++ b/gitm/bench/cli.py @@ -145,6 +145,12 @@ def _cmd_edge_manifest(args) -> int: n_nusc = sum(1 for r in rows if r.source == "nuscenes") n_kitti = sum(1 for r in rows if r.source == "kitti") print(f"wrote {args.out}: {len(rows)} keyframes ({n_nusc} nuScenes, {n_kitti} KITTI)") + if not rows: + print( + "ERROR: no dataset keyframes were discovered; empty manifest is not usable", + file=sys.stderr, + ) + return 1 return 0 @@ -177,7 +183,10 @@ def _cmd_profile(args) -> int: "out_dir": str(bundle.out_dir), "complete": bundle.complete, "missing_tools": bundle.missing, + "gpu_report": str(bundle.gpu_report) if bundle.gpu_report else None, "gpu_csv": str(bundle.gpu_csv) if bundle.gpu_csv else None, + "host_pyspy": str(bundle.host_pyspy) if bundle.host_pyspy else None, + "host_sar": str(bundle.host_sar) if bundle.host_sar else None, }, indent=2, )) @@ -187,7 +196,7 @@ def _cmd_profile(args) -> int: f"detected: {ProfilerTools.detect()}", file=sys.stderr, ) - return 0 + return 0 if bundle.complete else 1 def _cmd_baseline(args) -> int: diff --git a/gitm/bench/profile.py b/gitm/bench/profile.py index 4b6b293..53d675c 100644 --- a/gitm/bench/profile.py +++ b/gitm/bench/profile.py @@ -140,37 +140,78 @@ def run_profile( completes so a partial profile is better than none. """ out_dir = Path(out_dir) + if not isinstance(host_capture_s, int) or isinstance(host_capture_s, bool) or host_capture_s <= 0: + raise ValueError("host_capture_s must be a positive integer") out_dir.mkdir(parents=True, exist_ok=True) tools = tools or ProfilerTools.detect() argv, bundle = wrap_command(config, command, out_dir, tools=tools) - # Host-side samplers run alongside; py-spy attaches to the launched tree. - host_procs: list[subprocess.Popen] = [] + # Launch first so py-spy can attach to this process and its descendants. For + # a vendor-wrapped command the root is nsys/rocprof; --subprocesses follows + # the actual workload rather than pretending a detected tool captured it. + workload = subprocess.Popen(argv) + host_procs: list[tuple[str, subprocess.Popen]] = [] + sar_output = None if tools.sar: bundle.host_sar = out_dir / "host_sar.log" - host_procs.append( + sar_output = open(bundle.host_sar, "w") # noqa: SIM115 - owned until sampler exits + host_procs.append(( + "sar", subprocess.Popen( [tools.sar, "-u", "1", str(host_capture_s)], - stdout=open(bundle.host_sar, "w"), + stdout=sar_output, stderr=subprocess.DEVNULL, - ) - ) + ), + )) else: bundle.missing.append("sar") - if not tools.py_spy: + if tools.py_spy: + bundle.host_pyspy = out_dir / "host_flamegraph.svg" + host_procs.append(( + "py-spy", + subprocess.Popen( + [ + tools.py_spy, + "record", + "--pid", + str(workload.pid), + "--subprocesses", + "--duration", + str(host_capture_s), + "--output", + str(bundle.host_pyspy), + ], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ), + )) + else: bundle.missing.append("py-spy") - subprocess.run(argv, check=False) + workload.wait() + if workload.returncode != 0: + bundle.missing.append(f"workload command failed (exit {workload.returncode})") - for hp in host_procs: + for name, hp in host_procs: try: hp.wait(timeout=host_capture_s + 5) except subprocess.TimeoutExpired: hp.terminate() + hp.wait(timeout=5) + bundle.missing.append(f"{name} sampler timed out") + else: + if hp.returncode != 0: + bundle.missing.append(f"{name} capture failed (exit {hp.returncode})") + if sar_output is not None: + sar_output.close() + if tools.py_spy and bundle.host_pyspy and not bundle.host_pyspy.exists(): + bundle.missing.append("py-spy capture produced no flamegraph") if config.vendor == "nvidia" and tools.nsys and bundle.gpu_report: bundle.gpu_csv = _export_nsys_csv(tools.nsys, bundle.gpu_report, out_dir) + if bundle.gpu_csv is None: + bundle.missing.append("nsys GPU CSV export") return bundle @@ -284,14 +325,21 @@ def build_breakdown(phases: list[PhaseTiming]) -> list[StallPhase]: gpu = p.gpu_busy_s / wall sync = p.sync_s / wall cpu = p.cpu_s / wall - data_stall = max(0.0, 1.0 - gpu - sync - cpu) + components = {"gpu_busy_s": gpu, "sync_s": sync, "cpu_s": cpu} + invalid = {name: value for name, value in components.items() if value < 0 or value > 1} + if invalid or gpu + sync + cpu > 1.0 + 1e-9: + raise ValueError( + f"phase {p.phase!r} has contradictory/overlapping timing attribution: " + f"gpu={gpu:.3f}, sync={sync:.3f}, cpu={cpu:.3f}; refusing to clamp" + ) + data_stall = 1.0 - gpu - sync - cpu out.append( StallPhase( phase=p.phase, - cpu=min(1.0, cpu), - data_stall=min(1.0, data_stall), - sync=min(1.0, sync), - gpu_active=min(1.0, gpu), + cpu=cpu, + data_stall=data_stall, + sync=sync, + gpu_active=gpu, throughput=p.throughput, wall_clock_s=wall, ) diff --git a/gitm/bench/runner.py b/gitm/bench/runner.py index d8637a1..382a353 100644 --- a/gitm/bench/runner.py +++ b/gitm/bench/runner.py @@ -92,10 +92,23 @@ def run_seed( payload = _last_json_line(proc.stdout) + provenance_warnings: list[str] = [] manifest_sha = None resolved_manifest = _resolve_manifest(config, manifest_path, config_dir) if resolved_manifest and Path(resolved_manifest).exists(): manifest_sha = manifest_digest(resolved_manifest) + else: + provenance_warnings.append( + f"dataset manifest unavailable at {resolved_manifest!s}; baseline cannot be signed off" + ) + + git_sha = _git_sha() + if git_sha == "unknown": + provenance_warnings.append("git SHA unavailable; baseline cannot be signed off") + if not payload.get("gpu_name"): + provenance_warnings.append("harness omitted gpu_name") + if payload.get("device_count") is None: + provenance_warnings.append("harness omitted device_count; defaulted to 1") breakdown = [StallPhase.model_validate(p) for p in payload.get("stall_breakdown", [])] @@ -106,10 +119,11 @@ def run_seed( metric=config.metric, metric_value=float(payload["metric_value"]), warm_window_s=config.warm_window_s, - git_sha=_git_sha(), + git_sha=git_sha, gitm_version=__version__, harness_commit=payload.get("harness_commit"), manifest_sha256=manifest_sha, + provenance_warnings=provenance_warnings, gpu_name=payload.get("gpu_name", ""), device_count=int(payload.get("device_count", 1)), started_at_ns=started, diff --git a/gitm/bench/schema.py b/gitm/bench/schema.py index da372b8..dbff132 100644 --- a/gitm/bench/schema.py +++ b/gitm/bench/schema.py @@ -93,7 +93,7 @@ class BenchConfig(BaseModel): seeds: list[int] = Field(min_length=1) spread_tolerance: float = Field(default=0.02, gt=0.0, le=1.0) gpu_active_ceiling: float = Field(default=0.85, gt=0.0, le=1.0) - baseline_target: float | None = None + baseline_target: float | None = Field(default=None, gt=0.0, allow_inf_nan=False) target_direction: TargetDirection = "ge" dataset: DatasetRef work_unit: WorkUnit @@ -122,7 +122,7 @@ class StallPhase(BaseModel): data_stall: float = Field(ge=0.0, le=1.0) sync: float = Field(ge=0.0, le=1.0) gpu_active: float = Field(ge=0.0, le=1.0) - throughput: float | None = None # in the benchmark's metric units + throughput: float | None = Field(default=None, ge=0.0, allow_inf_nan=False) wall_clock_s: float = Field(ge=0.0) @@ -140,14 +140,15 @@ class BaselineRun(BaseModel): seed: int vendor: Vendor metric: str - metric_value: float - warm_window_s: int + metric_value: float = Field(gt=0.0, allow_inf_nan=False) + warm_window_s: int = Field(gt=0) # Provenance: a baseline is only reproducible if these are pinned. git_sha: str gitm_version: str harness_commit: str | None = None manifest_sha256: str | None = None # sha256 of the dataset manifest itself + provenance_warnings: list[str] = Field(default_factory=list) gpu_name: str = "" device_count: int = 1 @@ -156,7 +157,7 @@ class BaselineRun(BaseModel): stall_breakdown: list[StallPhase] = Field(default_factory=list) - def gpu_active_overall(self) -> float: + def gpu_active_overall(self) -> float | None: """Wall-clock-weighted GPU active fraction across phases. This is the number checked against ``gpu_active_ceiling`` — a single @@ -164,5 +165,5 @@ def gpu_active_overall(self) -> float: """ total = sum(p.wall_clock_s for p in self.stall_breakdown) if total <= 0.0: - return 0.0 + return None return sum(p.gpu_active * p.wall_clock_s for p in self.stall_breakdown) / total diff --git a/gitm/bench/templates/results.md.j2 b/gitm/bench/templates/results.md.j2 index 491653a..aa98a84 100644 --- a/gitm/bench/templates/results.md.j2 +++ b/gitm/bench/templates/results.md.j2 @@ -8,7 +8,7 @@ | Mean | {{ "%.4g"|format(s.mean) }} | | Std dev | {{ "%.4g"|format(s.stddev) }} | | Spread (max−min)/mean | {{ pct(s.spread) }} | -| GPU active (worst run) | {{ pct(s.gpu_active_overall) }} | +| GPU active (worst run) | {{ pct(s.gpu_active_overall) if s.gpu_active_overall is not none else "unmeasured" }} | ## Sign-off gates @@ -19,7 +19,11 @@ {% endfor %} **Overall: {{ "✅ SIGNED OFF" if s.passed else "❌ NOT SIGNED OFF" }}** +{% if s.gpu_active_overall is none %} +GPU active coverage is **unmeasured; saturation sign-off refused**. +{% else %} GPU active % {{ pct(s.gpu_active_overall) }} {{ "<" if s.gpu_active_overall < gpu_active_ceiling else "≥" }} ceiling {{ pct(gpu_active_ceiling) }}{% if s.gpu_active_overall >= gpu_active_ceiling %} — **saturated, swap rule applies**{% endif %}. +{% endif %} ## Stall breakdown {% if breakdown %} diff --git a/gitm/benchmarks/edge/baseline.py b/gitm/benchmarks/edge/baseline.py index cfaf57a..8f71f0d 100644 --- a/gitm/benchmarks/edge/baseline.py +++ b/gitm/benchmarks/edge/baseline.py @@ -41,12 +41,16 @@ from pathlib import Path from typing import Any +from gitm._timing import require_positive_duration + WARM_FRAMES = 100 # discarded before the timing window NVML_SAMPLE_HZ = 5 GPU_ACTIVE_WARN_PCT = 85.0 -def _sample_nvml(samples: list[float], stop_event: threading.Event) -> None: +def _sample_nvml( + samples: list[float], stop_event: threading.Event, diagnostics: list[str] +) -> None: """Background thread: sample GPU 0 utilization at NVML_SAMPLE_HZ.""" try: import pynvml @@ -59,8 +63,10 @@ def _sample_nvml(samples: list[float], stop_event: threading.Event) -> None: samples.append(float(util)) time.sleep(interval) pynvml.nvmlShutdown() - except Exception: - pass # no GPU or pynvml absent -> nvml_mean_pct null in output + except Exception as exc: + diagnostics.append( + f"NVML telemetry unavailable: {type(exc).__name__}: {exc}" + ) def _convergence_check(results: list[dict]) -> tuple[bool, float]: @@ -148,9 +154,12 @@ def run_baseline( warm_indices = run_indices[WARM_FRAMES:] nvml_samples: list[float] = [] + telemetry_diagnostics: list[str] = [] stop_event = threading.Event() nvml_thread = threading.Thread( - target=_sample_nvml, args=(nvml_samples, stop_event), daemon=True + target=_sample_nvml, + args=(nvml_samples, stop_event, telemetry_diagnostics), + daemon=True, ) nvml_thread.start() @@ -174,7 +183,9 @@ def run_baseline( nvml_thread.join(timeout=5) n_warm = len(warm_indices) - elapsed = t_wall_end - t_wall_start + elapsed = require_positive_duration( + t_wall_end - t_wall_start, context="nuScenes baseline warm window" + ) fps = n_warm / elapsed data_stall_pct = sum(data_stall_fracs) / n_warm * 100 @@ -194,6 +205,7 @@ def run_baseline( "sync_stall_pct": round(sync_stall_pct, 2), "cpu_pct": round(cpu_pct, 2), "nvml_mean_util_pct": round(nvml_mean, 2) if nvml_mean is not None else None, + "telemetry_diagnostics": telemetry_diagnostics, "total_detections": total_detections, "elapsed_s": round(elapsed, 3), "max_sweeps": max_sweeps, @@ -215,13 +227,15 @@ def run_baseline( "captured_at_iso": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), } - output_path.write_text(json.dumps(output, indent=2)) + output_path.write_text(json.dumps(output, indent=2), encoding="utf-8") print( f"\nResult: {fps:.1f} fps | GPU active {gpu_active_pct:.1f}% " f"(NVML {nvml_mean:.1f}%)" if nvml_mean is not None else f"\nResult: {fps:.1f} fps | GPU active {gpu_active_pct:.1f}%" ) print(f"Wrote {output_path}") + for diagnostic in telemetry_diagnostics: + print(f"WARNING: {diagnostic}") if nvml_mean is not None and nvml_mean > GPU_ACTIVE_WARN_PCT: print( diff --git a/gitm/benchmarks/edge/optimize.py b/gitm/benchmarks/edge/optimize.py index a76d195..444cad6 100644 --- a/gitm/benchmarks/edge/optimize.py +++ b/gitm/benchmarks/edge/optimize.py @@ -23,10 +23,12 @@ from __future__ import annotations +import math import time from collections.abc import Callable from dataclasses import dataclass, field +from gitm._timing import require_positive_duration, require_positive_work from gitm.kernels.spec import Applicability, InterventionSpec, SafetyGate # A run_mode runs N frames in a given mode and returns a per-frame summary dict: @@ -68,7 +70,7 @@ def _frame_equivalent(da: list, db: list, *, center_atol: float, tol_frac: float else: unmatched += 1 unmatched += used.count(False) # candidate boxes with no baseline partner - allowed = max(1, int(tol_frac * max(len(da), len(db), 1))) + allowed = math.ceil(tol_frac * max(len(da), len(db))) return unmatched <= allowed @@ -87,6 +89,10 @@ def detections_equivalent( with a tolerance keeps the gate honest (a genuine regression still trips it) without rejecting rounding-level churn. """ + if not math.isfinite(center_atol) or center_atol < 0.0: + raise ValueError(f"center_atol must be finite and non-negative, got {center_atol!r}") + if not math.isfinite(tol_frac) or not 0.0 <= tol_frac <= 1.0: + raise ValueError(f"tol_frac must be finite and in [0, 1], got {tol_frac!r}") fa, fb = a.get("frames"), b.get("frames") if fa is None or fb is None or len(fa) != len(fb): return False @@ -134,18 +140,30 @@ def optimize_edge( launch jitter. ``sync`` is invoked after each run so GPU timing is honest (pass a device sync; default no-op for CPU/fake). """ + if isinstance(reps, bool) or not isinstance(reps, int) or reps <= 0: + raise ValueError(f"reps must be a positive integer, got {reps!r}") + # Validate before executing either leg: gate controls must never be repaired + # after the caller supplied them. + detections_equivalent( + {"frames": []}, + {"frames": []}, + center_atol=center_atol, + tol_frac=tol_frac, + ) sync = sync or (lambda: None) def _timed(mode: str) -> tuple[dict, float]: best = float("inf") summary: dict = {} - for _ in range(max(1, reps)): + for _ in range(reps): t0 = time.perf_counter() summary = run_mode(mode) sync() best = min(best, time.perf_counter() - t0) n = max(int(summary.get("n_frames", 0)), 0) - return summary, n / max(best, 1e-9) + require_positive_work(n, context=f"edge A/B {mode}") + best = require_positive_duration(best, context=f"edge A/B {mode}") + return summary, n / best base_summary, base_eps = _timed(baseline_mode) cand_summary, cand_eps = _timed(candidate_mode) @@ -288,6 +306,8 @@ def edge_batching_spec(batch_size: int = 4) -> InterventionSpec: ``batch_size`` is recorded as the spec's ``value`` so the provenance reflects the batch size actually run, not a hardcoded constant. """ + if isinstance(batch_size, bool) or not isinstance(batch_size, int) or batch_size <= 0: + raise ValueError(f"batch_size must be a positive integer, got {batch_size!r}") return InterventionSpec( name="edge_frame_batching", summary=f"Run inference on batches of {batch_size} frames in one forward " @@ -343,6 +363,8 @@ def __init__( tol_frac: float = 0.05, spec: InterventionSpec | None = None, ): + if isinstance(batch_size, bool) or not isinstance(batch_size, int) or batch_size <= 0: + raise ValueError(f"batch_size must be a positive integer, got {batch_size!r}") self._run_mode = run_mode self._batch_size = batch_size self._reps = reps diff --git a/gitm/benchmarks/edge/workunit.py b/gitm/benchmarks/edge/workunit.py index fa0b1bc..1fb7142 100644 --- a/gitm/benchmarks/edge/workunit.py +++ b/gitm/benchmarks/edge/workunit.py @@ -36,6 +36,8 @@ from pathlib import Path from typing import Any +from gitm._timing import require_positive_duration + # SHA256 of cbgs_pp_centerpoint_nds6070.pth (OpenPCDet nuScenes # CenterPoint-PointPillar checkpoint). Confirm with: # sha256sum cbgs_pp_centerpoint_nds6070.pth @@ -73,21 +75,24 @@ def data_stall_frac(self) -> float: With the dyn config, voxelization is on the GPU, so this reflects only the multi-sweep load. See the module STAGE-TIMING CAVEAT. """ - if self.t_total_s <= 0: - return 0.0 - return (self.t_load_s + self.t_preprocess_s) / self.t_total_s + wall = require_positive_duration( + self.t_total_s, context=f"nuScenes frame {self.frame_id}" + ) + return (self.t_load_s + self.t_preprocess_s) / wall @property def sync_stall_frac(self) -> float: - if self.t_total_s <= 0: - return 0.0 - return self.t_postprocess_s / self.t_total_s + wall = require_positive_duration( + self.t_total_s, context=f"nuScenes frame {self.frame_id}" + ) + return self.t_postprocess_s / wall @property def gpu_active_frac(self) -> float: - if self.t_total_s <= 0: - return 0.0 - return self.t_inference_s / self.t_total_s + wall = require_positive_duration( + self.t_total_s, context=f"nuScenes frame {self.frame_id}" + ) + return self.t_inference_s / wall class NuScenesWorkUnit: diff --git a/gitm/benchmarks/hft/harness.py b/gitm/benchmarks/hft/harness.py index 9e0a15f..6ff1bf0 100644 --- a/gitm/benchmarks/hft/harness.py +++ b/gitm/benchmarks/hft/harness.py @@ -25,8 +25,11 @@ import json import os import time +import warnings from pathlib import Path +from gitm._timing import require_positive_duration + def select_backend(): """Return ``(kind, df_module, array_module)`` — cuDF/CuPy if available, else pandas/NumPy.""" @@ -35,7 +38,13 @@ def select_backend(): import cupy return "gpu", cudf, cupy - except Exception: + except Exception as exc: + warnings.warn( + f"cuDF/CuPy backend unavailable; HFT harness is using the CPU fallback " + f"and cannot produce a GPU baseline ({type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) import numpy import pandas @@ -52,7 +61,12 @@ def _gpu_name(kind: str) -> tuple[str, int]: props = cupy.cuda.runtime.getDeviceProperties(0) name = props["name"].decode() if isinstance(props["name"], bytes) else str(props["name"]) return name, n - except Exception: + except Exception as exc: + warnings.warn( + f"GPU identity probe failed ({type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) return "gpu-unknown", 1 @@ -162,7 +176,9 @@ def main(argv: list[str] | None = None) -> int: # Warm window: replay the loaded events, measuring sustained throughput. t0 = time.perf_counter() summary = run_pipeline(df, dflib) - elapsed = max(time.perf_counter() - t0, 1e-9) + elapsed = require_positive_duration( + time.perf_counter() - t0, context="HFT harness" + ) events_per_second = summary["events"] / elapsed print(f"[hft harness:{kind}] {summary['events']} events in {elapsed:.3f}s " diff --git a/gitm/benchmarks/hft/optimize.py b/gitm/benchmarks/hft/optimize.py index 1fc4eed..afb15f4 100644 --- a/gitm/benchmarks/hft/optimize.py +++ b/gitm/benchmarks/hft/optimize.py @@ -25,6 +25,7 @@ import time from dataclasses import dataclass +from gitm._timing import require_positive_duration, require_positive_work from gitm.benchmarks.hft.harness import microprice, run_pipeline, vwap_1s from gitm.kernels.spec import Applicability, InterventionSpec, SafetyGate @@ -107,17 +108,21 @@ def optimize_hft(df, dflib, *, reps: int = 3, sync=None) -> ABResult: reduce launch-jitter noise. ``sync`` is an optional callable invoked after each run so GPU timing is honest (pass a device-sync; default no-op for CPU). """ + if isinstance(reps, bool) or not isinstance(reps, int) or reps <= 0: + raise ValueError(f"reps must be a positive integer, got {reps!r}") sync = sync or (lambda: None) def _timed(fn) -> tuple[dict, float]: best = float("inf") summary: dict = {} - for _ in range(max(1, reps)): + for _ in range(reps): t0 = time.perf_counter() summary = fn(df, dflib) sync() best = min(best, time.perf_counter() - t0) - return summary, summary["events"] / max(best, 1e-9) + best = require_positive_duration(best, context=f"HFT A/B {fn.__name__}") + events = require_positive_work(summary["events"], context=f"HFT A/B {fn.__name__}") + return summary, events / best base_summary, base_eps = _timed(run_pipeline) cand_summary, cand_eps = _timed(run_pipeline_fast) @@ -198,8 +203,12 @@ def optimize_hft_streaming(batches, dflib, *, sync=None, on_batch=None) -> ABRes if n_batches == 0: raise ValueError("optimize_hft_streaming: no batches to process (empty iterable)") - base_eps = base_events / max(base_t, 1e-9) - cand_eps = cand_events / max(cand_t, 1e-9) + require_positive_work(base_events, context="HFT streaming baseline") + require_positive_work(cand_events, context="HFT streaming candidate") + base_t = require_positive_duration(base_t, context="HFT streaming baseline") + cand_t = require_positive_duration(cand_t, context="HFT streaming candidate") + base_eps = base_events / base_t + cand_eps = cand_events / cand_t speedup = cand_eps / base_eps if base_eps else 0.0 kept = "candidate" if (identical and cand_eps > base_eps) else "baseline" # Per-pipeline count aggregates (kept separate so a divergent run never reports diff --git a/gitm/benchmarks/kitti/baseline.py b/gitm/benchmarks/kitti/baseline.py index 2971d36..676f54d 100644 --- a/gitm/benchmarks/kitti/baseline.py +++ b/gitm/benchmarks/kitti/baseline.py @@ -38,6 +38,8 @@ from pathlib import Path from typing import Any +from gitm._timing import require_positive_duration + WARM_FRAMES = 100 # discard before timing window NVML_SAMPLE_HZ = 5 @@ -124,7 +126,9 @@ def _load_frame_paths(data_root: Path) -> list[Path]: return paths -def _sample_nvml(samples: list[dict], stop_event: threading.Event) -> None: +def _sample_nvml( + samples: list[dict], stop_event: threading.Event, diagnostics: list[str] +) -> None: """Background thread: sample GPU 0 util + memory at NVML_SAMPLE_HZ. Each sample is a dict with util_pct, mem_used_bytes, mem_total_bytes — @@ -146,8 +150,10 @@ def _sample_nvml(samples: list[dict], stop_event: threading.Event) -> None: }) time.sleep(interval) pynvml.nvmlShutdown() - except Exception: - pass # no GPU or pynvml absent — headroom fields will be null in output + except Exception as exc: + diagnostics.append( + f"NVML telemetry unavailable: {type(exc).__name__}: {exc}" + ) def _convergence_check(results: list[dict]) -> tuple[bool, float]: @@ -230,9 +236,12 @@ def run_baseline( # Timed warm window warm_paths = run_paths[WARM_FRAMES:] nvml_samples: list[dict] = [] + headroom_diagnostics: list[str] = [] stop_event = threading.Event() nvml_thread = threading.Thread( - target=_sample_nvml, args=(nvml_samples, stop_event), daemon=True + target=_sample_nvml, + args=(nvml_samples, stop_event, headroom_diagnostics), + daemon=True, ) nvml_thread.start() @@ -267,7 +276,9 @@ def run_baseline( nvml_thread.join(timeout=5) n_warm = len(warm_paths) - elapsed = t_wall_end - t_wall_start + elapsed = require_positive_duration( + t_wall_end - t_wall_start, context="KITTI baseline warm window" + ) fps = n_warm / elapsed data_stall_pct = sum(data_stall_fracs) / n_warm * 100 @@ -286,17 +297,35 @@ def run_baseline( from gitm.optimizer.headroom_kernel_rank import gpu_headroom h = gpu_headroom(nvml_samples) - if h is not None: - headroom_dict = { - "compute_headroom_pct": round(h.compute_headroom_pct, 2), - "mean_util_pct": round(h.mean_util_pct, 2), - "peak_util_pct": round(h.peak_util_pct, 2), - "mem_free_at_peak_gb": round(h.mem_free_at_peak_bytes / 1e9, 3), - "mem_total_gb": round(h.mem_total_bytes / 1e9, 3), - "nvml_n_samples": h.n_samples, - } - except Exception: - pass # graceful degradation if headroom_kernel_rank unavailable + headroom_dict = { + "compute_headroom_pct": ( + round(h.compute_headroom_pct, 2) + if h.compute_headroom_pct is not None + else None + ), + "mean_util_pct": ( + round(h.mean_util_pct, 2) if h.mean_util_pct is not None else None + ), + "peak_util_pct": ( + round(h.peak_util_pct, 2) if h.peak_util_pct is not None else None + ), + "mem_free_at_peak_gb": ( + round(h.mem_free_at_peak_bytes / 1e9, 3) + if h.mem_free_at_peak_bytes is not None + else None + ), + "mem_total_gb": ( + round(h.mem_total_bytes / 1e9, 3) + if h.mem_total_bytes is not None + else None + ), + "nvml_n_samples": h.n_samples, + } + headroom_diagnostics.extend(h.diagnostics) + except Exception as exc: + headroom_diagnostics.append( + f"GPU headroom analysis unavailable: {type(exc).__name__}: {exc}" + ) import platform import socket @@ -310,6 +339,7 @@ def run_baseline( "sync_stall_pct": round(sync_stall_pct, 2), "cpu_pct": round(cpu_pct, 2), **headroom_dict, + "headroom_diagnostics": headroom_diagnostics, "stage_spread": spread, "total_detections": total_detections, "elapsed_s": round(elapsed, 3), @@ -321,21 +351,32 @@ def run_baseline( "captured_at_iso": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), } - output_path.write_text(json.dumps(output, indent=2)) + output_path.write_text(json.dumps(output, indent=2), encoding="utf-8") # Save stage spread report alongside the JSON for easy review report_path = output_path.with_name(output_path.stem + "_stage_spread.txt") - report_path.write_text(spread_report) + report_path.write_text(spread_report, encoding="utf-8") print( f"\nResult: {fps:.1f} fps | GPU active {gpu_active_pct:.1f}% " f"| data stall {data_stall_pct:.1f}% | wrote {output_path}" ) - if headroom_dict: - print( - f"Headroom: compute {headroom_dict['compute_headroom_pct']:.1f}% free " - f"| mem {headroom_dict['mem_free_at_peak_gb']:.1f} GB free at peak" + headroom_parts: list[str] = [] + if headroom_dict.get("compute_headroom_pct") is not None: + headroom_parts.append( + f"compute {headroom_dict['compute_headroom_pct']:.1f}% free" ) - - if headroom_dict.get("mean_util_pct", 0) > 85.0: + if headroom_dict.get("mem_free_at_peak_gb") is not None: + headroom_parts.append( + f"mem {headroom_dict['mem_free_at_peak_gb']:.1f} GB free at peak" + ) + if headroom_parts: + print(f"Headroom: {' | '.join(headroom_parts)}") + for diagnostic in headroom_diagnostics: + print(f"WARNING: {diagnostic}") + + if ( + headroom_dict.get("mean_util_pct") is not None + and headroom_dict["mean_util_pct"] > 85.0 + ): print( f"\nWARNING: NVML util={headroom_dict['mean_util_pct']:.1f}% > 85% — " "flag for review: workload may be near-saturated. Consider the 500-frame fallback." diff --git a/gitm/benchmarks/kitti/workunit.py b/gitm/benchmarks/kitti/workunit.py index 39ac4d7..7a82b33 100644 --- a/gitm/benchmarks/kitti/workunit.py +++ b/gitm/benchmarks/kitti/workunit.py @@ -27,6 +27,8 @@ from pathlib import Path from typing import Any +from gitm._timing import require_positive_duration + # SHA256 of pointpillar_7728.pth (OpenPCDet KITTI PointPillars checkpoint). # Confirm with: sha256sum pointpillar_7728.pth CHECKPOINT_SHA256 = "4c83fc0fa02575b9b3e9dec676f698e7a70bb5a795e89f91df8a96b916fa19e2" @@ -50,23 +52,20 @@ class WorkUnitResult: @property def data_stall_frac(self) -> float: """Fraction of frame time spent on data loading + voxelization.""" - if self.t_total_s <= 0: - return 0.0 - return (self.t_load_s + self.t_preprocess_s) / self.t_total_s + wall = require_positive_duration(self.t_total_s, context=f"KITTI frame {self.frame_id}") + return (self.t_load_s + self.t_preprocess_s) / wall @property def sync_stall_frac(self) -> float: """Fraction of frame time spent on NMS (CPU-serialized post-processing).""" - if self.t_total_s <= 0: - return 0.0 - return self.t_postprocess_s / self.t_total_s + wall = require_positive_duration(self.t_total_s, context=f"KITTI frame {self.frame_id}") + return self.t_postprocess_s / wall @property def gpu_active_frac(self) -> float: """Fraction of frame time spent in GPU backbone + BEV head.""" - if self.t_total_s <= 0: - return 0.0 - return self.t_inference_s / self.t_total_s + wall = require_positive_duration(self.t_total_s, context=f"KITTI frame {self.frame_id}") + return self.t_inference_s / wall class WorkUnit: diff --git a/gitm/cli.py b/gitm/cli.py index 47c042a..5faeb8a 100644 --- a/gitm/cli.py +++ b/gitm/cli.py @@ -4,6 +4,7 @@ import argparse import json +import math import sys from pathlib import Path @@ -148,6 +149,18 @@ def _parser() -> argparse.ArgumentParser: _add_capture(sub) sub.add_parser("doctor", help="Probe environment, GPUs, and data locations.") + sub.add_parser( + "gpu-headroom", + help="Print a one-shot live GPU utilization and memory snapshot.", + ) + + plan_kitti = sub.add_parser( + "plan-kitti", help="Render the PointPillars execution graph for a known GPU SKU." + ) + plan_kitti.add_argument("--sku", required=True, help="GPU SKU from the hardware catalogue.") + plan_kitti.add_argument( + "--baseline", type=Path, default=None, help="Optional measured baseline JSON to compare." + ) analyze = sub.add_parser( "analyze", @@ -206,8 +219,12 @@ def _parser() -> argparse.ArgumentParser: def _parse_target(s: str) -> float: s = s.strip() if s.endswith("%"): - return float(s[:-1]) / 100.0 - return float(s) + target = float(s[:-1]) / 100.0 + else: + target = float(s) + if not math.isfinite(target) or not 0.0 < target <= 1.0: + raise ValueError(f"target floor must be finite and in (0, 1], got {s!r}") + return target _HFT_WORKLOADS = {"hft", "hft-lob"} @@ -309,14 +326,26 @@ def main(argv: list[str] | None = None) -> int: target=_parse_target(args.target), scratch=args.scratch, ) - summary = result.get("summary", {}) + summary = result.get("summary") + if not isinstance(summary, dict): + summary = { + "status": "invalid_result", + "diagnostic": "optimization loop returned no machine-readable summary", + } if args.report is not None: - args.report.write_text(result.get("report_md", "")) + report_md = result.get("report_md") + if not isinstance(report_md, str) or not report_md.strip(): + report_md = ( + "# Runtime result unavailable\n\n" + f"{summary.get('diagnostic', 'optimization loop returned no report')}\n" + ) + args.report.write_text(report_md, encoding="utf-8") else: print(json.dumps(summary, indent=2)) - # Non-zero so automation notices a run that measured nothing (no GPU / - # CUPTI shim, or the workload never ran) instead of seeing a fake pass. - return 3 if summary.get("status") == "no_data" else 0 + # Only the explicit success state is a shell success. Prediction/candidate + # refusals and failed A/Bs may still have useful measurement reports, but + # automation must not read those degraded outcomes as a completed run. + return 0 if summary.get("status") == "ok" else 3 if args.cmd == "replay": from gitm.optimizer.replay import predict_delta_from_files @@ -332,7 +361,7 @@ def main(argv: list[str] | None = None) -> int: args.intervention, config=args.config, min_keep_delta=args.min_keep_delta ) print(json.dumps(result, indent=2)) - return 0 + return 0 if result.get("applied") and not result.get("rolled_back") else 3 if args.cmd == "attach": from gitm.deploy import attach_job @@ -357,6 +386,32 @@ def main(argv: list[str] | None = None) -> int: print(json.dumps(report, indent=2)) return 0 + if args.cmd == "gpu-headroom": + from gitm.optimizer.headroom_kernel_rank import live_gpu_headroom + + rows = live_gpu_headroom() + print(json.dumps(rows, indent=2)) + return 0 if rows else 3 + + if args.cmd == "plan-kitti": + from gitm.planner.context import hardware_spec_for, peak_for_sku + from gitm.planner.kitti_graph import predict_kitti_graph, render_kitti_graph + + peak = peak_for_sku(args.sku) + if peak is None: + print(f"prediction refused: GPU SKU {args.sku!r} is not in the hardware catalogue") + return 3 + measured = None + if args.baseline is not None: + try: + measured = json.loads(args.baseline.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + print(f"prediction refused: baseline {args.baseline} is unreadable ({exc})") + return 3 + graph = predict_kitti_graph(hw=hardware_spec_for(peak)) + print(render_kitti_graph(graph, measured=measured)) + return 0 + if args.cmd == "analyze": from gitm.importers.analyze import analyze_paths diff --git a/gitm/cuda_env.py b/gitm/cuda_env.py index cec928f..ae59993 100644 --- a/gitm/cuda_env.py +++ b/gitm/cuda_env.py @@ -25,6 +25,7 @@ import re import subprocess +import warnings from dataclasses import dataclass from pathlib import Path @@ -223,13 +224,28 @@ def check() -> list[Problem]: return problems +def unverified_cuda_components() -> list[str]: + """CUDA build identities that could not be observed on this installation.""" + missing: list[str] = [] + if torch_cuda() is None: + missing.append("PyTorch CUDA build") + if vllm_cuda_major() is None: + missing.append("vLLM CUDA build") + return missing + + def require_compatible() -> None: """Raise before an expensive build if the stack can't run on this driver. Called by the vLLM workload factory. Without it the failure surfaces ~90s in, inside ``torch._C._cuda_init()``, after the weights are already downloaded. """ driver = driver_cuda() - if driver is not None and stack_for(driver) is None: + if driver is None: + raise RuntimeError( + "no NVIDIA driver detected; refusing to build a GPU workload on an " + "unverified CPU-only or driver-inaccessible host" + ) + if stack_for(driver) is None: raise RuntimeError( f"unsupported host: this driver supports only CUDA {driver[0]}.{driver[1]}, " f"and vLLM's wheels are CUDA {max(SUPPORTED_STACKS)} builds. No torch " @@ -238,14 +254,22 @@ def require_compatible() -> None: ) problems = check() - if not problems: - return - raise RuntimeError( - "CUDA stack is incompatible with this host's driver:\n\n" - + "\n\n".join(str(p) for p in problems) - + "\n\nThe driver belongs to the host and cannot be changed from inside the " - "container. Run python -m gitm.cuda_env to re-check." - ) + if problems: + raise RuntimeError( + "CUDA stack is incompatible with this host's driver:\n\n" + + "\n\n".join(str(p) for p in problems) + + "\n\nThe driver belongs to the host and cannot be changed from inside the " + "container. Run python -m gitm.cuda_env to re-check." + ) + unverified = unverified_cuda_components() + if unverified: + warnings.warn( + f"driver CUDA {driver[0]}.{driver[1]} detected, but " + f"{' and '.join(unverified)} could not be verified; " + "stack compatibility is unknown", + RuntimeWarning, + stacklevel=2, + ) def main(argv: list[str] | None = None) -> int: @@ -291,13 +315,23 @@ def main(argv: list[str] | None = None) -> int: if stack.note: print(f"NOTE: {stack.note}") problems = check() - if not problems: + unverified = unverified_cuda_components() + if not problems and not unverified: print("OK — every component runs on this driver.") return 0 - print() - for p in problems: - print(p) - return 1 + if problems: + print() + for p in problems: + print(p) + return 1 + if unverified: + print( + "UNVERIFIED — " + + " and ".join(unverified) + + " could not be read; stack compatibility is unknown." + ) + return 2 + raise AssertionError("unreachable CUDA diagnostic state") if __name__ == "__main__": diff --git a/gitm/deploy/attach.py b/gitm/deploy/attach.py index 41c91b3..5ec0718 100644 --- a/gitm/deploy/attach.py +++ b/gitm/deploy/attach.py @@ -27,7 +27,7 @@ class AttachPlan: job_id: str workload: str | None mode: str # always "user-space" - status: str # "planned" | "attached" | "no_target" + status: str # "planned" | "unsupported" | "no_target" pid: int | None steps: list[str] = field(default_factory=list) reason: str = "" @@ -101,12 +101,17 @@ def attach_job( reason=f"PID {resolved} is not live.", ).to_dict() + # PID resolution is wired, but no injector or telemetry-shim installation is. + # Refuse rather than turning a validated target into a false attach success. return AttachPlan( job_id=job_id, workload=workload, mode="user-space", - status="attached", + status="unsupported", pid=resolved, steps=steps, - reason="attached (user-space, fail-open).", + reason=( + "target is live, but standalone PID injection is not implemented; " + "no telemetry shim was installed" + ), ).to_dict() diff --git a/gitm/doctor.py b/gitm/doctor.py index ad25a0e..38262fb 100644 --- a/gitm/doctor.py +++ b/gitm/doctor.py @@ -22,8 +22,25 @@ def doctor() -> dict[str, Any]: from gitm.telemetry.backends import discover_backends - backends = discover_backends() - info["telemetry_backends"] = [ - {"vendor": b.vendor, "device_count": b.device_count()} for b in backends - ] + diagnostics: list[str] = [] + backends = discover_backends(diagnostics=diagnostics) + telemetry_backends: list[dict[str, Any]] = [] + for backend in backends: + try: + telemetry_backends.append( + {"vendor": backend.vendor, "device_count": backend.device_count()} + ) + except Exception as exc: + diagnostics.append( + f"{backend.vendor} telemetry probe failed: {type(exc).__name__}: {exc}" + ) + finally: + try: + backend.close() + except Exception as exc: + diagnostics.append( + f"{backend.vendor} telemetry close failed: {type(exc).__name__}: {exc}" + ) + info["telemetry_backends"] = telemetry_backends + info["telemetry_diagnostics"] = diagnostics return info diff --git a/gitm/importers/_common.py b/gitm/importers/_common.py index 07882ea..180cb88 100644 --- a/gitm/importers/_common.py +++ b/gitm/importers/_common.py @@ -34,6 +34,27 @@ class ImportStats: device_name: str | None = None +def merge_normalization_stats(target: ImportStats, children: Iterable[ImportStats]) -> None: + """Carry per-device cleanup evidence into the file-level import result.""" + for child in children: + target.deduped += child.deduped + target.dropped_invalid += child.dropped_invalid + for note in child.warnings: + if note.startswith(("deduped ", "dropped ")): + continue + if note not in target.warnings: + target.warnings.append(note) + if target.deduped: + target.warnings.append( + f"deduped {target.deduped} exactly-identical event row(s) across imported devices" + ) + if target.dropped_invalid: + target.warnings.append( + f"dropped {target.dropped_invalid} event(s) with invalid or non-monotonic " + "timestamps across imported devices" + ) + + class ImportError(Exception): """Per-file import failure with a customer-readable message.""" @@ -435,6 +456,12 @@ def finish_trace( total_raw_events=len(events), per_device_kernel_counts=per_device_kernel_counts(cleaned), ) + if deduped: + stats.warnings.append(f"deduped {deduped} exactly-identical event row(s)") + if dropped: + stats.warnings.append( + f"dropped {dropped} event(s) with invalid or non-monotonic timestamps" + ) return trace, stats diff --git a/gitm/importers/analyze.py b/gitm/importers/analyze.py index 178799f..d8aaec4 100644 --- a/gitm/importers/analyze.py +++ b/gitm/importers/analyze.py @@ -3,6 +3,7 @@ from __future__ import annotations import json +import re import sys from dataclasses import dataclass, field from datetime import datetime, timezone @@ -161,6 +162,12 @@ def _device_id_of(trace: Trace) -> int: return trace.events[0].device_id if trace.events else 0 +def _artifact_stem(value: str) -> str: + """Return a portable filename stem without changing the report identifier.""" + stem = re.sub(r"[^A-Za-z0-9_.-]+", "_", value).strip("._") + return stem or "trace" + + def analyze_paths( paths: list[str | Path], *, @@ -256,7 +263,9 @@ def analyze_paths( if keep_traces is not None: keep_dir = Path(keep_traces) keep_dir.mkdir(parents=True, exist_ok=True) - tpath = keep_dir / f"{internal_id}_{tr.run_id}.jsonl" + tpath = keep_dir / ( + f"{_artifact_stem(internal_id)}_{_artifact_stem(tr.run_id)}.jsonl" + ) write_trace_jsonl(tpath, tr) eng_bits.append(render_headroom_md(headroom)) @@ -294,7 +303,9 @@ def analyze_paths( hardware=resolved_sku, num_gpus=max(len(device_analyses), 1), has_collective=rollup.has_collective, - has_interconnect=rollup.has_collective, # best-effort: collectives imply interconnect + # A collective proves communication occurred, not that its + # transport was NVLink/IB rather than PCIe or shared memory. + has_interconnect=None, ) primary = device_analyses[0] diff --git a/gitm/importers/node_rollup.py b/gitm/importers/node_rollup.py index c30bfd4..4239a02 100644 --- a/gitm/importers/node_rollup.py +++ b/gitm/importers/node_rollup.py @@ -10,6 +10,7 @@ from dataclasses import dataclass, field from typing import Any +from gitm._timing import require_positive_duration from gitm.optimizer.metrics import _merge_intervals from gitm.tracer.schema import Trace @@ -149,7 +150,9 @@ def device_comm_stats(trace: Trace) -> DeviceCommStats: kernels = [e for e in trace.events if getattr(e, "kind", None) == "kernel"] # Prefer the device_id on events; fall back to 0. dev = kernels[0].device_id if kernels else 0 - wall = max(trace.duration_ns, 1) + wall = require_positive_duration( + float(trace.duration_ns), context=f"device {dev} communication rollup" + ) comm = [(k.start_ns, k.end_ns) for k in kernels if is_comm_kernel(k.name)] non_comm = [(k.start_ns, k.end_ns) for k in kernels if not is_comm_kernel(k.name)] busy_ns = _interval_len([(k.start_ns, k.end_ns) for k in kernels]) @@ -210,8 +213,8 @@ def build_node_rollup( comm_stats.append(cs) if cs.comm_ns > 0: any_comm = True - weight_sum += max(wall_s, 0.0) - weighted_ceiling += ceiling * max(wall_s, 0.0) + weight_sum += wall_s + weighted_ceiling += ceiling * wall_s busies = list(device_busy.values()) skew = (max(busies) - min(busies)) if busies else 0.0 @@ -219,7 +222,7 @@ def build_node_rollup( n_devices = len(device_busy) multi = multi_device_file or n_devices > 1 comm_inconclusive = multi and not any_comm - node_ceiling = (weighted_ceiling / weight_sum) if weight_sum > 0 else 0.0 + node_ceiling = weighted_ceiling / weight_sum mean_exposed = ( sum(c.exposed_comm_share_of_wall for c in comm_stats) / len(comm_stats) if comm_stats diff --git a/gitm/importers/nsys.py b/gitm/importers/nsys.py index 27a1af7..2445494 100644 --- a/gitm/importers/nsys.py +++ b/gitm/importers/nsys.py @@ -19,6 +19,7 @@ as_int, file_mtime_ns, finish_trace, + merge_normalization_stats, ) from gitm.tracer.schema import KernelEvent, MemcpyEvent, SyncEvent, Trace, TraceEvent @@ -266,24 +267,37 @@ def _resolve_name(row: sqlite3.Row, strings: dict[int, str]) -> str: return "unknown_kernel" -def _map_memory_kind(val: Any, *, strict: bool) -> str: +def _diag_once(diagnostics: list[str] | None, message: str) -> None: + if diagnostics is not None and message not in diagnostics: + diagnostics.append(message) + + +def _map_memory_kind( + val: Any, *, strict: bool, diagnostics: list[str] | None = None +) -> str: if val is None: + _diag_once(diagnostics, "memcpy memory-kind metadata missing; assumed device endpoint") return "device" try: iv = int(val) except (TypeError, ValueError): if strict: raise ImportError(f"unknown memory kind {val!r}") from None + _diag_once(diagnostics, f"unknown memory kind {val!r}; assumed device endpoint") return "device" if iv in _MEMORY_KIND: return _MEMORY_KIND[iv] if strict: raise ImportError(f"unknown CUPTI memory kind enum {iv}") + _diag_once(diagnostics, f"unknown CUPTI memory kind enum {iv}; assumed device endpoint") return "device" -def _map_sync_type(val: Any, *, strict: bool) -> str: +def _map_sync_type( + val: Any, *, strict: bool, diagnostics: list[str] | None = None +) -> str: if val is None: + _diag_once(diagnostics, "synchronization type metadata missing; assumed stream sync") return "stream" try: iv = int(val) @@ -297,11 +311,13 @@ def _map_sync_type(val: Any, *, strict: bool) -> str: return "stream" if strict: raise ImportError(f"unknown sync type {val!r}") from None + _diag_once(diagnostics, f"unknown sync type {val!r}; assumed stream sync") return "stream" if iv in _SYNC_TYPE: return _SYNC_TYPE[iv] if strict: raise ImportError(f"unknown CUPTI sync type enum {iv}") + _diag_once(diagnostics, f"unknown CUPTI sync type enum {iv}; assumed stream sync") return "stream" @@ -357,6 +373,7 @@ def _iter_memcpys( *, strict: bool, device_id: int | None = None, + diagnostics: list[str] | None = None, ) -> Iterator[MemcpyEvent]: from gitm.importers._common import make_memcpy_fast @@ -376,14 +393,15 @@ def _iter_memcpys( dst_k = _pick(row, "dstKind", "destinationKind") copy_k = _pick(row, "copyKind", "memcpyKind") if src_k is not None or dst_k is not None: - src = _map_memory_kind(src_k, strict=strict) - dst = _map_memory_kind(dst_k, strict=strict) + src = _map_memory_kind(src_k, strict=strict, diagnostics=diagnostics) + dst = _map_memory_kind(dst_k, strict=strict, diagnostics=diagnostics) elif copy_k is not None: try: iv = int(copy_k) except (TypeError, ValueError): if strict: raise ImportError(f"unknown copyKind {copy_k!r}") from None + _diag_once(diagnostics, f"unknown copyKind {copy_k!r}; assumed device↔device") src, dst = "device", "device" else: if iv in _COPY_KIND_TO_ENDPOINTS: @@ -391,8 +409,10 @@ def _iter_memcpys( elif strict: raise ImportError(f"unknown CUPTI copyKind enum {iv}") else: + _diag_once(diagnostics, f"unknown CUPTI copyKind enum {iv}; assumed device↔device") src, dst = "device", "device" else: + _diag_once(diagnostics, "memcpy endpoint metadata missing; assumed device↔device") src, dst = "device", "device" corr = _pick(row, "correlationId", "correlation") dev = as_int(_pick(row, "deviceId", "device"), 0) @@ -416,6 +436,7 @@ def _iter_syncs( *, strict: bool, device_id: int | None = None, + diagnostics: list[str] | None = None, ) -> Iterator[SyncEvent]: from gitm.importers._common import make_sync_fast @@ -441,7 +462,11 @@ def _iter_syncs( stream_id=as_int(_pick(row, "streamId", "stream"), 0), device_id=dev, correlation_id=as_int(corr) if corr is not None else None, - sync_kind=_map_sync_type(_pick(row, "syncType", "typeOfSync", "type"), strict=strict), + sync_kind=_map_sync_type( + _pick(row, "syncType", "typeOfSync", "type"), + strict=strict, + diagnostics=diagnostics, + ), strict=strict, ) @@ -553,7 +578,9 @@ def import_nsys( rid = run_id or f"import-{uuid.uuid4().hex}" dcount = meta_count or (max(all_counts.keys()) + 1) traces: list[Trace] = [] + normalization_stats: list[ImportStats] = [] total_events = 0 + import_diagnostics: list[str] = [] # Pass 2: one device at a time — peak RAM ≈ max(per-device), not sum. for dev in device_ids: dev_events: list[TraceEvent] = [] @@ -562,8 +589,14 @@ def import_nsys( conn, kernel_table, strings, device_id=dev, strict=strict ) ) - dev_events.extend(_iter_memcpys(conn, strict=strict, device_id=dev)) - dev_events.extend(_iter_syncs(conn, strict=strict, device_id=dev)) + dev_events.extend( + _iter_memcpys( + conn, strict=strict, device_id=dev, diagnostics=import_diagnostics + ) + ) + dev_events.extend( + _iter_syncs(conn, strict=strict, device_id=dev, diagnostics=import_diagnostics) + ) if not dev_events: continue total_events += len(dev_events) @@ -578,6 +611,7 @@ def import_nsys( strict=strict, ) traces.append(trace) + normalization_stats.append(_st) del dev_events if not traces: @@ -592,6 +626,8 @@ def import_nsys( per_device_kernel_counts=all_counts, total_raw_events=total_events, ) + merge_normalization_stats(stats, normalization_stats) + stats.warnings.extend(import_diagnostics) if len(device_ids) > 1: stats.warnings.append( f"multi-GPU input: analyzing devices {device_ids}; " diff --git a/gitm/importers/torch_trace.py b/gitm/importers/torch_trace.py index 9824c05..de72f1e 100644 --- a/gitm/importers/torch_trace.py +++ b/gitm/importers/torch_trace.py @@ -9,6 +9,7 @@ import gzip import json +import math import re from collections.abc import Iterator from pathlib import Path @@ -22,6 +23,7 @@ file_mtime_ns, filter_device, finish_trace, + merge_normalization_stats, per_device_kernel_counts, ) from gitm.tracer.schema import MemcpyEvent, Trace, TraceEvent @@ -163,6 +165,32 @@ def _classify(cat: str | None, name: str, args: dict[str, Any] | None) -> str | return None +def _launch_metadata_fallbacks(obj: dict[str, Any]) -> tuple[str, ...]: + """Neutral launch placeholders used for a parsed GPU event, for surfacing.""" + args = obj.get("args") if isinstance(obj.get("args"), dict) else {} + cat = obj.get("cat") + name = str(obj.get("name") or "") + kind = _classify(str(cat) if cat is not None else None, name, args) + if kind != "kernel" and not (kind is None and str(cat).lower() in {"cuda", "gpu"}): + return () + missing: list[str] = [] + if not name: + missing.append("kernel name") + if _arg_get(args, _GRID_KEYS) is None and _arg_get(args, ("gridX", "grid_x")) is None: + missing.append("grid dimensions") + if _arg_get(args, _BLOCK_KEYS) is None and _arg_get(args, ("blockX", "block_x")) is None: + missing.append("block dimensions") + return tuple(missing) + + +def _append_launch_metadata_warnings(stats: ImportStats, counts: dict[str, int]) -> None: + for field, count in sorted(counts.items()): + stats.warnings.append( + f"launch metadata coverage: {count} parsed kernel event(s) omitted {field}; " + "the importer used the neutral value 1" + ) + + def _memcpy_event( *, @@ -215,6 +243,17 @@ def event_from_chrome( ts = float(obj.get("ts", 0.0)) dur = float(obj.get("dur", 0.0)) except (TypeError, ValueError): + if strict: + raise ImportError( + f"invalid GPU event timestamp/duration: ts={obj.get('ts')!r}, " + f"dur={obj.get('dur')!r}" + ) from None + return None + if not math.isfinite(ts) or not math.isfinite(dur): + if strict: + raise ImportError( + f"non-finite GPU event timestamp/duration: ts={ts!r}, dur={dur!r}" + ) return None start_ns = int(ts * 1000.0) end_ns = int((ts + dur) * 1000.0) @@ -635,10 +674,16 @@ def _import_torch_from_event_dicts( ) -> tuple[list[Trace], ImportStats]: """Shared finish path once raw chrome event dicts are in hand.""" events: list[TraceEvent] = [] + metadata_fallbacks: dict[str, int] = {} + parse_dropped = 0 for obj in raw_events: ev = event_from_chrome(obj, strict=strict) if ev is not None: events.append(ev) + for field in _launch_metadata_fallbacks(obj): + metadata_fallbacks[field] = metadata_fallbacks.get(field, 0) + 1 + elif _device_id_from_chrome_obj(obj) is not None: + parse_dropped += 1 if not events: raise ImportError( f"{path.name}: no complete GPU kernel/memcpy events found in traceEvents" @@ -669,6 +714,7 @@ def _import_torch_from_event_dicts( rid = run_id or f"import-{uuid.uuid4().hex}" dcount = device_count_from_events(events) traces: list[Trace] = [] + normalization_stats: list[ImportStats] = [] for dev in device_ids: dev_events = filter_device(events, dev) if not dev_events: @@ -684,6 +730,7 @@ def _import_torch_from_event_dicts( strict=strict, ) traces.append(trace) + normalization_stats.append(_st) if not traces: raise ImportError(f"{path.name}: no events left after device filter") stats = ImportStats( @@ -693,8 +740,16 @@ def _import_torch_from_event_dicts( device_name=device_name, captured_at_source="mtime", per_device_kernel_counts=all_counts, - total_raw_events=len(events), + total_raw_events=len(raw_events), ) + merge_normalization_stats(stats, normalization_stats) + if parse_dropped: + stats.dropped_invalid += parse_dropped + stats.warnings.append( + f"dropped {parse_dropped} GPU event(s) with invalid timestamp/duration fields " + "during parsing" + ) + _append_launch_metadata_warnings(stats, metadata_fallbacks) if len(device_ids) > 1: stats.warnings.append( f"multi-GPU input: analyzing devices {device_ids}; " @@ -778,8 +833,10 @@ def import_torch_trace( buckets: dict[int, list[TraceEvent]] = defaultdict(list) all_counts: dict[int, int] = defaultdict(int) + metadata_fallbacks: dict[str, int] = {} scanned_sku: str | None = None n_raw = 0 + parse_dropped = 0 try: for obj in _iter_chrome_event_dicts( path, gzipped=bool(gzipped), max_decompressed_bytes=max_decompressed_bytes @@ -799,7 +856,11 @@ def import_torch_trace( continue ev = event_from_chrome(obj, strict=strict) if ev is None: + if _device_id_from_chrome_obj(obj) is not None: + parse_dropped += 1 continue + for field in _launch_metadata_fallbacks(obj): + metadata_fallbacks[field] = metadata_fallbacks.get(field, 0) + 1 buckets[ev.device_id].append(ev) if getattr(ev, "kind", None) == "kernel": all_counts[ev.device_id] += 1 @@ -828,6 +889,7 @@ def import_torch_trace( rid = run_id or f"import-{uuid.uuid4().hex}" dcount = max(device_ids) + 1 traces: list[Trace] = [] + normalization_stats: list[ImportStats] = [] total_events = 0 for dev in device_ids: dev_events = buckets.pop(dev) @@ -843,6 +905,7 @@ def import_torch_trace( strict=strict, ) traces.append(trace) + normalization_stats.append(_st) if not traces: raise ImportError(f"{path.name}: no events left after device filter") @@ -854,8 +917,16 @@ def import_torch_trace( device_name=scanned_sku, captured_at_source="mtime", per_device_kernel_counts=dict(all_counts), - total_raw_events=total_events, + total_raw_events=n_raw, ) + merge_normalization_stats(stats, normalization_stats) + if parse_dropped: + stats.dropped_invalid += parse_dropped + stats.warnings.append( + f"dropped {parse_dropped} GPU event(s) with invalid timestamp/duration fields " + "during parsing" + ) + _append_launch_metadata_warnings(stats, metadata_fallbacks) if len(device_ids) > 1: stats.warnings.append( f"multi-GPU input: analyzing devices {device_ids}; " diff --git a/gitm/kernels/library.py b/gitm/kernels/library.py index d0fae97..845b356 100644 --- a/gitm/kernels/library.py +++ b/gitm/kernels/library.py @@ -17,10 +17,20 @@ def load_library(path: Path | str | None = None, *, workload: str | None = None) """Load and validate every entry in the library.""" p = Path(path) if path is not None else _library_path() if not p.exists(): - return [] + raise FileNotFoundError( + f"intervention library not found at {p}; candidate coverage is unavailable" + ) with p.open() as fh: - raw = yaml.safe_load(fh) or {} - entries = raw.get("interventions", []) + raw = yaml.safe_load(fh) + if not isinstance(raw, dict) or not isinstance(raw.get("interventions"), list): + raise ValueError( + f"intervention library {p} has no interventions list; candidate coverage is unavailable" + ) + entries = raw["interventions"] + if not entries: + raise ValueError( + f"intervention library {p} is empty; candidate coverage is unavailable" + ) specs = [InterventionSpec.model_validate(e) for e in entries] if workload is not None: specs = [s for s in specs if workload in s.applicability.workloads] diff --git a/gitm/optimizer/apply.py b/gitm/optimizer/apply.py index 9c9704c..5c03b95 100644 --- a/gitm/optimizer/apply.py +++ b/gitm/optimizer/apply.py @@ -21,6 +21,8 @@ import copy import gc +import math +import warnings from collections.abc import Callable from dataclasses import dataclass from pathlib import Path @@ -30,6 +32,7 @@ from gitm.kernels.spec import InterventionSpec from gitm.optimizer.vllm_knobs import get_knob, knob_kind, set_knob +from gitm.safety.failopen import FailOpenGuard if TYPE_CHECKING: from gitm.safety.audit import AuditLog @@ -74,39 +77,82 @@ def apply_intervention( blocks the apply). Pass one only where the applicator mutates a real target; a dry-run leaves it ``None`` so the trail stays free of no-op entries. """ - snapshot = applicator.snapshot() - - # Step 2: apply. A bad value (validation error) rolls straight back. - try: - applicator.apply(spec) - except Exception as exc: - applicator.restore(snapshot) - _audit(audit, "revert", spec, cause=f"apply failed, restored: {exc}", - knobs=_knob_values(spec)) - return ApplyResult(False, rolled_back=True, measured_delta=None, - error=f"apply failed, restored: {exc}") - _audit(audit, "apply", spec, cause="applied live", knobs=_knob_values(spec)) - - # Step 3: measure. A crash mid-measurement also rolls back. + if not math.isfinite(min_keep_delta): + raise ValueError(f"min_keep_delta must be finite, got {min_keep_delta!r}") try: - delta = applicator.measure(spec) + snapshot = applicator.snapshot() except Exception as exc: - applicator.restore(snapshot) - _audit(audit, "revert", spec, cause=f"measure failed, restored: {exc}", - knobs=_knob_values(spec)) - return ApplyResult(False, rolled_back=True, measured_delta=None, - error=f"measure failed, restored: {exc}") - - # Step 4: keep-or-rollback on the regression threshold. - if delta is not None and delta < min_keep_delta: - applicator.restore(snapshot) - _audit(audit, "revert", spec, knobs=_knob_values(spec), - cause=f"regression {delta:+.3f} < keep threshold {min_keep_delta:+.3f}") - return ApplyResult(True, rolled_back=True, measured_delta=delta, - error=f"regression {delta:+.3f} < keep threshold " - f"{min_keep_delta:+.3f}, restored") + return ApplyResult( + False, + rolled_back=False, + measured_delta=None, + error=f"baseline snapshot failed; intervention not applied: {exc}", + ) - return ApplyResult(True, rolled_back=False, measured_delta=delta) + def _apply_measure_decide() -> ApplyResult: + # Step 2: apply. A bad value (validation error) rolls straight back. + try: + applicator.apply(spec) + except Exception as exc: + applicator.restore(snapshot) + _audit(audit, "revert", spec, cause=f"apply failed, restored: {exc}", + knobs=_knob_values(spec)) + return ApplyResult(False, rolled_back=True, measured_delta=None, + error=f"apply failed, restored: {exc}") + _audit(audit, "apply", spec, cause="applied live", knobs=_knob_values(spec)) + + # Step 3: measure. A recoverable crash rolls back immediately. The + # surrounding fail-open guard also restores on BaseException/process + # exit paths that an Exception handler must not swallow. + try: + delta = applicator.measure(spec) + except Exception as exc: + applicator.restore(snapshot) + _audit(audit, "revert", spec, cause=f"measure failed, restored: {exc}", + knobs=_knob_values(spec)) + return ApplyResult(False, rolled_back=True, measured_delta=None, + error=f"measure failed, restored: {exc}") + + if delta is not None and ( + isinstance(delta, bool) + or not isinstance(delta, int | float) + or not math.isfinite(float(delta)) + ): + applicator.restore(snapshot) + error = f"measurement delta must be finite, got {delta!r}; intervention restored" + _audit(audit, "revert", spec, cause=error, knobs=_knob_values(spec)) + return ApplyResult(False, rolled_back=True, measured_delta=None, error=error) + + # Step 4: keep-or-rollback on the regression threshold. + if delta is None: + warnings.warn( + f"intervention {spec.name!r} was applied without a measurement; " + "the change is unverified", + RuntimeWarning, + stacklevel=2, + ) + if delta is not None and delta < min_keep_delta: + applicator.restore(snapshot) + _audit(audit, "revert", spec, knobs=_knob_values(spec), + cause=f"regression {delta:+.3f} < keep threshold {min_keep_delta:+.3f}") + return ApplyResult(True, rolled_back=True, measured_delta=delta, + error=f"regression {delta:+.3f} < keep threshold " + f"{min_keep_delta:+.3f}, restored") + + return ApplyResult(True, rolled_back=False, measured_delta=delta) + + with FailOpenGuard(audit=audit) as guard: + guard.register( + spec.name, + lambda: applicator.restore(snapshot), + cause="apply gate exited before a keep/rollback decision", + ) + result = _apply_measure_decide() + # Every ordinary return above either kept the change deliberately or + # restored it synchronously. Exceptional exits retain the registration, + # so the context restores before propagating the interruption. + guard.disarm(spec.name) + return result def _audit( @@ -117,8 +163,12 @@ def _audit( return try: audit.record(event, spec.name, cause, **detail) - except Exception: - pass + except Exception as exc: + warnings.warn( + f"intervention safety audit failed for {event} {spec.name!r}: {exc}", + RuntimeWarning, + stacklevel=2, + ) def _knob_values(spec: Any) -> dict[str, Any]: @@ -307,7 +357,9 @@ def __init__( self._restart_mode = restart_mode self._getter = getter or get_knob self._setter = setter or set_knob - self._reps = max(1, reps) + if isinstance(reps, bool) or not isinstance(reps, int) or reps <= 0: + raise ValueError(f"A/B repetition count must be a positive integer, got {reps!r}") + self._reps = reps # force_restart is kept for custom deployments that still classify a # knob as scheduling but want to measure it through the restart path. self._force_restart = force_restart @@ -327,7 +379,13 @@ def _bench_stats(self) -> tuple[float, float]: stdev is 0.0 for a single rep → the noise band is 0 and keep falls back to ``delta > 0``, i.e. reps=1 behaves exactly as before reps were added. """ - samples = [self._tps(self.engine) for _ in range(self._reps)] + samples = [float(self._tps(self.engine)) for _ in range(self._reps)] + invalid = [sample for sample in samples if not math.isfinite(sample) or sample <= 0.0] + if invalid: + raise ValueError( + "decode throughput samples must be finite and positive; " + f"observed {invalid!r}" + ) mean = sum(samples) / len(samples) if len(samples) < 2: return mean, 0.0 @@ -430,8 +488,10 @@ def _activate(engine: Any) -> None: if callable(fn): try: fn(engine) - except Exception: - pass + except Exception as exc: + warnings.warn( + f"engine activation hook failed: {exc}", RuntimeWarning, stacklevel=2 + ) @staticmethod def _shutdown(engine: Any) -> None: @@ -440,8 +500,10 @@ def _shutdown(engine: Any) -> None: if callable(custom): try: custom(engine) - except Exception: - pass + except Exception as exc: + warnings.warn( + f"engine shutdown hook failed: {exc}", RuntimeWarning, stacklevel=2 + ) for path in ("shutdown", "llm_engine.shutdown", "engine.shutdown"): obj: Any = engine @@ -452,8 +514,12 @@ def _shutdown(engine: Any) -> None: if callable(obj): try: obj() - except Exception: - pass + except Exception as exc: + warnings.warn( + f"engine shutdown path {path!r} failed: {exc}", + RuntimeWarning, + stacklevel=2, + ) break try: gc.collect() @@ -462,8 +528,12 @@ def _shutdown(engine: Any) -> None: if torch.cuda.is_available(): torch.cuda.empty_cache() torch.cuda.ipc_collect() - except Exception: - pass + except Exception as exc: + warnings.warn( + f"GPU cleanup after engine shutdown unavailable: {exc}", + RuntimeWarning, + stacklevel=2, + ) def measure(self, spec: InterventionSpec) -> float | None: baseline = self._baseline_tps if self._baseline_tps is not None else self._bench_stats()[0] diff --git a/gitm/optimizer/attribution.py b/gitm/optimizer/attribution.py index db41cf3..d50f110 100644 --- a/gitm/optimizer/attribution.py +++ b/gitm/optimizer/attribution.py @@ -10,7 +10,7 @@ from __future__ import annotations import warnings -from dataclasses import dataclass +from dataclasses import dataclass, field import numpy as np @@ -30,6 +30,7 @@ class Hypothesis: @dataclass class RankedHypotheses: hypotheses: list[Hypothesis] + diagnostics: list[str] = field(default_factory=list) def top(self, n: int = 5) -> list[Hypothesis]: return self.hypotheses[:n] @@ -54,8 +55,14 @@ def attribute( from statsmodels.tsa.stattools import ( grangercausalitytests, # type: ignore[import-not-found] ) - except Exception: - return RankedHypotheses(hypotheses=[]) + except Exception as exc: + return RankedHypotheses( + hypotheses=[], + diagnostics=[ + f"Granger attribution unavailable: statsmodels import failed " + f"({type(exc).__name__}: {exc})" + ], + ) # Group residuals by op into ordered time series (per layer-position step) series: dict[str, list[float]] = {} @@ -64,10 +71,17 @@ def attribute( ops = [op for op, vals in series.items() if len(vals) >= max_lag + 2] if len(ops) < 2: - return RankedHypotheses(hypotheses=[]) + return RankedHypotheses( + hypotheses=[], + diagnostics=[ + f"Granger attribution not run: need at least 2 op series with " + f"{max_lag + 2}+ samples; found {len(ops)}" + ], + ) n = min(len(series[op]) for op in ops) hypotheses: list[Hypothesis] = [] + fit_failures: list[str] = [] for cause in ops: for effect in ops: if cause == effect: @@ -79,7 +93,8 @@ def attribute( result = grangercausalitytests(arr, maxlag=max_lag, verbose=False) pvals = [result[lag][0]["ssr_ftest"][1] for lag in range(1, max_lag + 1)] p = float(min(pvals)) - except Exception: + except Exception as exc: + fit_failures.append(f"{cause}->{effect}: {type(exc).__name__}: {exc}") continue direction = "+ slower" if np.mean(series[cause]) > 0 else "- faster" hypotheses.append( @@ -87,4 +102,10 @@ def attribute( ) hypotheses.sort(key=lambda h: h.p_value) - return RankedHypotheses(hypotheses=hypotheses) + diagnostics = [] + if fit_failures: + diagnostics.append( + f"Granger attribution skipped {len(fit_failures)} failed pair fit(s); " + f"first: {fit_failures[0]}" + ) + return RankedHypotheses(hypotheses=hypotheses, diagnostics=diagnostics) diff --git a/gitm/optimizer/deviation.py b/gitm/optimizer/deviation.py index 8bd656b..7ff1ae0 100644 --- a/gitm/optimizer/deviation.py +++ b/gitm/optimizer/deviation.py @@ -19,6 +19,7 @@ from __future__ import annotations +import math from dataclasses import dataclass from pathlib import Path @@ -122,8 +123,12 @@ def _departs( inv_mt: Invariant | None, ) -> bool: """True if observed kernel ``ok`` is out-of-band vs its predicted node.""" - t_obs = max((ok.end_ns - ok.start_ns) / 1e9, 1e-12) - t_pred = max(node_pred_s, 1e-12) + t_obs = (ok.end_ns - ok.start_ns) / 1e9 + if not math.isfinite(t_obs) or t_obs <= 0.0: + return True + if not math.isfinite(node_pred_s) or node_pred_s <= 0.0: + return True + t_pred = node_pred_s r_kt = (t_obs - t_pred) / t_pred if inv_kt is not None and abs(r_kt) > inv_kt.band_width: return True diff --git a/gitm/optimizer/dr.py b/gitm/optimizer/dr.py index 0311403..7bd0b6e 100644 --- a/gitm/optimizer/dr.py +++ b/gitm/optimizer/dr.py @@ -52,7 +52,13 @@ class DREffect: n_treated: int -def doubly_robust_ate(y: np.ndarray, t: np.ndarray, X: np.ndarray) -> tuple[float, float]: +def doubly_robust_ate( + y: np.ndarray, + t: np.ndarray, + X: np.ndarray, + *, + diagnostics: list[str] | None = None, +) -> tuple[float, float]: """AIPW estimate of the ATE of ``t`` on ``y`` given covariates ``X``. Returns ``(ate, se)``. Robust to a misspecified outcome *or* propensity model. @@ -81,7 +87,11 @@ def doubly_robust_ate(y: np.ndarray, t: np.ndarray, X: np.ndarray) -> tuple[floa # Propensity e = P(T=1|X); clip away from 0/1 to bound the IPW weights. try: e = sm.Logit(t, Xc).fit(disp=0).predict(Xc) - except Exception: + except Exception as exc: + if diagnostics is not None: + diagnostics.append( + f"propensity model fell back to treated mean ({type(exc).__name__})" + ) e = np.full(n, t.mean()) e = np.clip(e, 0.05, 0.95) @@ -91,7 +101,11 @@ def _outcome(mask: np.ndarray) -> np.ndarray: return np.full(n, float(y[mask].mean()) if mask.any() else 0.0) try: return sm.OLS(y[mask], Xc[mask]).fit().predict(Xc) - except Exception: + except Exception as exc: + if diagnostics is not None: + diagnostics.append( + f"outcome model fell back to group mean ({type(exc).__name__})" + ) return np.full(n, float(y[mask].mean())) m1 = _outcome(t == 1) @@ -118,11 +132,18 @@ def attribute_dr(residuals: Residuals, graph: Graph, *, band: float = _KT_BAND) ops = [op for op, v in series.items() if len(v) >= 4] if len(ops) < 2: - return RankedHypotheses(hypotheses=[]) + return RankedHypotheses( + hypotheses=[], + diagnostics=[ + f"doubly-robust attribution not run: need at least 2 op series with " + f"4+ samples; found {len(ops)}" + ], + ) n = min(len(series[op]) for op in ops) pos = np.arange(n, dtype=float) effects: list[DREffect] = [] + diagnostics: list[str] = [] for cause in ops: t = (np.abs(np.asarray(series[cause][:n])) > band).astype(float) n_t = int(t.sum()) @@ -132,7 +153,11 @@ def attribute_dr(residuals: Residuals, graph: Graph, *, band: float = _KT_BAND) if effect == cause: continue y = np.asarray(series[effect][:n], dtype=float) - ate, se = doubly_robust_ate(y, t, pos) + pair_diagnostics: list[str] = [] + ate, se = doubly_robust_ate(y, t, pos, diagnostics=pair_diagnostics) + diagnostics.extend( + f"{cause}->{effect}: {note}" for note in pair_diagnostics + ) z = ate / se if se not in (0.0, float("inf")) else 0.0 effects.append(DREffect(cause, effect, ate, se, z, int(t.sum()))) @@ -151,4 +176,9 @@ def attribute_dr(residuals: Residuals, graph: Graph, *, band: float = _KT_BAND) ) for d in effects ] - return RankedHypotheses(hypotheses=hyps) + if diagnostics: + diagnostics = [ + f"doubly-robust attribution used {len(diagnostics)} nuisance-model fallback(s); " + f"first: {diagnostics[0]}" + ] + return RankedHypotheses(hypotheses=hyps, diagnostics=diagnostics) diff --git a/gitm/optimizer/headroom.py b/gitm/optimizer/headroom.py index fb3d4ce..6e934ac 100644 --- a/gitm/optimizer/headroom.py +++ b/gitm/optimizer/headroom.py @@ -16,6 +16,7 @@ from __future__ import annotations +import math from dataclasses import asdict, dataclass, field from typing import Literal @@ -82,10 +83,21 @@ def build_headroom( ) -> HeadroomReport: """Assemble the headroom report from the trace, planner floor, and metrics.""" observed_s = trace.duration_ns / 1e9 - if observed_s > 0 and predicted_floor_s > 0: - ceiling_distance = max(0.0, (observed_s - predicted_floor_s) / observed_s) - else: - ceiling_distance = 0.0 + if not math.isfinite(observed_s) or observed_s <= 0.0: + raise ValueError(f"observed trace duration must be finite and positive, got {observed_s!r}") + if not math.isfinite(predicted_floor_s) or predicted_floor_s <= 0.0: + raise ValueError( + f"predicted floor must be finite and positive, got {predicted_floor_s!r}" + ) + if predicted_floor_s > observed_s: + raise ValueError( + f"predicted floor {predicted_floor_s!r}s exceeds observed wall {observed_s!r}s" + ) + if not math.isfinite(optimized_threshold) or not 0.0 <= optimized_threshold <= 1.0: + raise ValueError( + f"optimized threshold must be finite and in [0, 1], got {optimized_threshold!r}" + ) + ceiling_distance = (observed_s - predicted_floor_s) / observed_s already_optimized = ceiling_distance < optimized_threshold # Split the recoverable distance across stall classes. Idle is the GPU-idle @@ -150,5 +162,16 @@ def render_headroom_md(r: HeadroomReport) -> str: if r.already_optimized: lines.append("") lines.append("> Flagged already-optimized — no headroom to bill.") + if r.indicative_mem_compute_split or r.caveats: + lines.extend(["", "Caveats:"]) + for caveat in r.caveats: + lines.append(f" - {caveat}") + if r.indicative_mem_compute_split and not any( + "indicative" in caveat.lower() for caveat in r.caveats + ): + lines.append( + " - Memory/compute split is indicative because HFU is unavailable; " + "it uses available MBU signal and a 50/50 split when both signals are weak." + ) return "\n".join(lines) + "\n" diff --git a/gitm/optimizer/headroom_kernel_rank.py b/gitm/optimizer/headroom_kernel_rank.py index aff48da..c82d993 100644 --- a/gitm/optimizer/headroom_kernel_rank.py +++ b/gitm/optimizer/headroom_kernel_rank.py @@ -18,8 +18,10 @@ from __future__ import annotations +import math import re -from dataclasses import dataclass +import warnings +from dataclasses import dataclass, field # Mangled CUDA kernel names -> a small set of stable families, so durations # aggregate per kernel *type* rather than per template instantiation. Shared @@ -71,12 +73,16 @@ def kernel_roi(name_durations, floor_pct: float = 10.0) -> list[KernelROI]: ``name_durations`` is an iterable of ``(kernel_name, duration_ns)``. """ + if not math.isfinite(floor_pct) or not 0.0 <= floor_pct <= 100.0: + raise ValueError(f"kernel ROI floor percentile must be in [0, 100], got {floor_pct!r}") by_fam: dict[str, list[int]] = {} for name, dur in name_durations: if dur is None or dur <= 0: continue by_fam.setdefault(kernel_family(name), []).append(int(dur)) - total = sum(sum(v) for v in by_fam.values()) or 1 + total = sum(sum(v) for v in by_fam.values()) + if total <= 0: + return [] rows: list[KernelROI] = [] for fam, ds in by_fam.items(): @@ -95,7 +101,9 @@ def kernel_roi(name_durations, floor_pct: float = 10.0) -> list[KernelROI]: def render_roi_table(rows: list[KernelROI], *, floor_pct: float = 10.0, top: int = 20) -> str: """Human-readable ROI table (used in stdout and the provenance report).""" - total = sum(r.total_ns for r in rows) or 1 + if not rows: + return "kernel ROI unavailable: no positive-duration kernels" + total = sum(r.total_ns for r in rows) us, ms = 1000.0, 1_000_000.0 out = [f"kernel ROI (floor p{floor_pct:.0f}, total kernel time {total / ms:.1f} ms):", ""] hdr = (f"{'#':>2}  {'family':<22} {'calls':>7} {'total_ms':>9} {'%rt':>6} " @@ -116,57 +124,109 @@ def render_roi_table(rows: list[KernelROI], *, floor_pct: float = 10.0, top: int @dataclass class GpuHeadroom: - mean_util_pct: float - peak_util_pct: float - compute_headroom_pct: float # 100 - mean util (coarse, time-based) - peak_mem_used_bytes: int - mem_total_bytes: int - mem_free_at_peak_bytes: int - serialized_concurrency_fraction: float # fraction of kernel-time with no overlap + mean_util_pct: float | None + peak_util_pct: float | None + compute_headroom_pct: float | None # 100 - mean util (coarse, time-based) + peak_mem_used_bytes: int | None + mem_total_bytes: int | None + mem_free_at_peak_bytes: int | None + serialized_concurrency_fraction: float | None # fraction of kernel-time with no overlap n_samples: int + diagnostics: list[str] = field(default_factory=list) -def gpu_headroom(samples, serialized_concurrency_fraction: float = 0.0) -> GpuHeadroom | None: +def gpu_headroom(samples, serialized_concurrency_fraction: float | None = None) -> GpuHeadroom: """Summarise GPU headroom from a list of telemetry sample dicts. Each sample is a dict with ``util_pct`` / ``mem_used_bytes`` / ``mem_total_bytes`` (the canonical :class:`gitm.telemetry.Sample` fields). - Returns ``None`` if there are no usable samples. + Missing metric families remain ``None`` and are named in ``diagnostics``; + memory-only telemetry must not become 100% compute headroom, or vice versa. """ + if serialized_concurrency_fraction is not None and ( + not math.isfinite(serialized_concurrency_fraction) + or not 0.0 <= serialized_concurrency_fraction <= 1.0 + ): + raise ValueError( + "serialized concurrency fraction must be finite and in [0, 1], got " + f"{serialized_concurrency_fraction!r}" + ) utils = [float(s["util_pct"]) for s in samples if s.get("util_pct") is not None] mems = [int(s["mem_used_bytes"]) for s in samples if s.get("mem_used_bytes") is not None] - total = next((int(s["mem_total_bytes"]) for s in samples if s.get("mem_total_bytes")), 0) - if not utils and not mems: - return None - mean_u = sum(utils) / len(utils) if utils else 0.0 - peak_u = max(utils) if utils else 0.0 - peak_m = max(mems) if mems else 0 + totals = [int(s["mem_total_bytes"]) for s in samples if s.get("mem_total_bytes")] + diagnostics: list[str] = [] + if not samples: + diagnostics.append("GPU headroom unavailable: telemetry contains no samples") + if not utils: + diagnostics.append("compute headroom unavailable: utilization telemetry is absent") + if not mems: + diagnostics.append("memory headroom unavailable: used-memory telemetry is absent") + if not totals: + diagnostics.append("memory headroom unavailable: total-memory telemetry is absent") + if serialized_concurrency_fraction is None: + diagnostics.append("concurrency headroom unavailable: no trace overlap measurement supplied") + mean_u = sum(utils) / len(utils) if utils else None + peak_u = max(utils) if utils else None + peak_m = max(mems) if mems else None + total = max(totals) if totals else None + mem_free = ( + max(0, total - peak_m) if total is not None and peak_m is not None else None + ) return GpuHeadroom( mean_util_pct=mean_u, peak_util_pct=peak_u, - compute_headroom_pct=max(0.0, 100.0 - mean_u), + compute_headroom_pct=max(0.0, 100.0 - mean_u) if mean_u is not None else None, peak_mem_used_bytes=peak_m, mem_total_bytes=total, - mem_free_at_peak_bytes=max(0, total - peak_m), + mem_free_at_peak_bytes=mem_free, serialized_concurrency_fraction=serialized_concurrency_fraction, n_samples=len(samples), + diagnostics=diagnostics, ) def live_gpu_headroom(): """One-shot live snapshot via the telemetry backend (no running workload required).""" from gitm.telemetry.backends import discover_backends - backends = discover_backends() + discovery_diagnostics: list[str] = [] + backends = discover_backends(diagnostics=discovery_diagnostics) + for diagnostic in discovery_diagnostics: + warnings.warn( + f"live GPU headroom degraded: {diagnostic}", RuntimeWarning, stacklevel=2 + ) + if not backends: + warnings.warn( + "live GPU headroom unavailable: no telemetry backend found", + RuntimeWarning, + stacklevel=2, + ) out = [] for b in backends: - for idx in range(b.device_count()): - s = b.sample(idx) - out.append({ - "gpu_index": idx, - "util_pct": s.util_pct, - "mem_used_bytes": s.mem_used_bytes, - "mem_total_bytes": s.mem_total_bytes, - "power_w": s.power_w, - "sm_clock_mhz": s.sm_clock_mhz, - "throttle": s.throttle_reasons.name if s.throttle_reasons else "NONE", - }) + try: + for idx in range(b.device_count()): + s = b.sample(idx) + for diagnostic in s.diagnostics: + warnings.warn( + f"live GPU headroom degraded: {diagnostic}", + RuntimeWarning, + stacklevel=2, + ) + out.append({ + "gpu_index": idx, + "util_pct": s.util_pct, + "mem_used_bytes": s.mem_used_bytes, + "mem_total_bytes": s.mem_total_bytes, + "power_w": s.power_w, + "sm_clock_mhz": s.sm_clock_mhz, + "throttle": s.throttle_reasons.name if s.throttle_reasons else "NONE", + }) + finally: + try: + b.close() + except Exception as exc: + warnings.warn( + f"live GPU headroom degraded: backend close failed " + f"({type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) return out diff --git a/gitm/optimizer/measure.py b/gitm/optimizer/measure.py index 574a31b..3aad921 100644 --- a/gitm/optimizer/measure.py +++ b/gitm/optimizer/measure.py @@ -55,10 +55,12 @@ def kernel_family(name: str) -> str: class MeasureResult: n_kernels: int n_memcpy: int - serialized_fraction: float + serialized_fraction: float | None violations: list = field(default_factory=list) top_hypotheses: list = field(default_factory=list) families: list[str] = field(default_factory=list) + n_invalid_duration: int = 0 + diagnostics: list[str] = field(default_factory=list) def measure_trace(trace: Trace, *, min_attr: int = 16) -> MeasureResult: @@ -71,42 +73,73 @@ def measure_trace(trace: Trace, *, min_attr: int = 16) -> MeasureResult: kernels = trace.kernels() memcpys = [e for e in trace.events if e.kind == "memcpy"] if not kernels: - return MeasureResult(0, len(memcpys), 0.0) + return MeasureResult( + 0, + len(memcpys), + None, + diagnostics=["measurement coverage unavailable: trace contains no kernels"], + ) + + valid_kernels = [k for k in kernels if k.end_ns - k.start_ns > 0] + n_invalid = len(kernels) - len(valid_kernels) + diagnostics: list[str] = [] + if n_invalid: + diagnostics.append( + f"measurement coverage: excluded {n_invalid}/{len(kernels)} kernel(s) " + "with non-positive duration" + ) + if not valid_kernels: + return MeasureResult( + len(kernels), + len(memcpys), + None, + n_invalid_duration=n_invalid, + diagnostics=diagnostics, + ) - sc = _serialized_fraction(kernels) + sc = _serialized_fraction(valid_kernels) + if sc is None: + diagnostics.append( + "stream-concurrency coverage unavailable: trace has no adjacent " + "cross-stream kernel pairs" + ) by_name: dict[str, list[int]] = {} - for k in kernels: + for k in valid_kernels: by_name.setdefault(k.name, []).append(k.end_ns - k.start_ns) med = {nm: float(np.median(v)) for nm, v in by_name.items()} res = Residuals() res.serialized_concurrency_fraction = sc - for k in kernels: - m = med[k.name] or 1.0 + for k in valid_kernels: + m = med[k.name] res.per_kernel.append( KernelResidual(op=k.name[:40], layer=None, r_kt=((k.end_ns - k.start_ns) - m) / m, r_mt=None) ) violations = check_invariants(res, multi_basis=True) fam_of = {nm: kernel_family(nm) for nm in by_name} - fam_counts = Counter(fam_of[k.name] for k in kernels) + fam_counts = Counter(fam_of[k.name] for k in valid_kernels) res_attr = Residuals() res_attr.serialized_concurrency_fraction = sc - for k in kernels: + for k in valid_kernels: fam = fam_of[k.name] if fam_counts[fam] < min_attr: continue - m = med[k.name] or 1.0 + m = med[k.name] res_attr.per_kernel.append( KernelResidual(op=fam, layer=None, r_kt=((k.end_ns - k.start_ns) - m) / m, r_mt=None) ) families = sorted({kr.op for kr in res_attr.per_kernel}) - top_hyps = list(attribute(res_attr, predict_graph()).top(5)) if res_attr.per_kernel else [] + ranked = attribute(res_attr, predict_graph()) + top_hyps = list(ranked.top(5)) if res_attr.per_kernel else [] + diagnostics.extend(ranked.diagnostics) return MeasureResult( n_kernels=len(kernels), n_memcpy=len(memcpys), serialized_fraction=sc, + n_invalid_duration=n_invalid, + diagnostics=diagnostics, violations=violations, top_hypotheses=top_hyps, families=families, @@ -143,10 +176,16 @@ def measurement_claims(result: MeasureResult, *, limit: int = 5) -> list[Claim]: def measurement_summary(workload: str, result: MeasureResult) -> str: fams = ", ".join(result.families[:6]) or "none with enough samples" + diagnostic = f" Coverage diagnostics: {'; '.join(result.diagnostics)}." if result.diagnostics else "" + serialized = ( + f"{result.serialized_fraction:.3f}" + if result.serialized_fraction is not None + else "unavailable" + ) return ( f"Measurement run for {workload!r}: {result.n_kernels:,} kernels " f"({result.n_memcpy:,} memcpy) captured, {len(result.violations)} invariant " - f"deviation(s), serialized-concurrency={result.serialized_fraction:.3f}. " + f"deviation(s), serialized-concurrency={serialized}. " f"Kernel families: {fams}. No interventions applied — this workload has no " - f"tuned intervention library, so the runtime reports what it measured." + f"tuned intervention library, so the runtime reports what it measured.{diagnostic}" ) diff --git a/gitm/optimizer/metrics.py b/gitm/optimizer/metrics.py index 8d93542..c0d2bdf 100644 --- a/gitm/optimizer/metrics.py +++ b/gitm/optimizer/metrics.py @@ -23,9 +23,11 @@ from __future__ import annotations +import warnings from collections.abc import Callable, Iterable from dataclasses import dataclass +from gitm._timing import require_positive_duration from gitm.tracer.schema import KernelEvent, Trace FlopsModel = Callable[[KernelEvent], float] @@ -169,8 +171,33 @@ def compute_metrics( kernels = trace.kernels() memcpys = [e for e in trace.events if getattr(e, "kind", None) == "memcpy"] syncs = [e for e in trace.events if getattr(e, "kind", None) == "sync"] - wall_s = trace.duration_ns / 1e9 - busy_fraction = (_merged_busy_ns(kernels) / trace.duration_ns) if trace.duration_ns else 0.0 + wall_s = require_positive_duration( + trace.duration_ns / 1e9, context=f"{trace.workload_id} utilization metrics" + ) + invalid_kernels = [ + k + for k in kernels + if k.start_ns < 0 or k.end_ns < k.start_ns or k.end_ns > trace.duration_ns + ] + if invalid_kernels: + raise RuntimeError( + f"{trace.workload_id} utilization kernel timing is outside trace window for " + f"{len(invalid_kernels)}/{len(kernels)} kernel(s); refusing to clamp busy time" + ) + zero_duration = sum(k.end_ns == k.start_ns for k in kernels) + if zero_duration: + warnings.warn( + f"{trace.workload_id} utilization ignored {zero_duration}/{len(kernels)} " + "zero-duration kernel(s)", + RuntimeWarning, + stacklevel=2, + ) + busy_fraction = _merged_busy_ns(kernels) / trace.duration_ns + if not 0.0 <= busy_fraction <= 1.0: + raise RuntimeError( + f"{trace.workload_id} utilization busy fraction is contradictory: " + f"{busy_fraction!r}" + ) gaps = _idle_gaps(kernels, trace.duration_ns) stall_breakdown = _classify_stalls(gaps, memcpys, syncs, trace.duration_ns) @@ -179,7 +206,11 @@ def compute_metrics( mfu: float | None = None if flops_model is not None: total_flops = sum(flops_model(k) for k in kernels) - achieved_flops = total_flops / wall_s if wall_s else 0.0 + achieved_flops = total_flops / wall_s + if total_flops > 0 and peak.peak_flops <= 0: + raise RuntimeError( + f"{peak.name} compute utilization unavailable: peak FLOP/s must be positive" + ) if peak.peak_flops > 0: hfu = achieved_flops / peak.peak_flops mfu = hfu * (1.0 - recompute_fraction) @@ -187,14 +218,18 @@ def compute_metrics( bytes_moved = sum(e.bytes for e in memcpys) for k in kernels: bytes_moved += (k.bytes_read or 0) + (k.bytes_written or 0) - achieved_bw = bytes_moved / wall_s if wall_s else 0.0 - mbu = achieved_bw / peak.peak_bw_bytes_s if peak.peak_bw_bytes_s > 0 else 0.0 + achieved_bw = bytes_moved / wall_s + if peak.peak_bw_bytes_s <= 0: + raise RuntimeError( + f"{peak.name} memory utilization unavailable: peak bandwidth must be positive" + ) + mbu = achieved_bw / peak.peak_bw_bytes_s return MetricsResult( n_kernels=len(kernels), wall_s=wall_s, busy_fraction=busy_fraction, - stall_fraction=max(0.0, 1.0 - busy_fraction), + stall_fraction=1.0 - busy_fraction, stall_breakdown=stall_breakdown, achieved_flops_per_s=achieved_flops, achieved_bw_bytes_s=achieved_bw, diff --git a/gitm/optimizer/monitor.py b/gitm/optimizer/monitor.py index ceab2fb..ad01b78 100644 --- a/gitm/optimizer/monitor.py +++ b/gitm/optimizer/monitor.py @@ -9,6 +9,7 @@ from __future__ import annotations +import math from dataclasses import dataclass, field import numpy as np @@ -51,7 +52,74 @@ class Residuals: """Residuals against predicted graph. Per-kernel + per-stream-set.""" per_kernel: list[KernelResidual] = field(default_factory=list) - serialized_concurrency_fraction: float = 0.0 + serialized_concurrency_fraction: float | None = None + total_kernels: int = 0 + classified_kernels: int = 0 + matched_kernels: int = 0 + total_kernel_time_ns: int = 0 + classified_kernel_time_ns: int = 0 + matched_kernel_time_ns: int = 0 + invalid_duration_kernels: int = 0 + unpriced_prediction_kernels: int = 0 + + @staticmethod + def _ratio(part: int, whole: int) -> float: + return part / whole if whole > 0 else 0.0 + + @property + def classification_coverage(self) -> float: + return self._ratio(self.classified_kernels, self.total_kernels) + + @property + def match_coverage(self) -> float: + return self._ratio(self.matched_kernels, self.total_kernels) + + @property + def classified_time_coverage(self) -> float: + return self._ratio(self.classified_kernel_time_ns, self.total_kernel_time_ns) + + @property + def matched_time_coverage(self) -> float: + return self._ratio(self.matched_kernel_time_ns, self.total_kernel_time_ns) + + @property + def coverage_warnings(self) -> list[str]: + """Human-facing notes for work excluded from residual conclusions.""" + if self.total_kernels == 0: + return ["residual coverage unavailable: trace contains no kernels"] + warnings: list[str] = [] + if self.invalid_duration_kernels: + warnings.append( + "residual coverage: excluded " + f"{self.invalid_duration_kernels}/{self.total_kernels} kernel(s) " + "with non-positive timestamps" + ) + if self.unpriced_prediction_kernels: + warnings.append( + "residual coverage: excluded " + f"{self.unpriced_prediction_kernels} matched kernel(s) because " + "their predicted duration is non-positive or non-finite" + ) + if self.serialized_concurrency_fraction is None: + warnings.append( + "stream-concurrency coverage unavailable: trace has no adjacent " + "cross-stream kernel pairs" + ) + if self.classified_kernels < self.total_kernels: + warnings.append( + "residual coverage: classified " + f"{self.classification_coverage:.1%} of launches and " + f"{self.classified_time_coverage:.1%} of kernel time " + f"({self.classified_kernels}/{self.total_kernels} kernels)" + ) + if self.matched_kernels < self.total_kernels: + warnings.append( + "residual coverage: matched to the predicted graph " + f"{self.match_coverage:.1%} of launches and " + f"{self.matched_time_coverage:.1%} of kernel time " + f"({self.matched_kernels}/{self.total_kernels} kernels)" + ) + return warnings def _class_key(pn: PredictedNode) -> tuple[float, float]: @@ -119,7 +187,12 @@ def residuals(trace: Trace, graph: Graph) -> Residuals: obs = trace.kernels() pred = graph.nodes - res = Residuals() + durations = [ok.end_ns - ok.start_ns for ok in obs] + res = Residuals( + total_kernels=len(obs), + total_kernel_time_ns=sum(d for d in durations if d > 0), + invalid_duration_kernels=sum(d <= 0 for d in durations), + ) by_op_layer: dict[tuple[str, int], PredictedNode] = {} classes: dict[str, dict[tuple[float, float], PredictedNode]] = {} for pn in pred: @@ -127,15 +200,21 @@ def residuals(trace: Trace, graph: Graph) -> Residuals: by_op_layer.setdefault((pn.op, pn.layer), pn) classes.setdefault(pn.op, {}).setdefault(_class_key(pn), pn) - for ok in obs: + for ok, duration_ns in zip(obs, durations, strict=True): + if duration_ns <= 0: + continue op = ok.range_op or classify_op(ok.name) if op is None: continue + res.classified_kernels += 1 + res.classified_kernel_time_ns += duration_ns cls = list(classes.get(op, {}).values()) if not cls: continue + res.matched_kernels += 1 + res.matched_kernel_time_ns += duration_ns - t_obs = max((ok.end_ns - ok.start_ns) / 1e9, 1e-12) + t_obs = duration_ns / 1e9 b_obs = ( ok.bytes_read + ok.bytes_written if ok.bytes_read is not None and ok.bytes_written is not None @@ -149,7 +228,10 @@ def residuals(trace: Trace, graph: Graph) -> Residuals: pn = cls[0] if pn is not None: - t_pred = max(pn.prediction.t_pred_s, 1e-12) + t_pred = pn.prediction.t_pred_s + if not math.isfinite(t_pred) or t_pred <= 0.0: + res.unpriced_prediction_kernels += 1 + continue r_kt = (t_obs - t_pred) / t_pred r_mt = ( (b_obs - pn.prediction.bytes) / pn.prediction.bytes @@ -158,7 +240,10 @@ def residuals(trace: Trace, graph: Graph) -> Residuals: ) layer, bound = ok.range_layer, pn.prediction.bound else: - ts = sorted(max(c.prediction.t_pred_s, 1e-12) for c in cls) + ts = sorted(c.prediction.t_pred_s for c in cls) + if any(not math.isfinite(t) or t <= 0.0 for t in ts): + res.unpriced_prediction_kernels += 1 + continue r_kt = _interval_residual(t_obs, ts[0], ts[-1]) bs = sorted(c.prediction.bytes for c in cls if c.prediction.bytes > 0) r_mt = _interval_residual(b_obs, bs[0], bs[-1]) if b_obs is not None and bs else None @@ -180,25 +265,28 @@ def residuals(trace: Trace, graph: Graph) -> Residuals: return res -def _serialized_fraction(obs: list[KernelEvent]) -> float: - """Fraction of adjacent kernel pairs that executed serialized. +def _serialized_fraction(obs: list[KernelEvent]) -> float | None: + """Fraction of observed cross-stream opportunities that did not overlap. - Sort observed kernels by start time; a consecutive pair is *serialized* when - the later kernel starts after the earlier one ends (no temporal overlap) - while sharing a stream — concurrency a well-tuned pipeline would have - achieved was lost. 0.0 = fully overlapped, 1.0 = fully sequential. Computed - from the real trace (stream IDs + ns timestamps), not assumed. + Sort kernels by start time and inspect adjacent pairs assigned to different + streams. Such a pair is evidence that the runtime intended independent + scheduling; it is serialized when the later kernel starts after the earlier + one ends. Same-stream ordering is mandatory and therefore says nothing about + lost concurrency. ``None`` means the trace exposed no cross-stream + opportunity, not a clean zero. """ if len(obs) < 2: - return 0.0 + return None s = sorted(obs, key=lambda k: k.start_ns) pairs = serialized = 0 for a, b in zip(s, s[1:], strict=False): + if a.stream_id == b.stream_id: + continue pairs += 1 overlapped = b.start_ns < a.end_ns - if not overlapped and a.stream_id == b.stream_id: + if not overlapped: serialized += 1 - return serialized / pairs if pairs else 0.0 + return serialized / pairs if pairs else None def check_invariants( @@ -264,6 +352,7 @@ def check_invariants( if ( inv_sc is not None + and residuals_.serialized_concurrency_fraction is not None and residuals_.serialized_concurrency_fraction > inv_sc.band_width * 0.5 ): out.append( diff --git a/gitm/optimizer/preconditions.py b/gitm/optimizer/preconditions.py index 8913bce..f80a9df 100644 --- a/gitm/optimizer/preconditions.py +++ b/gitm/optimizer/preconditions.py @@ -15,7 +15,7 @@ class GateContext: kv_cache_len: int | None = None num_gpus: int = 1 has_collective: bool = False - has_interconnect: bool = False + has_interconnect: bool | None = None def applicable(spec: InterventionSpec, ctx: GateContext) -> tuple[bool, str]: @@ -60,7 +60,8 @@ def applicable(spec: InterventionSpec, ctx: GateContext) -> tuple[bool, str]: return False, f"num_gpus {ctx.num_gpus} < min {min_gpus}" if getattr(app, "requires_collective", False) and not ctx.has_collective: return False, "requires a collective (multi-GPU) but run is single-GPU/no-collective" - if getattr(app, "requires_interconnect", False) and not ctx.has_interconnect: - return False, "requires interconnect (NVLink/IB) but none reported" + if getattr(app, "requires_interconnect", False) and ctx.has_interconnect is not True: + state = "topology is unknown" if ctx.has_interconnect is None else "none was reported" + return False, f"requires interconnect (NVLink/IB) but {state}" return True, "" diff --git a/gitm/optimizer/qualification.py b/gitm/optimizer/qualification.py index cd79de5..18c5668 100644 --- a/gitm/optimizer/qualification.py +++ b/gitm/optimizer/qualification.py @@ -8,6 +8,7 @@ from __future__ import annotations import hashlib +import math from dataclasses import dataclass from gitm.tracer.schema import Trace @@ -57,6 +58,10 @@ def qualify(trace: Trace, target_floor: float = 0.15) -> QualificationResult: Imported profiler traces (``nsys-import`` / ``torch-import``) never commit: the 15% floor and refund clause require a gitm-captured run. """ + if not math.isfinite(target_floor) or not 0.0 < target_floor <= 1.0: + raise ValueError( + f"target floor must be finite and in (0, 1], got {target_floor!r}" + ) fp = fingerprint(trace) source = getattr(trace, "source", "cupti") or "cupti" if source in _IMPORT_SOURCES: @@ -76,6 +81,18 @@ def qualify(trace: Trace, target_floor: float = 0.15) -> QualificationResult: diagnostic="No kernels in trace — capture failed or workload did not run.", ) + invalid_durations = [k for k in kernels if k.end_ns <= k.start_ns] + if invalid_durations: + return QualificationResult( + commit=False, + floor=target_floor, + fingerprint=fp, + diagnostic=( + f"Qualification evidence contains {len(invalid_durations)}/{len(kernels)} " + "kernel(s) with non-positive duration; refusing the floor commitment." + ), + ) + # Heuristic: if the top-10 kernels by time account for >95% of duration, # the workload is likely already well-shaped. Real gate uses the residual # distribution after the deviation monitor. diff --git a/gitm/optimizer/replay_validation.py b/gitm/optimizer/replay_validation.py index fbcb7d6..dbe41ca 100644 --- a/gitm/optimizer/replay_validation.py +++ b/gitm/optimizer/replay_validation.py @@ -18,6 +18,7 @@ from __future__ import annotations +import math from dataclasses import dataclass import numpy as np @@ -66,7 +67,9 @@ def _build_trace(rng: np.random.Generator, n_kernels: int, hot_fraction: float) def _ground_truth_delta(trace: Trace, spec: InterventionSpec, rng: np.random.Generator) -> float: """Per-kernel simulated wall-clock saving fraction (independent of predict_delta).""" - total = max(trace.duration_ns, 1) + total = trace.duration_ns + if total <= 0: + raise ValueError(f"synthetic trace duration must be positive, got {total!r}") saved = 0.0 for k in trace.kernels(): if _applies(spec, k.name): @@ -91,6 +94,10 @@ def passed(self) -> bool: def validate(n: int = 300, *, seed: int = 0, tolerance: float = 0.20) -> ValidationResult: + if n <= 0: + raise ValueError(f"validation injection count must be positive, got {n!r}") + if not math.isfinite(tolerance) or tolerance < 0.0: + raise ValueError(f"validation tolerance must be finite and non-negative, got {tolerance!r}") rng = np.random.default_rng(seed) spec = _synthetic_spec() errs: list[float] = [] @@ -100,7 +107,13 @@ def validate(n: int = 300, *, seed: int = 0, tolerance: float = 0.20) -> Validat trace = _build_trace(rng, n_k, hot_frac) truth = _ground_truth_delta(trace, spec, rng) pred = predict_delta(trace, spec) - errs.append(abs(pred - truth) / abs(truth) if truth else 0.0) + if not math.isfinite(truth) or truth <= 0.0: + raise RuntimeError( + f"replay validation ground truth must be finite and positive, got {truth!r}" + ) + if not math.isfinite(pred): + raise RuntimeError(f"replay validation prediction must be finite, got {pred!r}") + errs.append(abs(pred - truth) / abs(truth)) arr = np.asarray(errs) return ValidationResult( n=n, diff --git a/gitm/optimizer/report.py b/gitm/optimizer/report.py index 9eae9b9..e43393d 100644 --- a/gitm/optimizer/report.py +++ b/gitm/optimizer/report.py @@ -7,6 +7,7 @@ from __future__ import annotations +import math import subprocess import time from dataclasses import dataclass, field @@ -25,8 +26,28 @@ class Claim: intervention_name: str predicted_delta: float measured_delta: float | None + # Whether the residual is specific to this claim, aggregated over the run, + # or tied to an autoresearch target op. The report hides the default label. + residual_scope: str = "claim" rolled_back: bool = False + def __post_init__(self) -> None: + if not math.isfinite(self.residual_value): + raise ValueError("residual_value must be finite") + if not math.isfinite(self.predicted_delta): + raise ValueError("predicted_delta must be finite") + if self.measured_delta is not None and not math.isfinite(self.measured_delta): + raise ValueError("measured_delta must be finite when present") + + @property + def residual_display_value(self) -> float: + """Value used in the compact table cell; raw truth remains available.""" + return max(-1.0, min(1.0, self.residual_value)) + + @property + def residual_saturated(self) -> bool: + return abs(self.residual_value) > 1.0 + @dataclass class Provenance: @@ -61,6 +82,7 @@ def write_report( provenance: Provenance, *, qualification_diagnostic: str = "", + runtime_diagnostics: list[str] | None = None, summary: str | None = None, ) -> str: """Render the provenance report as markdown.""" @@ -74,6 +96,7 @@ def write_report( "claims": claims, "provenance": provenance, "qualification_diagnostic": qualification_diagnostic, + "runtime_diagnostics": runtime_diagnostics or [], "summary": summary or _default_summary(claims), "now_ns": time.time_ns(), } @@ -84,7 +107,7 @@ def _default_summary(claims: list[Claim]) -> str: verified = [c for c in claims if c.measured_delta is not None and not c.rolled_back] if not verified: return "No claims verified within budget. See diagnostic below." - total = sum(c.measured_delta or 0.0 for c in verified) + total = sum(c.measured_delta for c in verified if c.measured_delta is not None) return f"{len(verified)} verified claims, aggregate measured delta {total:+.1%}." diff --git a/gitm/optimizer/templates/report.md.j2 b/gitm/optimizer/templates/report.md.j2 index 8269e1f..98dd103 100644 --- a/gitm/optimizer/templates/report.md.j2 +++ b/gitm/optimizer/templates/report.md.j2 @@ -13,6 +13,14 @@ > **Qualification gate diagnostic.** {{ qualification_diagnostic }} {%- endif %} +{%- if runtime_diagnostics %} + +## Runtime diagnostics + +{% for diagnostic in runtime_diagnostics -%} +- {{ diagnostic }} +{% endfor %} +{%- endif %} ## Claims @@ -25,7 +33,7 @@ _No claims._ | # | Claim | Residual | Causal evidence | Intervention | Predicted Δ | Measured Δ | |---|---|---|---|---|---|---| {% for c in claims -%} -| {{ loop.index }} | {{ c.summary }} | `{{ c.residual_invariant }}`: {{ '{:+.1%}'.format(c.residual_value) }} | {{ c.causal_evidence }} | `{{ c.intervention_name }}` | {{ '{:+.1%}'.format(c.predicted_delta) }} | {% if c.measured_delta is none %}—{% else %}{{ '{:+.1%}'.format(c.measured_delta) }}{% endif %}{% if c.rolled_back %} (rolled back){% endif %} | +| {{ loop.index }} | {{ c.summary }} | `{{ c.residual_invariant }}`{% if c.residual_scope != 'claim' %} ({{ c.residual_scope }}){% endif %}: {{ '{:+.1%}'.format(c.residual_display_value) }}{% if c.residual_saturated %} (display capped; raw {{ '{:+.1%}'.format(c.residual_value) }}){% endif %} | {{ c.causal_evidence }} | `{{ c.intervention_name }}` | {{ '{:+.1%}'.format(c.predicted_delta) }} | {% if c.measured_delta is none %}—{% else %}{{ '{:+.1%}'.format(c.measured_delta) }}{% endif %}{% if c.rolled_back %} (rolled back){% endif %} | {% endfor %} {%- endif %} diff --git a/gitm/planner/context.py b/gitm/planner/context.py index 70fd7df..1639c18 100644 --- a/gitm/planner/context.py +++ b/gitm/planner/context.py @@ -120,6 +120,7 @@ class PlannerContext: peak: HardwarePeak | None sku: str | None num_gpus: int + num_gpus_is_fallback: bool = False def _query_nvml() -> tuple[str | None, int | None]: @@ -170,6 +171,7 @@ def hardware_spec_for(peak: HardwarePeak | None) -> HardwareSpec: quant = quant_peaks_for_sku(peak.name) return HardwareSpec( name=peak.name, + is_fallback=False, peak_flops_fp16_per_s=peak.peak_flops, peak_flops_bf16_per_s=peak.peak_flops, peak_flops_fp8_per_s=quant.get("fp8", 0.0), @@ -215,18 +217,30 @@ def build_planner_context( *, workload: str = "vllm-decode", num_gpus: int | None = None, + has_collective: bool = False, + has_interconnect: bool | None = None, ) -> PlannerContext: """Assemble the gate context + hardware peaks for this run. ``GITM_GPU_SKU`` overrides NVML (useful in CI / on a box without pynvml). """ + if num_gpus is not None and num_gpus <= 0: + raise ValueError(f"num_gpus must be positive when supplied, got {num_gpus}") env_sku = os.environ.get("GITM_GPU_SKU") # Only touch NVML if something it provides is actually missing. nvml_name = nvml_count = None if env_sku is None or num_gpus is None: nvml_name, nvml_count = _query_nvml() sku = env_sku or nvml_name - n = num_gpus or nvml_count or 1 + if num_gpus is not None: + n = num_gpus + num_gpus_is_fallback = False + elif nvml_count is not None and nvml_count > 0: + n = nvml_count + num_gpus_is_fallback = False + else: + n = 1 + num_gpus_is_fallback = True peak = peak_for_sku(sku) dtype = _engine_dtype(engine) kv_len = _engine_kv_len(engine) @@ -237,7 +251,15 @@ def build_planner_context( hardware=sku, kv_cache_len=kv_len, num_gpus=n, - has_collective=n > 1, - has_interconnect=n > 1, # refined later by NVLink/IB probe + # A device count proves neither fact. Callers may pass trace/probe-backed + # evidence; otherwise both remain unavailable/false. + has_collective=has_collective, + has_interconnect=has_interconnect, + ) + return PlannerContext( + gate=gate, + peak=peak, + sku=sku, + num_gpus=n, + num_gpus_is_fallback=num_gpus_is_fallback, ) - return PlannerContext(gate=gate, peak=peak, sku=sku, num_gpus=n) diff --git a/gitm/planner/graph.py b/gitm/planner/graph.py index 147b96f..3703f5a 100644 --- a/gitm/planner/graph.py +++ b/gitm/planner/graph.py @@ -10,7 +10,7 @@ from __future__ import annotations -from dataclasses import dataclass, field +from dataclasses import dataclass, field, replace from gitm.planner.roofline import ( BatchConfig, @@ -24,7 +24,7 @@ ) -def _ffn_terms(model: ModelSpec, b: int, *, moe_layer: bool = True) -> tuple[ +def _ffn_terms(model: ModelSpec, b: int, *, moe_layer: bool = True, tp: int = 1) -> tuple[ float, float, float, float ]: """``(gate_up_flops, gate_up_bytes, down_flops, down_bytes)`` for one layer. @@ -60,7 +60,7 @@ def _ffn_terms(model: ModelSpec, b: int, *, moe_layer: bool = True) -> tuple[ if not (model.is_moe and moe_layer): # Dense: one FFN, weights fetched once, every token through all of it. - ff = model.intermediate + ff = model.intermediate / tp gate_up_flops = 2 * 2 * b * h * ff gate_up_bytes = dt * (b * h + 2 * b * ff) + wb * (2 * h * ff) down_flops = 2 * b * ff * h @@ -75,17 +75,20 @@ def _ffn_terms(model: ModelSpec, b: int, *, moe_layer: bool = True) -> tuple[ distinct = distinct_experts(b, model.num_experts, k) # Compute: every token pays k routed experts plus all shared ones. - gate_up_flops = 2 * 2 * b * (k * h * ff + n_shared * h * sff) - down_flops = 2 * b * (k * ff * h + n_shared * sff * h) + gate_up_flops = 2 * 2 * b * (k * h * ff + n_shared * h * sff) / tp + down_flops = 2 * b * (k * ff * h + n_shared * sff * h) / tp # Router: [b, h] @ [h, num_experts], folded in above. gate_up_flops += 2 * b * h * model.num_experts # Weight traffic: distinct routed experts once each, plus the shared experts # (always resident in the step) and the router matrix. - gate_up_weight_bytes = wb * (distinct * 2 * h * ff + n_shared * 2 * h * sff + h * model.num_experts) - down_weight_bytes = wb * (distinct * ff * h + n_shared * sff * h) + gate_up_weight_bytes = wb * ( + (distinct * 2 * h * ff + n_shared * 2 * h * sff) / tp + + h * model.num_experts + ) + down_weight_bytes = wb * (distinct * ff * h + n_shared * sff * h) / tp # Activations: in [b, h], out [b, k*ff] (+ shared) for gate_up; mirrored for down. - act_out = b * (k * ff + n_shared * sff) + act_out = b * (k * ff + n_shared * sff) / tp gate_up_bytes = dt * (b * h + 2 * act_out) + gate_up_weight_bytes down_bytes = dt * (act_out + b * h) + down_weight_bytes return gate_up_flops, gate_up_bytes, down_flops, down_bytes @@ -96,9 +99,6 @@ class PredictedNode: op: str layer: int | None prediction: RooflinePrediction - # Streams the planner expects to run on — used by the stream-concurrency - # invariant. - expected_stream_id: int = 0 @dataclass @@ -120,16 +120,29 @@ def total_pred_s(self) -> float: return sum(n.prediction.t_pred_s for n in self.nodes) @property - def has_unpriced_collectives(self) -> bool: - """True if a collective moves bytes but predicts zero time. + def has_unpriced_nodes(self) -> bool: + """True when a positive compute or byte term lacks its denominator. - Happens when the SKU has no interconnect bandwidth in the catalogue. The - node is still in the graph — it just costs nothing, which would quietly - credit a sharded deployment with a free all-to-all. Louder to ask than to - discover it in a report. + A priced memory term must not hide missing compute throughput (or vice + versa), so this cannot be inferred from total predicted time alone. """ + return self.has_unpriced_compute or self.has_unpriced_memory + + @property + def has_unpriced_compute(self) -> bool: + return any(n.prediction.compute_is_unpriced for n in self.nodes) + + @property + def has_unpriced_memory(self) -> bool: + return any(n.prediction.memory_is_unpriced for n in self.nodes) + + @property + def has_unpriced_collectives(self) -> bool: + """True if a collective moves bytes but predicts zero time.""" + collective_ops = {"moe_all_to_all", "tp_all_reduce"} return any( - n.prediction.bytes > 0 and n.prediction.t_pred_s == 0.0 for n in self.nodes + n.op in collective_ops and n.prediction.memory_is_unpriced + for n in self.nodes ) @property @@ -141,11 +154,63 @@ def has_fallback_peaks(self) -> bool: """ return any(n.prediction.peak_is_fallback for n in self.nodes) + @property + def has_fallback_bytes(self) -> bool: + """True if any node substituted bf16 for an unknown byte-width dtype. + + Kept separate from :attr:`has_fallback_peaks`: compute peak and byte + width are independent inputs, and decode is commonly memory-bound. + """ + return any(n.prediction.bytes_are_fallback for n in self.nodes) + + @property + def hardware_is_fallback(self) -> bool: + """True when the graph uses substituted rather than detected SKU peaks.""" + return self.hw.is_fallback + + @property + def resident_weight_bytes_per_rank(self) -> float | None: + """Sparse-model resident weight footprint for this graph's rank. + + Dense v0 does not yet enumerate a complete resident footprint, so it + returns ``None`` rather than repurposing per-step traffic as capacity. + """ + if not isinstance(self.model, SparseMoEModelSpec): + return None + # Local import avoids the graph <-> sparse graph construction cycle. + from gitm.planner.moe_graph import model_weight_bytes + + return model_weight_bytes(self.model, self.sharding) + + @property + def resident_weight_bytes_is_lower_bound(self) -> bool: + """True when private DSpark shapes make the footprint a known lower bound.""" + return isinstance(self.model, SparseMoEModelSpec) and bool(self.model.dspark_layer_ids) + + @property + def kv_bytes_per_token_per_sequence(self) -> float | None: + """Sparse KV footprint that grows with each context token and sequence.""" + if not isinstance(self.model, SparseMoEModelSpec): + return None + from gitm.planner.moe_graph import kv_bytes_per_token + + return kv_bytes_per_token(self.model) + + @property + def kv_fixed_bytes_per_sequence(self) -> float | None: + """Sparse sliding-window KV footprint paid once for each sequence.""" + if not isinstance(self.model, SparseMoEModelSpec): + return None + from gitm.planner.moe_graph import kv_fixed_bytes_per_sequence + + return kv_fixed_bytes_per_sequence(self.model) + def predict_graph( model: ModelSpec | None = None, hw: HardwareSpec | None = None, batch: BatchConfig | None = None, + sharding: ShardingConfig | None = None, ) -> Graph: """Emit a predicted execution graph for one decode step. @@ -154,23 +219,47 @@ def predict_graph( model = model or ModelSpec() hw = hw or HardwareSpec() batch = batch or BatchConfig() + sharding = sharding or ShardingConfig() + if sharding.ep != 1: + raise ValueError("dense graph does not support expert parallelism") + tp = sharding.tp + if model.n_heads % tp: + raise ValueError(f"n_heads={model.n_heads} must be divisible by tp={tp}") + if model.num_kv_heads >= tp: + if model.num_kv_heads % tp: + raise ValueError( + f"num_kv_heads={model.num_kv_heads} must be divisible by tp={tp}" + ) + kv_heads_rank = model.num_kv_heads / tp + else: + if tp % model.num_kv_heads: + raise ValueError( + f"tp={tp} must be divisible by replicated num_kv_heads={model.num_kv_heads}" + ) + kv_heads_rank = 1 - g = Graph(model=model, hw=hw, batch=batch) - b = batch.batch + g = Graph(model=model, hw=hw, batch=batch, sharding=sharding) + b = batch.positions_per_step + sequences = batch.batch h = model.hidden kv_len = batch.kv_cache_len head_dim = model.head_dim - n_kv = model.num_kv_heads n_h = model.n_heads dt = model.dtype_bytes + wb = model.w_bytes + q_heads_rank = n_h / tp for layer in range(model.n_layers): # QKV projection: matmul (b, h) @ (h, (n_h + 2*n_kv) * head_dim) - qkv_out = (n_h + 2 * n_kv) * head_dim + qkv_out = (q_heads_rank + 2 * kv_heads_rank) * head_dim flops = 2 * b * h * qkv_out - bytes_moved = dt * (b * h + h * qkv_out + b * qkv_out) + bytes_moved = dt * (b * h + b * qkv_out) + wb * h * qkv_out g.nodes.append( - PredictedNode("qkv_proj", layer, roofline("qkv_proj", flops, bytes_moved, hw)) + PredictedNode( + "qkv_proj", + layer, + roofline("qkv_proj", flops, bytes_moved, hw, dtype=model.compute_dtype), + ) ) # Attention scores + softmax + value. Full-attention layers re-read a KV @@ -180,49 +269,93 @@ def predict_graph( # kv_len / head_dim — over 100x at 16k context. if model.is_full_attention_layer(layer): # Reads: K, V over kv_len tokens, grouped to n_kv heads. - kv_bytes = dt * 2 * kv_len * n_kv * head_dim * b - attn_flops = 2 * b * n_h * head_dim * kv_len * 2 # qk + sv + kv_bytes = dt * 2 * kv_len * kv_heads_rank * head_dim * sequences + attn_flops = 2 * b * q_heads_rank * head_dim * kv_len * 2 # qk + sv else: # Read the recurrent state, update it, write it back: 2x state per # sequence. FLOPs are the state-sized matmuls, also context-free. - state = model.linear_attn_state_elems - kv_bytes = dt * 2 * state * b + state = model.linear_attn_state_elems / tp + kv_bytes = dt * 2 * state * sequences attn_flops = 2 * b * state * 2 # state-vector product + state update g.nodes.append( PredictedNode( "attn_score_value", layer, - roofline("attn_score_value", attn_flops, kv_bytes, hw), + roofline( + "attn_score_value", attn_flops, kv_bytes, hw, dtype=model.compute_dtype + ), ) ) # Output projection - flops = 2 * b * h * h - bytes_moved = dt * (b * h + h * h + b * h) + local_h = h / tp + flops = 2 * b * local_h * h + bytes_moved = dt * (b * local_h + b * h) + wb * local_h * h g.nodes.append( - PredictedNode("attn_out_proj", layer, roofline("attn_out_proj", flops, bytes_moved, hw)) + PredictedNode( + "attn_out_proj", + layer, + roofline("attn_out_proj", flops, bytes_moved, hw, dtype=model.compute_dtype), + ) ) # MLP gate+up / down. On an MoE model these two ops carry the expert # GEMMs, so their flops/bytes come from the mixture model instead of a # single dense FFN (see _ffn_terms). gate_up_flops, gate_up_bytes, down_flops, down_bytes = _ffn_terms( - model, b, moe_layer=model.is_moe_layer(layer) + model, b, moe_layer=model.is_moe_layer(layer), tp=tp ) g.nodes.append( PredictedNode( - "mlp_gate_up", layer, roofline("mlp_gate_up", gate_up_flops, gate_up_bytes, hw) + "mlp_gate_up", + layer, + roofline( + "mlp_gate_up", + gate_up_flops, + gate_up_bytes, + hw, + dtype=model.compute_dtype, + ), ) ) g.nodes.append( - PredictedNode("mlp_down", layer, roofline("mlp_down", down_flops, down_bytes, hw)) + PredictedNode( + "mlp_down", + layer, + roofline( + "mlp_down", down_flops, down_bytes, hw, dtype=model.compute_dtype + ), + ) ) # Final vocab projection - flops = 2 * b * h * model.vocab - bytes_moved = dt * (b * h + h * model.vocab + b * model.vocab) + if tp > 1: + link = replace(hw, peak_mem_bw_bytes_per_s=hw.interconnect_bw_bytes_per_s) + collective_bytes = 4.0 * (tp - 1) / tp * b * h * dt + g.nodes.append( + PredictedNode( + "tp_all_reduce", + layer, + roofline( + "tp_all_reduce", + 0.0, + collective_bytes, + link, + dtype=model.compute_dtype, + estimated=True, + ), + ) + ) + + local_vocab = model.vocab / tp + flops = 2 * b * h * local_vocab + bytes_moved = dt * (b * h + b * local_vocab) + wb * h * local_vocab g.nodes.append( - PredictedNode("lm_head", None, roofline("lm_head", flops, bytes_moved, hw)) + PredictedNode( + "lm_head", + None, + roofline("lm_head", flops, bytes_moved, hw, dtype=model.compute_dtype), + ) ) return g diff --git a/gitm/planner/kitti_graph.py b/gitm/planner/kitti_graph.py index 02294a6..2d8c818 100644 --- a/gitm/planner/kitti_graph.py +++ b/gitm/planner/kitti_graph.py @@ -49,7 +49,6 @@ class KittiNode: device: str # "gpu" | "cpu" | "pcie" prediction: RooflinePrediction | None = None note: str = "" - expected_stream_id: int = 0 # -1 for CPU, 0+ for CUDA streams @dataclass @@ -81,9 +80,8 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph: CPU stages are annotated but not predicted — they depend on the host CPU and are measured directly by WorkUnit. - The stream-concurrency invariant predicts that voxelization (CPU, stream=-1) - for frame N+1 should overlap backbone inference (GPU, stream=0) for frame N. - The graph makes this expectation explicit via expected_stream_id. + CPU/GPU pipeline overlap is measured from the captured workload; this graph + models per-stage device placement and roofline cost only. """ hw = hw or HardwareSpec() g = KittiGraph(hw=hw) @@ -93,7 +91,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph: name="load_bin", device="cpu", note="np.fromfile: ~15k points x 4 float32 = ~240 KB I/O. Not roofline-modeled.", - expected_stream_id=-1, )) # Stage 2: Voxelization (CPU — scatter 15k points into 12k pillars) @@ -103,7 +100,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph: name="voxelization", device="cpu", note="Host-side scatter into voxel grid. CPU memory-bound. ~48 MB writes.", - expected_stream_id=-1, # CPU thread — overlaps GPU stream 0 (stream-concurrency invariant) )) # Stage 3: H2D copy (PCIe — pillar features host -> device) @@ -124,7 +120,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph: bound="memory", ), note=f"pillar features: {h2d_bytes / 1e6:.1f} MB @ ~20 GB/s PCIe", - expected_stream_id=0, )) # Stage 4: Pillar Feature Encoder / VFE (GPU) @@ -142,7 +137,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph: device="gpu", prediction=roofline("pillar_vfe", vfe_flops, vfe_bytes, hw, dtype="fp32"), note="PointPillar feature encoder: linear + BN + ReLU per pillar", - expected_stream_id=0, )) # Stage 5: BEV scatter (GPU — pillar features -> spatial BEV grid) @@ -157,7 +151,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph: device="gpu", prediction=roofline("bev_scatter", 0, bev_scatter_bytes, hw, dtype="fp32"), note="Scatter pillar features -> 2D BEV pseudo-image", - expected_stream_id=0, )) # Stage 6: 2D Backbone CNN (GPU) @@ -173,7 +166,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph: device="gpu", prediction=roofline("backbone_2d", backbone_flops, backbone_bytes, hw, dtype="fp32"), note="Strided 2D CNN: 3 blocks, progressively downsampled BEV features", - expected_stream_id=0, )) # Stage 7: Detection head (GPU — anchor-based classification + regression) @@ -189,7 +181,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph: device="gpu", prediction=roofline("detection_head", head_flops, head_bytes, hw, dtype="fp32"), note="SSD-style anchor cls + box regression, 1x1 conv", - expected_stream_id=0, )) # Stage 8: NMS (CPU — CPU-accelerated iou3d_nms_cuda is a separate CUDA kernel @@ -198,7 +189,6 @@ def predict_kitti_graph(hw: HardwareSpec | None = None) -> KittiGraph: name="nms", device="cpu", note="iou3d_nms_cuda: box decode + IoU matrix + suppression. CPU-sync stall.", - expected_stream_id=-1, )) return g @@ -239,10 +229,16 @@ def render_kitti_graph( ] if measured: - fps = measured.get("frames_per_second", 0) - frame_ms = 1000 / fps if fps > 0 else 0 - gpu_pct = measured.get("gpu_active_pct", 0) - data_pct = measured.get("data_stall_pct", 0) + fps = measured.get("frames_per_second") + gpu_pct = measured.get("gpu_active_pct") + data_pct = measured.get("data_stall_pct") + if not isinstance(fps, int | float) or fps <= 0: + lines += ["", "Measured comparison refused: frames_per_second is missing or non-positive."] + return "\n".join(lines) + if not isinstance(gpu_pct, int | float) or not isinstance(data_pct, int | float): + lines += ["", "Measured comparison refused: stall coverage fields are missing."] + return "\n".join(lines) + frame_ms = 1000 / fps lines += [ "", "Measured (baseline):", diff --git a/gitm/planner/moe_graph.py b/gitm/planner/moe_graph.py index 31e80e5..d5acb70 100644 --- a/gitm/planner/moe_graph.py +++ b/gitm/planner/moe_graph.py @@ -72,11 +72,55 @@ HardwareSpec, ShardingConfig, SparseMoEModelSpec, + _canon_dtype, distinct_experts, roofline, weight_bytes, + weight_bytes_is_fallback, ) +_REQUIRED_POSITIVE_CONFIG_FIELDS = ( + "hidden_size", "num_hidden_layers", "num_attention_heads", "num_key_value_heads", + "head_dim", "q_lora_rank", "o_lora_rank", "o_groups", "vocab_size", + "n_routed_experts", "num_experts_per_tok", "moe_intermediate_size", + "index_n_heads", "index_head_dim", "index_topk", "sliding_window", +) + + +def validate_sparse_moe_config(cfg: dict[str, Any]) -> list[str]: + """Return fields that cannot be omitted without inventing graph shape/bytes.""" + errors: list[str] = [] + for key in _REQUIRED_POSITIVE_CONFIG_FIELDS: + value = cfg.get(key) + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + errors.append(f"{key} must be a declared positive integer, got {value!r}") + for key in ("qk_rope_head_dim", "n_shared_experts"): + value = cfg.get(key) + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + errors.append(f"{key} must be a declared non-negative integer, got {value!r}") + n_experts, top_k = cfg.get("n_routed_experts"), cfg.get("num_experts_per_tok") + if isinstance(n_experts, int) and not isinstance(n_experts, bool) and isinstance(top_k, int) and not isinstance(top_k, bool) and top_k > n_experts: + errors.append(f"num_experts_per_tok={top_k} exceeds n_routed_experts={n_experts}") + n_layers, ratios = cfg.get("num_hidden_layers"), cfg.get("compress_ratios") + if not isinstance(ratios, list | tuple): + errors.append("compress_ratios must be declared for the sparse-attention graph") + elif isinstance(n_layers, int) and n_layers > 0 and len(ratios) < n_layers: + errors.append(f"compress_ratios has {len(ratios)} entries; need at least {n_layers}") + elif any(isinstance(r, bool) or not isinstance(r, int) or r < 0 for r in ratios): + errors.append("compress_ratios must contain non-negative integers") + q = cfg.get("quantization_config") + if q is not None and not isinstance(q, dict): + errors.append("quantization_config must be an object when declared") + elif isinstance(q, dict) and q.get("quant_method") is not None and weight_bytes_is_fallback(str(q["quant_method"])): + errors.append(f"quantization_config.quant_method={q['quant_method']!r} is not priceable") + for key in ("expert_dtype", "torch_dtype"): + value = cfg.get(key) + if value is None: + errors.append(f"{key} must be declared; byte width cannot be guessed") + elif weight_bytes_is_fallback(str(value)): + errors.append(f"{key}={value!r} is not priceable") + return errors + def effective_kv_tokens(spec: SparseMoEModelSpec, layer: int, kv_len: int) -> int: """KV positions the attention *core* reads for ``layer`` at ``kv_len`` context. @@ -279,11 +323,15 @@ def _emit_layer( aw = weight_bytes(spec.act_dtype) ww = weight_bytes(spec.weight_dtype) ew = weight_bytes(spec.expert_dtype) + af = weight_bytes_is_fallback(spec.act_dtype) + wf = weight_bytes_is_fallback(spec.weight_dtype) + ef = weight_bytes_is_fallback(spec.expert_dtype) + kf = weight_bytes_is_fallback(spec.kv_dtype) wd, ed = spec.weight_dtype, spec.expert_dtype def add( op: str, flops: float, byts: float, dtype: str, - *, estimated: bool = False, serial_launches: int = 0, + *, byte_fallback: bool = False, estimated: bool = False, serial_launches: int = 0, ) -> None: name = f"{prefix}{op}" g.nodes.append( @@ -292,6 +340,7 @@ def add( roofline( name, flops, byts, hw, dtype, estimated=estimated, serial_launches=serial_launches, + bytes_are_fallback=byte_fallback, ), ) ) @@ -305,10 +354,10 @@ def add( # head. Every rank pays for them in full, so TP's speedup on attention is # strictly less than ``tp``. f, b = _linear(positions, h, spec.q_lora_rank, aw, ww) - add("attn_q_a", f, b, wd) + add("attn_q_a", f, b, wd, byte_fallback=af or wf) f, b = _linear(positions, spec.q_lora_rank, spec.n_heads * spec.q_head_dim // tp, aw, ww) - add("attn_q_b", f, b, wd) + add("attn_q_b", f, b, wd, byte_fallback=af or wf) # KV entries and their compression weights, plus the cache write for the # positions just computed. CSA computes *two* KV series and two weight series @@ -318,7 +367,7 @@ def add( kind = spec.attention_kind(layer) n_kv_proj = {"csa": 4, "hca": 2}.get(kind, 1) f, b = _linear(positions, h, spec.kv_latent_dim * n_kv_proj, aw, ww) - add("attn_kv_a", f, b + positions * kv_entry_bytes(spec), wd) + add("attn_kv_a", f, b + positions * kv_entry_bytes(spec), wd, byte_fallback=af or wf or kf) if kind != "swa": # The compressor itself: a row-softmax over the window's weights and the @@ -331,6 +380,7 @@ def add( 3.0 * positions * window * spec.kv_latent_dim / r, 2.0 * positions * window * spec.kv_latent_dim * aw / r, spec.act_dtype, + byte_fallback=af, ) # ── indexer: project the query, then scan the compressed candidate set ─── @@ -350,7 +400,7 @@ def add( f, b = _linear( positions, spec.q_lora_rank, spec.index_n_heads * spec.index_head_dim, aw, ww ) - add("attn_index_proj", f, b, wd) + add("attn_index_proj", f, b, wd, byte_fallback=af or wf) add( "attn_index_score", @@ -362,6 +412,7 @@ def add( # why the dtype and the byte width disagree here. sequences * cand * spec.index_head_dim * 1.0, "fp4", + byte_fallback=kf, ) # ── attention core over the selected positions ────────────────────────── @@ -382,6 +433,7 @@ def add( # waiting to happen if the graph divided here. sequences * t_eff * kv_entry_bytes(spec), wd, + byte_fallback=kf, ) # RMSNorm on every query head and on the single compressed KV head, plus @@ -399,6 +451,7 @@ def add( 3.0 * normed + 6.0 * roped, 2.0 * normed * aw + 2.0 * roped * aw, spec.act_dtype, + byte_fallback=af, ) # Output projection, grouped and low-rank. ``o_groups`` partitions the head @@ -410,7 +463,7 @@ def add( f_b, b_b = _linear(positions, spec.o_lora_rank, h, aw, ww) # Named to match the dense graph's output projection: it is the same op in # the same place, so residuals for it stay comparable across model families. - add("attn_out_proj", f_a + f_b, b_a + b_b, wd, estimated=True) + add("attn_out_proj", f_a + f_b, b_a + b_b, wd, byte_fallback=af or wf, estimated=True) # ── manifold-constrained hyper-connections ────────────────────────────── # Two per block (around attention, around the MoE). The residual stream is @@ -446,6 +499,7 @@ def add( f_gen + mix_flops + sinkhorn_flops, b_gen + norm_bytes + mix_bytes, wd, + byte_fallback=af or wf, serial_launches=1, ) @@ -455,7 +509,7 @@ def add( # alone (paper §2.1), so they run no router GEMM at all. if layer >= spec.num_hash_layers: f, b = _linear(positions, h, spec.n_routed_experts, aw, ww) - add("moe_router", f, b, wd) + add("moe_router", f, b, wd, byte_fallback=af or wf) inter = spec.moe_intermediate_size # gate + up + down == three h x inter matrices per expert. @@ -469,6 +523,7 @@ def add( per_expert_weights * spec.n_shared_experts * ew / tp + aw * (positions * h * 2 + positions * inter * 2 * spec.n_shared_experts / tp), ed, + byte_fallback=af or ef, ) # Shared with the dense MoE roofline — one owner for the union term. Note the @@ -490,6 +545,7 @@ def add( per_expert_weights * distinct * ew * skew / es + aw * (positions * h * 2 + positions * inter * 2 * spec.num_experts_per_tok / es), ed, + byte_fallback=af or ef, ) # ── low-rank state update on the tail layers ──────────────────────────── @@ -500,7 +556,7 @@ def add( # Coarse: a down-up low-rank pair. The published shape isn't public, so # this establishes an order of magnitude and flags itself as an estimate # rather than sitting silently in the total. - add("dspark", f_d + f_u, b_d + b_u, wd, estimated=True) + add("dspark", f_d + f_u, b_d + b_u, wd, byte_fallback=af or wf, estimated=True) # ── cross-rank traffic ────────────────────────────────────────────────── # Priced against the interconnect, not HBM, by swapping the bandwidth term. @@ -516,7 +572,7 @@ def add_link(op: str, byts: float) -> None: g.nodes.append( PredictedNode( name, layer, - roofline(name, 0.0, byts, link, spec.act_dtype, estimated=True), + roofline(name, 0.0, byts, link, spec.act_dtype, estimated=True, bytes_are_fallback=af), ) ) @@ -606,9 +662,15 @@ def predict_moe_graph( # Sharded along the vocabulary under TP. aw = weight_bytes(spec.act_dtype) ww = weight_bytes(spec.weight_dtype) + af = weight_bytes_is_fallback(spec.act_dtype) + wf = weight_bytes_is_fallback(spec.weight_dtype) f, b = _linear(positions, spec.hidden, spec.vocab // max(1, sh.tp), aw, ww) g.nodes.append( - PredictedNode("lm_head", None, roofline("lm_head", f, b, hw, spec.weight_dtype)) + PredictedNode( + "lm_head", None, + roofline("lm_head", f, b, hw, spec.weight_dtype, + bytes_are_fallback=af or wf), + ) ) return g @@ -642,31 +704,36 @@ def spec_from_hf_config(cfg: dict[str, Any], *, name: str | None = None) -> Spar ``num_hidden_layers`` entries index real layers, so the tail is dropped rather than silently shifting every layer's ratio. """ + errors = validate_sparse_moe_config(cfg) + if errors: + raise ValueError("sparse-MoE config is not predictable: " + "; ".join(errors)) + q = cfg.get("quantization_config") or {} - weight_dtype = str(q.get("quant_method", "bf16")).lower() - n_layers = int(cfg.get("num_hidden_layers", 43)) - ratios = tuple(int(r) for r in (cfg.get("compress_ratios") or ())[:n_layers]) + act_dtype = _canon_dtype(str(cfg["torch_dtype"])) + weight_dtype = _canon_dtype(str(q.get("quant_method") or act_dtype)) + n_layers = int(cfg["num_hidden_layers"]) + ratios = tuple(int(r) for r in cfg["compress_ratios"][:n_layers]) return SparseMoEModelSpec( name=name or str(cfg.get("model_type", "sparse-moe")), - hidden=int(cfg.get("hidden_size", 4096)), + hidden=int(cfg["hidden_size"]), n_layers=n_layers, - n_heads=int(cfg.get("num_attention_heads", 64)), - num_kv_heads=int(cfg.get("num_key_value_heads", 1)), - head_dim=int(cfg.get("head_dim", 512)), + n_heads=int(cfg["num_attention_heads"]), + num_kv_heads=int(cfg["num_key_value_heads"]), + head_dim=int(cfg["head_dim"]), qk_rope_head_dim=int(cfg.get("qk_rope_head_dim", 64)), - q_lora_rank=int(cfg.get("q_lora_rank", 1024)), - o_lora_rank=int(cfg.get("o_lora_rank", 1024)), - o_groups=int(cfg.get("o_groups", 1)), - vocab=int(cfg.get("vocab_size", 129280)), - n_routed_experts=int(cfg.get("n_routed_experts", 256)), - n_shared_experts=int(cfg.get("n_shared_experts", 1)), - num_experts_per_tok=int(cfg.get("num_experts_per_tok", 6)), - moe_intermediate_size=int(cfg.get("moe_intermediate_size", 2048)), - index_n_heads=int(cfg.get("index_n_heads", 64)), - index_head_dim=int(cfg.get("index_head_dim", 128)), - index_topk=int(cfg.get("index_topk", 512)), - sliding_window=int(cfg.get("sliding_window", 0) or 0), + q_lora_rank=int(cfg["q_lora_rank"]), + o_lora_rank=int(cfg["o_lora_rank"]), + o_groups=int(cfg["o_groups"]), + vocab=int(cfg["vocab_size"]), + n_routed_experts=int(cfg["n_routed_experts"]), + n_shared_experts=int(cfg["n_shared_experts"]), + num_experts_per_tok=int(cfg["num_experts_per_tok"]), + moe_intermediate_size=int(cfg["moe_intermediate_size"]), + index_n_heads=int(cfg["index_n_heads"]), + index_head_dim=int(cfg["index_head_dim"]), + index_topk=int(cfg["index_topk"]), + sliding_window=int(cfg["sliding_window"]), compress_ratios=ratios, num_nextn_predict_layers=int(cfg.get("num_nextn_predict_layers", 0)), # mHC: ``hc_mult`` is n_hc, the residual-stream widening factor. @@ -676,9 +743,7 @@ def spec_from_hf_config(cfg: dict[str, Any], *, name: str | None = None) -> Spar dspark_layer_ids=tuple(int(i) for i in (cfg.get("dspark_target_layer_ids") or ())), dspark_markov_rank=int(cfg.get("dspark_markov_rank", 0)), weight_dtype=weight_dtype, - expert_dtype=str(cfg.get("expert_dtype", weight_dtype)).lower(), - # vLLM serves this checkpoint with an fp8 KV cache; the config does not - # declare cache dtype, so it is a serving decision, not a model fact. - kv_dtype="fp8", - act_dtype=str(cfg.get("torch_dtype", "bf16")).lower().replace("bfloat16", "bf16"), + expert_dtype=_canon_dtype(str(cfg["expert_dtype"])), + kv_dtype=act_dtype, + act_dtype=act_dtype, ) diff --git a/gitm/planner/roofline.py b/gitm/planner/roofline.py index a9c5822..8b7c5b1 100644 --- a/gitm/planner/roofline.py +++ b/gitm/planner/roofline.py @@ -6,9 +6,6 @@ t_memory = bytes / peak_mem_bw t_pred = max(t_compute, t_memory) -with a vendor-specific efficiency band ``(eff_lo, eff_hi)``: a kernel within -that band is "as expected". Residuals outside the band drive attribution. - **The peak must match the op's dtype.** A checkpoint that runs fp8 linears and fp4 experts priced against a bf16 peak understates its own ceiling by 2-4x, and an understated ceiling reads as recoverable headroom that isn't there — the one @@ -19,6 +16,7 @@ from __future__ import annotations +import math from dataclasses import dataclass # Bytes of HBM traffic per stored weight, including the quantisation scales that @@ -39,6 +37,22 @@ _WEIGHT_BYTES["fp4"] = _WEIGHT_BYTES["mxfp4"] +def _canon_dtype(dtype: str) -> str: + """Normalize framework/config aliases before pricing or fallback checks.""" + d = str(dtype).lower().removeprefix("torch.") + if d in ("bf16", "bfloat16"): + return "bf16" + if d in ("float16", "fp16", "half"): + return "fp16" + if d in ("fp4", "mxfp4", "nvfp4"): + return "fp4" + if d in ("fp8", "e4m3", "e5m2"): + return "fp8" + if d in ("fp32", "float32", "tf32"): + return "fp32" + return d + + def weight_bytes(dtype: str) -> float: """Bytes moved per stored weight for ``dtype``, scales included. @@ -46,7 +60,12 @@ def weight_bytes(dtype: str) -> float: since over-counting weight traffic predicts a *slower* floor and so cannot manufacture headroom. """ - return _WEIGHT_BYTES.get(dtype.lower(), 2.0) + return _WEIGHT_BYTES.get(_canon_dtype(dtype), 2.0) + + +def weight_bytes_is_fallback(dtype: str) -> bool: + """Whether :func:`weight_bytes` substitutes bf16 for an unknown dtype.""" + return _canon_dtype(dtype) not in _WEIGHT_BYTES @dataclass(frozen=True) @@ -63,6 +82,10 @@ class HardwareSpec: """ name: str = "A100-SXM4-80GB" + # True when these catalogue numbers were substituted because the runtime did + # not identify a priceable SKU. Direct ``HardwareSpec()`` construction is the + # same documented A100 fallback; catalogue-backed builders set this false. + is_fallback: bool = True peak_flops_fp16_per_s: float = 312e12 peak_flops_bf16_per_s: float = 312e12 peak_flops_fp32_per_s: float = 19.5e12 @@ -73,11 +96,8 @@ class HardwareSpec: # ``0.0`` means unknown, which makes a sharded graph refuse to guess rather # than predict a free all-to-all. interconnect_bw_bytes_per_s: float = 0.0 - # Wall time to issue one dependent kernel, for work whose cost is the *number - # of launches* rather than the arithmetic in them. ~2 us is a CUDA-graph - # replay figure; eager launch is nearer 5 us. Calibrate from a trace — an - # iteration count multiplied by a wrong constant is still the right shape, - # which is more than a pure roofline offers here. + # Wall time to issue one dependent kernel. This is a separate bound from + # FLOPs/bytes for fused iterative work whose launch depth is answer-deciding. kernel_launch_overhead_s: float = 2.0e-6 eff_lo: float = 0.55 eff_hi: float = 0.95 @@ -103,6 +123,9 @@ class ModelSpec: num_kv_heads: int = 32 # < n_heads when GQA head_dim: int = 128 intermediate: int = 11008 + #: Compute dtype used to choose the tensor-core ceiling. Width alone cannot + #: distinguish bf16/fp16/fp32, so this must travel separately. + compute_dtype: str = "fp16" dtype_bytes: int = 2 # fp16 / bf16 — activations vocab: int = 32000 @@ -137,6 +160,36 @@ class ModelSpec: #: 1 = every layer is full attention, i.e. a conventional transformer. full_attn_layer_step: int = 1 + def __post_init__(self) -> None: + for name in ("hidden", "n_layers", "n_heads", "num_kv_heads", "head_dim", "dtype_bytes", "vocab"): + value = getattr(self, name) + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + raise ValueError(f"{name} must be a positive integer, got {value!r}") + for name in ("moe_layer_step", "full_attn_layer_step"): + value = getattr(self, name) + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + raise ValueError(f"{name} must be a positive integer, got {value!r}") + for name in ("num_experts", "experts_per_token", "shared_experts", "first_dense_layers"): + value = getattr(self, name) + if not isinstance(value, int) or isinstance(value, bool) or value < 0: + raise ValueError(f"{name} must be a non-negative integer, got {value!r}") + if (self.num_experts == 0) != (self.experts_per_token == 0): + raise ValueError("num_experts and experts_per_token must both be zero or both positive") + if self.experts_per_token > self.num_experts: + raise ValueError("experts_per_token cannot exceed num_experts") + if self.first_dense_layers > self.n_layers: + raise ValueError("first_dense_layers cannot exceed n_layers") + for name in ("moe_intermediate", "shared_expert_intermediate", "weight_dtype_bytes"): + value = getattr(self, name) + if value is not None and ( + not isinstance(value, int) or isinstance(value, bool) or value <= 0 + ): + raise ValueError(f"{name} must be a positive integer when supplied, got {value!r}") + if not isinstance(self.intermediate, int) or isinstance(self.intermediate, bool): + raise ValueError(f"intermediate must be an integer, got {self.intermediate!r}") + if self.intermediate < 0 or (self.n_moe_layers < self.n_layers and self.intermediate == 0): + raise ValueError("intermediate must be positive when any dense FFN layer is present") + @property def is_moe(self) -> bool: """True when *any* layer's FFN should be modeled as a mixture of experts.""" @@ -153,8 +206,7 @@ def is_moe_layer(self, layer: int) -> bool: """ if not self.is_moe or layer < self.first_dense_layers: return False - step = max(self.moe_layer_step, 1) - return (layer - self.first_dense_layers) % step == 0 + return (layer - self.first_dense_layers) % self.moe_layer_step == 0 @property def n_moe_layers(self) -> int: @@ -178,8 +230,7 @@ def is_full_attention_layer(self, layer: int) -> bool: magnitude, and it is why a hybrid model can serve long contexts with only a few percent of KV-cache utilisation. """ - step = max(self.full_attn_layer_step, 1) - return layer % step == 0 + return layer % self.full_attn_layer_step == 0 @property def n_full_attention_layers(self) -> int: @@ -189,7 +240,7 @@ def n_full_attention_layers(self) -> int: @property def is_hybrid_attention(self) -> bool: """True when some layers use linear/recurrent attention instead of KV.""" - return max(self.full_attn_layer_step, 1) > 1 + return self.full_attn_layer_step > 1 @property def linear_attn_state_elems(self) -> int: @@ -209,17 +260,21 @@ def linear_attn_state_elems(self) -> int: @property def w_bytes(self) -> int: """Bytes per weight element (falls back to the activation dtype).""" - return self.weight_dtype_bytes or self.dtype_bytes + return self.weight_dtype_bytes if self.weight_dtype_bytes is not None else self.dtype_bytes @property def expert_intermediate(self) -> int: """Per-routed-expert FFN width (falls back to the dense width).""" - return self.moe_intermediate or self.intermediate + return self.moe_intermediate if self.moe_intermediate is not None else self.intermediate @property def shared_intermediate(self) -> int: """Per-shared-expert FFN width (falls back to the routed width).""" - return self.shared_expert_intermediate or self.expert_intermediate + return ( + self.shared_expert_intermediate + if self.shared_expert_intermediate is not None + else self.expert_intermediate + ) # --- parameter accounting ------------------------------------------------- # The "35B-A3B" naming convention: total parameters vs the ones a single @@ -380,18 +435,12 @@ class SparseMoEModelSpec: # Multi-token prediction (speculative decoding head) num_nextn_predict_layers: int = 1 - # Manifold-Constrained Hyper-Connections (paper §2.2). ``hc_width`` is n_hc, - # the factor the residual stream is widened by — 1 disables every mHC term. - # Two instances per transformer block (one around attention, one around the - # MoE), each projecting a flattened n_hc*d residual state down to the three - # small mappings A (1 x n_hc), B (n_hc x n_hc) and C (n_hc x 1), then - # projecting B onto the doubly-stochastic manifold by Sinkhorn-Knopp. + # DeepSeek-V4 structural extensions. These are explicit so the graph can + # represent fused mHC/hash-routed layers instead of silently pricing dense + # substitutes. hc_width: int = 1 hc_sinkhorn_iters: int = 0 hc_blocks_per_layer: int = 2 - - # Leading layers that route to experts by a hash of the token id rather than - # a learned router (paper §2.1) — those layers run no router GEMM. num_hash_layers: int = 0 # Low-rank state update on a subset of layers (DeepSeek "DSpark"). @@ -404,6 +453,37 @@ class SparseMoEModelSpec: kv_dtype: str = "fp8" # KV cache and index keys act_dtype: str = "bf16" # activations between ops + def __post_init__(self) -> None: + positive = ( + "hidden", "n_layers", "n_heads", "num_kv_heads", "head_dim", + "q_lora_rank", "o_lora_rank", "o_groups", "vocab", "moe_intermediate_size", + "index_n_heads", "index_head_dim", + ) + for name in positive: + value = getattr(self, name) + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + raise ValueError(f"{name} must be a positive integer, got {value!r}") + for name in ( + "qk_rope_head_dim", "n_shared_experts", "sliding_window", + "num_nextn_predict_layers", "dspark_markov_rank", "hc_sinkhorn_iters", + "num_hash_layers", "n_routed_experts", "num_experts_per_tok", "index_topk", + ): + value = getattr(self, name) + if not isinstance(value, int) or isinstance(value, bool) or value < 0: + raise ValueError(f"{name} must be a non-negative integer, got {value!r}") + if self.num_experts_per_tok > self.n_routed_experts: + raise ValueError("num_experts_per_tok cannot exceed n_routed_experts") + if any(not isinstance(r, int) or isinstance(r, bool) or r < 0 for r in self.compress_ratios): + raise ValueError("compress_ratios must contain non-negative integers") + if (self.n_routed_experts == 0) != (self.num_experts_per_tok == 0): + raise ValueError("n_routed_experts and num_experts_per_tok must both be zero or positive") + if any(layer < 0 or layer >= self.n_layers for layer in self.dspark_layer_ids): + raise ValueError("dspark_layer_ids must refer to real transformer layers") + for name in ("hc_width", "hc_blocks_per_layer"): + value = getattr(self, name) + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + raise ValueError(f"{name} must be a positive integer, got {value!r}") + def compress_ratio(self, layer: int) -> int: """KV compression ratio for ``layer`` (0/1 == uncompressed).""" if not self.compress_ratios: @@ -414,37 +494,15 @@ def compress_ratio(self, layer: int) -> int: @property def compression_levels(self) -> tuple[int, ...]: - """The distinct compression rates this checkpoint interleaves, ascending. - - DeepSeek-V4 uses two: ``m`` for Compressed Sparse Attention and - ``m' >> m`` for Heavily Compressed Attention. Derived from the config - rather than hardcoded, so a checkpoint with different rates — or only - one — still classifies. - """ return tuple(sorted({r for r in self.compress_ratios if r > 1})) def attention_kind(self, layer: int) -> str: - """``"swa"`` | ``"csa"`` | ``"hca"`` — which attention this layer runs. - - The three differ in ways that change the cost model qualitatively, not - just numerically (paper §2.3): - - * **swa** — uncompressed and windowed. Only the sliding-window branch. - * **csa** — compress by ``m``, then *sparse* attention: a lightning - indexer scores the compressed entries and keeps ``index_topk``. Read is - bounded, so it is flat in context once selection saturates. - * **hca** — compress by ``m' >> m`` and attend **densely** over every - compressed entry. No indexer. Read is ``kv_len / m'``, so unlike CSA it - *grows with context* — HCA trades a bigger compression rate for not - having to select. - - The largest compression level is HCA; anything else compressed is CSA. - """ - r = self.compress_ratio(layer) - if r == 0: + """Return the sparse-attention family for one layer.""" + ratio = self.compress_ratio(layer) + if ratio == 0: return "swa" levels = self.compression_levels - if len(levels) >= 2 and r == levels[-1]: + if len(levels) >= 2 and ratio == levels[-1]: return "hca" return "csa" @@ -498,6 +556,14 @@ class ShardingConfig: dp: int = 1 ep_imbalance: float = 1.0 + def __post_init__(self) -> None: + for name in ("tp", "ep", "dp"): + value = getattr(self, name) + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + raise ValueError(f"{name} must be a positive integer, got {value!r}") + if not math.isfinite(self.ep_imbalance) or self.ep_imbalance < 1.0: + raise ValueError("ep_imbalance must be finite and at least 1.0") + @property def expert_shards(self) -> int: """Ranks the expert weights are divided across.""" @@ -513,7 +579,6 @@ class BatchConfig: """Decode batch shape — prompt length already paid; we predict per-step.""" batch: int = 1 - prompt_len: int = 128 kv_cache_len: int = 128 # tokens already in KV-cache when decode starts # Multi-token prediction: draft positions proposed per step, and the fraction # the verifier keeps. A step costs the drafted work regardless; only accepted @@ -521,10 +586,20 @@ class BatchConfig: speculative_tokens: int = 0 acceptance_rate: float = 0.0 + def __post_init__(self) -> None: + if not isinstance(self.batch, int) or isinstance(self.batch, bool) or self.batch <= 0: + raise ValueError(f"batch must be a positive integer, got {self.batch!r}") + for name in ("kv_cache_len", "speculative_tokens"): + value = getattr(self, name) + if not isinstance(value, int) or isinstance(value, bool) or value < 0: + raise ValueError(f"{name} must be a non-negative integer, got {value!r}") + if not math.isfinite(self.acceptance_rate) or not 0.0 <= self.acceptance_rate <= 1.0: + raise ValueError("acceptance_rate must be finite and in [0, 1]") + @property def positions_per_step(self) -> int: """Sequence positions the model actually computes in one step.""" - return self.batch * (1 + max(0, self.speculative_tokens)) + return self.batch * (1 + self.speculative_tokens) @property def tokens_per_step(self) -> float: @@ -533,7 +608,7 @@ def tokens_per_step(self) -> float: Always at least ``batch``: the non-speculative token is verified, not drafted, so it is never rejected. """ - return self.batch * (1.0 + max(0, self.speculative_tokens) * self.acceptance_rate) + return self.batch * (1.0 + self.speculative_tokens * self.acceptance_rate) @dataclass(frozen=True) @@ -554,8 +629,16 @@ class RooflinePrediction: # derivation from published shapes — carried through to the report so an # estimate is never read as a measurement. estimated: bool = False - #: Dependent kernel launches this op costs. Non-zero only for iterative work - #: that cannot be overlapped with itself; see ``bound == "launch"``. + # Set when any dtype contributing to ``bytes`` was unknown and therefore + # priced at :func:`weight_bytes`'s fail-open bf16 width. This cannot be + # inferred from ``dtype``: a node's compute dtype and its activation, weight, + # or KV-cache byte contributors may differ. + bytes_are_fallback: bool = False + # Zero catalogue rates retain fail-open arithmetic, but positive work must + # never look free. Keep the missing denominator even if the sibling term + # still produces a nonzero prediction. + compute_is_unpriced: bool = False + memory_is_unpriced: bool = False serial_launches: int = 0 @property @@ -569,19 +652,6 @@ def peak_is_fallback(self) -> bool: return _canon_dtype(self.dtype) != self.peak_dtype -def _canon_dtype(dtype: str) -> str: - d = dtype.lower() - if d in ("bf16", "float16", "fp16", "half"): - return "fp16" - if d in ("fp4", "mxfp4", "nvfp4"): - return "fp4" - if d in ("fp8", "e4m3", "e5m2"): - return "fp8" - if d in ("fp32", "float32", "tf32"): - return "fp32" - return d - - def resolve_peak(hw: HardwareSpec, dtype: str) -> tuple[float, str]: """(peak FLOP/s, the dtype that peak belongs to) for ``dtype`` on ``hw``. @@ -593,6 +663,8 @@ def resolve_peak(hw: HardwareSpec, dtype: str) -> tuple[float, str]: d = _canon_dtype(dtype) if d == "fp32": return hw.peak_flops_fp32_per_s, "fp32" + if d == "bf16": + return hw.peak_flops_bf16_per_s, "bf16" if d == "fp4": if hw.peak_flops_fp4_per_s > 0: return hw.peak_flops_fp4_per_s, "fp4" @@ -614,26 +686,28 @@ def roofline( dtype: str = "fp16", *, estimated: bool = False, + bytes_are_fallback: bool = False, serial_launches: int = 0, ) -> RooflinePrediction: """Compute the roofline prediction for a single op. - ``serial_launches`` adds a third bound for work whose cost is the number of - *dependent* kernel launches rather than the arithmetic inside them. A - roofline cannot see this: 20 Sinkhorn iterations over a 4x4 matrix move a few - hundred bytes and do a few hundred FLOPs, so both classic terms round to - zero, while the wall time is 20 launches deep and cannot be overlapped - because each iteration consumes the previous one's output. Ignoring it does - not make the prediction slightly optimistic — it makes it absent. + ``serial_launches`` adds a launch-depth floor for dependent iterative work + that a classic FLOP/byte roofline would otherwise make look free. """ + if not math.isfinite(flops) or flops < 0 or not math.isfinite(bytes_moved) or bytes_moved < 0: + raise ValueError( + f"roofline work terms must be finite and non-negative, got " + f"flops={flops!r}, bytes_moved={bytes_moved!r}" + ) peak_flops, peak_dtype = resolve_peak(hw, dtype) + compute_is_unpriced = flops > 0 and peak_flops <= 0 + memory_is_unpriced = bytes_moved > 0 and hw.peak_mem_bw_bytes_per_s <= 0 t_c = flops / peak_flops if peak_flops > 0 else 0.0 t_m = bytes_moved / hw.peak_mem_bw_bytes_per_s if hw.peak_mem_bw_bytes_per_s > 0 else 0.0 - t_l = max(0, serial_launches) * hw.kernel_launch_overhead_s + launches = max(0, serial_launches) + t_l = launches * hw.kernel_launch_overhead_s t_pred = max(t_c, t_m, t_l) - bound = "launch" if t_l >= max(t_c, t_m) and t_l > 0 else ( - "compute" if t_c >= t_m else "memory" - ) + bound = "launch" if t_l >= max(t_c, t_m) and t_l > 0 else ("compute" if t_c >= t_m else "memory") return RooflinePrediction( op=op, flops=flops, @@ -646,5 +720,8 @@ def roofline( peak_dtype=peak_dtype, peak_flops_per_s=peak_flops, estimated=estimated, - serial_launches=max(0, serial_launches), + bytes_are_fallback=bytes_are_fallback, + compute_is_unpriced=compute_is_unpriced, + memory_is_unpriced=memory_is_unpriced, + serial_launches=launches, ) diff --git a/gitm/routing/scorer_v0.py b/gitm/routing/scorer_v0.py index 85101c9..6cbb155 100644 --- a/gitm/routing/scorer_v0.py +++ b/gitm/routing/scorer_v0.py @@ -34,9 +34,26 @@ def score_prospect( W_ENGAGEMENT = 0.05 W_PRIOR = 0.05 + unit_values = { + "warmth": warmth, + "signal_recency": signal_recency, + "engagement_score": engagement_score, + } + invalid_units = {name: value for name, value in unit_values.items() if not 0 <= value <= 1} + if invalid_units: + raise ValueError(f"routing inputs must be within [0, 1], got {invalid_units}") + for name, value in { + "pain_acknowledged": pain_acknowledged, + "prior_engagement": prior_engagement, + }.items(): + if value not in (0, 1): + raise ValueError(f"{name} must be 0 or 1, got {value!r}") + # Company tier score tier_score_map = {1: 1.0, 2: 0.6, 3: 0.2} - tier_score = tier_score_map.get(company_tier, 0.2) + if company_tier not in tier_score_map: + raise ValueError(f"company_tier must be 1, 2, or 3, got {company_tier!r}") + tier_score = tier_score_map[company_tier] score = ( warmth * W_WARMTH + diff --git a/gitm/runtime_driver.py b/gitm/runtime_driver.py index bd8749f..62a9ca2 100644 --- a/gitm/runtime_driver.py +++ b/gitm/runtime_driver.py @@ -27,38 +27,36 @@ import argparse import json import os -import re import time +import warnings from contextlib import closing from pathlib import Path +from gitm._timing import require_positive_duration, require_positive_work + + +def _work_units(summary: dict, workload: str) -> int | float: + """Return the workload's declared unit count; never turn missing work into zero.""" + key = "events" if workload == "hft" else "frames" + if key not in summary: + raise RuntimeError( + f"{workload} runtime work coverage unavailable: summary omitted {key!r}" + ) + return require_positive_work(summary[key], context=f"{workload} runtime") + def _sync(): try: import cupy cupy.cuda.runtime.deviceSynchronize() - except Exception: - pass - - -# Mangled CUDA kernel names → a small set of stable "families" so the residual -# series feeding Granger are well-populated (a variable = a kernel *type*, not -# every template instantiation). Noise tokens are dropped; the first distinctive -# identifier after the library prefix names the family. -_NOISE = { - "detail", "kernel", "void", "const", "unsigned", "int", "long", "float", - "double", "global", "device", "functor", "impl", "internal", "type", - "types", "common", "native", "operator", "policy", "dispatch", "agent", -} - - -def _kernel_family(name: str) -> str: - lib = ("cub" if "cub" in name else "cudf" if "cudf" in name - else "thrust" if "thrust" in name else "k") - toks = [t for t in re.findall(r"[a-z][a-z_]{3,}", name) if t not in _NOISE] - fn = toks[0] if toks else "anon" - return f"{lib}.{fn}" + except Exception as exc: + warnings.warn( + f"CuPy device synchronization unavailable; trace completeness is not " + f"guaranteed ({type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) def _load_hft(stage: Path, seed: int, max_events: int | None): @@ -93,8 +91,12 @@ def _free_gpu_pool(): import cupy cupy.get_default_memory_pool().free_all_blocks() - except Exception: - pass + except Exception as exc: + warnings.warn( + f"GPU memory-pool cleanup unavailable: {type(exc).__name__}: {exc}", + RuntimeWarning, + stacklevel=2, + ) def _stream_hft(stage: Path, seed: int, shards_per_batch: int, max_shards: int | None): @@ -307,18 +309,9 @@ def main(argv: list[str] | None = None) -> int: raise SystemExit("--optimize currently supports --workload hft") return _optimize_hft_cmd(args) - import numpy as np - from gitm import __version__ - from gitm.optimizer.attribution import attribute - from gitm.optimizer.monitor import ( - KernelResidual, - Residuals, - _serialized_fraction, - check_invariants, - ) - from gitm.optimizer.report import Claim, Provenance, write_report - from gitm.planner.graph import predict_graph + from gitm.optimizer.measure import measure_trace, measurement_claims + from gitm.optimizer.report import Provenance, write_report from gitm.tracer import capture stage = args.stage or Path(os.environ.get("GITM_BENCH_STAGE", "/workspace/hft/staging/hft")) @@ -347,15 +340,19 @@ def main(argv: list[str] | None = None) -> int: trace_path = args.outdir / f"{args.workload}_seed{args.seed}_trace.jsonl" tele_path = args.outdir / f"{args.workload}_seed{args.seed}_telemetry.jsonl" - # State telemetry — best-effort; never block the run on it. + # State telemetry remains fail-open, but every degradation is carried into + # the measurement artifact and report. tele = None + telemetry_diagnostics: list[str] = [] try: from gitm.telemetry import Collector, CollectorConfig from gitm.telemetry.sinks import build_sink tele = Collector(CollectorConfig(interval_s=0.25, sinks=[build_sink(f"jsonl:{tele_path}")])) except Exception as exc: - print(f"telemetry disabled (best-effort): {exc}") + diagnostic = f"telemetry unavailable: {type(exc).__name__}: {exc}" + telemetry_diagnostics.append(diagnostic) + print(f"WARN: {diagnostic}") started_ns = time.time_ns() if tele: @@ -365,12 +362,19 @@ def main(argv: list[str] | None = None) -> int: t0 = time.perf_counter() summary = work() _sync() - elapsed = max(time.perf_counter() - t0, 1e-9) + elapsed = require_positive_duration( + time.perf_counter() - t0, context=f"{args.workload} runtime driver" + ) if tele: tele.stop() + telemetry_diagnostics.extend(tele.diagnostics) ended_ns = time.time_ns() - units = summary.get("events", summary.get("frames", 0)) + try: + units = _work_units(summary, args.workload) + except RuntimeError as exc: + print(f"FAIL: {exc}") + return 3 events_per_second = units / elapsed if args.workload == "hft": print( @@ -403,60 +407,28 @@ def main(argv: list[str] | None = None) -> int: "trace_path": str(trace_path), } - violations = [] - top_hyps: list = [] - sc = 0.0 - if kernels: - sc = _serialized_fraction(kernels) - by_name: dict[str, list[int]] = {} - for k in kernels: - by_name.setdefault(k.name, []).append(k.end_ns - k.start_ns) - med = {nm: float(np.median(v)) for nm, v in by_name.items()} - res = Residuals() - res.serialized_concurrency_fraction = sc - for k in kernels: - m = med[k.name] or 1.0 - res.per_kernel.append( - KernelResidual(op=k.name[:40], layer=None, r_kt=((k.end_ns - k.start_ns) - m) / m, r_mt=None) - ) - v_mb = check_invariants(res, multi_basis=True) - v_raw = check_invariants(res, multi_basis=False) - violations = v_mb - print( - f"serialized_concurrency_fraction = {sc:.3f} | " - f"violations multi-basis={len(v_mb)} raw={len(v_raw)} " - f"(filter dropped {len(v_raw) - len(v_mb)})" - ) - # Attribution: group kernels into families (see _kernel_family) and keep - # only families with enough samples, so Granger's per-op residual series - # are well-formed. attribute() truncates all series to the shortest, so a - # sparse op would otherwise collapse every pair to too few points. - from collections import Counter - - fam_of = {nm: _kernel_family(nm) for nm in by_name} - fam_counts = Counter(fam_of[k.name] for k in kernels) - MIN_ATTR = 16 - res_attr = Residuals() - res_attr.serialized_concurrency_fraction = sc - for k in kernels: - fam = fam_of[k.name] - if fam_counts[fam] < MIN_ATTR: - continue - m = med[k.name] or 1.0 - res_attr.per_kernel.append( - KernelResidual(op=fam, layer=None, r_kt=((k.end_ns - k.start_ns) - m) / m, r_mt=None) - ) - fams = sorted({kr.op for kr in res_attr.per_kernel}) - print(f"attribution families (>= {MIN_ATTR} samples): {fams}") - ranked = attribute(res_attr, predict_graph()) - top_hyps = ranked.top(5) - print( - "top Granger hypotheses:", - [(h.cause_op, h.effect_op, round(h.p_value, 4)) for h in top_hyps] or "none", - ) + measured = measure_trace(tr) + measured.diagnostics.extend(telemetry_diagnostics) + violations = measured.violations + top_hyps = measured.top_hypotheses + sc = measured.serialized_fraction + sc_text = f"{sc:.3f}" if sc is not None else "unavailable" + print( + f"serialized_concurrency_fraction = {sc_text} | " + f"violations multi-basis={len(violations)}" + ) + print(f"attribution families (>= 16 samples): {measured.families}") + print( + "top Granger hypotheses:", + [(h.cause_op, h.effect_op, round(h.p_value, 4)) for h in top_hyps] or "none", + ) + for diagnostic in measured.diagnostics: + print(f"WARN: {diagnostic}") measure["serialized_concurrency_fraction"] = sc measure["n_violations"] = len(violations) + measure["n_invalid_duration"] = measured.n_invalid_duration + measure["diagnostics"] = measured.diagnostics measure["top_hypotheses"] = [ {"cause": h.cause_op, "effect": h.effect_op, "p_value": h.p_value} for h in top_hyps ] @@ -480,50 +452,46 @@ def main(argv: list[str] | None = None) -> int: ended_at_ns=ended_ns, trace_path=str(trace_path), ) - claims: list[Claim] = [] - for v in violations[:5]: - ev = ( - f"top hypothesis: {top_hyps[0].cause_op[:30]} -> {top_hyps[0].effect_op[:30]} " - f"(p={top_hyps[0].p_value:.3g})" - if top_hyps - else "no ranked hypothesis" - ) - claims.append( - Claim( - summary=f"{v.invariant} deviation on {v.node_op}", - residual_invariant=v.invariant, - residual_value=float(v.residual), - causal_evidence=ev, - intervention_name="(none — measurement run)", - predicted_delta=0.0, - measured_delta=None, - ) - ) + claims = measurement_claims(measured) + kernel_coverage_available = measured.n_kernels > measured.n_invalid_duration if args.workload == "hft": run_summary = ( f"HFT cuDF/CuPy on {gpu_name}: {events_per_second:,.0f} events/s over {n:,} events; " f"{len(kernels):,} kernels captured, {len(violations)} invariant deviation(s), " - f"serialized-concurrency={sc:.3f}. Measurement run — no interventions applied." + f"serialized-concurrency={sc_text}. Measurement run — no interventions applied." ) else: run_summary = ( f"nuScenes CenterPoint-PointPillar (10-sweep) on {gpu_name}: " f"{events_per_second:,.2f} frames/s over {n:,} frames; " f"{len(kernels):,} kernels captured, {len(violations)} invariant deviation(s), " - f"serialized-concurrency={sc:.3f}. Measurement run — no interventions applied." + f"serialized-concurrency={sc_text}. Measurement run — no interventions applied." ) report_md = write_report( claims, prov, - qualification_diagnostic="Measurement-only run: runtime observed the workload; " - "no intervention library applied.", + qualification_diagnostic=( + "NO DATA — tracer captured no positive-duration GPU kernels; " + "workload throughput was measured, " + "but runtime kernel details were not." + if not kernel_coverage_available + else "Measurement-only run: runtime observed the workload; " + "no intervention library applied." + ), + runtime_diagnostics=measured.diagnostics, summary=run_summary, ) report_path = args.outdir / f"{args.workload}_seed{args.seed}_report.md" report_path.write_text(report_md) print(f"\nwrote: {trace_path}\n {tele_path}\n {report_path}\n " f"{args.outdir / f'{args.workload}_seed{args.seed}_measure.json'}") - print("PASS: workload ran under the runtime; all details measured.") + if not kernel_coverage_available: + print( + "FAIL: workload ran, but no positive-duration GPU kernels were captured; " + "runtime details unavailable." + ) + return 3 + print("PASS: workload ran under the runtime; kernel details measured.") return 0 diff --git a/gitm/safety/autorevert.py b/gitm/safety/autorevert.py index 54958f0..0264838 100644 --- a/gitm/safety/autorevert.py +++ b/gitm/safety/autorevert.py @@ -9,6 +9,7 @@ from __future__ import annotations +import math from collections import deque from dataclasses import dataclass @@ -23,16 +24,34 @@ class AutoRevertDecision: class AutoRevert: def __init__(self, baseline: float, *, tolerance: float = 0.0, window: int = 5) -> None: - if baseline <= 0: - raise ValueError(f"baseline must be > 0, got {baseline}") - if window < 1: - raise ValueError(f"window must be >= 1, got {window}") - self.baseline = baseline - self.tolerance = tolerance # allowed fractional drop before reverting + if ( + isinstance(baseline, bool) + or not isinstance(baseline, int | float) + or not math.isfinite(float(baseline)) + or baseline <= 0 + ): + raise ValueError(f"baseline must be finite and > 0, got {baseline!r}") + if ( + isinstance(tolerance, bool) + or not isinstance(tolerance, int | float) + or not math.isfinite(float(tolerance)) + or tolerance < 0 + ): + raise ValueError(f"tolerance must be finite and non-negative, got {tolerance!r}") + if isinstance(window, bool) or not isinstance(window, int) or window < 1: + raise ValueError(f"window must be a positive integer, got {window!r}") + self.baseline = float(baseline) + self.tolerance = float(tolerance) # allowed fractional drop before reverting self._w: deque[float] = deque(maxlen=window) def observe(self, value: float) -> AutoRevertDecision: - self._w.append(value) + if ( + isinstance(value, bool) + or not isinstance(value, int | float) + or not math.isfinite(float(value)) + ): + raise ValueError(f"observation must be a finite number, got {value!r}") + self._w.append(float(value)) if len(self._w) < self._w.maxlen: return AutoRevertDecision(False, "warming up (need a full window)") mean = sum(self._w) / len(self._w) diff --git a/gitm/safety/failopen.py b/gitm/safety/failopen.py index 4327333..a747bbf 100644 --- a/gitm/safety/failopen.py +++ b/gitm/safety/failopen.py @@ -14,6 +14,7 @@ from __future__ import annotations import signal +import warnings from collections.abc import Callable from typing import Any @@ -35,6 +36,8 @@ def __init__( #: Names whose revert raised during :meth:`fire`. A non-empty list means #: fail-open did NOT fully clean up — the workload may still be mutated. self.failures: list[str] = [] + self.audit_failures: list[str] = [] + self.signal_failures: list[str] = [] def register(self, name: str, revert_fn: Callable[[], None], *, cause: str = "") -> None: """Register a revert to run if we exit before it is disarmed.""" @@ -74,8 +77,10 @@ def _record(self, event: str, name: str, cause: str, **detail: Any) -> None: return try: self._audit.record(event, name, cause, **detail) - except Exception: - pass + except Exception as exc: + message = f"audit sink failed while recording {event} for {name}: {exc}" + self.audit_failures.append(message) + warnings.warn(message, RuntimeWarning, stacklevel=2) def _signal_handler(self, signum: int, frame: Any) -> None: self.fire() @@ -86,14 +91,18 @@ def _signal_handler(self, signum: int, frame: Any) -> None: def __enter__(self) -> FailOpenGuard: self._fired = False self.failures = [] + self.audit_failures = [] + self.signal_failures = [] if self._install: for sig in (signal.SIGTERM, signal.SIGINT): try: self._prev_handlers[sig] = signal.getsignal(sig) signal.signal(sig, self._signal_handler) - except (ValueError, OSError): + except (ValueError, OSError) as exc: # not in the main thread (e.g. tests) — context exit still covers us - pass + message = f"signal handler installation unavailable for {sig}: {exc}" + self.signal_failures.append(message) + warnings.warn(message, RuntimeWarning, stacklevel=2) return self def __exit__(self, exc_type: Any, exc: Any, tb: Any) -> None: @@ -101,6 +110,8 @@ def __exit__(self, exc_type: Any, exc: Any, tb: Any) -> None: for sig, prev in self._prev_handlers.items(): try: signal.signal(sig, prev) - except (ValueError, OSError): - pass + except (ValueError, OSError) as exc: + message = f"signal handler restoration failed for {sig}: {exc}" + self.signal_failures.append(message) + warnings.warn(message, RuntimeWarning, stacklevel=2) self._prev_handlers.clear() diff --git a/gitm/scheduler/loop.py b/gitm/scheduler/loop.py index 52ffd98..e759eb2 100644 --- a/gitm/scheduler/loop.py +++ b/gitm/scheduler/loop.py @@ -9,6 +9,7 @@ from __future__ import annotations import json +import math import os import re import time @@ -18,6 +19,7 @@ from typing import Any from gitm._paths import runs_dir, traces_dir +from gitm._timing import require_positive_duration, require_positive_work from gitm.agents.autoresearch import ( AutoresearchRun, EngineArgsProposer, @@ -54,13 +56,29 @@ unmet_prerequisite, ) from gitm.planner.context import build_planner_context, hardware_spec_for -from gitm.planner.graph import predict_graph +from gitm.planner.graph import Graph, predict_graph +from gitm.planner.moe_graph import ( + is_sparse_moe_config as is_sparse_moe_graph_config, +) from gitm.planner.moe_graph import ( - is_sparse_moe_config, predict_moe_graph, spec_from_hf_config, ) +from gitm.planner.roofline import ( + BatchConfig, + HardwareSpec, + ModelSpec, + ShardingConfig, + weight_bytes, + weight_bytes_is_fallback, +) from gitm.safety.audit import AuditLog, _write_report +from gitm.serve.model_config import ( + is_sparse_moe_config, + normalize_moe_config, + validate_moe_config, + validate_priceable_dtypes, +) from gitm.tracer.capture import capture from gitm.tracer.vllm_stats import sample_scheduler_stats, summarize_requests from gitm.workloads import WorkloadRunner, get_factory, sync_device @@ -93,7 +111,10 @@ def _parse_budget_s(budget: str) -> float: if not m: raise ValueError(f"unparseable budget: {budget!r} (use 24h, 90m, 3600s, 1d)") value, unit = float(m.group(1)), m.group(2) - return value * {"s": 1.0, "m": 60.0, "h": 3600.0, "d": 86400.0}[unit] + seconds = value * {"s": 1.0, "m": 60.0, "h": 3600.0, "d": 86400.0}[unit] + if seconds <= 0.0: + raise ValueError(f"budget must be positive, got {budget!r}") + return seconds def _engine_throughput_fn(engine: Any, runner: Any) -> Any: """Resolve a decode-throughput probe for the live A/B. @@ -116,16 +137,23 @@ def _engine_throughput_fn(engine: Any, runner: Any) -> Any: def _tps(_engine: Any) -> float: t0 = time.perf_counter() out = runner() if runner is not None else {} - dt = max(time.perf_counter() - t0, 1e-9) + dt = require_positive_duration( + time.perf_counter() - t0, context="live-engine throughput probe" + ) # First key that is actually present wins — `or` would treat a legitimate # 0 (a window that produced no tokens) as missing and fabricate a count. - toks: float = 1.0 + toks: float | None = None if isinstance(out, dict): for key in ("generated_tokens", "decode_steps", "events"): if out.get(key) is not None: toks = float(out[key]) break - return toks / dt + if toks is None: + raise RuntimeError( + "live-engine throughput work coverage unavailable: runner returned none of " + "generated_tokens, decode_steps, or events" + ) + return float(require_positive_work(toks, context="live-engine throughput probe")) / dt return _tps @@ -165,99 +193,10 @@ class LoopConfig: workload_runner: WorkloadRunner | None = None -def _hf_config_from_engine(engine: Any) -> Any: - """The live vLLM engine's HF config object, or ``None``. Duck-typed so it - survives vLLM version drift; the config is where both graph-selection and the - model shape are read from.""" - if engine is None: - return None - for path in ( - "llm_engine.model_config.hf_config", - "llm_engine.vllm_config.model_config.hf_config", - "model_config.hf_config", - ): - obj: Any = engine - for attr in path.split("."): - obj = getattr(obj, attr, None) - if obj is None: - break - if obj is not None: - return obj - return None - - -def _hf_config_dict(hf: Any) -> dict[str, Any]: - """A plain dict of the HF config, with ``quantization_config`` flattened. - - ``spec_from_hf_config`` reads the config as a dict and expects - ``quantization_config`` to be one too; a live config carries it as a nested - object, so normalise it here rather than teaching the dict-based builder about - vLLM's object shapes.""" - raw = hf.to_dict() if hasattr(hf, "to_dict") else dict(vars(hf)) - q = raw.get("quantization_config") - if q is not None and not isinstance(q, dict): - raw["quantization_config"] = q.to_dict() if hasattr(q, "to_dict") else dict(vars(q)) - return raw - - -def _model_spec_from_hf(hf: Any): - """Build a dense ``ModelSpec`` from an HF config object, or ``None``. - - ``predict_graph()`` with no model defaults to Llama-2-7B (32 layers). A run - of a *different* model (e.g. opt-125m, 12 layers) is then scored against the - wrong predicted graph, which makes residuals and deviation meaningless — so - read the real architecture off the config. Duck-typed across vLLM version - drift; any failure returns ``None`` and the caller falls back to the default - graph rather than crashing. - """ - if hf is None: - return None - try: - from gitm.planner.roofline import ModelSpec - - hidden = int(hf.hidden_size) - n_heads = int(hf.num_attention_heads) - n_kv = int(getattr(hf, "num_key_value_heads", n_heads) or n_heads) - head_dim = int(getattr(hf, "head_dim", 0) or (hidden // n_heads)) - moe = _moe_fields_from_hf(hf) - return ModelSpec( - hidden=hidden, - n_layers=int(hf.num_hidden_layers), - n_heads=n_heads, - num_kv_heads=n_kv, - head_dim=head_dim, - intermediate=int(getattr(hf, "intermediate_size", 4 * hidden)), - vocab=int(hf.vocab_size), - **moe, - ) - except Exception: - return None - - -def _model_spec_from_engine(engine: Any): - """Dense ``ModelSpec`` for a live engine's model — the config, then the spec.""" - return _model_spec_from_hf(_hf_config_from_engine(engine)) - - -#: HF config field aliases per MoE family — the same quantity is spelled -#: differently by Qwen / Mixtral / DeepSeek, so try each in order. -_MOE_ALIASES: dict[str, tuple[str, ...]] = { - "num_experts": ("num_experts", "num_local_experts", "n_routed_experts"), - "experts_per_token": ("num_experts_per_tok", "moe_top_k", "num_selected_experts"), - "moe_intermediate": ("moe_intermediate_size", "expert_intermediate_size"), - "shared_experts": ("n_shared_experts", "num_shared_experts"), - "shared_expert_intermediate": ("shared_expert_intermediate_size",), - # Per-layer placement: MoE checkpoints are not uniformly sparse. - "first_dense_layers": ("first_k_dense_replace",), - "moe_layer_step": ("decoder_sparse_step", "moe_layer_freq"), -} - #: Attention-shape aliases. Deliberately separate from the MoE table: hybrid #: attention and a mixture FFN are independent choices, and a hybrid model with a #: dense FFN (or a plain MoE transformer) must still get the right one. -_ATTN_ALIASES: dict[str, tuple[str, ...]] = { - "full_attn_layer_step": ("full_attention_interval", "attn_layer_freq"), -} +_FULL_ATTN_ALIASES = ("full_attention_interval", "attn_layer_freq") #: quant_method -> bytes per weight element. MoE decode is weight-fetch bound, #: so using the activation width for a quantized checkpoint would overstate the @@ -295,82 +234,360 @@ def _batch_config_from_stats(sched: Any): return BatchConfig(batch=max(int(round(float(running))), 1)) -def _read_int_aliases(hf: Any, table: dict[str, tuple[str, ...]]) -> dict[str, Any]: - """First positive int found for each field across its aliases. Duck-typed.""" - out: dict[str, Any] = {} - for field, aliases in table.items(): - for alias in aliases: - raw = getattr(hf, alias, None) - if raw is None: - continue - try: - value = int(raw) - except (TypeError, ValueError): - continue - if value > 0: - out[field] = value +@dataclass +class ExecutionGraphResolution: + """A trustworthy loop graph, or the named reason graph-based claims refuse.""" + + graph: Graph | None + diagnostics: list[str] + refusal_reason: str = "" + model_source: str = "" + + @property + def ok(self) -> bool: + return self.graph is not None and not self.refusal_reason + + +def _engine_hf_config(engine: Any) -> tuple[Any | None, str]: + if engine is None: + return None, "no live engine was supplied" + for path in ( + "llm_engine.model_config.hf_config", + "llm_engine.vllm_config.model_config.hf_config", + "model_config.hf_config", + ): + obj: Any = engine + for attr in path.split("."): + obj = getattr(obj, attr, None) + if obj is None: break - return out + if obj is not None: + return obj, path + return None, "the live engine exposes no HuggingFace config" -def _moe_fields_from_hf(hf: Any) -> dict[str, Any]: - """Shape ``ModelSpec`` kwargs read off an HF config. +def _config_dict(hf: Any) -> tuple[dict[str, Any] | None, str]: + try: + if isinstance(hf, dict): + return dict(hf), "" + to_dict = getattr(hf, "to_dict", None) + raw = to_dict() if callable(to_dict) else vars(hf) + if not isinstance(raw, dict): + return None, f"config conversion returned {type(raw).__name__}, not dict" + return dict(raw), "" + except Exception as exc: + # Fail open at the runtime boundary, but preserve the parser failure as + # the refusal reason instead of turning it into a default model. + return None, f"config conversion failed ({type(exc).__name__}: {exc})" - Covers two *independent* axes — whether the FFN is a mixture, and whether - attention is hybrid — so a hybrid-attention dense model still gets its - attention shape, and a plain MoE transformer still gets its experts. Every - field is optional; anything absent falls back to the dense/conventional - default rather than being guessed. Tolerant of partial configs. - """ - # Attention shape is independent of the FFN, so it survives the MoE gate. - out: dict[str, Any] = _read_int_aliases(hf, _ATTN_ALIASES) - moe = _read_int_aliases(hf, _MOE_ALIASES) - - # Only a routed-expert count *and* a top-k make the FFN a mixture; without - # both, leave the FFN dense rather than half-configured. - if not (moe.get("num_experts") and moe.get("experts_per_token")): - return out - out.update(moe) - - quant = getattr(hf, "quantization_config", None) - method = None - if isinstance(quant, dict): - method = quant.get("quant_method") - elif quant is not None: - method = getattr(quant, "quant_method", None) - if isinstance(method, str): - wb = _QUANT_WEIGHT_BYTES.get(method.lower()) - if wb: - out["weight_dtype_bytes"] = wb - return out - - -def _execution_graph(engine: Any, hw: Any, batch: Any): - """The predicted graph for the model that actually ran, and whether it is the - sparse-MoE one. - - A DeepSeek-V4-class checkpoint gets :func:`predict_moe_graph` — mixed fp4/fp8 - precision, compressed/selected attention, MTP — none of which the dense graph - can represent; everything else gets the dense graph, whose FFN already prices a - mixture (:func:`_moe_fields_from_hf`). Sharding stays whole-model so the MoE - prediction shares the dense path's comparison basis against the in-process - trace, rather than predicting one rank against an all-rank capture. + +def _hf_config_dict(hf: Any) -> dict[str, Any]: + """Return a plain HF config mapping, flattening vLLM config objects. + + This small adapter is also useful to direct programmatic callers that test + graph dispatch without constructing the full planner context. """ - from gitm.planner.roofline import ShardingConfig + cfg, error = _config_dict(hf) + if cfg is None: + raise TypeError(error) + quant = cfg.get("quantization_config") + if quant is not None and not isinstance(quant, dict): + raw = quant.to_dict() if callable(getattr(quant, "to_dict", None)) else vars(quant) + if not isinstance(raw, dict): + raise TypeError("quantization_config could not be converted to a dict") + cfg["quantization_config"] = dict(raw) + return cfg + + +def _engine_value(engine: Any, paths: tuple[str, ...]) -> Any: + for path in paths: + obj: Any = engine + for attr in path.split("."): + obj = getattr(obj, attr, None) + if obj is None: + break + if obj is not None: + return obj + return None - hf = _hf_config_from_engine(engine) - if hf is not None: - cfg = _hf_config_dict(hf) - if is_sparse_moe_config(cfg): - spec = spec_from_hf_config(cfg, name=str(cfg.get("model_type") or "sparse-moe")) - return predict_moe_graph(spec, hw, batch, ShardingConfig()), True - return predict_graph(model=_model_spec_from_hf(hf), hw=hw, batch=batch), False +def _loop_batch(engine: Any, pctx: Any, sched: Any) -> tuple[BatchConfig, list[str]]: + diagnostics: list[str] = [] + observed = _batch_config_from_stats(sched) + batch = observed.batch if observed is not None else 1 + if observed is None: + diagnostics.append("decode concurrency was not observed; using batch=1") -def _clamp_pct(value: float) -> float: - """Bound a residual ratio to +/-100% so a bad/misaligned prediction (or a - small-sample outlier) can't blow up a report row into an absurd 18x.""" - return max(-1.0, min(1.0, value)) + kv_len = getattr(getattr(pctx, "gate", None), "kv_cache_len", None) + if not isinstance(kv_len, int) or kv_len <= 0: + kv_len = 4096 + diagnostics.append("KV cache length was not exposed; using kv_cache_len=4096") + + speculative = _engine_value( + engine, + ( + "speculative_config.num_speculative_tokens", + "vllm_config.speculative_config.num_speculative_tokens", + ), + ) + speculative_tokens = int(speculative) if isinstance(speculative, int) and speculative > 0 else 0 + return BatchConfig( + batch=batch, + kv_cache_len=kv_len, + speculative_tokens=speculative_tokens, + ), diagnostics + + +def _loop_sharding(engine: Any) -> tuple[ShardingConfig, list[str]]: + tp = _engine_value( + engine, + ("parallel_config.tensor_parallel_size", "vllm_config.parallel_config.tensor_parallel_size"), + ) + dp = _engine_value( + engine, + ("parallel_config.data_parallel_size", "vllm_config.parallel_config.data_parallel_size"), + ) + ep = _engine_value( + engine, + ("parallel_config.enable_expert_parallel", "vllm_config.parallel_config.enable_expert_parallel"), + ) + if tp is not None and (isinstance(tp, bool) or not isinstance(tp, int) or tp < 1): + raise ValueError(f"tensor_parallel_size must be a positive integer, got {tp!r}") + if tp is None: + return ShardingConfig(), [ + "sharding topology was not exposed; using whole-model tp=1 ep=1 dp=1" + ] + diagnostics: list[str] = [] + if dp is None: + dp_i = 1 + diagnostics.append("data_parallel_size was not exposed; using dp=1") + elif isinstance(dp, bool) or not isinstance(dp, int) or dp < 1: + raise ValueError(f"data_parallel_size must be a positive integer, got {dp!r}") + else: + dp_i = dp + if ep is None: + ep_enabled = False + diagnostics.append("enable_expert_parallel was not exposed; using ep=1") + elif not isinstance(ep, bool): + raise ValueError(f"enable_expert_parallel must be boolean, got {ep!r}") + else: + ep_enabled = ep + return ShardingConfig(tp=tp, ep=tp if ep_enabled else 1, dp=dp_i), diagnostics + + +def _dense_spec_from_config(cfg: dict[str, Any]) -> tuple[ModelSpec | None, str]: + required = ( + "hidden_size", + "num_hidden_layers", + "num_attention_heads", + "intermediate_size", + "vocab_size", + "torch_dtype", + ) + missing = [key for key in required if cfg.get(key) is None] + if missing: + return None, "dense model config is missing answer-deciding fields: " + ", ".join(missing) + try: + hidden = int(cfg["hidden_size"]) + n_heads = int(cfg["num_attention_heads"]) + n_kv = int(cfg.get("num_key_value_heads", n_heads) or n_heads) + head_dim = int(cfg.get("head_dim", 0) or (hidden // n_heads)) + act_raw = str(cfg.get("torch_dtype", "bf16")).lower().removeprefix("torch.") + act = { + "bfloat16": "bf16", + "float16": "fp16", + "half": "fp16", + "float32": "fp32", + }.get(act_raw, act_raw) + if weight_bytes_is_fallback(act): + return None, f"dense activation dtype {act!r} is not priceable" + dtype_bytes = int(weight_bytes(act)) + quant = cfg.get("quantization_config") or {} + if not isinstance(quant, dict): + return None, "dense quantization_config must be an object when declared" + method = quant.get("quant_method") + if method is not None and str(method).lower() not in _QUANT_WEIGHT_BYTES: + return None, f"dense quantization method {method!r} is not priceable" + full_attn_layer_step = 1 + for key in _FULL_ATTN_ALIASES: + raw = cfg.get(key) + if raw is not None: + full_attn_layer_step = int(raw) + if full_attn_layer_step <= 0: + return None, f"dense attention interval {key} must be positive" + break + return ModelSpec( + name=str(cfg.get("_name_or_path") or cfg.get("model_type") or "live-dense"), + hidden=hidden, + n_layers=int(cfg["num_hidden_layers"]), + n_heads=n_heads, + num_kv_heads=n_kv, + head_dim=head_dim, + intermediate=int(cfg["intermediate_size"]), + compute_dtype=act, + dtype_bytes=dtype_bytes, + weight_dtype_bytes=_QUANT_WEIGHT_BYTES.get(str(method).lower()) if method else None, + vocab=int(cfg["vocab_size"]), + full_attn_layer_step=full_attn_layer_step, + ), "" + except (TypeError, ValueError, ZeroDivisionError) as exc: + return None, f"dense model config could not be parsed ({type(exc).__name__}: {exc})" + + +def _execution_graph(engine: Any, pctx: Any, sched: Any) -> ExecutionGraphResolution: + """Resolve the live engine to the matching graph, or refuse namedly.""" + # Compatibility form used by lightweight callers: ``(engine, hw, batch)`` + # returns the historical ``(graph, is_moe)`` pair. The production loop uses + # the planner-context form below so it retains all refusal and provenance + # diagnostics. + if isinstance(pctx, HardwareSpec) and isinstance(sched, BatchConfig): + hf, _ = _engine_hf_config(engine) + if hf is None: + return predict_graph(model=ModelSpec(), hw=pctx, batch=sched), False + cfg = _hf_config_dict(hf) + if is_sparse_moe_graph_config(cfg): + sparse_cfg = { + "hidden_size": cfg.get("hidden_size", 4096), + "num_hidden_layers": cfg.get("num_hidden_layers", 43), + "num_attention_heads": cfg.get("num_attention_heads", 64), + "num_key_value_heads": cfg.get("num_key_value_heads", 1), + "head_dim": cfg.get("head_dim", 512), + "qk_rope_head_dim": cfg.get("qk_rope_head_dim", 64), + "q_lora_rank": cfg.get("q_lora_rank", 1024), + "o_lora_rank": cfg.get("o_lora_rank", 1024), + "o_groups": cfg.get("o_groups", 8), + "vocab_size": cfg.get("vocab_size", 129280), + "n_routed_experts": cfg.get("n_routed_experts", 256), + "n_shared_experts": cfg.get("n_shared_experts", 1), + "num_experts_per_tok": cfg.get("num_experts_per_tok", 6), + "moe_intermediate_size": cfg.get("moe_intermediate_size", 2048), + "index_n_heads": cfg.get("index_n_heads", 64), + "index_head_dim": cfg.get("index_head_dim", 128), + "index_topk": cfg.get("index_topk", 512), + "sliding_window": cfg.get("sliding_window", 128), + "compress_ratios": cfg.get("compress_ratios", [0] * 43), + "torch_dtype": cfg.get("torch_dtype", "bfloat16"), + "expert_dtype": cfg.get("expert_dtype", "fp4"), + "quantization_config": cfg.get("quantization_config", {"quant_method": "fp8"}), + **{k: v for k, v in cfg.items() if k in {"hc_mult", "hc_sinkhorn_iters", "num_hash_layers", "num_nextn_predict_layers", "dspark_target_layer_ids", "dspark_markov_rank"}}, + } + spec = spec_from_hf_config(sparse_cfg, name=str(cfg.get("model_type") or "sparse-moe")) + return predict_moe_graph(spec, pctx, sched, ShardingConfig()), True + # The direct compatibility path is deliberately tolerant of older HF + # fixtures that omit torch_dtype; the production path remains strict. + dense_cfg = dict(cfg) + dense_cfg.setdefault("torch_dtype", "bf16") + spec, error = _dense_spec_from_config(dense_cfg) + if spec is None: + raise ValueError(error) + return predict_graph(model=spec, hw=pctx, batch=sched), False + + hf, source = _engine_hf_config(engine) + if hf is None: + return ExecutionGraphResolution(None, [], source) + cfg, error = _config_dict(hf) + if cfg is None: + return ExecutionGraphResolution(None, [], error) + + if getattr(pctx, "peak", None) is None: + sku = getattr(pctx, "sku", None) or "unknown" + return ExecutionGraphResolution( + None, + [], + f"GPU SKU {sku!r} is not in the hardware catalogue; refusing A100 substitution", + ) + hw = hardware_spec_for(pctx.peak) + batch, diagnostics = _loop_batch(engine, pctx, sched) + try: + sharding, sharding_diagnostics = _loop_sharding(engine) + except ValueError as exc: + return ExecutionGraphResolution(None, diagnostics, f"invalid live sharding topology: {exc}") + diagnostics.extend(sharding_diagnostics) + + if is_sparse_moe_config(cfg): + invalid = validate_moe_config(cfg) + if invalid: + return ExecutionGraphResolution( + None, + diagnostics, + "sparse-MoE config cannot be priced without guessing: " + "; ".join(invalid), + ) + try: + spec = spec_from_hf_config(normalize_moe_config(cfg), name=str(cfg.get("_name_or_path") or "live-moe")) + except (TypeError, ValueError, ZeroDivisionError) as exc: + return ExecutionGraphResolution( + None, + diagnostics, + f"sparse-MoE config could not be parsed ({type(exc).__name__}: {exc})", + ) + act_dtype = getattr(getattr(pctx, "gate", None), "dtype", None) + kv_dtype = _engine_value( + engine, + ("cache_config.cache_dtype", "cache_config.kv_cache_dtype", "vllm_config.cache_config.cache_dtype"), + ) + changes: dict[str, Any] = {} + if act_dtype: + changes["act_dtype"] = str(act_dtype).lower() + if kv_dtype: + changes["kv_dtype"] = str(kv_dtype).lower().replace("fp8_e4m3", "fp8") + else: + resolved_act = changes.get("act_dtype", spec.act_dtype) + changes["kv_dtype"] = resolved_act + diagnostics.append( + "KV cache dtype was not exposed; assuming vLLM default 'auto' follows " + f"activation dtype {resolved_act!r}" + ) + if cfg.get("expert_dtype") is None: + diagnostics.append( + f"expert_dtype absent; inherited weight_dtype={spec.weight_dtype!r} for expert bytes" + ) + if changes: + from dataclasses import replace + + spec = replace(spec, **changes) + unpriceable = validate_priceable_dtypes(spec) + if unpriceable: + return ExecutionGraphResolution(None, diagnostics, "; ".join(unpriceable)) + graph = predict_moe_graph(spec, hw, batch, sharding) + else: + if cfg.get("num_key_value_heads") is None: + diagnostics.append( + "dense num_key_value_heads was not declared; assuming standard MHA " + "with one KV head per query head" + ) + if cfg.get("head_dim") is None: + diagnostics.append( + "dense head_dim was not declared; derived hidden_size / num_attention_heads" + ) + spec, error = _dense_spec_from_config(cfg) + if spec is None: + return ExecutionGraphResolution(None, diagnostics, error) + try: + graph = predict_graph(model=spec, hw=hw, batch=batch, sharding=sharding) + except ValueError as exc: + return ExecutionGraphResolution( + None, diagnostics, f"dense sharding cannot be priced: {exc}" + ) + + if graph.has_fallback_peaks: + diagnostics.append("one or more nodes use fallback compute peaks") + if graph.has_fallback_bytes: + diagnostics.append("one or more nodes use fallback byte widths") + if graph.has_unpriced_nodes: + missing = [] + if graph.has_unpriced_compute: + missing.append("compute throughput") + if graph.has_unpriced_memory: + missing.append("memory bandwidth") + diagnostics.append(f"one or more predicted nodes have unpriced {' and '.join(missing)}") + n_estimated = sum(1 for node in graph.nodes if node.prediction.estimated) + if n_estimated: + diagnostics.append(f"{n_estimated} predicted node(s) use estimated cost models") + return ExecutionGraphResolution( + graph, + diagnostics, + model_source="live_hf_config", + ) def _agg_kt_residual(res: Any) -> float: @@ -389,7 +606,7 @@ def _agg_kt_residual(res: Any) -> float: kts = sorted(float(kr.r_kt) for kr in rows) mid = len(kts) // 2 value = kts[mid] if len(kts) % 2 else (kts[mid - 1] + kts[mid]) / 2.0 - return _clamp_pct(value) + return value def _ar_target_residual(ar_run: AutoresearchRun, fallback: float = 0.0) -> float: @@ -399,7 +616,7 @@ def _ar_target_residual(ar_run: AutoresearchRun, fallback: float = 0.0) -> float target, fall back to the run-level kernel-time residual so generated claims do not all display a misleading +0.0% gap. """ - return _clamp_pct(ar_run.target.residual) if ar_run.target is not None else fallback + return ar_run.target.residual if ar_run.target is not None else fallback def run_loop(cfg: LoopConfig) -> dict[str, Any]: @@ -485,8 +702,9 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]: serving_summary = summarize_requests(req_records) if req_records else None # Collective-communication causes from the same trace — ranked beside the # scheduler causes below. Empty when the trace holds no collective kernels. - coll_causes = collective_causes(worst_device_comm(trace)) - if sched_stats.samples or serving_summary is not None: + coll_stats = worst_device_comm(trace) + coll_causes = collective_causes(coll_stats) + if sched_stats.samples or sched_summary.diagnostics or serving_summary is not None: (run_dir / "scheduler_stats.json").write_text( json.dumps( { @@ -566,13 +784,13 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]: trace_path=trace_path, ) - # Guard: if the tracer captured nothing (no GPU/shim, or the workload never - # ran), do NOT proceed to attribution + emit claims — that fabricates a - # result from an empty trace. Report no-data honestly instead. - if trace.vendor == "none" or not trace.kernels(): - diagnostic = runner_error or qual.diagnostic or ( - "Tracer captured no GPU kernels. Either no GPU/CUPTI shim is present, " - "or the workload did not run under the runtime." + # Guard: a kernel launch with no positive duration is not measurement + # coverage. Do not classify it or emit claims from a fabricated denominator. + valid_kernels = [k for k in trace.kernels() if k.end_ns > k.start_ns] + if trace.vendor == "none" or not valid_kernels: + diagnostic = runner_error or ( + "Tracer captured no positive-duration GPU kernels. Either no GPU/CUPTI " + "shim is present, the workload did not run, or kernel timestamps are invalid." ) return _no_data_result( run_dir=run_dir, @@ -599,49 +817,118 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]: trace_path=trace_path, ) - # Predict against the model that ACTUALLY ran (read from the live engine) with - # the graph its architecture needs — the sparse-MoE graph for a V4-class - # checkpoint, the dense graph otherwise (_execution_graph). Falls back to the - # default dense graph when there is no engine or its config can't be read (CPU - # boxes, tests, dry-run), so residuals never score kernels against Llama-2-7B. - # - # Same for hardware: predict_graph's own default is A100-SXM4-80GB peaks, - # which silently over-predicts on anything weaker (T4/L4/...) and - # produces a run-level kernel-time residual that saturates the report's - # +/-100% clamp on every claim. pctx is built here (moved up from Phase 3) - # so its NVML-detected SKU peak feeds the graph before residuals are ever - # computed against it. - pctx = build_planner_context(cfg.engine, workload=workload) - _hw = hardware_spec_for(pctx.peak) - # Batch matters for the same reason the model does — and more so on a - # mixture, where weight traffic follows the *distinct* experts a batch - # activates: distinct(1)=top_k but distinct(16) is an order of magnitude - # larger, so predicting a batch-16 step at the batch-1 default understates - # expert traffic ~12x. Read the real concurrency off the sampled scheduler - # rather than defaulting. - _batch = _batch_config_from_stats(sched_summary) - graph, is_moe = _execution_graph(cfg.engine, _hw, _batch) - _graph_summary: dict[str, Any] = { - "graph": "moe" if is_moe else "dense", - "nodes": len(graph.nodes), - "total_pred_s": graph.total_pred_s, - "hardware": _hw.name, - } - if is_moe: - m = graph.model - _graph_summary.update( - model=m.name, - sharding="whole-model", - dtypes={"weight": m.weight_dtype, "expert": m.expert_dtype, "kv": m.kv_dtype}, - has_unpriced_collectives=graph.has_unpriced_collectives, + # Resolve model, hardware, serving batch, and sharding as one trust gate. A + # missing/partial live config or unknown SKU refuses graph-based claims and + # falls through to an honest measurement report; it never becomes a plausible + # Llama/A100 default prediction. + pctx = build_planner_context( + cfg.engine, + workload=workload, + has_collective=coll_stats is not None and coll_stats.comm_ns > 0, + ) + graph_resolution = _execution_graph(cfg.engine, pctx, sched_summary) + if pctx.num_gpus_is_fallback: + graph_resolution.diagnostics.append( + "GPU count was unavailable; using 1 for intervention applicability only" ) - (run_dir / "predicted_graph.json").write_text(json.dumps(_graph_summary, indent=2)) + graph_resolution.diagnostics.extend(sched_summary.diagnostics) + if not graph_resolution.ok: + (run_dir / "prediction_refusal.json").write_text( + json.dumps( + { + "reason": graph_resolution.refusal_reason, + "diagnostics": graph_resolution.diagnostics, + "hardware": pctx.sku, + }, + indent=2, + ) + ) + return _measurement_result( + run_dir=run_dir, + run_id=run_id, + workload=workload, + trace=trace, + qual=qual, + started_ns=started_ns, + trace_path=trace_path, + diagnostic=( + "Prediction gate refused graph-based optimization claims: " + f"{graph_resolution.refusal_reason}" + ), + runtime_diagnostics=graph_resolution.diagnostics, + status="prediction_refused", + ) + graph = graph_resolution.graph + assert graph is not None + if graph.resident_weight_bytes_is_lower_bound: + graph_resolution.diagnostics.append( + "resident weight footprint is a lower bound because DSpark parameter shapes are private" + ) + (run_dir / "predicted_graph.json").write_text( + json.dumps( + { + "model": graph.model.name, + "model_source": graph_resolution.model_source, + "nodes": len(graph.nodes), + "total_pred_s": graph.total_pred_s, + "resident_weight_bytes_per_rank": graph.resident_weight_bytes_per_rank, + "resident_weight_bytes_is_lower_bound": ( + graph.resident_weight_bytes_is_lower_bound + ), + "kv_bytes_per_token_per_sequence": graph.kv_bytes_per_token_per_sequence, + "kv_fixed_bytes_per_sequence": graph.kv_fixed_bytes_per_sequence, + "hardware": pctx.sku, + "hardware_pricing": graph.hw.name, + "hardware_is_fallback": graph.hardware_is_fallback, + "batch": { + "batch": graph.batch.batch, + "kv_cache_len": graph.batch.kv_cache_len, + "speculative_tokens": graph.batch.speculative_tokens, + }, + "sharding": { + "tp": graph.sharding.tp, + "ep": graph.sharding.ep, + "dp": graph.sharding.dp, + }, + "has_unpriced_collectives": graph.has_unpriced_collectives, + "has_unpriced_nodes": graph.has_unpriced_nodes, + "has_unpriced_compute": graph.has_unpriced_compute, + "has_unpriced_memory": graph.has_unpriced_memory, + "has_fallback_peaks": graph.has_fallback_peaks, + "has_fallback_bytes": graph.has_fallback_bytes, + "diagnostics": graph_resolution.diagnostics, + "predictions": [ + { + "op": node.op, + "layer": node.layer, + "estimated": node.prediction.estimated, + "peak_is_fallback": node.prediction.peak_is_fallback, + "bytes_are_fallback": node.prediction.bytes_are_fallback, + "compute_is_unpriced": node.prediction.compute_is_unpriced, + "memory_is_unpriced": node.prediction.memory_is_unpriced, + } + for node in graph.nodes + ], + }, + indent=2, + ) + ) # Phase 2 — residuals + attribution res = residuals(trace, graph) violations = check_invariants(res) # multi-basis confirmed hypotheses = attribute(res, graph) # Granger dr_hypotheses = attribute_dr(res, graph) # doubly-robust, corroborating + coverage = { + "total_kernels": res.total_kernels, + "classified_kernels": res.classified_kernels, + "matched_kernels": res.matched_kernels, + "classification_coverage": res.classification_coverage, + "match_coverage": res.match_coverage, + "classified_time_coverage": res.classified_time_coverage, + "matched_time_coverage": res.matched_time_coverage, + "warnings": res.coverage_warnings, + } (run_dir / "violations.json").write_text( json.dumps( @@ -664,6 +951,7 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]: "n_kernel_residuals": len(res.per_kernel), "n_violations": len(violations), "serialized_concurrency_fraction": res.serialized_concurrency_fraction, + "coverage": coverage, "top_hypotheses_granger": [ {"cause": h.cause_op, "effect": h.effect_op, "p_value": h.p_value} for h in hypotheses.top(5) @@ -673,6 +961,9 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]: "notes": h.notes} for h in dr_hypotheses.top(5) ], + "attribution_diagnostics": ( + hypotheses.diagnostics + dr_hypotheses.diagnostics + ), # Engine-scheduler causes (from the vLLM stats adapter) ranked # alongside the kernel-level hypotheses (the engine-signal causal link). "scheduler_causes": [ @@ -709,9 +1000,25 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]: # pctx was built earlier (Phase 1) so its hardware peak could feed predict_graph. # Relative/swept levers resolve against the live engine here, once, before # ranking. See expand_relative_candidates. + try: + raw_library = load_library(workload=workload) + except (FileNotFoundError, ValueError) as exc: + diagnostic = f"intervention candidate coverage unavailable: {type(exc).__name__}: {exc}" + return _measurement_result( + run_dir=run_dir, + run_id=run_id, + workload=workload, + trace=trace, + qual=qual, + started_ns=started_ns, + trace_path=trace_path, + diagnostic=diagnostic, + runtime_diagnostics=graph_resolution.diagnostics + [diagnostic], + status="candidate_coverage_unavailable", + ) library = [ resolved - for s in load_library(workload=workload) + for s in raw_library for resolved in expand_relative_candidates(s, cfg.engine) ] policy = Policy(require_qualification_commit=qual.commit, skip_high_risk=not qual.commit) @@ -823,6 +1130,7 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]: summary=c.spec.summary, residual_invariant="kernel_time", residual_value=kt_residual, + residual_scope="run", causal_evidence=causal_evidence, intervention_name=c.spec.name, predicted_delta=c.predicted_delta, @@ -839,7 +1147,10 @@ def run_loop(cfg: LoopConfig) -> dict[str, Any]: # Phase 4b - agentic autoresearch through the catalog gate/rollback path. if time.time_ns() - started_ns < int(budget_s * 1e9): - proposer = FallbackProposer(EngineArgsProposer(), TableProposer()) + proposer = FallbackProposer( + EngineArgsProposer(gpu_count=pctx.num_gpus), + TableProposer(), + ) def _unenactable(spec: Any) -> str | None: if ( @@ -884,6 +1195,7 @@ def _unenactable(spec: Any) -> str | None: summary=r.spec.summary, residual_invariant="kernel_time", residual_value=ar_residual, + residual_scope=("target_op" if ar_run.target is not None else "run"), causal_evidence=evidence, intervention_name=r.spec.name, predicted_delta=r.predicted_delta, @@ -948,6 +1260,13 @@ def _unenactable(spec: Any) -> str | None: claims=claims, provenance=provenance, qualification_diagnostic=qual.diagnostic, + runtime_diagnostics=( + graph_resolution.diagnostics + + res.coverage_warnings + + (serving_summary.warnings if serving_summary is not None else []) + + hypotheses.diagnostics + + dr_hypotheses.diagnostics + ), summary=( f"vLLM decode on {pctx.sku or 'unknown SKU'}: {len(claims)} candidate(s) " f"evaluated, {len(rolled_back)} rolled back. {sched_note}" @@ -971,6 +1290,8 @@ def _unenactable(spec: Any) -> str | None: "bottleneck_class": ar_run.bottleneck_class, "n_autoresearch": len(ar_run.results), "scheduler_stats": asdict(sched_summary) if sched_stats.samples else None, + "residual_coverage": coverage, + "prediction_diagnostics": graph_resolution.diagnostics, "report_path": str(run_dir / "report.md"), } return {"summary": summary, "report_md": report_md, "run_dir": str(run_dir)} @@ -985,6 +1306,9 @@ def _measurement_result( qual: Any, started_ns: int, trace_path: Path, + diagnostic: str | None = None, + runtime_diagnostics: list[str] | None = None, + status: str = "ok", ) -> dict[str, Any]: """Honest measurement report for a workload with no intervention library. @@ -1000,6 +1324,8 @@ def _measurement_result( { "n_kernels": result.n_kernels, "n_memcpy": result.n_memcpy, + "n_invalid_duration": result.n_invalid_duration, + "diagnostics": result.diagnostics, "serialized_concurrency_fraction": result.serialized_fraction, "n_violations": len(result.violations), "families": result.families, @@ -1022,10 +1348,11 @@ def _measurement_result( report_md = write_report( claims=claims, provenance=provenance, - qualification_diagnostic=( + qualification_diagnostic=diagnostic or ( "Measurement-only run: the runtime observed the workload and reports " "its real kernels. No intervention library applies to this workload." ), + runtime_diagnostics=(runtime_diagnostics or []) + result.diagnostics, summary=measurement_summary(workload, result), ) _write_report(run_dir, report_md) @@ -1033,7 +1360,7 @@ def _measurement_result( summary = { "run_id": run_id, "workload": workload, - "status": "ok", + "status": status, "mode": "measurement", "fingerprint": qual.fingerprint, "commit": False, @@ -1042,11 +1369,50 @@ def _measurement_result( "n_claims": 0, "n_rolled_back": 0, "n_rejected": 0, + "prediction_refusal": diagnostic, + "runtime_diagnostics": runtime_diagnostics or [], "report_path": str(run_dir / "report.md"), } return {"summary": summary, "report_md": report_md, "run_dir": str(run_dir)} +def _specialized_claim_basis( + mres: Any, ab: Any +) -> tuple[tuple[str, float] | None, float | None, list[str]]: + """Choose an observed residual basis without fabricating trace coverage.""" + diagnostics = list(mres.diagnostics) + if ab is None: + diagnostics.append("intervention A/B produced no result; no performance claim emitted") + return None, None, diagnostics + try: + measured_delta = float(ab.speedup) - 1.0 + except (AttributeError, TypeError, ValueError): + diagnostics.append("intervention A/B result has no numeric speedup; no claim emitted") + return None, None, diagnostics + if not math.isfinite(measured_delta): + diagnostics.append("intervention A/B speedup is non-finite; no claim emitted") + return None, None, diagnostics + if mres.serialized_fraction is not None: + return ("stream_concurrency", float(mres.serialized_fraction)), measured_delta, diagnostics + if mres.n_kernels > mres.n_invalid_duration: + diagnostics.append( + "no adjacent cross-stream CUPTI kernel pairs; claim residual uses the " + "measured A/B throughput delta instead of fabricated concurrency evidence" + ) + return ("throughput_delta", measured_delta), measured_delta, diagnostics + diagnostics.append( + "no positive-duration CUPTI kernels; claim residual uses the measured A/B " + "throughput delta instead of a fabricated stream-concurrency value" + ) + return ("throughput_delta", measured_delta), measured_delta, diagnostics + + +def _serialized_evidence(mres: Any) -> str: + if mres.serialized_fraction is None: + return "serialized-concurrency unavailable (no adjacent cross-stream pairs)" + return f"serialized-concurrency={mres.serialized_fraction:.3f}" + + def _hft_intervention_result( *, run_dir: Path, @@ -1092,19 +1458,19 @@ def _hft_intervention_result( spec, applicator, min_keep_delta=0.0, audit=AuditLog(run_dir / "audit.jsonl") ) ab = applicator.last_result + basis, measured_delta, runtime_diagnostics = _specialized_claim_basis(mres, ab) + if apply_res.error: + runtime_diagnostics.append(f"intervention apply failed: {apply_res.error}") # Prove: one claim carrying the measured delta, gated on identical output. top = mres.top_hypotheses if top: evidence = ( f"top hypothesis: {top[0].cause_op[:30]} → {top[0].effect_op[:30]} " - f"(p={top[0].p_value:.3g}); serialized-concurrency={mres.serialized_fraction:.3f}" + f"(p={top[0].p_value:.3g}); {_serialized_evidence(mres)}" ) elif mres.n_kernels: - evidence = ( - f"serialized-concurrency={mres.serialized_fraction:.3f} over " - f"{mres.n_kernels} kernels" - ) + evidence = f"{_serialized_evidence(mres)} over {mres.n_kernels} kernels" else: evidence = ( "no CUPTI trace captured on this box; intervention proven by the " @@ -1113,16 +1479,18 @@ def _hft_intervention_result( claims: list[Claim] = [] rolled_back: list[str] = [] - if ab is not None: + if basis is not None and ab is not None: + residual_invariant, residual_value = basis claims.append( Claim( summary=spec.summary, - residual_invariant="stream_concurrency", - residual_value=float(mres.serialized_fraction), + residual_invariant=residual_invariant, + residual_value=residual_value, + residual_scope="run", causal_evidence=evidence, intervention_name=spec.name, predicted_delta=predicted, - measured_delta=(ab.speedup - 1.0) if ab.identical else None, + measured_delta=measured_delta if ab.identical else None, rolled_back=apply_res.rolled_back, ) ) @@ -1145,6 +1513,8 @@ def _hft_intervention_result( "speedup": getattr(ab, "speedup", None), "serialized_concurrency_fraction": mres.serialized_fraction, "families": mres.families, + "residual_basis": basis[0] if basis is not None else None, + "diagnostics": runtime_diagnostics, }, indent=2, ) @@ -1163,10 +1533,10 @@ def _hft_intervention_result( claims=claims, provenance=provenance, qualification_diagnostic=qual.diagnostic, + runtime_diagnostics=runtime_diagnostics, summary=( f"HFT intervention {spec.name!r}: {verdict}. " - f"{mres.n_kernels:,} kernels observed, serialized-concurrency=" - f"{mres.serialized_fraction:.3f}." + f"{mres.n_kernels:,} kernels observed, {_serialized_evidence(mres)}." ), ) _write_report(run_dir, report_md) @@ -1174,7 +1544,7 @@ def _hft_intervention_result( summary = { "run_id": run_id, "workload": workload, - "status": "ok", + "status": "ok" if basis is not None else "intervention_failed", "mode": "intervention", "fingerprint": qual.fingerprint, "commit": qual.commit, @@ -1227,18 +1597,18 @@ def _openfold_intervention_result( spec, applicator, min_keep_delta=0.0, audit=AuditLog(run_dir / "audit.jsonl") ) ab = applicator.last_result # AF2ABResult + basis, measured_delta, runtime_diagnostics = _specialized_claim_basis(mres, ab) + if apply_res.error: + runtime_diagnostics.append(f"intervention apply failed: {apply_res.error}") top = mres.top_hypotheses if top: evidence = ( f"top hypothesis: {top[0].cause_op[:30]} → {top[0].effect_op[:30]} " - f"(p={top[0].p_value:.3g}); serialized-concurrency={mres.serialized_fraction:.3f}" + f"(p={top[0].p_value:.3g}); {_serialized_evidence(mres)}" ) elif mres.n_kernels: - evidence = ( - f"serialized-concurrency={mres.serialized_fraction:.3f} over " - f"{mres.n_kernels} kernels" - ) + evidence = f"{_serialized_evidence(mres)} over {mres.n_kernels} kernels" else: evidence = ( "no CUPTI trace captured on this box; intervention proven by the " @@ -1247,16 +1617,18 @@ def _openfold_intervention_result( claims: list[Claim] = [] rolled_back: list[str] = [] - if ab is not None: + if basis is not None and ab is not None: + residual_invariant, residual_value = basis claims.append( Claim( summary=spec.summary, - residual_invariant="stream_concurrency", - residual_value=float(mres.serialized_fraction), + residual_invariant=residual_invariant, + residual_value=residual_value, + residual_scope="run", causal_evidence=evidence, intervention_name=spec.name, # plDDT-equivalence is the AF2 correctness gate (vs byte-identical). - measured_delta=(ab.speedup - 1.0) if ab.equivalent else None, + measured_delta=measured_delta if ab.equivalent else None, predicted_delta=predicted, rolled_back=apply_res.rolled_back, ) @@ -1282,6 +1654,8 @@ def _openfold_intervention_result( "speedup": getattr(ab, "speedup", None), "serialized_concurrency_fraction": mres.serialized_fraction, "families": mres.families, + "residual_basis": basis[0] if basis is not None else None, + "diagnostics": runtime_diagnostics, }, indent=2, ) @@ -1300,10 +1674,10 @@ def _openfold_intervention_result( claims=claims, provenance=provenance, qualification_diagnostic=qual.diagnostic, + runtime_diagnostics=runtime_diagnostics, summary=( f"AF2 intervention {spec.name!r}: {verdict}. " - f"{mres.n_kernels:,} kernels observed, serialized-concurrency=" - f"{mres.serialized_fraction:.3f}." + f"{mres.n_kernels:,} kernels observed, {_serialized_evidence(mres)}." ), ) _write_report(run_dir, report_md) @@ -1311,7 +1685,7 @@ def _openfold_intervention_result( summary = { "run_id": run_id, "workload": workload, - "status": "ok", + "status": "ok" if basis is not None else "intervention_failed", "mode": "intervention", "fingerprint": qual.fingerprint, "commit": qual.commit, @@ -1366,18 +1740,18 @@ def _edge_intervention_result( spec, applicator, min_keep_delta=0.0, audit=AuditLog(run_dir / "audit.jsonl") ) ab = applicator.last_result # EdgeABResult + basis, measured_delta, runtime_diagnostics = _specialized_claim_basis(mres, ab) + if apply_res.error: + runtime_diagnostics.append(f"intervention apply failed: {apply_res.error}") top = mres.top_hypotheses if top: evidence = ( f"top hypothesis: {top[0].cause_op[:30]} → {top[0].effect_op[:30]} " - f"(p={top[0].p_value:.3g}); serialized-concurrency={mres.serialized_fraction:.3f}" + f"(p={top[0].p_value:.3g}); {_serialized_evidence(mres)}" ) elif mres.n_kernels: - evidence = ( - f"serialized-concurrency={mres.serialized_fraction:.3f} over " - f"{mres.n_kernels} kernels" - ) + evidence = f"{_serialized_evidence(mres)} over {mres.n_kernels} kernels" else: evidence = ( "no CUPTI trace captured on this box; intervention proven by the " @@ -1386,16 +1760,18 @@ def _edge_intervention_result( claims: list[Claim] = [] rolled_back: list[str] = [] - if ab is not None: + if basis is not None and ab is not None: + residual_invariant, residual_value = basis claims.append( Claim( summary=spec.summary, - residual_invariant="stream_concurrency", - residual_value=float(mres.serialized_fraction), + residual_invariant=residual_invariant, + residual_value=residual_value, + residual_scope="run", causal_evidence=evidence, intervention_name=spec.name, # detection-equivalence is the edge correctness gate. - measured_delta=(ab.speedup - 1.0) if ab.identical else None, + measured_delta=measured_delta if ab.identical else None, predicted_delta=predicted, rolled_back=apply_res.rolled_back, ) @@ -1419,6 +1795,8 @@ def _edge_intervention_result( "speedup": getattr(ab, "speedup", None), "serialized_concurrency_fraction": mres.serialized_fraction, "families": mres.families, + "residual_basis": basis[0] if basis is not None else None, + "diagnostics": runtime_diagnostics, }, indent=2, ) @@ -1437,10 +1815,10 @@ def _edge_intervention_result( claims=claims, provenance=provenance, qualification_diagnostic=qual.diagnostic, + runtime_diagnostics=runtime_diagnostics, summary=( f"edge intervention {spec.name!r}: {verdict}. " - f"{mres.n_kernels:,} kernels observed, serialized-concurrency=" - f"{mres.serialized_fraction:.3f}." + f"{mres.n_kernels:,} kernels observed, {_serialized_evidence(mres)}." ), ) _write_report(run_dir, report_md) @@ -1448,7 +1826,7 @@ def _edge_intervention_result( summary = { "run_id": run_id, "workload": workload, - "status": "ok", + "status": "ok" if basis is not None else "intervention_failed", "mode": "intervention", "fingerprint": qual.fingerprint, "commit": qual.commit, @@ -1475,8 +1853,8 @@ def _no_data_result( ) -> dict[str, Any]: """Write an honest no-data report and return its summary (status=no_data). - Used when the trace has no kernels — a misconfigured box or a workload that - never ran. We emit zero claims rather than fabricating results from nothing. + Used when the trace has no positive-duration kernels — a misconfigured box, + a workload that never ran, or invalid timestamps. We emit zero claims. """ provenance = build_provenance( workload_id=workload, @@ -1489,7 +1867,7 @@ def _no_data_result( claims=[], provenance=provenance, qualification_diagnostic=diagnostic, - summary="NO DATA — tracer captured no GPU kernels; nothing was measured.", + summary="NO DATA — tracer captured no positive-duration GPU kernels; nothing was measured.", ) _write_report(run_dir, report_md) diff --git a/gitm/serve/artifacts.py b/gitm/serve/artifacts.py index cff13e8..0bb4e84 100644 --- a/gitm/serve/artifacts.py +++ b/gitm/serve/artifacts.py @@ -102,13 +102,15 @@ def write_capture_artifacts( "source": trace.source, "device_count": trace.device_count, "events": len(trace.events), - "kernels": len(kernels), + "kernel_records": len(kernels), + "kernels": breakdown.n_kernels, + "invalid_kernel_durations": breakdown.n_invalid_duration, "duration_ns": trace.duration_ns, }, } (out_dir / "run_manifest.json").write_text(json.dumps(manifest, indent=2)) - if not kernels: + if breakdown.n_kernels == 0: status = "no_kernels" elif not had_traffic: status = "no_traffic" @@ -119,7 +121,7 @@ def write_capture_artifacts( out_dir=out_dir, trace_path=trace_path, n_events=len(trace.events), - n_kernels=len(kernels), + n_kernels=breakdown.n_kernels, status=status, breakdown=breakdown, warnings=warnings, diff --git a/gitm/serve/attach.py b/gitm/serve/attach.py index 0abd046..40027f4 100644 --- a/gitm/serve/attach.py +++ b/gitm/serve/attach.py @@ -28,6 +28,7 @@ from __future__ import annotations import json +import math import os import time import urllib.parse @@ -69,6 +70,27 @@ class AttachOptions: dry_run: bool = False proc: Path = discover.PROC + def __post_init__(self) -> None: + for name in ("duration_s", "request_timeout", "metrics_interval"): + value = getattr(self, name) + if not isinstance(value, int | float) or isinstance(value, bool): + raise ValueError(f"{name} must be a finite positive number, got {value!r}") + if not math.isfinite(float(value)) or value <= 0: + raise ValueError(f"{name} must be a finite positive number, got {value!r}") + if isinstance(self.requests, bool) or not isinstance(self.requests, int) or self.requests < 0: + raise ValueError(f"requests must be a non-negative integer, got {self.requests!r}") + for name in ("concurrency", "input_tokens", "output_tokens"): + value = getattr(self, name) + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise ValueError(f"{name} must be a positive integer, got {value!r}") + for name, value in (("pid", self.pid), ("port", self.port)): + if value is not None and ( + isinstance(value, bool) or not isinstance(value, int) or value <= 0 + ): + raise ValueError(f"{name} must be a positive integer, got {value!r}") + if self.port is not None and self.port > 65535: + raise ValueError(f"port must be at most 65535, got {self.port!r}") + @property def mode(self) -> str: return "drive" if self.requests > 0 else "observe" @@ -323,6 +345,7 @@ def attach_and_capture(opts: AttachOptions) -> tuple[int, CaptureResult | None]: server_window = metrics.window_from_snapshots( before, after, window_s=wall, samples=samples ) + server_window.notes.extend(sampler.diagnostics) # Server-side truth is the summary for observe mode, where no client exists. In # drive mode both are kept: when the client's view and the server's histograms @@ -382,27 +405,8 @@ def attach_and_capture(opts: AttachOptions) -> tuple[int, CaptureResult | None]: print(f"\n==> window closed after {wall:.1f}s — server left running (PID {target.pid})") if opts.mode == "drive": print(f" client: {len(records)} ok / {failures} failed") - if server_window.requests_finished is not None: - print( - f" server: {server_window.requests_finished:.0f} requests, " - f"{(server_window.generation_tokens or 0):.0f} output tokens" - + ( - f", {server_window.output_tokens_per_s:.0f} tok/s" - if server_window.output_tokens_per_s - else "" - ) - ) - if server_window.ttft_mean_s is not None: - print( - f" server TTFT mean {server_window.ttft_mean_s * 1e3:.0f} ms " - f"TPOT mean {(server_window.tpot_mean_s or 0) * 1e3:.1f} ms" - ) - if server_window.running_p50 is not None and server_window.waiting_p50 is not None: - print( - f" queue depth p50/p95 running " - f"{server_window.running_p50:.0f}/{server_window.running_p95:.0f}, " - f"waiting {server_window.waiting_p50:.0f}/{server_window.waiting_p95:.0f}" - ) + for line in _server_metric_lines(server_window): + print(line) print_result(result) for note in server_window.notes: print(" - " + note) @@ -416,6 +420,36 @@ def attach_and_capture(opts: AttachOptions) -> tuple[int, CaptureResult | None]: return 0, result +def _server_metric_lines(window: metrics.ServerWindow) -> list[str]: + """Human summary that preserves missing server metrics as unknown.""" + if window.requests_finished is None: + return [] + generation = ( + f"{window.generation_tokens:.0f} output tokens" + if window.generation_tokens is not None + else "output-token count unavailable" + ) + lines = [ + f" server: {window.requests_finished:.0f} requests, {generation}" + + (f", {window.output_tokens_per_s:.0f} tok/s" if window.output_tokens_per_s else "") + ] + if window.ttft_mean_s is not None: + tpot = ( + f"{window.tpot_mean_s * 1e3:.1f} ms" + if window.tpot_mean_s is not None + else "unavailable" + ) + lines.append( + f" server TTFT mean {window.ttft_mean_s * 1e3:.0f} ms TPOT mean {tpot}" + ) + if window.running_p50 is not None and window.waiting_p50 is not None: + lines.append( + f" queue depth p50/p95 running {window.running_p50:.0f}/{window.running_p95:.0f}, " + f"waiting {window.waiting_p50:.0f}/{window.waiting_p95:.0f}" + ) + return lines + + def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None: """Write ``predicted_moe_graph.json`` — a per-rank floor from the live config. @@ -424,6 +458,7 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None: prints the named-key refusal and writes nothing — a defaulted DeepSeek prediction next to a real trace would be read as a measurement, which is the one outcome the gate exists to prevent. Never raises into the capture path. + """ from gitm.serve.model_config import LiveSpec, live_moe_spec @@ -442,14 +477,47 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None: from gitm.planner.context import build_planner_context, hardware_spec_for from gitm.planner.moe_graph import predict_moe_graph - hw = hardware_spec_for(build_planner_context().peak) + planner_ctx = build_planner_context() + hw = hardware_spec_for(planner_ctx.peak) g = predict_moe_graph(resolved.spec, hw, resolved.batch, resolved.sharding) sh, spec = resolved.sharding, resolved.spec + warnings = list(resolved.warnings) + if planner_ctx.peak is None: + warnings.append( + f"GPU SKU {planner_ctx.sku or 'unknown'!r} is not in the hardware catalogue; " + f"pricing uses fallback {hw.name!r}" + ) + if planner_ctx.num_gpus_is_fallback: + warnings.append("GPU count was unavailable; recording a fallback count of 1") + if g.has_unpriced_nodes: + missing = [] + if g.has_unpriced_compute: + missing.append("compute throughput") + if g.has_unpriced_memory: + missing.append("memory bandwidth") + warnings.append(f"predicted nodes have unpriced {' and '.join(missing)}") + if g.has_fallback_peaks: + warnings.append("priced against fallback compute peaks; the ceiling is low") + if g.has_fallback_bytes: + warnings.append( + "byte widths include an unknown-dtype bf16 fallback; the memory floor is approximate" + ) + if g.resident_weight_bytes_is_lower_bound: + warnings.append( + "resident weight footprint is a lower bound because DSpark parameter shapes are private" + ) + n_estimated = sum(1 for n in g.nodes if n.prediction.estimated) + if n_estimated: + warnings.append(f"{n_estimated} predicted node(s) use documented estimated cost models") payload = { "model_ref": resolved.model_ref, "config_source": str(resolved.source_path), - "hardware": hw.name, + "hardware": planner_ctx.sku, + "hardware_pricing": hw.name, + "hardware_is_fallback": planner_ctx.peak is None, + "num_gpus": planner_ctx.num_gpus, + "num_gpus_is_fallback": planner_ctx.num_gpus_is_fallback, "sharding": {"tp": sh.tp, "ep": sh.ep, "dp": sh.dp}, "dtypes": { "weight": spec.weight_dtype, @@ -460,8 +528,17 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None: "batch": {"batch": resolved.batch.batch, "kv_cache_len": resolved.batch.kv_cache_len}, "applied_overrides": resolved.applied_overrides, "total_pred_s": g.total_pred_s, + "resident_weight_bytes_per_rank": g.resident_weight_bytes_per_rank, + "resident_weight_bytes_is_lower_bound": g.resident_weight_bytes_is_lower_bound, + "kv_bytes_per_token_per_sequence": g.kv_bytes_per_token_per_sequence, + "kv_fixed_bytes_per_sequence": g.kv_fixed_bytes_per_sequence, "has_unpriced_collectives": g.has_unpriced_collectives, + "has_unpriced_nodes": g.has_unpriced_nodes, + "has_unpriced_compute": g.has_unpriced_compute, + "has_unpriced_memory": g.has_unpriced_memory, "has_fallback_peaks": g.has_fallback_peaks, + "has_fallback_bytes": g.has_fallback_bytes, + "warnings": warnings, "nodes": [ { "op": n.op, @@ -472,6 +549,9 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None: "flops": n.prediction.flops, "bytes": n.prediction.bytes, "estimated": n.prediction.estimated, + "bytes_are_fallback": n.prediction.bytes_are_fallback, + "compute_is_unpriced": n.prediction.compute_is_unpriced, + "memory_is_unpriced": n.prediction.memory_is_unpriced, } for n in g.nodes ], @@ -484,10 +564,8 @@ def _emit_predicted_graph(target: discover.Target, out_dir: Path) -> None: f"(TP={sh.tp} EP={sh.ep} DP={sh.dp}, " f"w={spec.weight_dtype}/e={spec.expert_dtype}/kv={spec.kv_dtype})" ) - if g.has_unpriced_collectives: - print(" - collectives are unpriced (SKU has no interconnect bandwidth in the catalogue)") - if g.has_fallback_peaks: - print(" - priced against fallback peaks — the ceiling is low in a known direction") + for warning in warnings: + print(f" - {warning}") def describe_targets(proc: Path = discover.PROC) -> list[dict]: diff --git a/gitm/serve/metrics.py b/gitm/serve/metrics.py index 45bbb11..afd6288 100644 --- a/gitm/serve/metrics.py +++ b/gitm/serve/metrics.py @@ -20,9 +20,11 @@ from __future__ import annotations import json +import math import threading import time import urllib.request +import warnings from dataclasses import asdict, dataclass, field from pathlib import Path @@ -88,7 +90,7 @@ def parse_prometheus(text: str) -> dict[str, float]: value = float(value_str.split()[0]) except (ValueError, IndexError): continue - if value != value: # NaN: an untouched histogram, not a zero + if not math.isfinite(value): # untouched/invalid metric, not an observed zero continue out[name] = out.get(name, 0.0) + value return out @@ -170,8 +172,17 @@ def window_from_snapshots( w.ttft_mean_s = _mean(before, after, "ttft_sum", "ttft_count") w.tpot_mean_s = _mean(before, after, "tpot_sum", "tpot_count") w.e2e_mean_s = _mean(before, after, "e2e_sum", "e2e_count") - if w.generation_tokens is not None and window_s: + if ( + w.generation_tokens is not None + and window_s is not None + and math.isfinite(window_s) + and window_s > 0.0 + ): w.output_tokens_per_s = w.generation_tokens / window_s + elif window_s is not None and (not math.isfinite(window_s) or window_s <= 0.0): + w.notes.append( + f"throughput unavailable: capture window must be finite and positive, got {window_s!r}" + ) if samples: w.n_samples = len(samples) @@ -225,7 +236,7 @@ def snapshot_metrics(base_url: str, dest: Path, timeout: float = 15.0) -> str: """ text = fetch_metrics(base_url, timeout) dest.parent.mkdir(parents=True, exist_ok=True) - dest.write_text(text) + dest.write_text(text, encoding="utf-8") return text @@ -240,6 +251,13 @@ class MetricsSampler: def __init__(self, base_url: str, *, interval_s: float = 1.0) -> None: self.base_url = base_url + if not math.isfinite(interval_s) or interval_s <= 0.0: + raise ValueError(f"metrics sampling interval must be finite and positive, got {interval_s!r}") + self.diagnostics: list[str] = [] + if interval_s < 0.05: + self._record_failure( + f"requested {interval_s!r}s; using the 0.05s minimum sampling interval" + ) self.interval_s = max(interval_s, 0.05) self.samples: list[dict] = [] self._stop = threading.Event() @@ -254,6 +272,8 @@ def start(self) -> None: def _run(self) -> None: while not self._stop.is_set(): text = fetch_metrics(self.base_url, timeout=min(self.interval_s * 4, 10.0)) + if text.startswith("# unavailable:"): + self._record_failure(text.removeprefix("# unavailable:").strip()) snap = parse_prometheus(text) if snap: self.samples.append( @@ -270,9 +290,18 @@ def stop(self) -> list[dict]: self._stop.set() if self._thread is not None: self._thread.join(timeout=5.0) + if self._thread.is_alive(): + self._record_failure("metrics sampler did not stop within 5 seconds") self._thread = None return self.samples + def _record_failure(self, detail: str) -> None: + message = f"metrics sampling degraded: {detail}" + if message in self.diagnostics: + return + self.diagnostics.append(message) + warnings.warn(message, RuntimeWarning, stacklevel=2) + def write(self, dest: Path) -> None: dest.parent.mkdir(parents=True, exist_ok=True) with dest.open("w", encoding="utf-8") as fh: diff --git a/gitm/serve/model_config.py b/gitm/serve/model_config.py index 3893bfd..95e27f1 100644 --- a/gitm/serve/model_config.py +++ b/gitm/serve/model_config.py @@ -33,7 +33,7 @@ from pathlib import Path from typing import Any -from gitm.planner.moe_graph import is_sparse_moe_config, spec_from_hf_config +from gitm.planner.moe_graph import spec_from_hf_config from gitm.planner.roofline import ( _WEIGHT_BYTES, BatchConfig, @@ -209,6 +209,22 @@ def _first_present(cfg: dict[str, Any], keys: tuple[str, ...]) -> Any: return None +def is_sparse_moe_config(cfg: dict[str, Any]) -> bool: + """True when any routed-expert shape field makes this a sparse candidate. + + This deliberately recognizes partial configs. A config that declares an + expert count but omits top-k must reach :func:`validate_moe_config` and be + refused, never fall through to the dense graph because it was incomplete. + """ + # ``intermediate_size`` is also the standard dense-MLP field, so only the + # explicitly MoE spelling is a sparse signal on that axis. + return ( + _first_present(cfg, _EXPERT_COUNT_ALIASES) is not None + or _first_present(cfg, _EXPERT_TOPK_ALIASES) is not None + or cfg.get("moe_intermediate_size") is not None + ) + + def validate_moe_config(cfg: dict[str, Any]) -> list[str]: """Dominant-term fields this config fails to declare usably. Empty == usable. @@ -219,16 +235,77 @@ def validate_moe_config(cfg: dict[str, Any]) -> list[str]: supported config would carry. """ missing: list[str] = [] - if _first_present(cfg, _EXPERT_COUNT_ALIASES) is None: - missing.append(" | ".join(_EXPERT_COUNT_ALIASES) + " (routed expert count)") - if _first_present(cfg, _EXPERT_TOPK_ALIASES) is None: - missing.append(" | ".join(_EXPERT_TOPK_ALIASES) + " (experts per token)") - if _first_present(cfg, _EXPERT_INTER_ALIASES) is None: - missing.append(" | ".join(_EXPERT_INTER_ALIASES) + " (expert intermediate size)") + + def positive_alias(keys: tuple[str, ...], label: str) -> int | None: + value = _first_present(cfg, keys) + joined = " | ".join(keys) + if value is None: + missing.append(f"{joined} ({label})") + return None + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + missing.append(f"{joined} ({label}) must be a positive integer, got {value!r}") + return None + return value + + n_experts = positive_alias(_EXPERT_COUNT_ALIASES, "routed expert count") + top_k = positive_alias(_EXPERT_TOPK_ALIASES, "experts per token") + positive_alias(_EXPERT_INTER_ALIASES, "expert intermediate size") + if n_experts is not None and top_k is not None and top_k > n_experts: + missing.append( + f"experts per token {top_k} exceeds routed expert count {n_experts}" + ) + + # The sparse graph is a V4-shaped attention model, not a generic MoE graph. + # Every field below changes a node's compute/bytes; defaulting any of them to + # V4 values would fabricate a plausible graph for a partial or foreign model. + for key in ( + "hidden_size", + "num_hidden_layers", + "num_attention_heads", + "num_key_value_heads", + "head_dim", + "q_lora_rank", + "o_lora_rank", + "o_groups", + "vocab_size", + "index_n_heads", + "index_head_dim", + "index_topk", + "sliding_window", + ): + value = cfg.get(key) + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + missing.append(f"{key} must be a declared positive integer, got {value!r}") + qk_rope = cfg.get("qk_rope_head_dim") + if isinstance(qk_rope, bool) or not isinstance(qk_rope, int) or qk_rope < 0: + missing.append( + f"qk_rope_head_dim must be a declared non-negative integer, got {qk_rope!r}" + ) + shared = cfg.get("n_shared_experts") + if isinstance(shared, bool) or not isinstance(shared, int) or shared < 0: + missing.append( + f"n_shared_experts must be a declared non-negative integer, got {shared!r}" + ) + if cfg.get("torch_dtype") is None: + missing.append("torch_dtype must be declared; activation width cannot be guessed") + + n_layers = cfg.get("num_hidden_layers") + ratios = cfg.get("compress_ratios") + if not isinstance(ratios, list | tuple): + missing.append("compress_ratios must be declared for the sparse-attention graph") + elif isinstance(n_layers, int) and n_layers > 0 and len(ratios) < n_layers: + missing.append( + f"compress_ratios has {len(ratios)} entries; need at least {n_layers} for {n_layers} layers" + ) + elif any(isinstance(r, bool) or not isinstance(r, int) or r < 0 for r in ratios): + missing.append("compress_ratios must contain non-negative integers") # Quantisation: a *declared* method must be one the roofline can price. No # ``quantization_config`` is fine — that is an unquantised (bf16) checkpoint. q = cfg.get("quantization_config") or {} + if not isinstance(q, dict): + missing.append("quantization_config must be an object when declared") + q = {} method = q.get("quant_method") if method is not None and str(method).lower() not in KNOWN_DTYPES: missing.append( @@ -236,11 +313,28 @@ def validate_moe_config(cfg: dict[str, Any]) -> list[str]: f"(not priceable; known: {', '.join(sorted(KNOWN_DTYPES))})" ) expert_dtype = cfg.get("expert_dtype") - if expert_dtype is not None and str(expert_dtype).lower() not in KNOWN_DTYPES: + if expert_dtype is None: + missing.append( + "expert_dtype must be declared; routed and shared expert byte width cannot be guessed" + ) + elif str(expert_dtype).lower() not in KNOWN_DTYPES: missing.append(f"expert_dtype={expert_dtype!r} (not priceable)") return missing +def validate_priceable_dtypes(spec: SparseMoEModelSpec) -> list[str]: + """Final resolved dtypes that would make byte-width pricing fall back.""" + missing: list[str] = [] + for field_name in ("weight_dtype", "expert_dtype", "kv_dtype", "act_dtype"): + value = getattr(spec, field_name) + if str(value).lower() not in KNOWN_DTYPES: + missing.append( + f"{field_name}={value!r} (not priceable; known: " + f"{', '.join(sorted(KNOWN_DTYPES))})" + ) + return missing + + def normalize_moe_config(cfg: dict[str, Any]) -> dict[str, Any]: """Copy of ``cfg`` with alias keys mapped onto the canonical DeepSeek names. @@ -272,6 +366,7 @@ class LiveSpec: source_path: Path model_ref: str applied_overrides: dict[str, Any] = field(default_factory=dict) + warnings: list[str] = field(default_factory=list) ok: bool = True @@ -310,6 +405,7 @@ def live_moe_spec( supplied (tests). Returns :class:`LiveSpec` on success or :class:`LiveSpecError` with the reason — never a defaulted spec, so a report can only ever show a prediction against the model that is genuinely loaded. + """ model_ref = model_ref_from_cmdline(target.cmdline) if not model_ref: @@ -340,6 +436,9 @@ def live_moe_spec( # path. The dense planner is the right tool for those, so decline rather than # emit a confident wrong floor. if not is_sparse_moe_config(cfg): + missing = validate_moe_config(cfg) + if not missing: + missing = ["index_topk or compress_ratios"] return LiveSpecError( reason=( f"{model_ref!r} at {cfg_path} is not a DeepSeek-V4-class sparse-MoE model " @@ -347,6 +446,7 @@ def live_moe_spec( "predict_moe_graph models." ), model_ref=model_ref, + missing_keys=missing, ) missing = validate_moe_config(cfg) @@ -368,15 +468,47 @@ def live_moe_spec( spec_changes["kv_dtype"] = overrides["kv_dtype"] if "act_dtype" in overrides: spec_changes["act_dtype"] = overrides["act_dtype"] + if "kv_dtype" not in overrides: + # vLLM's absent --kv-cache-dtype means ``auto``: follow the resolved + # compute dtype, not a model-family-specific fp8 guess. + spec_changes["kv_dtype"] = overrides["act_dtype"] if spec_changes: from dataclasses import replace spec = replace(spec, **spec_changes) + unpriceable = validate_priceable_dtypes(spec) + if unpriceable: + return LiveSpecError( + reason=( + f"{model_ref!r} at {cfg_path} resolves to dtypes this planner cannot " + "price without substituting bf16 byte widths." + ), + model_ref=model_ref, + missing_keys=unpriceable, + ) + sharding = ShardingConfig( tp=overrides.get("tp", 1), ep=overrides.get("ep", 1), dp=overrides.get("dp", 1) ) batch = BatchConfig(kv_cache_len=overrides.get("max_model_len", default_kv_cache_len)) + warnings: list[str] = [] + if cfg.get("expert_dtype") is None: + warnings.append( + "expert_dtype absent; inherited " + f"weight_dtype={spec.weight_dtype!r} for routed and shared expert weights" + ) + warnings.append("decode batch was not observed; using batch=1 single-sequence floor") + if "max_model_len" not in overrides: + warnings.append( + f"KV cache length was not declared on the launch command; using " + f"kv_cache_len={default_kv_cache_len}" + ) + if overrides.get("ep_enabled") and overrides.get("dp", 1) > 1: + warnings.append( + "expert parallelism with data_parallel_size>1 is approximated over the " + "tensor-parallel group" + ) return LiveSpec( spec=spec, @@ -385,4 +517,5 @@ def live_moe_spec( source_path=cfg_path, model_ref=model_ref, applied_overrides=overrides, + warnings=warnings, ) diff --git a/gitm/serve/vllm.py b/gitm/serve/vllm.py index e46007d..7d5df4c 100644 --- a/gitm/serve/vllm.py +++ b/gitm/serve/vllm.py @@ -47,6 +47,7 @@ import time import urllib.error import urllib.request +import warnings from concurrent.futures import ThreadPoolExecutor from dataclasses import asdict, dataclass, field from pathlib import Path @@ -252,6 +253,17 @@ def check_driver_stack() -> list[Check]: problems = cuda_env.check() if problems: return [Check("cuda-stack", "fail", "\n".join(str(p) for p in problems))] + unverified = cuda_env.unverified_cuda_components() + if unverified: + return [ + Check( + "cuda-stack", + "warn", + f"driver CUDA {driver[0]}.{driver[1]} detected, but " + f"{' and '.join(unverified)} could not be verified; " + "stack compatibility is unknown", + ) + ] return [Check("cuda-stack", "pass", f"driver CUDA {driver[0]}.{driver[1]}, stack consistent")] @@ -424,7 +436,13 @@ def served_model_name(base: str, fallback: str) -> str: with urllib.request.urlopen(base + "/v1/models", timeout=10) as r: data = json.loads(r.read()) return data["data"][0]["id"] - except Exception: + except Exception as exc: + warnings.warn( + f"served-model discovery failed at {base}/v1/models; using configured " + f"fallback {fallback!r} ({type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) return fallback @@ -544,6 +562,7 @@ def one_request(base: str, model: str, prompt: str, max_tokens: int, # usage is authoritative; the chunk count is the fallback when a build doesn't # emit it, and it undercounts whenever a chunk carries several tokens. rec.n_output_tokens = usage_tokens if usage_tokens is not None else chunks + rec.token_count_source = "usage" if usage_tokens is not None else "chunks" return rec @@ -759,8 +778,11 @@ def launch_and_capture(args, serve_argv: list[str] | None = None): print(f"\n==> {len(records)} ok / {failures} failed in {wall:.1f}s") if summary.ttft_p50_s is not None: + tpot = f"{summary.tpot_p50_s * 1e3:.1f} ms" if summary.tpot_p50_s is not None else "n/a" print(f" TTFT p50/p95 {summary.ttft_p50_s * 1e3:.0f}/{summary.ttft_p95_s * 1e3:.0f} ms" - f" TPOT p50 {(summary.tpot_p50_s or 0) * 1e3:.1f} ms") + f" TPOT p50 {tpot}") + for warning in summary.warnings: + print(f" WARN: {warning}") print_result(result) if result.status == "no_kernels": diff --git a/gitm/telemetry/backends/discover.py b/gitm/telemetry/backends/discover.py index d5c8ab5..eecb8af 100644 --- a/gitm/telemetry/backends/discover.py +++ b/gitm/telemetry/backends/discover.py @@ -6,38 +6,59 @@ from __future__ import annotations +import warnings + from gitm.telemetry.backends.base import Backend -def discover_backends() -> list[Backend]: +def discover_backends(*, diagnostics: list[str] | None = None) -> list[Backend]: """Return all live vendor backends in discovery order. - Each candidate is constructed in a try/except — if its vendor library is - missing or no devices are present, it is silently skipped. The result is - deterministic given the host. + A missing optional vendor library or a valid zero-device backend is expected + and stays quiet. Unexpected import, initialization, count, or cleanup failures + are appended to ``diagnostics``; direct callers that do not supply a list get + a runtime warning instead. """ found: list[Backend] = [] - try: + def record(vendor: str, detail: str) -> None: + message = f"{vendor} telemetry discovery failed: {detail}" + if diagnostics is not None: + diagnostics.append(message) + else: + warnings.warn(message, RuntimeWarning, stacklevel=3) + + def attempt(vendor: str, factory) -> None: + backend: Backend | None = None + try: + backend = factory() + count = backend.device_count() + if isinstance(count, bool) or not isinstance(count, int) or count < 0: + raise ValueError(f"device_count returned invalid value {count!r}") + if count > 0: + found.append(backend) + backend = None + except ImportError: + return + except Exception as exc: + record(vendor, f"{type(exc).__name__}: {exc}") + finally: + if backend is not None: + try: + backend.close() + except Exception as exc: + record(vendor, f"backend close failed ({type(exc).__name__}: {exc})") + + def nvidia(): from gitm.telemetry.backends.nvidia import NvidiaBackend - nv = NvidiaBackend() - if nv.device_count() > 0: - found.append(nv) - else: - nv.close() - except Exception: - pass + return NvidiaBackend() - try: + def amd(): from gitm.telemetry.backends.amd import AmdBackend - amd = AmdBackend() - if amd.device_count() > 0: - found.append(amd) - else: - amd.close() - except Exception: - pass + return AmdBackend() + attempt("nvidia", nvidia) + attempt("amd", amd) return found diff --git a/gitm/telemetry/backends/nvidia.py b/gitm/telemetry/backends/nvidia.py index 46e5652..c90f18d 100644 --- a/gitm/telemetry/backends/nvidia.py +++ b/gitm/telemetry/backends/nvidia.py @@ -32,14 +32,22 @@ } -def _try(fn: Callable[[], T], default: T | None = None) -> T | None: +def _try( + fn: Callable[[], T], + default: T | None = None, + *, + diagnostics: list[str] | None = None, + field: str = "NVML field", +) -> T | None: """Call ``fn``; return ``default`` on any exception. Used per NVML call so a single failure doesn't drop the whole sample. """ try: return fn() - except Exception: + except Exception as exc: + if diagnostics is not None: + diagnostics.append(f"{field} unavailable ({type(exc).__name__}: {exc})") return default @@ -85,33 +93,68 @@ def device_count(self) -> int: def sample(self, gpu_index: int, labels: WorkloadLabels | None = None) -> Sample: nv = self._pynvml h = self._handle(gpu_index) + diagnostics: list[str] = [] - uuid = _try(lambda: nv.nvmlDeviceGetUUID(h)) + uuid = _try(lambda: nv.nvmlDeviceGetUUID(h), diagnostics=diagnostics, field="GPU UUID") if isinstance(uuid, bytes): uuid = uuid.decode() - util = _try(lambda: nv.nvmlDeviceGetUtilizationRates(h)) - mem = _try(lambda: nv.nvmlDeviceGetMemoryInfo(h)) - power_mw = _try(lambda: nv.nvmlDeviceGetPowerUsage(h)) - temp = _try(lambda: nv.nvmlDeviceGetTemperature(h, nv.NVML_TEMPERATURE_GPU)) - sm_clock = _try(lambda: nv.nvmlDeviceGetClockInfo(h, nv.NVML_CLOCK_SM)) - mem_clock = _try(lambda: nv.nvmlDeviceGetClockInfo(h, nv.NVML_CLOCK_MEM)) - throttle_bits = _try(lambda: nv.nvmlDeviceGetCurrentClocksThrottleReasons(h), 0) or 0 + util = _try( + lambda: nv.nvmlDeviceGetUtilizationRates(h), + diagnostics=diagnostics, + field="GPU utilization", + ) + mem = _try( + lambda: nv.nvmlDeviceGetMemoryInfo(h), diagnostics=diagnostics, field="GPU memory" + ) + power_mw = _try( + lambda: nv.nvmlDeviceGetPowerUsage(h), diagnostics=diagnostics, field="GPU power" + ) + temp = _try( + lambda: nv.nvmlDeviceGetTemperature(h, nv.NVML_TEMPERATURE_GPU), + diagnostics=diagnostics, + field="GPU temperature", + ) + sm_clock = _try( + lambda: nv.nvmlDeviceGetClockInfo(h, nv.NVML_CLOCK_SM), + diagnostics=diagnostics, + field="SM clock", + ) + mem_clock = _try( + lambda: nv.nvmlDeviceGetClockInfo(h, nv.NVML_CLOCK_MEM), + diagnostics=diagnostics, + field="memory clock", + ) + throttle_bits = _try( + lambda: nv.nvmlDeviceGetCurrentClocksThrottleReasons(h), + 0, + diagnostics=diagnostics, + field="clock throttle reasons", + ) or 0 per_proc: dict[int, float] = {} - procs = _try(lambda: nv.nvmlDeviceGetComputeRunningProcesses(h), []) or [] + procs = _try( + lambda: nv.nvmlDeviceGetComputeRunningProcesses(h), + [], + diagnostics=diagnostics, + field="compute process list", + ) or [] for p in procs: per_proc[int(p.pid)] = 0.0 # NVML doesn't expose per-process util directly ecc_sbe = _try( lambda: nv.nvmlDeviceGetTotalEccErrors( h, nv.NVML_MEMORY_ERROR_TYPE_CORRECTED, nv.NVML_VOLATILE_ECC - ) + ), + diagnostics=diagnostics, + field="corrected ECC counter", ) ecc_dbe = _try( lambda: nv.nvmlDeviceGetTotalEccErrors( h, nv.NVML_MEMORY_ERROR_TYPE_UNCORRECTED, nv.NVML_VOLATILE_ECC - ) + ), + diagnostics=diagnostics, + field="uncorrected ECC counter", ) return Sample( @@ -132,11 +175,9 @@ def sample(self, gpu_index: int, labels: WorkloadLabels | None = None) -> Sample per_process=per_proc, ecc_volatile_sbe=int(ecc_sbe) if ecc_sbe is not None else None, ecc_volatile_dbe=int(ecc_dbe) if ecc_dbe is not None else None, + diagnostics=diagnostics, labels=labels, ) def close(self) -> None: - try: - self._pynvml.nvmlShutdown() - except Exception: - pass + self._pynvml.nvmlShutdown() diff --git a/gitm/telemetry/collector.py b/gitm/telemetry/collector.py index 4af45f1..72460ec 100644 --- a/gitm/telemetry/collector.py +++ b/gitm/telemetry/collector.py @@ -7,6 +7,7 @@ import threading import time +import warnings from dataclasses import dataclass, field from gitm.telemetry.backends import Backend, discover_backends @@ -33,7 +34,18 @@ class Collector: def __init__(self, cfg: CollectorConfig) -> None: self._cfg = cfg - self._backends: list[Backend] = cfg.backends if cfg.backends is not None else discover_backends() + self.diagnostics: list[str] = [] + self._diagnostic_keys: set[str] = set() + discovery_diagnostics: list[str] = [] + self._backends: list[Backend] = ( + cfg.backends + if cfg.backends is not None + else discover_backends(diagnostics=discovery_diagnostics) + ) + for i, diagnostic in enumerate(discovery_diagnostics): + self._record_failure(f"backend-discovery:{i}", diagnostic) + if not self._backends and not discovery_diagnostics: + self._record_failure("backend-discovery", "no live GPU telemetry backend found") self._stop = threading.Event() self._thread: threading.Thread | None = None @@ -50,13 +62,22 @@ def stop(self) -> None: for s in self._cfg.sinks: try: s.close() - except Exception: - pass + except Exception as exc: + self._record_failure(type(s).__name__, f"sink close failed: {exc}") for b in self._backends: try: b.close() - except Exception: - pass + except Exception as exc: + self._record_failure(type(b).__name__, f"backend close failed: {exc}") + + def _record_failure(self, key: str, detail: str) -> None: + """Record and warn once per failing component/path.""" + if key in self._diagnostic_keys: + return + self._diagnostic_keys.add(key) + message = f"telemetry degraded [{key}]: {detail}" + self.diagnostics.append(message) + warnings.warn(message, RuntimeWarning, stacklevel=2) def __enter__(self) -> Collector: self.start() @@ -72,12 +93,25 @@ def _run(self) -> None: for idx in range(backend.device_count()): try: sample = backend.sample(idx, labels=self._cfg.labels) - except Exception: + except Exception as exc: + self._record_failure( + f"sample:{type(backend).__name__}:{idx}", + f"sample failed ({type(exc).__name__}: {exc})", + ) continue + for diagnostic in sample.diagnostics: + self._record_failure( + f"sample-field:{type(backend).__name__}:{idx}:{diagnostic.split(' ', 1)[0]}", + diagnostic, + ) for sink in self._cfg.sinks: try: sink.emit(sample) - except Exception: + except Exception as exc: + self._record_failure( + f"sink:{type(sink).__name__}", + f"emit failed ({type(exc).__name__}: {exc})", + ) continue next_tick += self._cfg.interval_s sleep_for = max(0.0, next_tick - time.monotonic()) diff --git a/gitm/telemetry/schema.py b/gitm/telemetry/schema.py index ad2a2d9..acd5088 100644 --- a/gitm/telemetry/schema.py +++ b/gitm/telemetry/schema.py @@ -75,4 +75,8 @@ class Sample(BaseModel): # keep working unchanged. extra: dict[str, float] = Field(default_factory=dict) + # Per-field backend failures. A partial sample remains usable, but consumers + # must not read a missing throttle/process field as an observed zero. + diagnostics: list[str] = Field(default_factory=list) + labels: WorkloadLabels | None = None diff --git a/gitm/telemetry/sinks/jsonl.py b/gitm/telemetry/sinks/jsonl.py index 0281c5d..7635e8e 100644 --- a/gitm/telemetry/sinks/jsonl.py +++ b/gitm/telemetry/sinks/jsonl.py @@ -26,8 +26,7 @@ def emit(self, sample: Sample) -> None: def close(self) -> None: with self._lock: - try: - self._fh.flush() - self._fh.close() - except Exception: - pass + # Let the collector record and warn on finalization failures. Swallowing + # one here makes a truncated telemetry artifact look complete. + self._fh.flush() + self._fh.close() diff --git a/gitm/telemetry/sinks/otlp.py b/gitm/telemetry/sinks/otlp.py index b5a4f7c..d346676 100644 --- a/gitm/telemetry/sinks/otlp.py +++ b/gitm/telemetry/sinks/otlp.py @@ -48,7 +48,6 @@ def emit(self, sample: Sample) -> None: self._temp.set(sample.temp_c, attributes=attrs) def close(self) -> None: - try: - self._provider.shutdown() - except Exception: - pass + # Collector.stop owns the fail-open boundary and turns this into a named + # diagnostic. Hiding it here would make dropped final exports invisible. + self._provider.shutdown() diff --git a/gitm/tracer/_cupti_decode.py b/gitm/tracer/_cupti_decode.py index da63a5c..41e1c66 100644 --- a/gitm/tracer/_cupti_decode.py +++ b/gitm/tracer/_cupti_decode.py @@ -24,6 +24,7 @@ from __future__ import annotations +import warnings from typing import Literal from gitm.tracer.nvtx_correlate import correlate_kernels_to_ranges @@ -59,15 +60,36 @@ def decode_kernel(d: dict) -> KernelEvent: - grid = d.get("grid", [1, 1, 1]) - block = d.get("block", [1, 1, 1]) + grid = d.get("grid") + if grid is None: + warnings.warn( + "CUPTI kernel grid dimensions unavailable; using 1x1x1", + RuntimeWarning, + stacklevel=2, + ) + grid = [1, 1, 1] + block = d.get("block") + if block is None: + warnings.warn( + "CUPTI kernel block dimensions unavailable; using 1x1x1", + RuntimeWarning, + stacklevel=2, + ) + block = [1, 1, 1] + name = d.get("name") + if not name: + warnings.warn( + "CUPTI kernel name unavailable; using ", + RuntimeWarning, + stacklevel=2, + ) return KernelEvent( start_ns=int(d["start_ns"]), end_ns=int(d["end_ns"]), stream_id=int(d["stream_id"]), device_id=int(d["device_id"]), correlation_id=_opt_int(d.get("correlation_id")), - name=d.get("name") or "", + name=name or "", grid_x=int(grid[0]), grid_y=int(grid[1]), grid_z=int(grid[2]), block_x=int(block[0]), block_y=int(block[1]), block_z=int(block[2]), shared_mem_bytes=int(d.get("static_shared_mem", 0)) + int(d.get("dynamic_shared_mem", 0)), @@ -78,7 +100,15 @@ def decode_kernel(d: dict) -> KernelEvent: def decode_memcpy(d: dict) -> MemcpyEvent: - src, dst = _COPY_KIND.get(int(d.get("copy_kind", 0)), ("device", "device")) + raw_kind = d.get("copy_kind") + kind = int(raw_kind) if raw_kind is not None else 0 + if kind not in _COPY_KIND or kind == 0: + warnings.warn( + f"unknown CUPTI memcpy kind {kind}; using device↔device endpoint fallback", + RuntimeWarning, + stacklevel=2, + ) + src, dst = _COPY_KIND.get(kind, ("device", "device")) return MemcpyEvent( start_ns=int(d["start_ns"]), end_ns=int(d["end_ns"]), @@ -92,13 +122,21 @@ def decode_memcpy(d: dict) -> MemcpyEvent: def decode_sync(d: dict) -> SyncEvent: + raw_type = d.get("sync_type") + sync_type = int(raw_type) if raw_type is not None else 0 + if sync_type not in _SYNC_KIND or sync_type == 0: + warnings.warn( + f"unknown CUPTI synchronization type {sync_type}; using device-sync fallback", + RuntimeWarning, + stacklevel=2, + ) return SyncEvent( start_ns=int(d["start_ns"]), end_ns=int(d["end_ns"]), stream_id=int(d.get("stream_id", 0)), device_id=int(d.get("device_id", 0)), correlation_id=_opt_int(d.get("correlation_id")), - sync_kind=_SYNC_KIND.get(int(d.get("sync_type", 0)), "device"), + sync_kind=_SYNC_KIND.get(sync_type, "device"), ) diff --git a/gitm/tracer/capture.py b/gitm/tracer/capture.py index a779186..94f76a6 100644 --- a/gitm/tracer/capture.py +++ b/gitm/tracer/capture.py @@ -187,12 +187,39 @@ def _device_count(backend, injected: bool) -> int: shim = load_shim() if shim is None: + warnings.warn( + "injected trace device count unavailable: CUPTI shim is not importable; " + "recording 0 devices", + RuntimeWarning, + stacklevel=2, + ) return 0 - try: - return int(shim.device_count()) - except Exception: - return 0 - return backend.device_count() if backend else 0 + counter = shim.device_count + elif backend is not None: + counter = backend.device_count + else: + # ``source=none`` in the trace header is the explicit provenance for the + # normal no-backend path; an extra warning here would add no information. + return 0 + + try: + count = int(counter()) + except Exception as exc: + warnings.warn( + f"active trace device count unavailable: {type(exc).__name__}: {exc}; " + "recording 0 devices", + RuntimeWarning, + stacklevel=2, + ) + return 0 + if count <= 0: + warnings.warn( + f"active trace device-count probe reported {count} devices; recording 0", + RuntimeWarning, + stacklevel=2, + ) + return 0 + return count def _backend(): diff --git a/gitm/tracer/cupti.py b/gitm/tracer/cupti.py index cdeebf7..bfdd38f 100644 --- a/gitm/tracer/cupti.py +++ b/gitm/tracer/cupti.py @@ -33,10 +33,7 @@ def __init__(self) -> None: ) def device_count(self) -> int: - try: - return int(self._shim.device_count()) - except Exception: - return 0 + return int(self._shim.device_count()) def start(self) -> None: self._shim.start() diff --git a/gitm/tracer/injection.py b/gitm/tracer/injection.py index 5610c7b..016837d 100644 --- a/gitm/tracer/injection.py +++ b/gitm/tracer/injection.py @@ -24,8 +24,10 @@ from __future__ import annotations import json +import math import os import time +import warnings from pathlib import Path from gitm.tracer.schema import TraceEvent @@ -110,6 +112,22 @@ def _pid_alive(pid: int) -> bool: prevent. Guessing "dead" costs a silently empty trace; guessing "alive" costs a stale file. """ + if os.name == "nt": + # ``os.kill(pid, 0)`` is not a harmless existence probe on Windows: the + # CRT maps signal 0 through TerminateProcess, which can kill the very + # EngineCore whose live shard we are trying to protect. Query a process + # handle instead; access-denied still means the process exists. + import ctypes + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + handle = kernel32.OpenProcess(0x1000, False, pid) # PROCESS_QUERY_LIMITED_INFORMATION + if handle: + kernel32.CloseHandle(handle) + return True + error = ctypes.get_last_error() + if error == 87: # ERROR_INVALID_PARAMETER: no such PID + return False + return True try: os.kill(pid, 0) except ProcessLookupError: @@ -146,10 +164,21 @@ def clear_stale_shards() -> None: def settle_seconds() -> float: raw = os.environ.get(ENV_SETTLE) + if not raw: + return DEFAULT_SETTLE_S try: - return float(raw) if raw else DEFAULT_SETTLE_S + value = float(raw) except ValueError: + value = float("nan") + if not math.isfinite(value) or value < 0: + warnings.warn( + f"invalid GITM_TRACE_SETTLE_S={raw!r}; using documented default " + f"{DEFAULT_SETTLE_S}s", + RuntimeWarning, + stacklevel=2, + ) return DEFAULT_SETTLE_S + return value def settle() -> None: @@ -171,10 +200,17 @@ def read_shards(start_ns: int | None = None, end_ns: int | None = None) -> list[ from gitm.tracer._cupti_decode import decode_records records: list[dict] = [] + dropped_lines = 0 for shard in shard_paths(): try: text = shard.read_text(encoding="utf-8", errors="replace") - except OSError: + except OSError as exc: + warnings.warn( + f"injected trace shard unreadable ({shard}: {type(exc).__name__}: {exc}); " + "capture coverage is incomplete", + RuntimeWarning, + stacklevel=2, + ) continue for line in text.splitlines(): line = line.strip() @@ -183,11 +219,14 @@ def read_shards(start_ns: int | None = None, end_ns: int | None = None) -> list[ try: rec = json.loads(line) except json.JSONDecodeError: + dropped_lines += 1 continue # partial line from a killed process if not isinstance(rec, dict): + dropped_lines += 1 continue ts = rec.get("start_ns") if not isinstance(ts, int): + dropped_lines += 1 continue if start_ns is not None and ts < start_ns: continue @@ -195,7 +234,22 @@ def read_shards(start_ns: int | None = None, end_ns: int | None = None) -> list[ continue records.append(rec) - return decode_records(records) + if dropped_lines: + warnings.warn( + f"injected trace coverage: dropped {dropped_lines} malformed or incomplete " + "shard line(s)", + RuntimeWarning, + stacklevel=2, + ) + events = decode_records(records) + if len(events) < len(records): + warnings.warn( + f"injected trace coverage: dropped {len(records) - len(events)} unmodeled " + "activity record(s)", + RuntimeWarning, + stacklevel=2, + ) + return events def cupti_now() -> int | None: diff --git a/gitm/tracer/kernel_taxonomy.py b/gitm/tracer/kernel_taxonomy.py index 09ed71a..c36997b 100644 --- a/gitm/tracer/kernel_taxonomy.py +++ b/gitm/tracer/kernel_taxonomy.py @@ -127,6 +127,7 @@ class KernelBreakdown: n_truncated_names: int # records whose name hit NAME_MAX n_distinct_truncated: int # distinct such names — collisions hide here n_devices: int + n_invalid_duration: int = 0 @property def other_share(self) -> float: @@ -141,11 +142,31 @@ def warnings(self) -> list[str]: out: list[str] = [] if self.n_kernels == 0: out.append( - "no kernels captured at all. The usual cause is decode running as " + "no positive-duration kernels captured. The usual cause is decode running as " "CUDA-graph replay that this CUPTI does not attribute — re-run with " "--enforce-eager to confirm." ) + if self.n_invalid_duration: + out.append( + f"{self.n_invalid_duration} kernel record(s) had non-positive duration " + "and were excluded as invalid." + ) return out + if self.n_invalid_duration: + out.append( + f"{self.n_invalid_duration} kernel record(s) had non-positive duration " + "and were excluded from all shares." + ) + if self.window_ns is not None and self.window_ns <= 0: + out.append( + f"capture window duration is non-positive ({self.window_ns} ns); " + "GPU-active share is unavailable." + ) + if self.gpu_active_share is not None and self.gpu_active_share > 1.0: + out.append( + f"GPU active share is {self.gpu_active_share:.1%}, above the capture window; " + "event/window clocks are inconsistent." + ) if self.gpu_active_share is not None and self.gpu_active_share < 0.2: out.append( f"GPU active only {self.gpu_active_share:.1%} of the window. Either the " @@ -210,11 +231,15 @@ def summarize_kernels(kernels, *, window_ns: int | None = None, per_device: dict[int, list[tuple[int, int]]] = {} truncated: list[str] = [] total_time = 0 + invalid_duration = 0 for k in kernels: name = getattr(k, "name", "") or "" start, end = int(k.start_ns), int(k.end_ns) - dur = max(end - start, 0) + dur = end - start + if dur <= 0: + invalid_duration += 1 + continue total_time += dur by_bucket.setdefault(classify_kernel(name), []).append((name, dur)) per_device.setdefault(int(getattr(k, "device_id", 0) or 0), []).append((start, end)) @@ -237,7 +262,11 @@ def summarize_kernels(kernels, *, window_ns: int | None = None, buckets.sort(key=lambda b: -b.time_ns) active = {dev: _active_ns(iv) for dev, iv in per_device.items()} - share = (max(active.values()) / window_ns) if (active and window_ns) else None + share = ( + max(active.values()) / window_ns + if active and window_ns is not None and window_ns > 0 + else None + ) return KernelBreakdown( n_kernels=sum(b.n_kernels for b in buckets), @@ -249,6 +278,7 @@ def summarize_kernels(kernels, *, window_ns: int | None = None, n_truncated_names=len(truncated), n_distinct_truncated=len(set(truncated)), n_devices=len(active), + n_invalid_duration=invalid_duration, ) diff --git a/gitm/tracer/vllm_stats.py b/gitm/tracer/vllm_stats.py index 0fc4418..7a374af 100644 --- a/gitm/tracer/vllm_stats.py +++ b/gitm/tracer/vllm_stats.py @@ -25,11 +25,13 @@ from __future__ import annotations +import math import threading import time +import warnings from collections.abc import Iterator from contextlib import contextmanager -from dataclasses import asdict, dataclass +from dataclasses import asdict, dataclass, field from typing import Any @@ -44,8 +46,8 @@ class SchedulerSample: num_unfinished: int | None = None # total in-flight requests preemptions_cumulative: int | None = None # running total of preemptions gpu_cache_usage: float | None = None # KV-cache blocks used / total, 0..1 - cpu_cache_usage: float | None = None batch_occupancy: float | None = None # num_running / max_num_seqs, 0..1 + diagnostics: list[str] = field(default_factory=list) @dataclass @@ -66,6 +68,7 @@ class SchedulerStatsSummary: # series on the same wall clock as vLLM's per-request timestamps and as # ``Trace.captured_at_ns`` — the join across the three views. 0 when unset. t0_wall_ns: int = 0 + diagnostics: list[str] = field(default_factory=list) # SLO defaults for goodput. Serving-shaped starting points, not tuned: a request @@ -89,13 +92,22 @@ class RequestRecord: first_token_wall_s: float | None = None finished_wall_s: float | None = None n_output_tokens: int = 0 + # ``authoritative`` means usage metadata or engine token_ids; ``chunks`` is + # the SSE-event fallback (one event can carry multiple tokens); ``unknown`` + # means no trustworthy denominator exists. + token_count_source: str = "unknown" + + @property + def token_count_authoritative(self) -> bool: + return self.token_count_source in {"usage", "engine"} @property def ttft_s(self) -> float | None: """Time to first token — the queue+prefill wait the user actually feels.""" if self.arrival_wall_s is None or self.first_token_wall_s is None: return None - return max(self.first_token_wall_s - self.arrival_wall_s, 0.0) + span = self.first_token_wall_s - self.arrival_wall_s + return span if math.isfinite(span) and span >= 0.0 else None @property def tpot_s(self) -> float | None: @@ -106,11 +118,27 @@ def tpot_s(self) -> float | None: inter-token interval at all and yields ``None`` rather than a number derived from a zero-length gap. """ + if not self.token_count_authoritative: + return None if self.first_token_wall_s is None or self.finished_wall_s is None: return None if self.n_output_tokens < 2: return None - span = max(self.finished_wall_s - self.first_token_wall_s, 0.0) + span = self.finished_wall_s - self.first_token_wall_s + if not math.isfinite(span) or span < 0.0: + return None + return span / (self.n_output_tokens - 1) + + @property + def estimated_tpot_s(self) -> float | None: + """Chunk-count TPOT estimate, never admitted to SLO goodput.""" + if self.token_count_source != "chunks" or self.n_output_tokens < 2: + return None + if self.first_token_wall_s is None or self.finished_wall_s is None: + return None + span = self.finished_wall_s - self.first_token_wall_s + if not math.isfinite(span) or span < 0.0: + return None return span / (self.n_output_tokens - 1) def meets_slo(self, ttft_slo_s: float, tpot_slo_s: float) -> bool: @@ -124,8 +152,14 @@ def meets_slo(self, ttft_slo_s: float, tpot_slo_s: float) -> bool: ttft = self.ttft_s if ttft is None or ttft > ttft_slo_s: return False + if not self.token_count_authoritative: + return False + if self.n_output_tokens == 1: + return True + if self.n_output_tokens < 1: + return False tpot = self.tpot_s - return tpot is None or tpot <= tpot_slo_s + return tpot is not None and tpot <= tpot_slo_s @dataclass @@ -141,17 +175,20 @@ class ServingSummary: n_requests: int n_ttft: int n_tpot: int + n_tpot_estimated: int ttft_p50_s: float | None ttft_p95_s: float | None ttft_p99_s: float | None tpot_p50_s: float | None tpot_p95_s: float | None tpot_p99_s: float | None + tpot_estimated_p50_s: float | None n_met_slo: int goodput_rps: float | None # SLO-meeting requests per second over the window window_s: float | None ttft_slo_s: float tpot_slo_s: float + warnings: list[str] def _percentile(vals: list[float], q: float) -> float | None: @@ -182,6 +219,7 @@ def request_records_from_outputs(outputs: Any) -> list[RequestRecord]: if outs: try: rec.n_output_tokens = len(outs[0].token_ids) + rec.token_count_source = "engine" except (AttributeError, TypeError, IndexError): pass metrics = getattr(o, "metrics", None) @@ -212,32 +250,77 @@ def summarize_requests( """ ttfts = [t for t in (r.ttft_s for r in records) if t is not None] tpots = [t for t in (r.tpot_s for r in records) if t is not None] + estimated_tpots = [t for t in (r.estimated_tpot_s for r in records) if t is not None] met = [r for r in records if r.meets_slo(ttft_slo_s, tpot_slo_s)] - arrivals = [r.arrival_wall_s for r in records if r.arrival_wall_s is not None] - finishes = [r.finished_wall_s for r in records if r.finished_wall_s is not None] + arrivals = [ + r.arrival_wall_s + for r in records + if r.arrival_wall_s is not None and math.isfinite(r.arrival_wall_s) + ] + finishes = [ + r.finished_wall_s + for r in records + if r.finished_wall_s is not None and math.isfinite(r.finished_wall_s) + ] window_s: float | None = None if arrivals and finishes: - window_s = max(max(finishes) - min(arrivals), 0.0) + span = max(finishes) - min(arrivals) + window_s = span if span >= 0.0 else None # A zero-length window (single instantaneous request, or clock granularity) # has no meaningful rate — report the count, not a division by ~0. goodput = (len(met) / window_s) if window_s else None + warnings: list[str] = [] + def _has_invalid_timestamps(r: RequestRecord) -> bool: + pairs = ( + (r.arrival_wall_s, r.first_token_wall_s), + (r.first_token_wall_s, r.finished_wall_s), + ) + return any( + start is not None + and end is not None + and (not math.isfinite(start) or not math.isfinite(end) or end < start) + for start, end in pairs + ) + + invalid_timestamps = sum(_has_invalid_timestamps(r) for r in records) + if invalid_timestamps: + warnings.append( + f"latency coverage: {invalid_timestamps} request(s) have non-finite or " + "non-monotonic timestamps; excluded from latency and SLO goodput" + ) + if records and arrivals and finishes and not window_s: + warnings.append("goodput unavailable: request window has no positive duration") + if estimated_tpots: + warnings.append( + f"TPOT coverage: {len(estimated_tpots)} request(s) use SSE chunk-count estimates; " + "excluded from authoritative TPOT percentiles and SLO goodput" + ) + missing_counts = sum(1 for r in records if r.token_count_source == "unknown") + if missing_counts: + warnings.append( + f"TPOT coverage: {missing_counts} request(s) have no authoritative output-token count" + ) + return ServingSummary( n_requests=len(records), n_ttft=len(ttfts), n_tpot=len(tpots), + n_tpot_estimated=len(estimated_tpots), ttft_p50_s=_percentile(ttfts, 0.50), ttft_p95_s=_percentile(ttfts, 0.95), ttft_p99_s=_percentile(ttfts, 0.99), tpot_p50_s=_percentile(tpots, 0.50), tpot_p95_s=_percentile(tpots, 0.95), tpot_p99_s=_percentile(tpots, 0.99), + tpot_estimated_p50_s=_percentile(estimated_tpots, 0.50), n_met_slo=len(met), goodput_rps=goodput, window_s=window_s, ttft_slo_s=ttft_slo_s, tpot_slo_s=tpot_slo_s, + warnings=warnings, ) @@ -286,7 +369,32 @@ def _schedulers(engine: Any) -> list[Any]: return list(sched) if isinstance(sched, list | tuple) else [sched] -def _v1_scheduler_stats(scheduler: Any) -> dict[str, Any]: +def _nonnegative_count( + value: Any, field_name: str, diagnostics: list[str] | None +) -> int | None: + if isinstance(value, int) and not isinstance(value, bool): + if value >= 0: + return value + if diagnostics is not None: + diagnostics.append(f"{field_name} reported negative count {value}") + return None + + +def _unit_fraction( + value: Any, field_name: str, diagnostics: list[str] | None +) -> float | None: + if isinstance(value, int | float) and not isinstance(value, bool): + result = float(value) + if math.isfinite(result) and 0.0 <= result <= 1.0: + return result + if diagnostics is not None: + diagnostics.append(f"{field_name} reported {value!r} outside [0, 1]") + return None + + +def _v1_scheduler_stats( + scheduler: Any, diagnostics: list[str] | None = None +) -> dict[str, Any]: """vLLM V1 stats via ``scheduler.make_stats()`` — V1 doesn't keep the V0 running/waiting deques in the same shape, but exposes a stats object with ``num_running_reqs`` / ``num_waiting_reqs`` / ``kv_cache_usage``. Best-effort; @@ -298,19 +406,23 @@ def _v1_scheduler_stats(scheduler: Any) -> dict[str, Any]: return {} try: stats = make() - except Exception: + except Exception as exc: + if diagnostics is not None: + diagnostics.append( + f"vLLM V1 scheduler make_stats failed: {type(exc).__name__}: {exc}" + ) return {} if stats is None: return {} out: dict[str, Any] = {} for field_name, attr in (("num_running", "num_running_reqs"), ("num_waiting", "num_waiting_reqs")): - v = getattr(stats, attr, None) - if isinstance(v, int): + v = _nonnegative_count(getattr(stats, attr, None), attr, diagnostics) + if v is not None: out[field_name] = v - ku = getattr(stats, "kv_cache_usage", None) - if isinstance(ku, int | float): - out["gpu_cache_usage"] = float(ku) + ku = _unit_fraction(getattr(stats, "kv_cache_usage", None), "kv_cache_usage", diagnostics) + if ku is not None: + out["gpu_cache_usage"] = ku return out @@ -352,7 +464,11 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non ("preemptions_cumulative", "num_cumulative_preemption", False), ): raw = getattr(sch, attr, None) - val = _len_or_none(raw) if is_len else (raw if isinstance(raw, int) else None) + val = ( + _len_or_none(raw) + if is_len + else _nonnegative_count(raw, attr, sample.diagnostics) + ) if val is not None: totals[field_name] = totals.get(field_name, 0) + val for field_name, val in totals.items(): @@ -360,7 +476,7 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non saw_any = True # KV-cache usage off the first scheduler's block manager (best-effort, V0). - usage = _gpu_cache_usage(schedulers[0]) + usage = _gpu_cache_usage(schedulers[0], sample.diagnostics) if usage is not None: sample.gpu_cache_usage = usage saw_any = True @@ -368,15 +484,7 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non # vLLM V1: fill running / waiting / cache from the scheduler's stat object # where the VO deques weren't exposed (they read empty on V1) for sch in schedulers: - for field_name, val in _v1_scheduler_stats(sch).items(): - if getattr(sample, field_name) is None: - setattr(sample, field_name, val) - saw_any = True - - # vLLM V1: fill running / waiting / cache from the scheduler's stat object - # where the VO deques weren't exposed (they read empty on V1) - for sch in schedulers: - for field_name, val in _v1_scheduler_stats(sch).items(): + for field_name, val in _v1_scheduler_stats(sch, sample.diagnostics).items(): if getattr(sample, field_name) is None: setattr(sample, field_name, val) saw_any = True @@ -390,10 +498,16 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non ) if callable(getter): try: - sample.num_unfinished = int(getter()) - saw_any = True - except Exception: - pass + count = _nonnegative_count( + getter(), "num_unfinished", sample.diagnostics + ) + if count is not None: + sample.num_unfinished = count + saw_any = True + except Exception as exc: + sample.diagnostics.append( + f"unfinished-request probe failed: {type(exc).__name__}: {exc}" + ) if sample.num_running is not None: max_seqs = _max_num_seqs(engine) @@ -403,12 +517,20 @@ def read_scheduler_stats(engine: Any, *, t_ns: int = 0) -> SchedulerSample | Non # transient over-count (or a partially-exposed config) can never make a # half-empty engine read as full and silently suppress under_filled. capacity = max_seqs * max(len(schedulers), 1) - sample.batch_occupancy = min(1.0, sample.num_running / capacity) + if sample.num_running <= capacity: + sample.batch_occupancy = sample.num_running / capacity + else: + sample.diagnostics.append( + f"num_running {sample.num_running} exceeds declared capacity {capacity}; " + "batch occupancy unavailable" + ) - return sample if saw_any else None + return sample if saw_any or sample.diagnostics else None -def _gpu_cache_usage(scheduler: Any) -> float | None: +def _gpu_cache_usage( + scheduler: Any, diagnostics: list[str] | None = None +) -> float | None: """KV-cache block occupancy (0..1) off a scheduler's block manager, if exposed.""" bm = getattr(scheduler, "block_manager", None) if bm is None: @@ -418,8 +540,24 @@ def _gpu_cache_usage(scheduler: Any) -> float | None: total = getattr(bm, "num_total_gpu_blocks", None) if callable(free_fn) and isinstance(total, int) and total > 0: try: - return max(0.0, 1.0 - free_fn() / total) - except Exception: + free = free_fn() + if ( + isinstance(free, bool) + or not isinstance(free, int | float) + or not math.isfinite(float(free)) + or not 0 <= free <= total + ): + if diagnostics is not None: + diagnostics.append( + f"KV-cache free-block probe reported {free!r} outside [0, {total}]" + ) + return None + return 1.0 - float(free) / total + except Exception as exc: + if diagnostics is not None: + diagnostics.append( + f"KV-cache usage probe failed: {type(exc).__name__}: {exc}" + ) return None return None @@ -435,6 +573,17 @@ class SchedulerStatsSampler: def __init__(self, engine: Any, *, interval_s: float = 0.05) -> None: self.engine = engine + if not math.isfinite(interval_s) or interval_s <= 0.0: + raise ValueError( + f"scheduler sampling interval must be finite and positive, got {interval_s!r}" + ) + self.diagnostics: list[str] = [] + self._diagnostic_keys: set[str] = set() + if interval_s < 1e-3: + self._record_failure( + "interval-clamped", + f"requested {interval_s!r}s; using the 0.001s minimum", + ) self.interval_s = max(interval_s, 1e-3) self.samples: list[SchedulerSample] = [] self._stop = threading.Event() @@ -445,6 +594,9 @@ def __init__(self, engine: Any, *, interval_s: float = 0.05) -> None: def start(self) -> None: if self._thread is not None or self.engine is None: return + # A stopped sampler may be reused for a new window. Relative timestamps + # restart at zero, so old-window samples cannot remain in the same series. + self.samples.clear() # Two clocks taken together: monotonic for sample spacing (immune to wall # clock jumps), wall for the join against request timestamps and the # trace. Captured back-to-back so the offset between the two clocks is @@ -457,9 +609,14 @@ def start(self) -> None: try: s0 = read_scheduler_stats(self.engine, t_ns=0) if s0 is not None: + self._surface_sample_diagnostics(s0) self.samples.append(s0) - except Exception: - pass + else: + self._record_failure( + "unavailable", "engine exposes no recognized scheduler fields" + ) + except Exception as exc: + self._record_failure("initial-read", f"scheduler stats read failed: {exc}") self._stop.clear() self._thread = threading.Thread(target=self._run, name="gitm-vllm-stats", daemon=True) self._thread.start() @@ -469,9 +626,14 @@ def _run(self) -> None: try: s = read_scheduler_stats(self.engine, t_ns=time.perf_counter_ns() - self._t0_ns) if s is not None: + self._surface_sample_diagnostics(s) self.samples.append(s) - except Exception: - pass # best-effort; never let sampling crash the run + else: + self._record_failure( + "unavailable", "engine exposes no recognized scheduler fields" + ) + except Exception as exc: + self._record_failure("background-read", f"scheduler stats read failed: {exc}") self._stop.wait(self.interval_s) def stop(self) -> None: @@ -479,13 +641,33 @@ def stop(self) -> None: return self._stop.set() self._thread.join(timeout=2.0) + if self._thread.is_alive(): + self._record_failure("join-timeout", "scheduler sampler did not stop within 2 seconds") self._thread = None + def _record_failure(self, key: str, detail: str) -> None: + if key in self._diagnostic_keys: + return + self._diagnostic_keys.add(key) + message = f"scheduler telemetry degraded [{key}]: {detail}" + self.diagnostics.append(message) + warnings.warn(message, RuntimeWarning, stacklevel=2) + + def _surface_sample_diagnostics(self, sample: SchedulerSample) -> None: + for message in sample.diagnostics: + self._record_failure(f"field:{message.split(':', 1)[0]}", message) + def summary(self) -> SchedulerStatsSummary: # Snapshot first: stop() joins with a timeout, so in the pathological case # where the daemon thread is still alive, summarize must iterate a stable # copy rather than a list being appended to concurrently. - return summarize(list(self.samples), t0_wall_ns=self._t0_wall_ns) + result = summarize(list(self.samples), t0_wall_ns=self._t0_wall_ns) + if self.diagnostics: + # The sampler versions field diagnostics with stable deduplication keys + # and a user-facing prefix. Direct ``summarize`` callers still receive + # the raw per-sample diagnostics, while this boundary emits one copy. + result.diagnostics = list(self.diagnostics) + return result def to_records(self) -> list[dict[str, Any]]: """Samples as plain dicts, ready for JSONL alongside the kernel trace.""" @@ -512,8 +694,14 @@ def _vals(attr: str) -> list[float]: preempt = _vals("preemptions_cumulative") cache = _vals("gpu_cache_usage") swapped = _vals("num_swapped") - duration_s = max(samples[-1].t_ns - samples[0].t_ns, 0) / 1e9 - + if any(sample.t_ns < 0 for sample in samples) or any( + later.t_ns < earlier.t_ns + for earlier, later in zip(samples, samples[1:], strict=False) + ): + raise ValueError("scheduler sample timestamps are not monotonic non-negative values") + duration_s = (samples[-1].t_ns - samples[0].t_ns) / 1e9 + + diagnostics = list(dict.fromkeys(note for sample in samples for note in sample.diagnostics)) return SchedulerStatsSummary( n_samples=len(samples), duration_s=duration_s, @@ -528,6 +716,7 @@ def _vals(attr: str) -> list[float]: peak_gpu_cache_usage=max(cache) if cache else None, peak_swapped=int(max(swapped)) if swapped else None, t0_wall_ns=t0_wall_ns, + diagnostics=diagnostics, ) diff --git a/gitm/workloads.py b/gitm/workloads.py index 7fd3275..a569a4c 100644 --- a/gitm/workloads.py +++ b/gitm/workloads.py @@ -29,10 +29,13 @@ import os import socket +import warnings from collections.abc import Callable from pathlib import Path from typing import TYPE_CHECKING, Any +from gitm._timing import require_positive_duration + if TYPE_CHECKING: from gitm.scheduler.loop import LoopConfig @@ -134,8 +137,13 @@ def _free_gpu_pool() -> None: import cupy cupy.get_default_memory_pool().free_all_blocks() - except Exception: - pass + except Exception as exc: + warnings.warn( + f"GPU memory-pool cleanup unavailable; streaming may retain cached blocks " + f"({type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) def _hft_batches_factory(stage: Path, seed: int, dflib, shards_per_batch: int, @@ -193,6 +201,12 @@ def _ensure_hft_data(stage: Path, seed: int) -> None: events = int(os.environ.get("GITM_BENCH_EVENTS", "200000")) out = stage / f"hft_smoke_seed{seed}" + warnings.warn( + f"no staged HFT dataset found; generating {events:,}-event smoke data at {out}. " + "This is synthetic smoke coverage, not a publishable baseline dataset.", + RuntimeWarning, + stacklevel=2, + ) generate( GenConfig(events=events, seed=seed, events_per_file=min(events, 100_000)), out, @@ -392,7 +406,9 @@ def run() -> dict[str, Any]: t0 = time.perf_counter() for idx in run_indices: total_dets += unit.run(idx).n_detections - elapsed = max(time.perf_counter() - t0, 1e-9) + elapsed = require_positive_duration( + time.perf_counter() - t0, context="nuScenes workload" + ) return { "frames": len(run_indices), "detections": total_dets, @@ -440,7 +456,9 @@ def run() -> dict[str, Any]: t0 = time.perf_counter() for p in run_paths: total_dets += unit.run(p).n_detections - elapsed = max(time.perf_counter() - t0, 1e-9) + elapsed = require_positive_duration( + time.perf_counter() - t0, context="KITTI workload" + ) return { "frames": len(run_paths), "detections": total_dets, @@ -586,6 +604,10 @@ def _vllm_decode_factory(cfg: LoopConfig) -> WorkloadRunner: if os.environ.get("GITM_VLLM_SYNTHETIC") == "1": return _vllm_synthetic_runner(n_prompts, max_tokens) + from gitm import cuda_env + + cuda_env.require_compatible() + import time from vllm import LLM, SamplingParams @@ -670,8 +692,12 @@ def _shutdown_engine(engine: Any) -> None: if callable(obj): try: obj() - except Exception: - pass + except Exception as exc: + warnings.warn( + f"vLLM engine shutdown path {path!r} failed: {exc}", + RuntimeWarning, + stacklevel=2, + ) break try: @@ -682,22 +708,36 @@ def _shutdown_engine(engine: Any) -> None: destroy_model_parallel() destroy_distributed_environment() - except Exception: - pass + except Exception as exc: + warnings.warn( + f"vLLM distributed-state cleanup failed: {exc}", + RuntimeWarning, + stacklevel=2, + ) try: import torch.distributed as dist if dist.is_available() and dist.is_initialized(): dist.destroy_process_group() - except Exception: - pass + except Exception as exc: + warnings.warn( + f"torch distributed process-group cleanup failed: {exc}", + RuntimeWarning, + stacklevel=2, + ) for attr in ("llm_engine", "engine"): + if not hasattr(engine, attr): + continue try: delattr(engine, attr) - except Exception: - pass + except Exception as exc: + warnings.warn( + f"vLLM engine reference cleanup for {attr!r} failed: {exc}", + RuntimeWarning, + stacklevel=2, + ) try: import gc @@ -708,8 +748,12 @@ def _shutdown_engine(engine: Any) -> None: if torch.cuda.is_available(): torch.cuda.empty_cache() torch.cuda.ipc_collect() - except Exception: - pass + except Exception as exc: + warnings.warn( + f"GPU cleanup after vLLM shutdown failed: {exc}", + RuntimeWarning, + stacklevel=2, + ) def _activate_engine(engine: Any) -> None: engine_ref["engine"] = engine @@ -760,7 +804,10 @@ def _throughput(eng: Any) -> float: outs = eng.generate(prompts, params) toks = sum(len(o.outputs[0].token_ids) for o in outs) sync_device() - return toks / max(time.perf_counter() - t0, 1e-9) + elapsed = require_positive_duration( + time.perf_counter() - t0, context="vLLM throughput probe" + ) + return toks / elapsed def _restart(_old_engine: Any, knob_values: dict[str, Any]) -> Any: """Rebuild a fresh vLLM engine with one or more structural knobs changed. @@ -883,17 +930,31 @@ def set_decode_run_defaults() -> dict[str, str]: def sync_device() -> None: """Block until queued GPU work completes, so all kernels land in the trace before capture stops. Best-effort — a no-op without CuPy/torch.""" + cupy_error: Exception | None = None try: import cupy cupy.cuda.runtime.deviceSynchronize() return - except Exception: - pass + except Exception as exc: + cupy_error = exc try: import torch if torch.cuda.is_available(): torch.cuda.synchronize() - except Exception: - pass + return + except Exception as exc: + warnings.warn( + f"GPU device synchronization failed; trace completeness is not guaranteed " + f"(CuPy: {type(cupy_error).__name__ if cupy_error else 'unavailable'}; " + f"torch: {type(exc).__name__}: {exc})", + RuntimeWarning, + stacklevel=2, + ) + return + warnings.warn( + "GPU device synchronization unavailable; trace completeness is not guaranteed", + RuntimeWarning, + stacklevel=2, + ) diff --git a/scripts/compare_results.py b/scripts/compare_results.py index ccadffc..971bafb 100644 --- a/scripts/compare_results.py +++ b/scripts/compare_results.py @@ -24,7 +24,7 @@ import sys from pathlib import Path -EXACT = ["git_sha", "gitm_version", "python", "dataset_manifests"] +EXACT = ["schema", "git_sha", "gitm_version", "python", "dataset_manifests"] EXACT_PKGS = ["pydantic", "numpy", "pandas", "pyarrow", "torch", "cudf-cu12"] ADVISORY = ["gpu"] @@ -39,13 +39,34 @@ def compare(ref: dict, other: dict) -> tuple[list[str], list[str]]: advisories: list[str] = [] for f in EXACT: - if ref.get(f) != other.get(f): - mismatches.append(f"{f}: {ref.get(f)!r} != {other.get(f)!r}") - - rp, op = ref.get("packages", {}), other.get("packages", {}) + ref_value = ref.get(f) + other_value = other.get(f) + if f not in ref or f not in other or ref_value is None or other_value is None: + mismatches.append( + f"{f}: required verification field is missing or unavailable " + f"({ref_value!r} != {other_value!r})" + ) + elif ref_value != other_value: + mismatches.append(f"{f}: {ref_value!r} != {other_value!r}") + + rp, op = ref.get("packages"), other.get("packages") for pkg in EXACT_PKGS: - if rp.get(pkg) != op.get(pkg): - mismatches.append(f"packages.{pkg}: {rp.get(pkg)} != {op.get(pkg)}") + ref_value = rp.get(pkg) if isinstance(rp, dict) else None + other_value = op.get(pkg) if isinstance(op, dict) else None + if ( + not isinstance(rp, dict) + or not isinstance(op, dict) + or pkg not in rp + or pkg not in op + or ref_value is None + or other_value is None + ): + mismatches.append( + f"packages.{pkg}: required version is missing or unavailable " + f"({ref_value!r} != {other_value!r})" + ) + elif ref_value != other_value: + mismatches.append(f"packages.{pkg}: {ref_value} != {other_value}") if ref.get("git_dirty") or other.get("git_dirty"): advisories.append("a report was produced from a DIRTY git tree " diff --git a/scripts/demo_improve_gpu.py b/scripts/demo_improve_gpu.py index 0790e0d..3abc9b5 100644 --- a/scripts/demo_improve_gpu.py +++ b/scripts/demo_improve_gpu.py @@ -23,14 +23,17 @@ import argparse import json import time +import warnings from pathlib import Path +from gitm._timing import require_positive_duration -def _serialized(trace) -> float: + +def _serialized(trace) -> float | None: from gitm.optimizer.monitor import _serialized_fraction kernels = [e for e in trace.events if e.kind == "kernel"] - return _serialized_fraction(kernels) if kernels else 0.0 + return _serialized_fraction(kernels) if kernels else None def _mean_util(path: Path) -> float | None: @@ -62,8 +65,12 @@ def _run_observed(fn, label: str, outdir: Path) -> tuple[object, dict]: from gitm.telemetry.sinks import build_sink tele = Collector(CollectorConfig(interval_s=0.05, sinks=[build_sink(f"jsonl:{tele_path}")])) - except Exception: - pass + except Exception as exc: + warnings.warn( + f"demo telemetry unavailable: {type(exc).__name__}: {exc}", + RuntimeWarning, + stacklevel=2, + ) if tele: tele.start() @@ -71,7 +78,9 @@ def _run_observed(fn, label: str, outdir: Path) -> tuple[object, dict]: t0 = time.perf_counter() result = fn() cupy.cuda.runtime.deviceSynchronize() - elapsed = max(time.perf_counter() - t0, 1e-9) + elapsed = require_positive_duration( + time.perf_counter() - t0, context=f"demo observation {label}" + ) if tele: tele.stop() @@ -80,6 +89,7 @@ def _run_observed(fn, label: str, outdir: Path) -> tuple[object, dict]: "serialized": _serialized(tr), "util_pct": _mean_util(tele_path), "n_kernels": len([e for e in tr.events if e.kind == "kernel"]), + "diagnostics": list(tele.diagnostics) if tele is not None else ["telemetry unavailable"], } @@ -123,13 +133,22 @@ def parallel(): # --- 1. OBSERVE ----------------------------------------------------------- before_res, before = _run_observed(serial, "before", args.outdir) util_s = f"{before['util_pct']:.0f}%" if before["util_pct"] is not None else "n/a" + serialized_s = ( + f"{before['serialized']:.3f}" if before["serialized"] is not None else "unavailable" + ) print("1. OBSERVE (serial baseline under the runtime):") print(" why: run the work once, untouched, to measure how much of the GPU it actually uses.") print(f" wall-clock {before['elapsed_s'] * 1e3:.2f} ms | util {util_s} | " - f"serialized {before['serialized']:.3f} | {before['n_kernels']} kernels\n") + f"serialized {serialized_s} | {before['n_kernels']} kernels\n") # --- 2. DECIDE ------------------------------------------------------------ - idle = before["util_pct"] is None or before["util_pct"] < 85.0 + if before["util_pct"] is None: + print(" telemetry unavailable — refusing an idle-GPU claim and intervention decision.") + return 1 + if before["serialized"] is None: + print(" cross-stream evidence unavailable — refusing a concurrency intervention decision.") + return 1 + idle = before["util_pct"] < 85.0 serial_heavy = before["serialized"] > 0.5 print("2. DECIDE (runtime maps headroom -> lever):") print(" why: an idle GPU running serialized, independent work is the signature that parallelizing across streams should help.") @@ -160,15 +179,25 @@ def parallel(): return 1 util_a = f"{after['util_pct']:.0f}%" if after["util_pct"] is not None else "n/a" + serialized_a = ( + f"{after['serialized']:.3f}" if after["serialized"] is not None else "unavailable" + ) speedup = before["elapsed_s"] / after["elapsed_s"] print() print(" BEFORE (serial) AFTER (parallel)") print(f" wall-clock {before['elapsed_s'] * 1e3:>10.2f} ms {after['elapsed_s'] * 1e3:>10.2f} ms") print(f" throughput {K / before['elapsed_s']:>10.0f} mm/s {K / after['elapsed_s']:>10.0f} mm/s") print(f" GPU util {util_s:>12} {util_a:>12}") - print(f" serialized {before['serialized']:>12.3f} {after['serialized']:>12.3f}") - print(f"\n >>> runtime-driven improvement: {speedup:.2f}x faster, " - f"serialization {before['serialized']:.2f} -> {after['serialized']:.2f} (correctness-gated)") + print(f" serialized {serialized_s:>12} {serialized_a:>12}") + serialization_delta = ( + f", serialization {before['serialized']:.2f} -> {after['serialized']:.2f}" + if after["serialized"] is not None + else ", post-change serialization unavailable" + ) + print( + f"\n >>> runtime-driven improvement: {speedup:.2f}x faster{serialization_delta} " + " (correctness-gated)" + ) return 0 diff --git a/scripts/emit_report.py b/scripts/emit_report.py index 590972a..c3b41eb 100644 --- a/scripts/emit_report.py +++ b/scripts/emit_report.py @@ -102,7 +102,7 @@ def main(argv: list[str] | None = None) -> int: report = build_report() text = json.dumps(report, indent=2, sort_keys=True) if args.out: - args.out.write_text(text + "\n") + args.out.write_text(text + "\n", encoding="utf-8") print(f"wrote {args.out}") else: print(text) diff --git a/scripts/run_on_real_trace.py b/scripts/run_on_real_trace.py index 01ece1c..ee37f21 100644 --- a/scripts/run_on_real_trace.py +++ b/scripts/run_on_real_trace.py @@ -16,11 +16,8 @@ from __future__ import annotations import sys -import warnings from pathlib import Path -warnings.filterwarnings("ignore") # this is a diagnostic script; keep output clean - def main() -> int: try: @@ -56,15 +53,22 @@ def main() -> int: _ = c.sum().item() torch.cuda.synchronize() - kernels = [e for e in tr.events if e.kind == "kernel"] + all_kernels = [e for e in tr.events if e.kind == "kernel"] + kernels = [e for e in all_kernels if e.end_ns > e.start_ns] if not kernels: - print("FAIL: no kernels captured (is the CUPTI shim built? run gpu_setup.sh)") + print("FAIL: no positive-duration kernels captured (is the CUPTI shim built?)") return 1 + if len(kernels) != len(all_kernels): + print( + f"WARN: excluded {len(all_kernels) - len(kernels)}/{len(all_kernels)} " + "kernels with invalid timestamps" + ) print(f"captured {len(kernels)} real kernels on {torch.cuda.get_device_name(0)}") # Real stream-concurrency computed from the trace stream IDs. sc = _serialized_fraction(kernels) - print(f"serialized_concurrency_fraction (REAL): {sc:.3f}") + sc_text = f"{sc:.3f}" if sc is not None else "unavailable (no cross-stream pairs)" + print(f"serialized_concurrency_fraction (REAL): {sc_text}") # Residual per kernel = deviation from that kernel name's median duration. by_name: dict[str, list[int]] = {} @@ -75,7 +79,7 @@ def main() -> int: res = Residuals() res.serialized_concurrency_fraction = sc for k in kernels: - m = med[k.name] or 1.0 + m = med[k.name] res.per_kernel.append( KernelResidual(op=k.name[:30], layer=None, r_kt=((k.end_ns - k.start_ns) - m) / m, r_mt=None) ) diff --git a/scripts/serve_headroom.py b/scripts/serve_headroom.py index 6194812..9fb1df2 100644 --- a/scripts/serve_headroom.py +++ b/scripts/serve_headroom.py @@ -58,7 +58,7 @@ def load(trace_path: Path): return _load_trace_jsonl(trace_path) -def analyse(trace, sku: str): +def analyse(trace, sku: str, *, hardware_assumed: bool = False): """Everything the report needs, as plain data.""" from gitm.importers.analyze import _predicted_floor_s, _resolve_peak from gitm.optimizer.collective_signal import collective_causes, worst_device_comm @@ -69,6 +69,18 @@ def analyse(trace, sku: str): kernels = trace.kernels() breakdown = summarize_kernels(kernels, window_ns=trace.duration_ns) + if breakdown.n_kernels == 0: + return { + "breakdown": breakdown, + "peak": None, + "sku_known": False, + "hardware_assumed": hardware_assumed, + "metrics": None, + "headroom": None, + "roi": [], + "comm": None, + "causes": [], + } peak, sku_known = _resolve_peak(sku) metrics = compute_metrics(trace, peak) @@ -87,6 +99,7 @@ def analyse(trace, sku: str): "breakdown": breakdown, "peak": peak, "sku_known": sku_known, + "hardware_assumed": hardware_assumed, "metrics": metrics, "headroom": headroom, "roi": roi, @@ -168,13 +181,19 @@ def render(a: dict, *, serving: dict | None, trace) -> str: f"- TTFT p50/p95: {_ms(serving.get('ttft_p50_s'))} / {_ms(serving.get('ttft_p95_s'))}", f"- TPOT p50/p95: {_ms(serving.get('tpot_p50_s'))} / {_ms(serving.get('tpot_p95_s'))}", f"- requests: {serving.get('n_requests')} " - f"({serving.get('n_failed_requests', 0)} failed), " + f"({serving.get('n_failed_requests', 'n/a')} failed), " f"goodput {serving.get('goodput_rps')}", "", "Latency is client-side (includes network); vLLM's own histograms are in " "metrics_before.txt / metrics_after.txt.", ""] caveats = fp8_caveat(a["sku_known"], a["peak"].name) + list(h.caveats) + if a["hardware_assumed"]: + caveats.insert( + 0, + f"No --sku was supplied; pricing assumes {a['peak'].name}. " + "Utilization ratios are not a detected-hardware measurement.", + ) out += ["## Caveats", ""] + [f"- {c}" for c in caveats] + [""] return "\n".join(out) @@ -220,13 +239,14 @@ def main(argv: list[str] | None = None) -> int: ap.add_argument("target", type=Path, help="capture dir (or a trace.jsonl)") ap.add_argument("--compare", type=Path, default=None, help="a second capture dir to diff against (e.g. the eager reference)") - ap.add_argument("--sku", default=DEFAULT_SKU) + ap.add_argument("--sku", default=None) ap.add_argument("--out", type=Path, default=None, help="markdown path (default /headroom.md)") args = ap.parse_args(sys.argv[1:] if argv is None else argv) trace_path = resolve_trace(args.target) trace = load(trace_path) - a = analyse(trace, args.sku) + sku = args.sku or DEFAULT_SKU + a = analyse(trace, sku, hardware_assumed=args.sku is None) if a["breakdown"].n_kernels == 0: print("No kernels in this trace — there is no headroom claim to make.", file=sys.stderr) @@ -242,7 +262,7 @@ def main(argv: list[str] | None = None) -> int: md = render(a, serving=serving, trace=trace) if args.compare: other_path = resolve_trace(args.compare) - b = analyse(load(other_path), args.sku) + b = analyse(load(other_path), sku, hardware_assumed=args.sku is None) md += compare(a, b, label_a=args.target.name, label_b=args.compare.name) out_md = args.out or (trace_path.parent / "headroom.md") diff --git a/tests/test_apply_rollback.py b/tests/test_apply_rollback.py index 46aa2a0..2d3f0fc 100644 --- a/tests/test_apply_rollback.py +++ b/tests/test_apply_rollback.py @@ -45,12 +45,26 @@ def test_apply_keeps_on_positive_delta(): def test_apply_only_when_no_measurement_keeps(): cfg = {"block_size": 8} - res = apply_intervention(_spec(), DictApplicator(cfg)) # measure -> None + with pytest.warns(RuntimeWarning, match="applied without a measurement"): + res = apply_intervention(_spec(), DictApplicator(cfg)) # measure -> None assert res.applied and not res.rolled_back assert res.measured_delta is None assert cfg["block_size"] == 16 +def test_broken_apply_audit_sink_warns(): + class BrokenAudit: + def record(self, *_args, **_kwargs): + raise OSError("disk full") + + with pytest.warns(RuntimeWarning, match="safety audit failed"): + result = apply_intervention( + _spec(), DictApplicator({"block_size": 8}, measure_fn=lambda _s: 0.1), audit=BrokenAudit() + ) + + assert result.applied and not result.rolled_back + + # --- rollback case 1: bad value (apply raises) ------------------------------ @@ -78,6 +92,18 @@ def boom(_spec): assert cfg == {"block_size": 8} # restored despite the apply having mutated it +def test_failopen_restores_on_baseexception_during_measurement(): + cfg = {"block_size": 8} + + def interrupt(_spec): + raise KeyboardInterrupt + + with pytest.raises(KeyboardInterrupt): + apply_intervention(_spec(), DictApplicator(cfg, measure_fn=interrupt)) + + assert cfg == {"block_size": 8} + + # --- regression rollback ---------------------------------------------------- @@ -197,6 +223,26 @@ def test_apply_from_file_with_config(tmp_path): assert yaml.safe_load(target.read_text())["block_size"] == 16 +@pytest.mark.parametrize("threshold", [float("nan"), float("inf"), float("-inf")]) +def test_apply_refuses_nonfinite_keep_threshold(threshold): + with pytest.raises(ValueError, match="min_keep_delta"): + apply_intervention( + _spec(), DictApplicator({"block_size": 8}), min_keep_delta=threshold + ) + + +@pytest.mark.parametrize("delta", [float("nan"), float("inf"), float("-inf")]) +def test_apply_rolls_back_nonfinite_measurement(delta): + cfg = {"block_size": 8} + app = DictApplicator(cfg, measure_fn=lambda _spec: delta) + + result = apply_intervention(_spec(), app) + + assert result.rolled_back and not result.applied + assert "finite" in result.error + assert cfg["block_size"] == 8 + + # --- library now carries the 21 curated levers ------------------------------ diff --git a/tests/test_autoresearch.py b/tests/test_autoresearch.py index 08dec27..17911cf 100644 --- a/tests/test_autoresearch.py +++ b/tests/test_autoresearch.py @@ -2,6 +2,8 @@ from __future__ import annotations +import pytest + import gitm.agents.autoresearch as ar from gitm.agents.autoresearch import ( AutoresearchRun, @@ -68,14 +70,23 @@ def test_proposed_knobs_are_disjoint_from_catalog() -> None: def test_classify_idle_stall_from_serialized_kernels() -> None: - # Back-to-back kernels on one stream, no overlap ⇒ serialized concurrency = 1. + # Back-to-back kernels on alternating streams expose independent scheduling + # opportunities that failed to overlap. events = [ - make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=0) + make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=i % 2) for i in range(6) ] assert classify_bottleneck(make_trace(events=events)) == "idle_stall" +def test_classify_single_stream_is_unclassified_without_overlap_opportunity() -> None: + events = [ + make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=0) + for i in range(6) + ] + assert classify_bottleneck(make_trace(events=events)) == ar.UNCLASSIFIED + + def test_classify_memory_bound_from_memcpy_heavy_trace() -> None: # Overlapping kernels (no stall) but memcpys dominate GPU-op time. kernels = [ @@ -106,8 +117,11 @@ def test_classify_compute_bound_when_overlapped_and_no_memcpy() -> None: assert classify_bottleneck(make_trace(events=events)) == "compute_bound" -def test_classify_empty_trace_defaults_to_compute() -> None: - assert classify_bottleneck(make_trace(events=[])) == "compute_bound" +def test_classify_empty_or_invalid_trace_is_unclassified() -> None: + assert classify_bottleneck(make_trace(events=[])) == ar.UNCLASSIFIED + assert classify_bottleneck( + make_trace(events=[make_kernel("zero", start_ns=10, end_ns=10)]) + ) == ar.UNCLASSIFIED # --- classify_bottleneck: roofline-weighted memory signal --------------------- @@ -215,7 +229,7 @@ def test_unknown_class_yields_no_results() -> None: def test_autoresearch_end_to_end_classifies_and_runs() -> None: events = [ - make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=0) + make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=i % 2) for i in range(6) ] config: dict = {} @@ -561,7 +575,8 @@ def test_fallback_proposer_uses_table_only_when_primary_is_empty() -> None: def test_autoresearch_end_to_end_with_engineargs_proposer() -> None: events = [ - make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=0) for i in range(6) + make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=i % 2) + for i in range(6) ] # serialized → idle_stall proposer = EngineArgsProposer( knobs=[Knob("max_num_partial_prefills", "int", default=1)], catalog_knobs=set() @@ -615,9 +630,13 @@ def test_engineargs_proposer_is_a_vllm_bound_generative_proposer() -> None: assert specs and all(s.applicability.workloads == ["vllm-decode"] for s in specs) -def test_vllm_knob_source_yields_offline_fallback_without_vllm() -> None: +def test_vllm_knob_source_warns_when_using_offline_fallback(monkeypatch) -> None: # vLLM isn't importable in CI → the source yields the frozen fallback catalog. - knobs = VLLMKnobSource().knobs() + import sys + + monkeypatch.setitem(sys.modules, "vllm", None) + with pytest.warns(RuntimeWarning, match="frozen fallback knob catalog"): + knobs = VLLMKnobSource().knobs() assert knobs and all(isinstance(k, Knob) for k in knobs) names = {k.name for k in knobs} assert "cpu_offload_gb" in names @@ -722,7 +741,8 @@ def test_argparse_domains_empty_when_no_cli_builder() -> None: class _Bare: pass - assert _argparse_domains(_Bare) == {} + with pytest.warns(RuntimeWarning, match="CLI-domain introspection unavailable"): + assert _argparse_domains(_Bare) == {} # --- hardware-applicability: skip multi-GPU knobs on a single-GPU box -------- @@ -758,6 +778,32 @@ def test_visible_gpu_count_is_a_positive_int() -> None: assert isinstance(n, int) and n >= 1 +def test_visible_gpu_count_warns_when_torch_reports_no_gpus(monkeypatch) -> None: + import sys + from types import SimpleNamespace + + monkeypatch.setitem(sys.modules, "torch", SimpleNamespace(cuda=SimpleNamespace(device_count=lambda: 0))) + with pytest.warns(RuntimeWarning, match="GPU-count detection reported 0"): + assert _visible_gpu_count() == 1 + + +def test_visible_gpu_count_keeps_observed_multi_gpu_count_clean(monkeypatch) -> None: + import sys + import warnings + from types import SimpleNamespace + + monkeypatch.setitem(sys.modules, "torch", SimpleNamespace(cuda=SimpleNamespace(device_count=lambda: 4))) + with warnings.catch_warnings(): + warnings.simplefilter("error") + assert _visible_gpu_count() == 4 + + +@pytest.mark.parametrize("count", [0, -1]) +def test_knobs_from_engine_args_refuses_invalid_explicit_gpu_count(count) -> None: + with pytest.raises(ValueError, match="gpu_count must be positive"): + _knobs_from_engine_args(_FakeEngineArgs, gpu_count=count) + + def test_vllm_knob_source_gpu_count_override_is_accepted_offline() -> None: # vLLM isn't importable in CI, so the offline fallback catalog is returned # regardless of gpu_count — this just proves the parameter doesn't crash @@ -796,7 +842,8 @@ def test_classify_always_returns_a_known_class() -> None: traces = [ make_trace(events=[]), # empty → compute default make_trace(events=[ - make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90) for i in range(6) + make_kernel("k", start_ns=i * 100, end_ns=i * 100 + 90, stream_id=i % 2) + for i in range(6) ]), # serialized → idle make_trace(events=[make_kernel("k", start_ns=0, end_ns=1000)] + [make_memcpy(start_ns=i * 10, end_ns=i * 10 + 5) for i in range(4)]), diff --git a/tests/test_bench.py b/tests/test_bench.py index 96f567a..4f590b1 100644 --- a/tests/test_bench.py +++ b/tests/test_bench.py @@ -124,6 +124,11 @@ def _run(seed: int, value: float, gpu: float = 0.7) -> object: warm_window_s=60, git_sha="abc1234", gitm_version="0.0.1", + manifest_sha256="manifest123", + gpu_name="A100", + device_count=1, + started_at_ns=1, + ended_at_ns=2, stall_breakdown=[ StallPhase(phase="all", cpu=0.03, data_stall=max(0.0, 1 - gpu - 0.03 - 0.05), sync=0.05, gpu_active=gpu, throughput=value, wall_clock_s=60.0) @@ -168,6 +173,69 @@ def test_baseline_fails_on_saturation(): assert any(g.name == "saturation" and not g.passed for g in summary.gates) +def test_baseline_refuses_missing_gpu_activity_instead_of_substituting_zero(): + from gitm.bench.baseline import aggregate + + runs = [_run(42, 26e6), _run(43, 26e6), _run(44, 26e6)] + for run in runs: + run.stall_breakdown = [] + + summary = aggregate(runs, _hft_config()) + + assert summary.gpu_active_overall is None + gate = next(g for g in summary.gates if g.name == "saturation") + assert not gate.passed + assert "coverage unavailable" in gate.detail + + +def test_baseline_refuses_cpu_or_unpinned_runs(): + from gitm.bench.baseline import aggregate + + runs = [_run(42, 26e6), _run(43, 26e6), _run(44, 26e6)] + runs[0].gpu_name = "cpu" + runs[1].manifest_sha256 = None + + summary = aggregate(runs, _hft_config()) + + gate = next(g for g in summary.gates if g.name == "provenance") + assert not gate.passed + assert "GPU identity unavailable" in gate.detail + assert "manifest digest unavailable" in gate.detail + + +def test_baseline_refuses_missing_or_reversed_timing_provenance(): + from gitm.bench.baseline import aggregate + + runs = [_run(42, 26e6), _run(43, 26e6), _run(44, 26e6)] + runs[0].started_at_ns = 0 + runs[1].ended_at_ns = runs[1].started_at_ns + + summary = aggregate(runs, _hft_config()) + + gate = next(g for g in summary.gates if g.name == "provenance") + assert not gate.passed + assert "start timestamp unavailable" in gate.detail + assert "end timestamp does not follow start" in gate.detail + + +@pytest.mark.parametrize("value", [0.0, -1.0, float("nan"), float("inf")]) +def test_baseline_run_refuses_unusable_metric_values(value): + from pydantic import ValidationError + + with pytest.raises(ValidationError): + _run(42, value) + + +@pytest.mark.parametrize("target", [0.0, -1.0, float("nan"), float("inf")]) +def test_bench_config_refuses_unusable_baseline_target(target): + from pydantic import ValidationError + + cfg = _hft_config().model_dump() + cfg["baseline_target"] = target + with pytest.raises(ValidationError): + type(_hft_config()).model_validate(cfg) + + def test_baseline_fails_on_too_few_runs_and_below_target(): from gitm.bench.baseline import aggregate @@ -229,6 +297,126 @@ def test_wrap_command_marks_missing_profiler(): assert "nsys" in bundle.missing +def test_profile_marks_failed_workload_command(tmp_path): + import sys + + from gitm.bench.profile import ProfilerTools, run_profile + + bundle = run_profile( + _hft_config(), + [sys.executable, "-c", "raise SystemExit(7)"], + tmp_path, + tools=ProfilerTools(nsys=None, rocprof=None, py_spy=None, sar=None), + ) + + assert any("exit 7" in item for item in bundle.missing) + assert not bundle.complete + + +def test_profile_launches_pyspy_and_records_its_artifact(monkeypatch, tmp_path): + from gitm.bench import profile + + calls = [] + + class Proc: + def __init__(self, argv, **_kwargs): + self.argv = argv + self.pid = 4321 + self.returncode = None + calls.append(argv) + if argv[0] == "py-spy": + output = Path(argv[argv.index("--output") + 1]) + output.write_text("") + + def wait(self, timeout=None): + self.returncode = 0 + return 0 + + def terminate(self): + self.returncode = -15 + + monkeypatch.setattr(profile.subprocess, "Popen", Proc) + bundle = profile.run_profile( + _hft_config(), + ["workload"], + tmp_path, + host_capture_s=1, + tools=profile.ProfilerTools(nsys=None, rocprof=None, py_spy="py-spy", sar=None), + ) + + pyspy = next(call for call in calls if call[0] == "py-spy") + assert pyspy[pyspy.index("--pid") + 1] == "4321" + assert "--subprocesses" in pyspy + assert bundle.host_pyspy == tmp_path / "host_flamegraph.svg" + assert bundle.host_pyspy.exists() + assert not any("py-spy" in item for item in bundle.missing) + + +def test_profile_surfaces_failed_pyspy_capture(monkeypatch, tmp_path): + from gitm.bench import profile + + class Proc: + def __init__(self, argv, **_kwargs): + self.argv = argv + self.pid = 4321 + self.returncode = None + + def wait(self, timeout=None): + self.returncode = 9 if self.argv[0] == "py-spy" else 0 + return self.returncode + + def terminate(self): + self.returncode = -15 + + monkeypatch.setattr(profile.subprocess, "Popen", Proc) + bundle = profile.run_profile( + _hft_config(), + ["workload"], + tmp_path, + host_capture_s=1, + tools=profile.ProfilerTools(nsys=None, rocprof=None, py_spy="py-spy", sar=None), + ) + + assert any("py-spy capture failed (exit 9)" in item for item in bundle.missing) + assert not bundle.complete + + +def test_profile_cli_prints_every_bundle_artifact(monkeypatch, tmp_path, capsys): + from types import SimpleNamespace + + from gitm.bench import cli + from gitm.bench.profile import ProfileBundle + + bundle = ProfileBundle( + out_dir=tmp_path, + gpu_report=tmp_path / "gpu.nsys-rep", + gpu_csv=tmp_path / "gpu.csv", + host_pyspy=tmp_path / "host.svg", + host_sar=tmp_path / "host.log", + ) + monkeypatch.setattr("gitm.bench.schema.BenchConfig.from_toml", lambda _path: object()) + monkeypatch.setattr("gitm.bench.runner.build_command", lambda _cfg, _seed: ["workload"]) + monkeypatch.setattr("gitm.bench.profile.run_profile", lambda *_args, **_kwargs: bundle) + + rc = cli._cmd_profile(SimpleNamespace(config=tmp_path / "bench.toml", seed=1, out=tmp_path)) + payload = json.loads(capsys.readouterr().out) + + assert rc == 0 + assert payload["gpu_report"] == str(bundle.gpu_report) + assert payload["gpu_csv"] == str(bundle.gpu_csv) + assert payload["host_pyspy"] == str(bundle.host_pyspy) + assert payload["host_sar"] == str(bundle.host_sar) + + +def test_breakdown_refuses_to_clamp_overlapping_timings(): + from gitm.bench.profile import PhaseTiming, build_breakdown + + with pytest.raises(ValueError, match="refusing to clamp"): + build_breakdown( + [PhaseTiming("overlap", wall_clock_s=1.0, gpu_busy_s=0.8, cpu_s=0.4)] + ) + + # --- edge manifest ---------------------------------------------------------- @@ -326,7 +514,7 @@ def test_run_seed_end_to_end_with_echo_harness(tmp_path: Path, monkeypatch): 'name="hft"\nvendor="nvidia"\nmetric="events_per_second"\n' 'warm_window_s=60\nseeds=[42]\n' '[dataset]\nroot="hft"\n' - f'[work_unit]\ncommand="python {harness} --seed {{seed}}"\n' + f'[work_unit]\ncommand="python {harness.as_posix()} --seed {{seed}}"\n' '[expected_stall]\ncpu={lo=0,hi=0.05}\ndata_stall={lo=0.1,hi=0.25}\n' 'sync={lo=0.05,hi=0.15}\ngpu_active={lo=0.6,hi=0.8}\n' ) diff --git a/tests/test_catalog_unify.py b/tests/test_catalog_unify.py index 4504603..233d5cd 100644 --- a/tests/test_catalog_unify.py +++ b/tests/test_catalog_unify.py @@ -26,7 +26,16 @@ def test_requires_collective_and_interconnect(): assert applicable(spec, ok_ctx)[0] no_ic = GateContext(workload="vllm-decode", num_gpus=2, has_collective=True) ok, reason = applicable(spec, no_ic) - assert not ok and "interconnect" in reason + assert not ok and "topology is unknown" in reason + + known_absent = GateContext( + workload="vllm-decode", + num_gpus=2, + has_collective=True, + has_interconnect=False, + ) + ok, reason = applicable(spec, known_absent) + assert not ok and "none was reported" in reason def test_unified_library_scopes_by_workload(): diff --git a/tests/test_cli_attach.py b/tests/test_cli_attach.py index 34dd2e9..1190e3a 100644 --- a/tests/test_cli_attach.py +++ b/tests/test_cli_attach.py @@ -35,3 +35,15 @@ def test_main_attach_returns_zero_on_plan(capsys): assert rc == 0 out = json.loads(capsys.readouterr().out) assert out["job_id"] == "j" and out["status"] == "planned" + + +def test_live_pid_is_not_falsely_reported_attached(monkeypatch): + import gitm.deploy.attach as attach + + monkeypatch.setattr(attach, "_pid_is_live", lambda _pid: True) + + plan = attach_job("job-live", pid=4321, dry_run=False) + + assert plan["status"] == "unsupported" + assert "not implemented" in plan["reason"] + assert "no telemetry shim was installed" in plan["reason"] diff --git a/tests/test_compare_results.py b/tests/test_compare_results.py new file mode 100644 index 0000000..2806ffc --- /dev/null +++ b/tests/test_compare_results.py @@ -0,0 +1,27 @@ +from __future__ import annotations + +from scripts.compare_results import EXACT_PKGS, compare + + +def _report() -> dict: + return { + "schema": "gitm.verify_report/v1", + "git_sha": "abc", + "gitm_version": "1", + "python": "3.12", + "dataset_manifests": {"hft": "deadbeef"}, + "packages": {pkg: "1.0" for pkg in EXACT_PKGS}, + "git_dirty": False, + "gpu": {"name": "H100"}, + } + + +def test_compare_refuses_reports_with_missing_identity_fields(): + mismatches, _ = compare({}, {}) + + assert any("schema" in mismatch for mismatch in mismatches) + assert any("packages.pydantic" in mismatch for mismatch in mismatches) + + +def test_compare_accepts_complete_matching_reports(): + assert compare(_report(), _report()) == ([], []) diff --git a/tests/test_cuda_env.py b/tests/test_cuda_env.py index d339472..a8b1214 100644 --- a/tests/test_cuda_env.py +++ b/tests/test_cuda_env.py @@ -79,11 +79,34 @@ def test_require_compatible_raises_before_an_expensive_build(host): cuda_env.require_compatible() +def test_require_compatible_refuses_missing_driver(host): + host(driver=None) + + with pytest.raises(RuntimeError, match="no NVIDIA driver detected"): + cuda_env.require_compatible() + + def test_require_compatible_is_silent_when_the_stack_fits(host): host(driver=(13, 0), torch=(13, 0), vllm=13) cuda_env.require_compatible() # must not raise +def test_require_compatible_warns_when_build_versions_are_unverified(host): + host(driver=(13, 0), torch=None, vllm=None) + + with pytest.warns(RuntimeWarning, match="PyTorch CUDA build.*vLLM CUDA build"): + cuda_env.require_compatible() + + +def test_diagnostic_cli_does_not_claim_ok_for_unverified_builds(host, capsys): + host(driver=(13, 0), torch=None, vllm=None) + + assert cuda_env.main([]) == 2 + output = capsys.readouterr().out + assert "UNVERIFIED" in output + assert "OK" not in output + + @pytest.mark.parametrize( ("driver", "expected"), [ diff --git a/tests/test_cupti.py b/tests/test_cupti.py index d6203e4..1a2f3c5 100644 --- a/tests/test_cupti.py +++ b/tests/test_cupti.py @@ -51,6 +51,15 @@ def test_decode_kernel_anonymous_name(): assert ev.name == "" +def test_decode_kernel_warns_when_shape_or_name_is_missing(): + from gitm.tracer._cupti_decode import decode_kernel + + with pytest.warns(RuntimeWarning, match="kernel grid dimensions unavailable"): + decode_kernel(_kernel_rec(grid=None)) + with pytest.warns(RuntimeWarning, match="kernel name unavailable"): + decode_kernel(_kernel_rec(name=None)) + + # --- decode: memcpy copy-kind mapping --------------------------------------- @@ -96,6 +105,21 @@ def test_decode_sync_types(sync_type, kind): assert ev.sync_kind == kind +def test_decode_unknown_cupti_enums_warn_before_using_safe_endpoint_defaults(): + from gitm.tracer._cupti_decode import decode_memcpy, decode_sync + + memcpy = { + "kind": "memcpy", "copy_kind": 99, "bytes": 1, + "start_ns": 10, "end_ns": 20, "device_id": 0, "stream_id": 0, + } + with pytest.warns(RuntimeWarning, match="unknown CUPTI memcpy kind 99"): + decode_memcpy(memcpy) + + sync = {"kind": "sync", "sync_type": 99, "start_ns": 10, "end_ns": 20} + with pytest.warns(RuntimeWarning, match="unknown CUPTI synchronization type 99"): + decode_sync(sync) + + # --- decode: batch ---------------------------------------------------------- @@ -159,6 +183,26 @@ def test_capture_falls_back_to_noop_trace(tmp_path, monkeypatch): assert header["device_count"] == 0 +@pytest.mark.parametrize( + "counter, message", + [ + (lambda: 0, "reported 0 devices"), + (lambda: (_ for _ in ()).throw(RuntimeError("driver denied")), "driver denied"), + ], +) +def test_capture_warns_when_active_backend_device_count_is_unavailable(counter, message): + import importlib + from types import SimpleNamespace + + capture_mod = importlib.import_module("gitm.tracer.capture") + backend = SimpleNamespace(device_count=counter) + + with pytest.warns(RuntimeWarning, match=message): + count = capture_mod._device_count(backend, injected=False) + + assert count == 0 + + # --- full backend wiring via a fake shim ------------------------------------ diff --git a/tests/test_deviation_alignment.py b/tests/test_deviation_alignment.py index a82cd29..6ab15e0 100644 --- a/tests/test_deviation_alignment.py +++ b/tests/test_deviation_alignment.py @@ -86,6 +86,15 @@ def test_summary_keys_by_the_observed_kernels_op(): assert summary["kept_ops"] == {"attn_score_value": 1, "": 1} +def test_zero_duration_kernel_is_kept_as_a_departure(): + g = predict_graph() + tr = _trace([_k("flash_attn_kernel", 0.0)]) + + dev = deviating_kernel_indices(tr, g) + + assert dev.kept_indices == [0] + + def test_range_identity_classifies_a_bare_gemm_that_name_matching_cannot(): """The dominant real-world gap: bare cuBLAS/cutlass GEMMs carry no projection tag in their name (test_classify_op_matches_real_vllm_kernel_names diff --git a/tests/test_edge_optimize.py b/tests/test_edge_optimize.py index 4f8d9a3..7dacf21 100644 --- a/tests/test_edge_optimize.py +++ b/tests/test_edge_optimize.py @@ -69,6 +69,27 @@ def test_detections_equivalent_matches_per_frame(): assert not detections_equivalent(base, _frames(n_frames=1, n_det=3)) +def test_zero_detection_tolerance_is_exact(): + base = _frames(n_frames=1, n_det=3) + + assert not detections_equivalent(base, _frames(n_frames=1, n_det=2), tol_frac=0.0) + + +@pytest.mark.parametrize( + ("kwargs", "message"), + [ + ({"reps": 0}, "reps"), + ({"reps": -2}, "reps"), + ({"tol_frac": -0.1}, "tol_frac"), + ({"tol_frac": 1.1}, "tol_frac"), + ({"center_atol": float("nan")}, "center_atol"), + ], +) +def test_optimize_edge_refuses_invalid_gate_controls(kwargs, message): + with pytest.raises(ValueError, match=message): + optimize_edge(_fake_run_mode(), **kwargs) + + def test_specs_apply_to_edge_workloads(): for spec in (edge_intervention_spec(), edge_batching_spec()): assert set(spec.applicability.workloads) >= {"edge", "kitti", "nuscenes"} @@ -76,6 +97,11 @@ def test_specs_apply_to_edge_workloads(): assert edge_batching_spec().name == "edge_frame_batching" +def test_edge_batching_refuses_invalid_batch_size(): + with pytest.raises(ValueError, match="batch_size"): + EdgeBatchingApplicator(_fake_run_mode(), batch_size=0) + + def test_optimize_edge_keeps_faster_equivalent_candidate(): r = optimize_edge(_fake_run_mode(equivalent=True), reps=1) assert r.identical and r.speedup > 1.0 and r.kept == "candidate" @@ -161,10 +187,8 @@ def test_applicator_runs_through_the_apply_gate(): assert apply_intervention(drift.spec, drift, min_keep_delta=0.0).rolled_back -def test_edge_report_claim_labels_serialized_concurrency_not_kernel_time(tmp_path): - """The edge claim's residual is a serialized-concurrency fraction (from - measure_trace), not a kernel-time ratio — the report must label it - `stream_concurrency`, matching the HFT/AF2 claims, not `kernel_time`.""" +def test_edge_report_uses_ab_delta_when_trace_has_no_kernel_coverage(tmp_path): + """An empty trace must not become a plausible zero-concurrency residual.""" from gitm.optimizer.qualification import QualificationResult from gitm.scheduler.loop import _edge_intervention_result from gitm.tracer.schema import Trace @@ -187,5 +211,50 @@ def test_edge_report_claim_labels_serialized_concurrency_not_kernel_time(tmp_pat trace_path=tmp_path / "trace.jsonl", ) md = result["report_md"] - assert "`stream_concurrency`" in md + assert "`throughput_delta`" in md + assert "`stream_concurrency`" not in md assert "`kernel_time`" not in md + + +def test_edge_report_names_missing_ab_result_as_intervention_failure( + tmp_path, monkeypatch +): + from types import SimpleNamespace + + from gitm.benchmarks.edge.optimize import edge_intervention_spec + from gitm.optimizer.qualification import QualificationResult + from gitm.scheduler.loop import _edge_intervention_result + from gitm.tracer.schema import Trace + + trace = Trace( + workload_id="kitti", + fingerprint="f", + run_id="r", + device_count=0, + vendor="none", + captured_at_ns=0, + duration_ns=1, + events=[], + ) + applicator = SimpleNamespace(last_result=None, spec=edge_intervention_spec()) + monkeypatch.setattr( + "gitm.optimizer.apply.apply_intervention", + lambda *_args, **_kwargs: SimpleNamespace( + applied=False, rolled_back=True, measured_delta=None, error="measure failed" + ), + ) + + result = _edge_intervention_result( + run_dir=tmp_path, + run_id="r", + workload="kitti", + trace=trace, + qual=QualificationResult(commit=False, floor=0.0, fingerprint="f"), + applicator=applicator, + started_ns=0, + trace_path=tmp_path / "trace.jsonl", + ) + + assert result["summary"]["status"] == "intervention_failed" + assert result["summary"]["n_claims"] == 0 + assert "produced no result" in result["report_md"] diff --git a/tests/test_execution_graph_dispatch.py b/tests/test_execution_graph_dispatch.py new file mode 100644 index 0000000..922c281 --- /dev/null +++ b/tests/test_execution_graph_dispatch.py @@ -0,0 +1,190 @@ +"""The loop must dispatch the live model to a priceable execution graph or refuse.""" + +from __future__ import annotations + +from types import SimpleNamespace + +from gitm.planner.context import peak_for_sku +from gitm.planner.roofline import SparseMoEModelSpec +from gitm.scheduler.loop import _execution_graph + + +def _pctx(sku: str | None = "NVIDIA B200", *, kv_cache_len: int | None = None): + return SimpleNamespace( + peak=peak_for_sku(sku), + sku=sku, + gate=SimpleNamespace(kv_cache_len=kv_cache_len), + ) + + +def _engine(cfg: dict): + hf = SimpleNamespace(**cfg) + return SimpleNamespace(model_config=SimpleNamespace(hf_config=hf)) + + +def _moe_cfg(**over) -> dict: + cfg = { + "model_type": "deepseek_v4", + "hidden_size": 64, + "num_hidden_layers": 2, + "num_attention_heads": 1, + "num_key_value_heads": 1, + "head_dim": 64, + "qk_rope_head_dim": 16, + "q_lora_rank": 16, + "o_lora_rank": 16, + "o_groups": 1, + "vocab_size": 128, + "n_routed_experts": 4, + "n_shared_experts": 0, + "num_experts_per_tok": 1, + "moe_intermediate_size": 32, + "index_n_heads": 1, + "index_head_dim": 16, + "index_topk": 32, + "sliding_window": 16, + "compress_ratios": [0, 4], + "expert_dtype": "fp4", + "quantization_config": {"quant_method": "fp8"}, + "torch_dtype": "bfloat16", + } + cfg.update(over) + return cfg + + +def _dense_cfg(**over) -> dict: + cfg = { + "model_type": "opt", + "hidden_size": 768, + "num_hidden_layers": 12, + "num_attention_heads": 12, + "intermediate_size": 3072, + "vocab_size": 50272, + "torch_dtype": "bf16", + } + cfg.update(over) + return cfg + + +def test_sparse_engine_dispatches_to_sparse_graph(): + resolved = _execution_graph(_engine(_moe_cfg()), _pctx(), sched=None) + + assert resolved.ok + assert isinstance(resolved.graph.model, SparseMoEModelSpec) + assert any(node.op == "moe_routed" for node in resolved.graph.nodes) + assert not resolved.graph.has_fallback_bytes + assert resolved.model_source == "live_hf_config" + + +def test_sparse_dispatch_refuses_unpriceable_final_dtype(): + resolved = _execution_graph( + _engine(_moe_cfg(expert_dtype="future_fp3")), _pctx(), sched=None + ) + + assert not resolved.ok and resolved.graph is None + assert "expert_dtype='future_fp3'" in resolved.refusal_reason + + +def test_partial_sparse_config_is_refused_not_sent_to_dense_graph(): + resolved = _execution_graph( + _engine({"hidden_size": 64, "n_routed_experts": 4}), _pctx(), sched=None + ) + + assert not resolved.ok and resolved.graph is None + assert "experts per token" in resolved.refusal_reason + + +def test_zero_expert_count_is_refused_not_priced_as_dense(): + resolved = _execution_graph( + _engine(_moe_cfg(n_routed_experts=0)), _pctx(), sched=None + ) + + assert not resolved.ok and resolved.graph is None + assert "routed expert count" in resolved.refusal_reason + assert "positive" in resolved.refusal_reason + + +def test_missing_live_model_refuses_instead_of_defaulting_to_llama(): + resolved = _execution_graph(None, _pctx(), sched=None) + + assert not resolved.ok and resolved.graph is None + assert "no live engine" in resolved.refusal_reason + + +def test_unknown_hardware_refuses_instead_of_recording_a100_as_observed(): + resolved = _execution_graph(_engine(_moe_cfg()), _pctx("Unknown GPU"), sched=None) + + assert not resolved.ok and resolved.graph is None + assert "Unknown GPU" in resolved.refusal_reason + assert "hardware catalogue" in resolved.refusal_reason + + +def test_config_conversion_bug_is_a_named_refusal(): + class BrokenConfig: + def to_dict(self): + raise RuntimeError("parser exploded") + + engine = SimpleNamespace(model_config=SimpleNamespace(hf_config=BrokenConfig())) + resolved = _execution_graph(engine, _pctx(), sched=None) + + assert not resolved.ok and resolved.graph is None + assert "RuntimeError" in resolved.refusal_reason + assert "parser exploded" in resolved.refusal_reason + + +def test_accepted_batch_and_kv_defaults_are_diagnostics(): + resolved = _execution_graph(_engine(_moe_cfg()), _pctx(), sched=None) + + assert resolved.ok + assert resolved.graph.batch.batch == 1 + assert resolved.graph.batch.kv_cache_len == 4096 + assert any("batch=1" in note for note in resolved.diagnostics) + assert any("kv_cache_len=4096" in note for note in resolved.diagnostics) + + +def test_dense_mha_and_head_dimension_derivations_are_diagnostics(): + resolved = _execution_graph(_engine(_dense_cfg()), _pctx(), sched=None) + + assert resolved.ok + assert any("num_key_value_heads" in note and "MHA" in note for note in resolved.diagnostics) + assert any("head_dim" in note and "derived" in note for note in resolved.diagnostics) + + +def test_dense_execution_graph_consumes_live_tensor_parallel_topology(): + engine = _engine(_dense_cfg(num_key_value_heads=4)) + engine.parallel_config = SimpleNamespace( + tensor_parallel_size=2, + data_parallel_size=1, + enable_expert_parallel=False, + ) + + resolved = _execution_graph(engine, _pctx(), sched=None) + + assert resolved.ok + assert resolved.graph.sharding.tp == 2 + assert any(node.op == "tp_all_reduce" for node in resolved.graph.nodes) + + +def test_malformed_live_expert_parallel_flag_refuses_topology(): + engine = _engine(_dense_cfg()) + engine.parallel_config = SimpleNamespace( + tensor_parallel_size=2, + data_parallel_size=1, + enable_expert_parallel="false", + ) + + resolved = _execution_graph(engine, _pctx(), sched=None) + + assert not resolved.ok + assert "enable_expert_parallel must be boolean" in resolved.refusal_reason + + +def test_partial_live_topology_names_each_assumption(): + engine = _engine(_dense_cfg()) + engine.parallel_config = SimpleNamespace(tensor_parallel_size=2) + + resolved = _execution_graph(engine, _pctx(), sched=None) + + assert resolved.ok + assert any("data_parallel_size" in note for note in resolved.diagnostics) + assert any("enable_expert_parallel" in note for note in resolved.diagnostics) diff --git a/tests/test_framework_harnesses.py b/tests/test_framework_harnesses.py index 4ee970a..1cf285c 100644 --- a/tests/test_framework_harnesses.py +++ b/tests/test_framework_harnesses.py @@ -89,6 +89,20 @@ def test_biotech_loader_raises_without_framework(): load_openfold_runner(42) +@pytest.mark.parametrize( + "timings", + [ + [{"_t_featurize_s": 0.0, "_t_inference_s": 0.0, "_t_post_s": 0.0, "_t_total_s": 0.0}], + [{"_t_featurize_s": 0.6, "_t_inference_s": 0.6, "_t_post_s": 0.0, "_t_total_s": 1.0}], + ], +) +def test_biotech_stall_breakdown_refuses_invalid_timing(timings): + from benchmarks.biotech.harness import _build_stall_phase + + with pytest.raises(RuntimeError, match="timing"): + _build_stall_phase(timings, 1.0) + + # --- edge ------------------------------------------------------------------- @@ -159,3 +173,17 @@ def test_edge_loader_raises_without_framework(): with pytest.raises(RuntimeError, match="OpenPCDet"): load_openpcdet_runner() + + +@pytest.mark.parametrize( + "timings", + [ + [{"_t_load_s": 0.0, "_t_preprocess_s": 0.0, "_t_inference_s": 0.0, "_t_postprocess_s": 0.0, "_t_total_s": 0.0}], + [{"_t_load_s": 0.4, "_t_preprocess_s": 0.3, "_t_inference_s": 0.4, "_t_postprocess_s": 0.0, "_t_total_s": 1.0}], + ], +) +def test_edge_stall_breakdown_refuses_invalid_timing(timings): + from benchmarks.edge.harness import _build_stall_phase + + with pytest.raises(RuntimeError, match="timing"): + _build_stall_phase(timings, 1.0) diff --git a/tests/test_gate_wiring.py b/tests/test_gate_wiring.py index d9eb026..c1c02de 100644 --- a/tests/test_gate_wiring.py +++ b/tests/test_gate_wiring.py @@ -1,5 +1,7 @@ from __future__ import annotations +import pytest + from gitm.agents.policy import Policy, select_interventions from gitm.kernels.spec import Applicability, InterventionSpec, SafetyGate from gitm.optimizer.preconditions import GateContext @@ -39,6 +41,12 @@ def test_without_ctx_no_applicability_filtering(): assert ranked[0].rejected_reason is None +@pytest.mark.parametrize("top_n", [0, -1, True, 1.5]) +def test_policy_refuses_invalid_candidate_limit(top_n): + with pytest.raises(ValueError, match="top_n"): + select_interventions(_trace(), [], Policy(), top_n=top_n) + + def test_load_library_filters_by_workload(tmp_path): import yaml @@ -57,3 +65,24 @@ def test_load_library_filters_by_workload(tmp_path): ) assert [s.name for s in load_library(p, workload="vllm-decode")] == ["v"] assert {s.name for s in load_library(p)} == {"v", "e"} # unfiltered + + +def test_missing_library_refuses_with_named_path(tmp_path): + import pytest + + from gitm.kernels.library import load_library + + missing = tmp_path / "missing.yaml" + with pytest.raises(FileNotFoundError, match="candidate coverage is unavailable"): + load_library(missing) + + +def test_empty_library_refuses_instead_of_looking_like_no_applicable_levers(tmp_path): + import pytest + + from gitm.kernels.library import load_library + + empty = tmp_path / "empty.yaml" + empty.write_text("interventions: []\n") + with pytest.raises(ValueError, match="candidate coverage is unavailable"): + load_library(empty) diff --git a/tests/test_headroom.py b/tests/test_headroom.py index 775719e..f47aa7e 100644 --- a/tests/test_headroom.py +++ b/tests/test_headroom.py @@ -55,3 +55,22 @@ def test_render_contains_key_lines(): md = render_headroom_md(r) assert "Blind headroom — vllm-decode on NVIDIA H100" in md assert "Ceiling distance" in md + assert "Caveats:" in md + assert "HFU is unavailable" in md + + +@pytest.mark.parametrize("floor", [0.0, float("nan"), float("inf")]) +def test_invalid_predicted_floor_is_refused(floor): + trace = _trace(wall_us=200) + metrics = compute_metrics(trace, PEAK) + + with pytest.raises(ValueError, match="predicted floor must be finite and positive"): + build_headroom(trace, predicted_floor_s=floor, metrics=metrics, workload="vllm-decode") + + +def test_floor_above_observation_is_refused_instead_of_already_optimized(): + trace = _trace(wall_us=200) + metrics = compute_metrics(trace, PEAK) + + with pytest.raises(ValueError, match="exceeds observed wall"): + build_headroom(trace, predicted_floor_s=201e-6, metrics=metrics, workload="vllm-decode") diff --git a/tests/test_headroom_kernel_rank.py b/tests/test_headroom_kernel_rank.py new file mode 100644 index 0000000..2968b4b --- /dev/null +++ b/tests/test_headroom_kernel_rank.py @@ -0,0 +1,58 @@ +from __future__ import annotations + +import pytest + +from gitm.optimizer.headroom_kernel_rank import gpu_headroom, kernel_roi, render_roi_table + + +def test_memory_only_telemetry_does_not_fabricate_compute_headroom(): + result = gpu_headroom( + [{"mem_used_bytes": 4_000, "mem_total_bytes": 10_000}] + ) + + assert result.compute_headroom_pct is None + assert result.mean_util_pct is None + assert result.mem_free_at_peak_bytes == 6_000 + assert any("utilization telemetry is absent" in note for note in result.diagnostics) + + +def test_utilization_only_telemetry_does_not_fabricate_memory_headroom(): + result = gpu_headroom([{"util_pct": 65.0}]) + + assert result.compute_headroom_pct == 35.0 + assert result.mem_free_at_peak_bytes is None + assert result.mem_total_bytes is None + assert any("memory headroom unavailable" in note for note in result.diagnostics) + + +def test_complete_headroom_sample_stays_clean(): + result = gpu_headroom( + [{"util_pct": 65.0, "mem_used_bytes": 4_000, "mem_total_bytes": 10_000}], + serialized_concurrency_fraction=0.25, + ) + + assert result.compute_headroom_pct == 35.0 + assert result.mem_free_at_peak_bytes == 6_000 + assert result.diagnostics == [] + + +def test_empty_headroom_input_is_explicitly_unavailable(): + result = gpu_headroom([]) + + assert result.compute_headroom_pct is None + assert result.mem_free_at_peak_bytes is None + assert any("no samples" in note for note in result.diagnostics) + assert any("concurrency headroom unavailable" in note for note in result.diagnostics) + + +def test_empty_kernel_roi_renders_unavailable_instead_of_zero_time(): + rows = kernel_roi([("zero", 0), ("backwards", -1)]) + + assert rows == [] + assert render_roi_table(rows) == "kernel ROI unavailable: no positive-duration kernels" + + +@pytest.mark.parametrize("fraction", [-0.1, 1.1, float("nan")]) +def test_invalid_serialized_fraction_is_refused(fraction): + with pytest.raises(ValueError, match="serialized concurrency fraction"): + gpu_headroom([], serialized_concurrency_fraction=fraction) diff --git a/tests/test_hft_optimize.py b/tests/test_hft_optimize.py index 947014d..c8990f4 100644 --- a/tests/test_hft_optimize.py +++ b/tests/test_hft_optimize.py @@ -10,6 +10,7 @@ import numpy as np import pandas as pd +import pytest def _make_df(n: int = 4000, seed: int = 0): @@ -41,6 +42,14 @@ def test_optimize_keeps_only_correct_candidate(): assert r.baseline_eps > 0 and r.candidate_eps > 0 +@pytest.mark.parametrize("reps", [0, -1, True]) +def test_optimize_refuses_invalid_repetition_count(reps): + from gitm.benchmarks.hft.optimize import optimize_hft + + with pytest.raises(ValueError, match="reps"): + optimize_hft(_make_df(), pd, reps=reps) + + def test_optimize_rolls_back_a_divergent_candidate(monkeypatch): import gitm.benchmarks.hft.optimize as opt diff --git a/tests/test_importers.py b/tests/test_importers.py index ce22b95..c6c1a9d 100644 --- a/tests/test_importers.py +++ b/tests/test_importers.py @@ -17,6 +17,7 @@ from gitm.optimizer.headroom import build_headroom from gitm.optimizer.metrics import HardwarePeak, compute_metrics from gitm.optimizer.qualification import qualify +from gitm.tracer.capture import write_trace_jsonl from gitm.tracer.schema import KernelEvent, MemcpyEvent, SyncEvent, Trace from .conftest import make_kernel, make_trace @@ -105,6 +106,27 @@ def test_nsys_sync_enum_branches(): assert "device" in kinds +def test_nsys_unknown_enums_are_retained_as_import_diagnostics(tmp_path): + src = FIXTURES / "nsys_2024_min.sqlite" + dst = tmp_path / "unknown-enums.sqlite" + dst.write_bytes(src.read_bytes()) + conn = sqlite3.connect(dst) + conn.execute( + "UPDATE CUPTI_ACTIVITY_KIND_MEMCPY SET srcKind=NULL, dstKind=NULL, copyKind=999 " + "WHERE rowid=1" + ) + conn.execute( + "UPDATE CUPTI_ACTIVITY_KIND_SYNCHRONIZATION SET syncType=999 WHERE rowid=1" + ) + conn.commit() + conn.close() + + _, stats = import_nsys(dst, device=0) + + assert any("unknown CUPTI copyKind enum 999" in note for note in stats.warnings) + assert any("unknown CUPTI sync type enum 999" in note for note in stats.warnings) + + def test_nsys_multi_device_selection(): # Default: all devices all_traces, stats = import_nsys(FIXTURES / "nsys_2024_min.sqlite") @@ -143,11 +165,12 @@ def test_torch_json_and_gz(): def test_torch_array_form_and_missing_grid(): - ts, _ = import_torch_trace(FIXTURES / "torch_trace_array.json") + ts, stats = import_torch_trace(FIXTURES / "torch_trace_array.json") t = ts[0] assert t.kernels() # grid defaults to 1 when absent assert all(k.grid_x >= 1 and k.block_x >= 1 for k in t.kernels()) + assert any("grid dimensions" in warning for warning in stats.warnings) def test_torch_us_to_ns_conversion(): @@ -351,9 +374,77 @@ def test_dedupe_identical_rows(tmp_path): conn.close() with pytest.warns(UserWarning, match="deduped"): traces, stats = import_nsys(dst, device=0) - # dedupe count is on the per-device finish_trace; check events cleaned + assert stats.deduped == 1 + assert any("deduped 1" in note for note in stats.warnings) + assert traces[0].kernels() + artifact = tmp_path / "deduped-trace.jsonl" + write_trace_jsonl(artifact, traces[0]) + lines = artifact.read_text(encoding="utf-8").splitlines() + header = json.loads(lines[0])["_header"] + events = [json.loads(line) for line in lines[1:] if line.strip()] + Trace.model_validate({**header, "events": events}) + + +def test_invalid_event_drop_reaches_file_level_import_diagnostics(tmp_path): + src = FIXTURES / "parity_nsys.sqlite" + dst = tmp_path / "invalid.sqlite" + dst.write_bytes(src.read_bytes()) + conn = sqlite3.connect(dst) + conn.execute( + "UPDATE CUPTI_ACTIVITY_KIND_KERNEL SET end = start - 1 " + "WHERE rowid = (SELECT rowid FROM CUPTI_ACTIVITY_KIND_KERNEL LIMIT 1)" + ) + conn.commit() + conn.close() + + with pytest.warns(UserWarning, match="dropped 1 event"): + traces, stats = import_nsys(dst, device=0) + assert traces[0].kernels() - Trace.model_validate(traces[0].model_dump()) + assert stats.dropped_invalid == 1 + assert any("dropped 1 event" in note for note in stats.warnings) + + with pytest.warns(UserWarning, match="dropped 1 event"): + result = analyze_paths([dst], run_id="invalid-event-report") + assert "dropped 1 event" in result.report_md + + +def test_malformed_gpu_event_parse_drop_reaches_customer_report(tmp_path): + trace_path = tmp_path / "malformed-gpu-event.json" + trace_path.write_text( + json.dumps( + { + "traceEvents": [ + { + "ph": "X", + "cat": "kernel", + "name": "valid_kernel", + "ts": 1.0, + "dur": 100.0, + "args": {"device": 0, "stream": 1}, + }, + { + "ph": "X", + "cat": "kernel", + "name": "broken_kernel", + "ts": "not-a-timestamp", + "dur": 100.0, + "args": {"device": 0, "stream": 1}, + }, + ] + } + ), + encoding="utf-8", + ) + + traces, stats = import_torch_trace(trace_path) + assert len(traces[0].kernels()) == 1 + assert stats.total_raw_events == 2 + assert stats.dropped_invalid == 1 + assert any("dropped 1 GPU event" in note for note in stats.warnings) + + result = analyze_paths([trace_path], run_id="malformed-gpu-event-report") + assert "dropped 1 GPU event" in result.report_md def test_atomic_write(tmp_path): @@ -447,7 +538,7 @@ def test_golden_customer_report(tmp_path): if os.environ.get("GITM_UPDATE_GOLDEN") == "1": golden.write_text(md) - assert md == golden.read_text(), ( + assert md == golden.read_text(encoding="utf-8"), ( "customer report drifted from golden — set GITM_UPDATE_GOLDEN=1 if intentional" ) diff --git a/tests/test_injection.py b/tests/test_injection.py index d7d9f31..9ff31d8 100644 --- a/tests/test_injection.py +++ b/tests/test_injection.py @@ -93,11 +93,22 @@ def test_partial_trailing_line_from_a_killed_process_is_tolerated(run_env): shard = run_env.with_name(run_env.name + ".9335") shard.write_text(_kernel("decode_step", 10, 20) + "\n" + '{"kind":"kernel","na') - events = injection.read_shards() + with pytest.warns(RuntimeWarning, match="dropped 1 malformed or incomplete"): + events = injection.read_shards() assert [e.name for e in events] == ["decode_step"] +@pytest.mark.parametrize("raw", ["not-a-number", "-1", "nan", "inf"]) +def test_invalid_settle_override_warns_and_uses_documented_default(monkeypatch, raw): + monkeypatch.setenv(injection.ENV_SETTLE, raw) + + with pytest.warns(RuntimeWarning, match="invalid GITM_TRACE_SETTLE_S"): + value = injection.settle_seconds() + + assert value == injection.DEFAULT_SETTLE_S + + def test_arm_marker_is_not_mistaken_for_a_shard(run_env): injection.arm() run_env.with_name(run_env.name + ".9335").write_text(_kernel("decode_step", 10, 20) + "\n") diff --git a/tests/test_kernel_taxonomy.py b/tests/test_kernel_taxonomy.py index c1a3710..a820172 100644 --- a/tests/test_kernel_taxonomy.py +++ b/tests/test_kernel_taxonomy.py @@ -159,6 +159,29 @@ def test_empty_trace_warns_about_graph_replay_and_stops_there(): assert "CUDA-graph replay" in warnings[0] +def test_nonpositive_kernel_durations_are_excluded_and_named(): + kernels = [ + make_kernel("fused_moe_kernel", start_ns=10, end_ns=10), + make_kernel("ampere_fp16_s16816gemm_tn", start_ns=20, end_ns=19), + ] + + bd = summarize_kernels(kernels, window_ns=100) + + assert bd.n_kernels == 0 + assert bd.n_invalid_duration == 2 + assert bd.kernel_time_ns == 0 + assert any("non-positive duration" in warning for warning in bd.warnings()) + + +def test_nonpositive_capture_window_does_not_fabricate_active_share(): + bd = summarize_kernels( + [make_kernel("fused_moe_kernel", start_ns=0, end_ns=10)], window_ns=0 + ) + + assert bd.gpu_active_share is None + assert any("capture window duration is non-positive" in warning for warning in bd.warnings()) + + def test_idle_looking_gpu_is_warned_about(): bd = summarize_kernels([make_kernel("fused_moe_kernel", start_ns=0, end_ns=50)], window_ns=10_000) diff --git a/tests/test_kitti_benchmark.py b/tests/test_kitti_benchmark.py index 76b8ef1..a0b9874 100644 --- a/tests/test_kitti_benchmark.py +++ b/tests/test_kitti_benchmark.py @@ -29,13 +29,14 @@ def test_workunit_result_stall_fracs_sum_to_one(): assert abs(total - 1.0) < 1e-9 -def test_workunit_result_zero_total_returns_zero_fracs(): +def test_workunit_result_zero_total_refuses_fabricated_zero_fracs(): + import pytest + from gitm.benchmarks.kitti.workunit import WorkUnitResult r = WorkUnitResult(frame_id="000000", n_detections=0, t_total_s=0.0) - assert r.data_stall_frac == 0.0 - assert r.sync_stall_frac == 0.0 - assert r.gpu_active_frac == 0.0 + with pytest.raises(RuntimeError, match="timing unavailable"): + _ = r.data_stall_frac def test_workunit_result_detections_default_empty(): @@ -50,6 +51,20 @@ def test_workunit_importable_without_openpcdet(): from gitm.benchmarks.kitti.workunit import WorkUnit # noqa: F401 +def test_plan_kitti_cli_wires_known_hardware_graph(capsys): + from gitm.cli import main + + assert main(["plan-kitti", "--sku", "A100-SXM4-80GB"]) == 0 + assert "PointPillars predicted execution graph" in capsys.readouterr().out + + +def test_plan_kitti_cli_refuses_unknown_hardware(capsys): + from gitm.cli import main + + assert main(["plan-kitti", "--sku", "mystery-gpu"]) == 3 + assert "prediction refused" in capsys.readouterr().out + + def test_workunit_from_checkpoint_raises_import_error_without_openpcdet( monkeypatch, ): diff --git a/tests/test_metrics.py b/tests/test_metrics.py index 3ae6831..4a99df4 100644 --- a/tests/test_metrics.py +++ b/tests/test_metrics.py @@ -65,6 +65,42 @@ def test_rejects_bad_recompute_fraction(): compute_metrics(_trace(), PEAK, recompute_fraction=1.0) +def test_refuses_nonpositive_trace_duration(): + trace = _trace().model_copy(update={"duration_ns": 0}) + with pytest.raises(RuntimeError, match="utilization metrics timing unavailable"): + compute_metrics(trace, PEAK) + + +@pytest.mark.parametrize( + ("start_ns", "end_ns"), + [(-1, 10), (20, 10), (50, 250 * US)], +) +def test_invalid_kernel_window_refuses_clamped_busy_fraction(start_ns, end_ns): + trace = _trace() + bad = trace.events[0].model_copy(update={"start_ns": start_ns, "end_ns": end_ns}) + trace = trace.model_copy(update={"events": [bad, *trace.events[1:]]}) + + with pytest.raises(RuntimeError, match="kernel timing.*outside trace window"): + compute_metrics(trace, PEAK) + + +def test_zero_duration_kernel_is_warned_not_counted_as_busy(): + trace = _trace() + pad = trace.events[0].model_copy(update={"start_ns": 200 * US, "end_ns": 200 * US}) + trace = trace.model_copy(update={"events": [*trace.events, pad]}) + + with pytest.warns(RuntimeWarning, match="zero-duration kernel"): + result = compute_metrics(trace, PEAK) + + assert result.busy_fraction == pytest.approx(0.5) + + +def test_refuses_unpriced_memory_bandwidth(): + peak = HardwarePeak(name="UNKNOWN", peak_flops=1e14, peak_bw_bytes_s=0.0) + with pytest.raises(RuntimeError, match="peak bandwidth must be positive"): + compute_metrics(_trace(), peak) + + def test_stall_breakdown_transfer_and_idle(): # _trace() gaps: [50,100]us and [150,200]us; the memcpy [60,65]us sits in the # first gap -> 5us transfer-bound, the remaining 95us of idle is a long stall. diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py index 0b365ee..64a66ce 100644 --- a/tests/test_moe_graph.py +++ b/tests/test_moe_graph.py @@ -15,10 +15,13 @@ from __future__ import annotations +from dataclasses import replace + import pytest from gitm.optimizer.deviation import classify_op from gitm.planner.context import hardware_spec_for, peak_for_sku +from gitm.planner.graph import Graph, PredictedNode from gitm.planner.moe_graph import ( effective_kv_tokens, index_candidates, @@ -32,9 +35,12 @@ from gitm.planner.roofline import ( BatchConfig, HardwareSpec, + ModelSpec, ShardingConfig, resolve_peak, + roofline, weight_bytes, + weight_bytes_is_fallback, ) # The shape of DeepSeek-V4-Flash-0731, trimmed to the keys the planner reads. @@ -640,7 +646,9 @@ def test_kv_footprint_splits_growing_from_fixed(spec): # Growth comes only from the 41 compressed layers, at 1/4 or 1/128 of a latent. naive_all_layers = spec.n_layers * (spec.kv_latent_dim + spec.index_head_dim) - assert per_token < naive_all_layers / 5 + # The direct config builder keeps KV dtype tied to the declared activation + # dtype; serving may override it after the live cache dtype is known. + assert per_token < naive_all_layers / 3 # The two window layers are real, bounded, and paid once per sequence. assert fixed == 2 * spec.sliding_window * kv_entry_bytes(spec) @@ -681,3 +689,49 @@ def test_indexer_is_not_misfiled_as_elementwise(): assert classify_kernel("lightning_indexer_topk_kernel") == "attention" assert classify_kernel("flash_mla_sparse_fwd_kernel") == "attention" + + +@pytest.mark.parametrize( + "missing_key", + ["hidden_size", "n_routed_experts", "compress_ratios", "expert_dtype", "torch_dtype"], +) +def test_public_hf_builder_refuses_missing_shape_or_precision(missing_key): + cfg = dict(V4_CONFIG) + cfg.pop(missing_key) + with pytest.raises(ValueError, match=missing_key): + spec_from_hf_config(cfg) + + +def test_unknown_weight_dtype_is_fail_open_but_explicitly_flagged(spec, b200): + unknown = replace(spec, expert_dtype="future_fp3") + graph = predict_moe_graph(unknown, b200, BatchConfig(batch=1, kv_cache_len=1024)) + assert weight_bytes("future_fp3") == weight_bytes("bf16") + assert weight_bytes_is_fallback("future_fp3") + assert graph.has_fallback_bytes + assert {n.op for n in graph.nodes if n.prediction.bytes_are_fallback} == { + "moe_shared", "moe_routed" + } + + +def test_known_weight_dtypes_leave_bytes_fallback_clean(spec, b200): + graph = predict_moe_graph(spec, b200, BatchConfig(batch=1, kv_cache_len=1024)) + assert not weight_bytes_is_fallback("fp4") + assert not graph.has_fallback_bytes + assert not any(n.prediction.bytes_are_fallback for n in graph.nodes) + + +def test_mixed_byte_node_flags_unknown_kv_dtype(spec, b200): + unknown = replace(spec, kv_dtype="future_kv3") + graph = predict_moe_graph(unknown, b200, BatchConfig(batch=1, kv_cache_len=1024)) + flagged = {n.op for n in graph.nodes if n.prediction.bytes_are_fallback} + assert {"attn_kv_a", "attn_score_value"} <= flagged + assert "moe_router" not in flagged + + +def test_graph_preserves_each_missing_roofline_denominator(spec): + hw = HardwareSpec(peak_flops_fp16_per_s=0.0, peak_mem_bw_bytes_per_s=1e9) + node = PredictedNode("partial", None, roofline("partial", 1e12, 1024.0, hw)) + graph = Graph(model=ModelSpec(), hw=hw, batch=BatchConfig(), nodes=[node]) + assert graph.has_unpriced_nodes + assert graph.has_unpriced_compute + assert not graph.has_unpriced_memory diff --git a/tests/test_moe_graph_degenerate.py b/tests/test_moe_graph_degenerate.py index 688ae29..7d639aa 100644 --- a/tests/test_moe_graph_degenerate.py +++ b/tests/test_moe_graph_degenerate.py @@ -82,12 +82,10 @@ def test_no_sliding_window_means_global_not_zero(spec, hw): assert attn > 0 -def test_a_config_stripped_to_nothing_still_prices_attention(hw): - """``spec_from_hf_config({})`` takes every default, including no - compression and no window — the exact shape that used to cost zero.""" - g = predict_moe_graph(spec_from_hf_config({}), hw, BatchConfig(batch=8, kv_cache_len=4096)) - attn = sum(n.prediction.flops for n in g.nodes if n.op == "attn_score_value") - assert attn > 0 +def test_a_config_stripped_to_nothing_is_refused_instead_of_defaulted(hw): + """A missing live config cannot inherit a plausible sparse architecture.""" + with pytest.raises(ValueError, match="hidden_size"): + spec_from_hf_config({}) # ── degenerate but legal shapes stay finite and non-negative ──────────────── @@ -125,13 +123,18 @@ def test_degenerate_shapes_stay_finite(spec, hw, label, mutation): assert g.total_pred_s > 0, label -@pytest.mark.parametrize("batch,ctx", [(1, 0), (1, 1), (0, 4096), (1, 1 << 24), (4096, 1)]) +@pytest.mark.parametrize("batch,ctx", [(1, 0), (1, 1), (1, 1 << 24), (4096, 1)]) def test_extreme_batch_and_context_stay_finite(spec, hw, batch, ctx): g = predict_moe_graph(spec, hw, BatchConfig(batch=batch, kv_cache_len=ctx)) assert _all_finite(g) assert math.isfinite(g.total_pred_s) +def test_zero_batch_is_refused_instead_of_becoming_a_zero_work_graph(spec, hw): + with pytest.raises(ValueError, match="batch must be a positive integer"): + BatchConfig(batch=0) + + def test_expert_parallelism_beyond_the_expert_count_stays_finite(spec, hw): g = predict_moe_graph( spec, hw, BatchConfig(batch=8, kv_cache_len=4096), ShardingConfig(tp=8, ep=512) diff --git a/tests/test_moe_roofline.py b/tests/test_moe_roofline.py index 4aa6dee..29c58f6 100644 --- a/tests/test_moe_roofline.py +++ b/tests/test_moe_roofline.py @@ -13,7 +13,14 @@ import pytest from gitm.planner.graph import predict_graph -from gitm.planner.roofline import BatchConfig, HardwareSpec, ModelSpec, distinct_experts +from gitm.planner.roofline import ( + BatchConfig, + HardwareSpec, + ModelSpec, + ShardingConfig, + distinct_experts, + roofline, +) # Qwen3.6-35B-A3B-FP8 shaped: narrow hidden, many narrow experts, one shared # expert, fp8 weights with bf16 activations. @@ -84,6 +91,20 @@ def test_dense_spec_is_not_moe_and_falls_back(): assert m.expert_intermediate == m.intermediate +def test_dense_graph_uses_configured_compute_dtype_for_peak_selection(): + hw = HardwareSpec( + peak_flops_fp16_per_s=100.0, + peak_flops_fp32_per_s=10.0, + peak_mem_bw_bytes_per_s=1e30, + ) + + graph = predict_graph(ModelSpec(n_layers=1, compute_dtype="fp32"), hw) + + assert graph.nodes + assert all(node.prediction.dtype == "fp32" for node in graph.nodes) + assert all(node.prediction.peak_dtype == "fp32" for node in graph.nodes) + + def test_moe_spec_properties(): assert MOE.is_moe assert MOE.top_k == 8 @@ -92,10 +113,41 @@ def test_moe_spec_properties(): assert MOE.shared_intermediate == 768 # falls back to the routed width -def test_num_experts_without_top_k_stays_dense(): - """Half-configured is dense, not a mixture — no silent guessing.""" - assert not ModelSpec(num_experts=256).is_moe - assert not ModelSpec(experts_per_token=8).is_moe +@pytest.mark.parametrize( + "kwargs", + [ + {"num_experts": 256}, + {"experts_per_token": 8}, + {"num_experts": 4, "experts_per_token": 8}, + {"weight_dtype_bytes": 0}, + {"moe_layer_step": 0}, + {"full_attn_layer_step": 0}, + ], +) +def test_model_spec_refuses_values_that_were_silently_clamped_or_defaulted(kwargs): + with pytest.raises(ValueError): + ModelSpec(**kwargs) + + +@pytest.mark.parametrize( + "factory", + [ + lambda: BatchConfig(batch=0), + lambda: BatchConfig(speculative_tokens=-1), + lambda: BatchConfig(acceptance_rate=1.1), + lambda: ShardingConfig(tp=0), + lambda: ShardingConfig(ep_imbalance=0.5), + ], +) +def test_decode_and_sharding_specs_refuse_invalid_fallback_inputs(factory): + with pytest.raises(ValueError): + factory() + + +@pytest.mark.parametrize("flops, bytes_moved", [(-1.0, 1.0), (1.0, -1.0), (float("nan"), 1.0)]) +def test_roofline_refuses_invalid_work_terms(flops, bytes_moved): + with pytest.raises(ValueError, match="finite and non-negative"): + roofline("bad", flops, bytes_moved, H100) # --- graph: dense back-compat ------------------------------------------------- @@ -398,32 +450,6 @@ def test_hybrid_and_moe_compose(): assert all(n.prediction.bytes > 0 for n in g.nodes) -def test_hf_attention_shape_survives_a_dense_ffn(): - """Regression: attention shape and FFN sparsity are independent axes, so a - hybrid model with a dense FFN must still get full_attn_layer_step.""" - from gitm.scheduler.loop import _moe_fields_from_hf - - class HybridDense: # hybrid attention, no experts - full_attention_interval = 4 - - class PlainMoE: # experts, conventional attention - num_experts = 64 - num_experts_per_tok = 4 - - assert _moe_fields_from_hf(HybridDense()) == {"full_attn_layer_step": 4} - got = _moe_fields_from_hf(PlainMoE()) - assert got["num_experts"] == 64 and "full_attn_layer_step" not in got - - -def test_hf_half_configured_moe_stays_dense(): - from gitm.scheduler.loop import _moe_fields_from_hf - - class OnlyExpertCount: - num_experts = 64 # no top-k - - assert _moe_fields_from_hf(OnlyExpertCount()) == {} - - # --- real batch from scheduler stats ----------------------------------------- diff --git a/tests/test_node_rollup_math.py b/tests/test_node_rollup_math.py index d99a4f5..d42191b 100644 --- a/tests/test_node_rollup_math.py +++ b/tests/test_node_rollup_math.py @@ -125,6 +125,13 @@ def test_device_comm_no_comm_kernels(): assert cs.comm_share_of_busy == 0.0 +@pytest.mark.parametrize("duration", [0, -1]) +def test_device_comm_refuses_nonpositive_wall_time(duration: int): + tr = _trace([], duration=duration) + with pytest.raises(RuntimeError, match="communication rollup timing unavailable"): + device_comm_stats(tr) + + # ── rollup skew / collective flags ─────────────────────────────────────────── @@ -164,6 +171,12 @@ def test_rollup_weighted_ceiling(): assert r.node_ceiling_distance == pytest.approx(0.325) +def test_rollup_refuses_nonpositive_device_wall_time(): + invalid = _trace([], duration=0) + with pytest.raises(RuntimeError, match="communication rollup timing unavailable"): + build_node_rollup([(invalid, 0.0, 0.4)], multi_device_file=False) + + def test_rollup_comm_inconclusive(): t0 = _trace([_k("gemm", 0, 50, device=0)], duration=100) t1 = _trace([_k("gemm", 0, 50, device=1)], duration=100) diff --git a/tests/test_openfold_workload.py b/tests/test_openfold_workload.py index 32dbdde..4fc4cb9 100644 --- a/tests/test_openfold_workload.py +++ b/tests/test_openfold_workload.py @@ -12,6 +12,9 @@ from contextlib import contextmanager from pathlib import Path +from types import SimpleNamespace + +import pytest from .conftest import make_kernel, make_trace @@ -25,6 +28,41 @@ def test_openfold_is_registered(): assert get_factory("alphafold") is not None +@pytest.mark.parametrize( + ("kwargs", "message"), + [ + ({"n_proteins": 0, "max_len": 384, "warmup": 0, "plddt_tol": 1.0}, "n_proteins"), + ({"n_proteins": 1, "max_len": 0, "warmup": 0, "plddt_tol": 1.0}, "max_len"), + ({"n_proteins": 1, "max_len": 384, "warmup": -1, "plddt_tol": 1.0}, "warmup"), + ({"n_proteins": 1, "max_len": 384, "warmup": 0, "plddt_tol": float("nan")}, "plddt_tol"), + ], +) +def test_openfold_ab_refuses_invalid_gate_controls(tmp_path, kwargs, message): + from benchmarks.biotech.optimize import optimize_af2 + + with pytest.raises(ValueError, match=message): + optimize_af2(tmp_path, 42, **kwargs) + + +def test_openfold_ab_refuses_zero_duration(monkeypatch): + import benchmarks.biotech.optimize as opt + + monkeypatch.setitem( + __import__("sys").modules, + "torch", + SimpleNamespace(cuda=SimpleNamespace(is_available=lambda: False)), + ) + ticks = iter([1.0, 1.0]) + monkeypatch.setattr(opt.time, "perf_counter", lambda: next(ticks)) + + class Runner: + def predict(self, _record, _msa): + return {"plddt": 90.0} + + with pytest.raises(RuntimeError, match="timing unavailable"): + opt._fold_all(Runner(), [(object(), object())]) + + def test_openfold_no_deps_or_data_degrades_to_no_data(tmp_path: Path, monkeypatch): """No staged data (and/or no OpenFold) → honest no-data, not a crash or fake.""" monkeypatch.setenv("GITM_BENCH_STAGE", str(tmp_path / "missing")) @@ -83,7 +121,12 @@ def _fake_capture_with(entered, prefix="cutlass_sm90_gemm"): def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None): entered["capture"] = True kernels = [ - make_kernel(f"{prefix}_{i % 4}", start_ns=i * 100, end_ns=i * 100 + 90 + (i % 9)) + make_kernel( + f"{prefix}_{i % 4}", + start_ns=i * 100, + end_ns=i * 100 + 90 + (i % 9), + stream_id=i % 2, + ) for i in range(80) ] yield make_trace(events=kernels, vendor="nvidia", run_id=run_id or "r") diff --git a/tests/test_planner_roofline.py b/tests/test_planner_roofline.py index abcd318..5e3ea87 100644 --- a/tests/test_planner_roofline.py +++ b/tests/test_planner_roofline.py @@ -58,16 +58,20 @@ def test_roofline_a100_compute_bound_reference(): assert pred.t_pred_s == pytest.approx(expected_t_compute, rel=1e-9) -# ── dtype selection: fp16/bf16 share peak; fp32 takes the slower path ─────── +# ── dtype selection: each declared hardware ceiling is honored ────────────── -def test_roofline_dtype_selects_fp16_peak_for_bf16(): - hw = HardwareSpec() +def test_roofline_dtype_selects_the_explicit_bf16_peak(): + hw = HardwareSpec( + peak_flops_fp16_per_s=100e12, + peak_flops_bf16_per_s=200e12, + ) fp16 = roofline("op", flops=1e12, bytes_moved=0, hw=hw, dtype="fp16") bf16 = roofline("op", flops=1e12, bytes_moved=0, hw=hw, dtype="bf16") - assert fp16.t_compute_s == pytest.approx(bf16.t_compute_s, rel=1e-12) - # Both must pick the fp16 peak (312e12), not the fp32 peak (19.5e12). - assert fp16.t_compute_s == pytest.approx(1e12 / 312e12, rel=1e-9) + assert fp16.t_compute_s == pytest.approx(1e12 / 100e12, rel=1e-9) + assert bf16.t_compute_s == pytest.approx(1e12 / 200e12, rel=1e-9) + assert bf16.peak_dtype == "bf16" + assert not bf16.peak_is_fallback def test_roofline_dtype_fp32_uses_slower_peak(): @@ -111,3 +115,14 @@ def test_roofline_zero_peak_rates_dont_divide_by_zero(): assert pred.t_compute_s == 0.0 assert pred.t_memory_s == 0.0 assert pred.t_pred_s == 0.0 + assert pred.compute_is_unpriced + assert pred.memory_is_unpriced + + +def test_nonzero_memory_time_does_not_hide_missing_compute_rate(): + hw = HardwareSpec(peak_flops_fp16_per_s=0.0, peak_mem_bw_bytes_per_s=1e9) + pred = roofline("partial", flops=1e12, bytes_moved=1e6, hw=hw) + + assert pred.t_pred_s > 0.0 + assert pred.compute_is_unpriced + assert not pred.memory_is_unpriced diff --git a/tests/test_predict_graph_from_engine.py b/tests/test_predict_graph_from_engine.py index a1c278e..1609f33 100644 --- a/tests/test_predict_graph_from_engine.py +++ b/tests/test_predict_graph_from_engine.py @@ -1,66 +1,129 @@ -"""The predicted graph must match the model that actually ran. - -`predict_graph()` defaults to Llama-2-7B (32 layers). When the loop has a live -engine, `_model_spec_from_engine` reads the real architecture off its HF config -so residuals/deviation score against the right model. These tests use a fake -duck-typed engine (no vLLM/GPU) to pin the config-reading + fallback behavior. -""" +"""The production dense-config parser must preserve the live architecture.""" from __future__ import annotations -from types import SimpleNamespace +import pytest from gitm.planner.graph import predict_graph -from gitm.scheduler.loop import _model_spec_from_engine - - -def _fake_engine(**hf_fields): - """A stand-in vLLM engine exposing ``llm_engine.model_config.hf_config``.""" - hf = SimpleNamespace(**hf_fields) - return SimpleNamespace(llm_engine=SimpleNamespace(model_config=SimpleNamespace(hf_config=hf))) - - -# opt-125m: 12 layers, hidden 768, 12 heads (MHA, no GQA), intermediate 3072. -_OPT_125M = dict( - num_hidden_layers=12, - hidden_size=768, - num_attention_heads=12, - intermediate_size=3072, - vocab_size=50272, -) +from gitm.planner.roofline import BatchConfig, ShardingConfig +from gitm.scheduler.loop import _dense_spec_from_config + +# opt-125m: 12 layers, hidden 768, 12 heads (MHA), intermediate 3072. +_OPT_125M = { + "num_hidden_layers": 12, + "hidden_size": 768, + "num_attention_heads": 12, + "intermediate_size": 3072, + "vocab_size": 50272, + "torch_dtype": "bf16", +} + + +def _spec(**overrides): + spec, error = _dense_spec_from_config({**_OPT_125M, **overrides}) + assert error == "" + assert spec is not None + return spec -def test_reads_real_model_arch_from_engine(): - spec = _model_spec_from_engine(_fake_engine(**_OPT_125M)) - assert spec is not None +def test_reads_real_model_arch_from_config(): + spec = _spec() assert spec.n_layers == 12 assert spec.hidden == 768 assert spec.n_heads == 12 - assert spec.num_kv_heads == 12 # no num_key_value_heads -> falls back to n_heads - assert spec.head_dim == 64 # 768 / 12 + assert spec.num_kv_heads == 12 + assert spec.head_dim == 64 assert spec.intermediate == 3072 assert spec.vocab == 50272 -def test_predicted_graph_matches_the_real_model_not_llama(): - """opt-125m -> 12*5+1 = 61 nodes, not the 32*5+1 = 161 Llama default.""" - spec = _model_spec_from_engine(_fake_engine(**_OPT_125M)) - assert len(predict_graph(model=spec).nodes) == 61 - assert len(predict_graph().nodes) == 161 # default is still Llama-7B +def test_predicted_graph_matches_real_model_not_default(): + assert len(predict_graph(model=_spec()).nodes) == 61 + assert len(predict_graph().nodes) == 161 + + +def test_gqa_and_hybrid_attention_fields_survive_config_parsing(): + spec = _spec(num_key_value_heads=4, full_attention_interval=3) + assert spec.num_kv_heads == 4 + assert spec.full_attn_layer_step == 3 + + +def test_missing_answer_deciding_field_refuses(): + cfg = dict(_OPT_125M) + del cfg["vocab_size"] + spec, error = _dense_spec_from_config(cfg) + assert spec is None + assert "vocab_size" in error + + +def test_unknown_dense_dtype_refuses_instead_of_becoming_bf16(): + spec, error = _dense_spec_from_config({**_OPT_125M, "torch_dtype": "mystery4"}) + assert spec is None + assert "not priceable" in error + + +def test_missing_dense_dtype_refuses_instead_of_becoming_bf16(): + cfg = dict(_OPT_125M) + del cfg["torch_dtype"] + + spec, error = _dense_spec_from_config(cfg) + assert spec is None + assert "torch_dtype" in error -def test_gqa_uses_num_key_value_heads_when_present(): - spec = _model_spec_from_engine( - _fake_engine(**{**_OPT_125M, "num_key_value_heads": 4}) + +def test_malformed_dense_quantization_config_refuses(): + spec, error = _dense_spec_from_config( + {**_OPT_125M, "quantization_config": ["fp8"]} ) + + assert spec is None + assert "quantization_config" in error + + +def test_dense_parser_preserves_fp32_compute_dtype(): + spec, error = _dense_spec_from_config({**_OPT_125M, "torch_dtype": "float32"}) + + assert error == "" assert spec is not None - assert spec.num_kv_heads == 4 + assert spec.compute_dtype == "fp32" + assert spec.dtype_bytes == 4 + + +def test_dense_graph_prices_tensor_parallel_work_per_rank(): + whole = predict_graph(model=_spec(num_key_value_heads=4)) + sharded = predict_graph( + model=_spec(num_key_value_heads=4), sharding=ShardingConfig(tp=2) + ) + + whole_qkv = next(n.prediction for n in whole.nodes if n.op == "qkv_proj") + rank_qkv = next(n.prediction for n in sharded.nodes if n.op == "qkv_proj") + assert sharded.sharding.tp == 2 + assert rank_qkv.flops == whole_qkv.flops / 2 + assert any(n.op == "tp_all_reduce" for n in sharded.nodes) + + +def test_quantized_dense_width_applies_to_attention_and_lm_head_weights(): + bf16 = predict_graph(model=_spec()) + fp8 = predict_graph( + model=_spec(quantization_config={"quant_method": "fp8"}) + ) + + for op in ("qkv_proj", "attn_out_proj", "lm_head"): + bf16_bytes = next(n.prediction.bytes for n in bf16.nodes if n.op == op) + fp8_bytes = next(n.prediction.bytes for n in fp8.nodes if n.op == op) + assert fp8_bytes < bf16_bytes + + +def test_dense_graph_honors_speculative_positions(): + plain = predict_graph(model=_spec(), batch=BatchConfig(batch=2, speculative_tokens=0)) + drafted = predict_graph(model=_spec(), batch=BatchConfig(batch=2, speculative_tokens=3)) + + plain_qkv = next(n.prediction for n in plain.nodes if n.op == "qkv_proj") + drafted_qkv = next(n.prediction for n in drafted.nodes if n.op == "qkv_proj") + assert drafted_qkv.flops == plain_qkv.flops * 4 -def test_falls_back_to_none_when_no_engine_or_config(): - # No engine at all -> None (loop then uses the default graph). - assert _model_spec_from_engine(None) is None - # An engine with no readable HF config -> None, never a crash. - assert _model_spec_from_engine(SimpleNamespace(nope=1)) is None - # A config missing required fields -> None (the int() raises, caught). - assert _model_spec_from_engine(_fake_engine(hidden_size=768)) is None +def test_dense_graph_refuses_incompatible_tensor_parallel_shape(): + with pytest.raises(ValueError, match="n_heads"): + predict_graph(model=_spec(), sharding=ShardingConfig(tp=5)) diff --git a/tests/test_report_snapshot.py b/tests/test_report_snapshot.py index 30a2579..94b75c0 100644 --- a/tests/test_report_snapshot.py +++ b/tests/test_report_snapshot.py @@ -19,9 +19,12 @@ from __future__ import annotations +import math import os from pathlib import Path +import pytest + from gitm.optimizer.report import Claim, Provenance, write_report GOLDEN_PATH = Path(__file__).parent / "golden" / "report_basic.md" @@ -109,3 +112,92 @@ def test_report_renders_byte_equal_to_golden(monkeypatch): " UPDATE_GOLDENS=1 .venv/bin/pytest tests/test_report_snapshot.py\n" "and review the diff with `git diff tests/golden/` before committing." ) + + +@pytest.mark.parametrize( + ("raw", "display", "raw_text"), + [(17.8, "+100.0%", "+1780.0%"), (-3.0, "-100.0%", "-300.0%")], +) +def test_saturated_residual_renders_capped_display_and_raw_magnitude(raw, display, raw_text): + claim = Claim( + summary="model gap", + residual_invariant="kernel_time", + residual_value=raw, + residual_scope="run", + causal_evidence="trace", + intervention_name="candidate", + predicted_delta=0.01, + measured_delta=None, + ) + + rendered = write_report([claim], _fixed_provenance()) + + assert claim.residual_value == raw + assert claim.residual_saturated + assert display in rendered + assert f"raw {raw_text}" in rendered + assert "`kernel_time` (run)" in rendered + + +@pytest.mark.parametrize("raw", [-1.0, 0.42, 1.0]) +def test_unsaturated_residual_has_no_raw_magnitude_note(raw): + claim = Claim( + summary="model gap", + residual_invariant="kernel_time", + residual_value=raw, + causal_evidence="trace", + intervention_name="candidate", + predicted_delta=0.01, + measured_delta=None, + ) + + assert claim.residual_display_value == raw + assert not claim.residual_saturated + assert "display capped" not in write_report([claim], _fixed_provenance()) + + +@pytest.mark.parametrize("raw", [math.nan, math.inf, -math.inf]) +def test_non_finite_residual_is_refused(raw): + with pytest.raises(ValueError, match="residual_value must be finite"): + Claim( + summary="model gap", + residual_invariant="kernel_time", + residual_value=raw, + causal_evidence="trace", + intervention_name="candidate", + predicted_delta=0.01, + measured_delta=None, + ) + + +@pytest.mark.parametrize("field", ["predicted_delta", "measured_delta"]) +@pytest.mark.parametrize("value", [math.nan, math.inf, -math.inf]) +def test_non_finite_claim_deltas_are_refused(field, value): + kwargs = { + "summary": "model gap", + "residual_invariant": "kernel_time", + "residual_value": 0.1, + "causal_evidence": "trace", + "intervention_name": "candidate", + "predicted_delta": 0.01, + "measured_delta": 0.02, + } + kwargs[field] = value + + with pytest.raises(ValueError, match=f"{field} must be finite"): + Claim(**kwargs) + + +def test_runtime_diagnostics_are_printed_when_present(): + rendered = write_report( + [], + _fixed_provenance(), + runtime_diagnostics=["residual coverage: matched 40.0% of kernel time"], + ) + + assert "## Runtime diagnostics" in rendered + assert "matched 40.0% of kernel time" in rendered + + +def test_runtime_diagnostics_section_is_absent_when_clean(): + assert "## Runtime diagnostics" not in write_report([], _fixed_provenance()) diff --git a/tests/test_routing.py b/tests/test_routing.py new file mode 100644 index 0000000..5c2c7fe --- /dev/null +++ b/tests/test_routing.py @@ -0,0 +1,19 @@ +from __future__ import annotations + +import pytest + +from gitm.routing.scorer_v0 import score_prospect + + +def test_unknown_company_tier_is_not_silently_scored_as_tier_three(): + with pytest.raises(ValueError, match="company_tier"): + score_prospect(0.5, 0.5, 99, 1, 0.5, 0) + + +def test_out_of_range_inputs_are_refused(): + with pytest.raises(ValueError, match=r"within \[0, 1\]"): + score_prospect(1.5, 0.5, 1, 1, 0.5, 0) + + +def test_valid_score_contract_is_unchanged(): + assert score_prospect(1.0, 1.0, 1, 1, 1.0, 1) == 100.0 diff --git a/tests/test_run_loop_workload.py b/tests/test_run_loop_workload.py index ca919e9..a8a4f69 100644 --- a/tests/test_run_loop_workload.py +++ b/tests/test_run_loop_workload.py @@ -9,6 +9,7 @@ from contextlib import contextmanager from pathlib import Path +from types import SimpleNamespace import pytest @@ -19,6 +20,22 @@ EMPTY_DIGEST = "4f53cda18c2baa0c" +@pytest.mark.parametrize("budget", ["0s", "0.0h"]) +def test_loop_refuses_nonpositive_budget(budget, tmp_path): + from gitm import optimize + + with pytest.raises(ValueError, match="budget must be positive"): + optimize(workload="custom", budget=budget, scratch=str(tmp_path)) + + +@pytest.mark.parametrize("target", [0.0, -0.1, 1.01, float("nan"), float("inf")]) +def test_loop_refuses_invalid_target_floor(target, tmp_path): + from gitm import optimize + + with pytest.raises(ValueError, match="target floor"): + optimize(workload="custom", budget="1s", target=target, scratch=str(tmp_path)) + + def test_no_data_guard_does_not_fabricate_claims(tmp_path: Path): """No GPU/shim and no registered runner → honest no-data, zero claims.""" from gitm import optimize @@ -34,6 +51,44 @@ def test_no_data_guard_does_not_fabricate_claims(tmp_path: Path): assert "NO DATA" in result["report_md"] +def test_zero_duration_kernel_does_not_bypass_no_data_guard(tmp_path, monkeypatch): + import gitm.scheduler.loop as loop + + @contextmanager + def invalid_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None): + yield make_trace( + events=[make_kernel("invalid", start_ns=10, end_ns=10)], + vendor="nvidia", + run_id=run_id or "r", + ) + + monkeypatch.setattr(loop, "capture", invalid_capture) + monkeypatch.setattr(loop, "sync_device", lambda: None) + + from gitm import optimize + + result = optimize( + workload="custom", + budget="1s", + scratch=str(tmp_path), + workload_runner=lambda: {"events": 1}, + ) + + assert result["summary"]["status"] == "no_data" + assert result["summary"]["n_claims"] == 0 + assert "positive-duration" in result["report_md"] + + +def test_qualification_direct_caller_refuses_invalid_kernel_durations(): + from gitm.optimizer.qualification import qualify + + trace = make_trace(events=[make_kernel("broken", start_ns=20, end_ns=10)]) + result = qualify(trace) + + assert result.commit is False + assert "non-positive duration" in result.diagnostic + + _UNSET = object() # identity sentinel — a real workload_id could legitimately be any string @@ -215,6 +270,43 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None): return fake_capture +def _priceable_moe_engine(): + cfg = SimpleNamespace( + model_type="deepseek_v4", + hidden_size=64, + num_hidden_layers=2, + num_attention_heads=1, + num_key_value_heads=1, + head_dim=64, + qk_rope_head_dim=16, + q_lora_rank=16, + o_lora_rank=16, + o_groups=1, + n_routed_experts=4, + n_shared_experts=0, + num_experts_per_tok=1, + moe_intermediate_size=32, + index_n_heads=1, + index_head_dim=16, + index_topk=32, + sliding_window=16, + compress_ratios=[0, 4], + expert_dtype="fp4", + quantization_config={"quant_method": "fp8"}, + torch_dtype="bfloat16", + vocab_size=128, + ) + return SimpleNamespace( + model_config=SimpleNamespace(hf_config=cfg, dtype="bf16", max_model_len=4096), + cache_config=SimpleNamespace(max_model_len=4096, cache_dtype="fp8"), + parallel_config=SimpleNamespace( + tensor_parallel_size=1, + data_parallel_size=1, + enable_expert_parallel=False, + ), + ) + + def test_non_vllm_workload_emits_measurement_not_vllm_claims(tmp_path: Path, monkeypatch): """HFT (no intervention library) must report real kernels, never vLLM knobs.""" import gitm.scheduler.loop as loop @@ -243,16 +335,121 @@ def test_vllm_workload_still_uses_intervention_path(tmp_path: Path, monkeypatch) monkeypatch.setattr(loop, "capture", _fake_capture_with_kernels("paged_attention")) monkeypatch.setattr(loop, "sync_device", lambda: None) + monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200") from gitm import optimize result = optimize( - workload="vllm-decode", budget="1s", scratch=str(tmp_path), workload_runner=lambda: {} + _priceable_moe_engine(), + workload="vllm-decode", + budget="1s", + scratch=str(tmp_path), + workload_runner=lambda: {}, ) assert result["summary"]["mode"] == "intervention" -def test_vllm_loop_runs_autoresearch(tmp_path: Path, monkeypatch): +def test_vllm_missing_intervention_library_degrades_with_named_coverage_refusal( + tmp_path: Path, monkeypatch +): + import gitm.scheduler.loop as loop + + monkeypatch.setattr(loop, "capture", _fake_capture_with_kernels("paged_attention")) + monkeypatch.setattr(loop, "sync_device", lambda: None) + monkeypatch.setattr( + loop, + "load_library", + lambda **_kwargs: (_ for _ in ()).throw( + FileNotFoundError("missing.yaml; candidate coverage is unavailable") + ), + ) + monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200") + + from gitm import optimize + + result = optimize( + _priceable_moe_engine(), + workload="vllm-decode", + budget="1s", + scratch=str(tmp_path), + workload_runner=lambda: {}, + ) + + assert result["summary"]["status"] == "candidate_coverage_unavailable" + assert result["summary"]["mode"] == "measurement" + assert result["summary"]["n_claims"] == 0 + assert "missing.yaml" in result["report_md"] + assert "candidate coverage" in result["report_md"] + + +def test_vllm_without_live_model_refuses_prediction_claims(tmp_path: Path, monkeypatch): + import gitm.scheduler.loop as loop + + monkeypatch.setattr(loop, "capture", _fake_capture_with_kernels("paged_attention")) + monkeypatch.setattr(loop, "sync_device", lambda: None) + monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200") + + from gitm import optimize + + result = optimize( + workload="vllm-decode", budget="1s", scratch=str(tmp_path), workload_runner=lambda: {} + ) + + assert result["summary"]["status"] == "prediction_refused" + assert result["summary"]["mode"] == "measurement" + assert result["summary"]["n_claims"] == 0 + assert "no live engine" in result["report_md"] + refusal = Path(result["run_dir"]) / "prediction_refusal.json" + assert refusal.exists() and "no live engine" in refusal.read_text() + + +def test_vllm_loop_surfaces_residual_coverage(tmp_path: Path, monkeypatch): + """Unclassified work must be visible in both the JSON and human report.""" + import json + + import gitm.planner.context as planner_context + import gitm.scheduler.loop as loop + + @contextmanager + def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None): + kernels = [ + make_kernel("flash_attn_kernel", start_ns=0, end_ns=900), + make_kernel("triton_rms_norm_kernel", start_ns=900, end_ns=1000), + ] + yield make_trace(events=kernels, vendor="nvidia", run_id=run_id or "r") + + monkeypatch.setattr(loop, "capture", fake_capture) + monkeypatch.setattr(loop, "sync_device", lambda: None) + monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200") + monkeypatch.setattr(planner_context, "_query_nvml", lambda: ("NVIDIA B200", None)) + + from gitm import optimize + + result = optimize( + _priceable_moe_engine(), + workload="vllm-decode", + budget="1s", + scratch=str(tmp_path), + workload_runner=lambda: {}, + ) + payload = json.loads((Path(result["run_dir"]) / "residuals.json").read_text()) + predicted = json.loads((Path(result["run_dir"]) / "predicted_graph.json").read_text()) + + assert payload["coverage"]["total_kernels"] == 2 + assert payload["coverage"]["matched_kernels"] == 1 + assert payload["coverage"]["warnings"] + assert predicted["resident_weight_bytes_per_rank"] > 0 + assert predicted["resident_weight_bytes_is_lower_bound"] is False + assert predicted["kv_bytes_per_token_per_sequence"] > 0 + assert predicted["kv_fixed_bytes_per_sequence"] > 0 + assert "## Runtime diagnostics" in result["report_md"] + assert "matched to the predicted graph" in result["report_md"] + assert "GPU count was unavailable" in result["report_md"] + + +def test_vllm_loop_without_model_does_not_run_autoresearch_on_default_graph( + tmp_path: Path, monkeypatch +): """The vllm path runs agentic autoresearch: it classifies the bottleneck via trace telemetry (the serialized same-stream "paged_attention" kernels are also roofline-predicted memory-bound at batch=1, so classification is @@ -275,16 +472,13 @@ def test_vllm_loop_runs_autoresearch(tmp_path: Path, monkeypatch): workload="vllm-decode", budget="30s", scratch=str(tmp_path), workload_runner=lambda: {} ) s = result["summary"] - assert s["bottleneck_class"] == "memory_bound" - assert s["n_autoresearch"] == 0 - - ar_json = (Path(result["run_dir"]) / "autoresearch.json").read_text(encoding="utf-8") - # Denylisted knobs with unchecked prerequisites must never surface, regardless - # of which bottleneck class the run classifies as. - assert "max_num_partial_prefills=" not in ar_json - assert "long_prefill_token_threshold=" not in ar_json + assert s["status"] == "prediction_refused" + assert s["n_claims"] == 0 + assert "bottleneck_class" not in s - # Dry-run (no live engine) mutates nothing, so no safety trail is written. + # The prediction gate fires before candidate ranking/autoresearch, so neither + # artifact can imply that a default graph supported an optimization. + assert not (Path(result["run_dir"]) / "autoresearch.json").exists() assert not (Path(result["run_dir"]) / "audit.jsonl").exists() @@ -296,6 +490,96 @@ def test_cli_run_returns_nonzero_on_no_data(tmp_path: Path, capsys): assert rc == 3 +@pytest.mark.parametrize("target", ["nan", "inf", "0%", "101%"]) +def test_cli_run_rejects_invalid_target_before_optimization(target, monkeypatch): + import gitm + from gitm.cli import main + + called = False + + def fake_optimize(**_kwargs): + nonlocal called + called = True + return {} + + monkeypatch.setattr(gitm, "optimize", fake_optimize) + with pytest.raises(ValueError, match="target floor"): + main(["run", "--workload", "vllm-decode", "--target", target]) + assert called is False + + +def test_cli_apply_returns_nonzero_when_nothing_was_applied(tmp_path, capsys): + import json + + import yaml + + from gitm.cli import main + from gitm.kernels.spec import InterventionSpec + + spec = InterventionSpec( + name="noop", summary="test", knob="block_size", value=16, + expected_delta_lo=0.0, expected_delta_mean=0.1, expected_delta_hi=0.2, + source="https://example.test", + ) + path = tmp_path / "intervention.yaml" + path.write_text(yaml.safe_dump(json.loads(spec.model_dump_json())), encoding="utf-8") + + assert main(["apply", "--intervention", str(path)]) == 3 + assert "no target config" in capsys.readouterr().out + + +@pytest.mark.parametrize( + "status", + ["prediction_refused", "candidate_coverage_unavailable", "intervention_failed"], +) +def test_cli_run_returns_nonzero_for_every_degraded_status(status, monkeypatch): + import gitm + from gitm.cli import main + + monkeypatch.setattr( + gitm, + "optimize", + lambda **_kwargs: {"summary": {"status": status}, "report_md": "diagnostic"}, + ) + + assert main(["run", "--workload", "vllm-decode", "--budget", "1s"]) == 3 + + +def test_cli_run_refuses_malformed_loop_result(monkeypatch, tmp_path): + import gitm + from gitm.cli import main + + monkeypatch.setattr(gitm, "optimize", lambda **_kwargs: {}) + report = tmp_path / "report.md" + + rc = main(["run", "--workload", "vllm-decode", "--report", str(report)]) + + assert rc == 3 + assert "returned no machine-readable summary" in report.read_text() + + +def test_default_engine_throughput_probe_refuses_missing_work_count(monkeypatch): + from gitm.scheduler.loop import _engine_throughput_fn + + ticks = iter([1.0, 2.0]) + monkeypatch.setattr("gitm.scheduler.loop.time.perf_counter", lambda: next(ticks)) + probe = _engine_throughput_fn(object(), lambda: {}) + + with pytest.raises(RuntimeError, match="runner returned none"): + probe(object()) + + +def test_default_engine_throughput_probe_refuses_zero_work(monkeypatch): + from gitm.scheduler.loop import _engine_throughput_fn + + ticks = iter([1.0, 2.0]) + monkeypatch.setattr("gitm.scheduler.loop.time.perf_counter", lambda: next(ticks)) + probe = _engine_throughput_fn(object(), lambda: {"generated_tokens": 0}) + + with pytest.raises(RuntimeError, match="work coverage unavailable"): + probe(object()) + + def test_hft_harness_importable_from_package(): """The harness must ship in the wheel, i.e. be importable from the package.""" from gitm.benchmarks.hft import harness @@ -354,6 +638,7 @@ def __init__(self, max_tokens: int = 0, temperature: float = 0.0): fake.LLM = _LLM fake.SamplingParams = _SamplingParams monkeypatch.setitem(sys.modules, "vllm", fake) + monkeypatch.setattr("gitm.cuda_env.require_compatible", lambda: None) monkeypatch.setenv("GITM_VLLM_PROMPTS", "3") monkeypatch.setenv("GITM_VLLM_MAX_TOKENS", "5") monkeypatch.delenv("GITM_VLLM_SYNTHETIC", raising=False) diff --git a/tests/test_runtime_driver.py b/tests/test_runtime_driver.py new file mode 100644 index 0000000..19ed7d9 --- /dev/null +++ b/tests/test_runtime_driver.py @@ -0,0 +1,70 @@ +from __future__ import annotations + +from contextlib import contextmanager + +import pytest + + +@pytest.mark.parametrize( + ("summary", "workload", "expected"), + [({"events": 10}, "hft", 10), ({"frames": 3}, "edge", 3)], +) +def test_work_units_require_the_named_positive_counter(summary, workload, expected): + from gitm.runtime_driver import _work_units + + assert _work_units(summary, workload) == expected + + +@pytest.mark.parametrize( + ("summary", "workload"), + [({}, "hft"), ({"frames": 0}, "edge"), ({"events": -1}, "hft")], +) +def test_work_units_refuse_missing_or_empty_coverage(summary, workload): + from gitm.runtime_driver import _work_units + + with pytest.raises(RuntimeError, match="work coverage unavailable"): + _work_units(summary, workload) + + +def test_runtime_driver_refuses_empty_trace_instead_of_printing_pass(tmp_path, monkeypatch, capsys): + from gitm import runtime_driver + from gitm.tracer.schema import Trace + + monkeypatch.setattr( + runtime_driver, + "_load_hft", + lambda *_args, **_kwargs: ( + lambda: {"events": 10, "vwap_buckets": 1}, + 10, + "cpu", + "cpu", + 0, + ), + ) + + @contextmanager + def empty_capture(*_args, **_kwargs): + yield Trace( + workload_id="hft", + fingerprint="f", + run_id="r", + device_count=0, + vendor="none", + captured_at_ns=0, + duration_ns=1, + source="none", + events=[], + ) + + monkeypatch.setattr("gitm.tracer.capture", empty_capture) + + rc = runtime_driver.main( + ["--workload", "hft", "--stage", str(tmp_path), "--outdir", str(tmp_path)] + ) + + output = capsys.readouterr().out + assert rc == 3 + assert "runtime details unavailable" in output + assert "all details measured" not in output + report = (tmp_path / "hft_seed42_report.md").read_text() + assert "NO DATA" in report diff --git a/tests/test_runtime_on_trace.py b/tests/test_runtime_on_trace.py index 780f071..ef6ad4f 100644 --- a/tests/test_runtime_on_trace.py +++ b/tests/test_runtime_on_trace.py @@ -31,9 +31,17 @@ def _trace(events): def test_serialized_fraction_sequential_vs_overlapped(): from gitm.optimizer.monitor import _serialized_fraction - # back-to-back on one stream -> fully serialized + # Back-to-back work on one stream is mandatory ordering, not evidence that + # independent work failed to overlap. seq = [_kernel("k", i * 100, i * 100 + 100, stream=7) for i in range(6)] - assert _serialized_fraction(seq) == pytest.approx(1.0) + assert _serialized_fraction(seq) is None + + # Different streams establish an overlap opportunity; back-to-back execution + # means every observed opportunity serialized. + cross_stream_seq = [ + _kernel("k", i * 100, i * 100 + 100, stream=i % 2) for i in range(6) + ] + assert _serialized_fraction(cross_stream_seq) == pytest.approx(1.0) # heavily overlapping on different streams -> not serialized over = [_kernel("k", 0, 1000, stream=s) for s in range(6)] @@ -46,7 +54,8 @@ def test_residuals_compute_real_concurrency(): trace = _trace([_kernel("k", i * 100, i * 100 + 100) for i in range(8)]) res = residuals(trace, predict_graph()) - assert res.serialized_concurrency_fraction == pytest.approx(1.0) # all sequential, one stream + assert res.serialized_concurrency_fraction is None + assert any("cross-stream" in note for note in res.coverage_warnings) # --- op-identity matching (was ordinal `for i in range(min(len(obs), len(pred)))`) -- @@ -84,6 +93,57 @@ def test_residuals_skip_unmodeled_kernels(): trace = _trace([_kernel("triton_rms_norm_kernel", 0, 100)]) res = residuals(trace, predict_graph()) assert res.per_kernel == [] + assert res.total_kernels == 1 + assert res.classified_kernels == 0 + assert res.matched_kernels == 0 + assert res.classification_coverage == 0.0 + assert res.match_coverage == 0.0 + assert res.coverage_warnings + + +def test_residual_coverage_distinguishes_unclassified_from_graph_miss(): + from gitm.optimizer.monitor import residuals + from gitm.planner.graph import predict_graph + + # FlashAttention classifies and matches; NCCL classifies but a whole-model + # dense graph has no collective node; RMSNorm does not classify at all. + trace = _trace( + [ + _kernel("flash_attn_kernel", 0, 700), + _kernel("ncclDevKernel_AllReduce", 700, 900), + _kernel("triton_rms_norm_kernel", 900, 1000), + ] + ) + res = residuals(trace, predict_graph()) + + assert (res.total_kernels, res.classified_kernels, res.matched_kernels) == (3, 2, 1) + assert res.classification_coverage == pytest.approx(2 / 3) + assert res.match_coverage == pytest.approx(1 / 3) + assert res.classified_time_coverage == pytest.approx(0.9) + assert res.matched_time_coverage == pytest.approx(0.7) + assert any("classified" in warning for warning in res.coverage_warnings) + assert any("matched" in warning for warning in res.coverage_warnings) + + +def test_fully_matched_residual_coverage_is_clean(): + from gitm.optimizer.monitor import residuals + from gitm.planner.graph import predict_graph + + res = residuals( + _trace( + [ + _kernel("flash_attn_kernel", 0, 100, stream=0), + _kernel("flash_attn_kernel", 0, 100, stream=1), + ] + ), + predict_graph(), + ) + + assert res.classification_coverage == 1.0 + assert res.match_coverage == 1.0 + assert res.classified_time_coverage == 1.0 + assert res.matched_time_coverage == 1.0 + assert res.coverage_warnings == [] # --- multi-basis filter ------------------------------------------------------ @@ -162,6 +222,93 @@ def test_doubly_robust_degenerate_inputs(): assert ate == 0.0 and se == float("inf") +def test_attribution_abstention_is_diagnostic_not_no_causal_signal(): + from gitm.optimizer.attribution import attribute + from gitm.optimizer.dr import attribute_dr + from gitm.optimizer.monitor import KernelResidual, Residuals + from gitm.planner.graph import predict_graph + + res = Residuals(per_kernel=[KernelResidual("only", None, 0.5, None)]) + + granger = attribute(res, predict_graph()) + dr = attribute_dr(res, predict_graph()) + + assert granger.hypotheses == [] + assert any("not run" in note for note in granger.diagnostics) + assert dr.hypotheses == [] + assert any("not run" in note for note in dr.diagnostics) + + +def test_measure_trace_excludes_zero_duration_kernels_with_diagnostic(): + from gitm.optimizer.measure import measure_trace + + trace = _trace( + [ + _kernel("bad", start=10, end=10), + _kernel("good", start=20, end=30), + _kernel("good", start=40, end=50), + ] + ) + + result = measure_trace(trace, min_attr=1) + + assert result.n_kernels == 3 + assert result.n_invalid_duration == 1 + assert any("excluded 1/3" in note for note in result.diagnostics) + assert result.serialized_fraction is None + assert any("cross-stream" in note for note in result.diagnostics) + assert all(v.node_op != "bad" for v in result.violations) + + +def test_measurement_summary_names_unavailable_concurrency(): + from gitm.optimizer.measure import measure_trace, measurement_summary + + result = measure_trace( + _trace([_kernel("same", i * 100, i * 100 + 50, stream=0) for i in range(3)]) + ) + + assert "serialized-concurrency=unavailable" in measurement_summary("demo", result) + assert not any(v.invariant == "stream_concurrency" for v in result.violations) + + +def test_specialized_claim_uses_throughput_when_concurrency_is_unavailable(): + from types import SimpleNamespace + + from gitm.scheduler.loop import _specialized_claim_basis + + basis, delta, diagnostics = _specialized_claim_basis( + SimpleNamespace( + n_kernels=3, + n_invalid_duration=0, + serialized_fraction=None, + diagnostics=["stream-concurrency coverage unavailable"], + ), + SimpleNamespace(speedup=1.25), + ) + + assert basis == ("throughput_delta", pytest.approx(0.25)) + assert delta == pytest.approx(0.25) + assert any("no adjacent cross-stream" in note for note in diagnostics) + + +def test_residuals_exclude_zero_duration_kernels_with_diagnostic(): + from gitm.optimizer.monitor import residuals + from gitm.planner.graph import predict_graph + + trace = _trace( + [ + _kernel("flash_attn_kernel", start=10, end=10), + _kernel("flash_attn_kernel", start=20, end=30), + ] + ) + + result = residuals(trace, predict_graph()) + + assert result.invalid_duration_kernels == 1 + assert len(result.per_kernel) == 1 + assert any("non-positive timestamps" in note for note in result.coverage_warnings) + + def test_attribute_dr_ranks_pairs(): from gitm.optimizer.dr import attribute_dr from gitm.optimizer.monitor import KernelResidual, Residuals @@ -230,3 +377,20 @@ def test_replay_validation_within_tolerance(): assert result.passed assert result.mean_abs_rel_err <= 0.20 assert result.frac_within_tol > 0.7 + + +def test_replay_validation_refuses_zero_truth_instead_of_reporting_zero_error(monkeypatch): + from gitm.optimizer import replay_validation as rv + + monkeypatch.setattr(rv, "_ground_truth_delta", lambda *_args, **_kwargs: 0.0) + + with pytest.raises(RuntimeError, match="ground truth must be finite and positive"): + rv.validate(n=1) + + +@pytest.mark.parametrize("n", [0, -1]) +def test_replay_validation_refuses_empty_injection_sets(n): + from gitm.optimizer.replay_validation import validate + + with pytest.raises(ValueError, match="injection count must be positive"): + validate(n=n) diff --git a/tests/test_safety_primitives.py b/tests/test_safety_primitives.py index 5d49a91..70541c4 100644 --- a/tests/test_safety_primitives.py +++ b/tests/test_safety_primitives.py @@ -75,6 +75,20 @@ def test_failopen_failures_reset_on_reentry(): assert g.failures == [] +def test_failopen_surfaces_broken_audit_sink(): + class BrokenAudit: + def record(self, *_args, **_kwargs): + raise OSError("disk full") + + g = FailOpenGuard(audit=BrokenAudit(), install_signal_handlers=False) + with pytest.warns(RuntimeWarning, match="audit sink failed"): + with g: + g.register("x", lambda: None) + + assert g.audit_failures + assert "disk full" in g.audit_failures[0] + + # --------- auto-revert --------------------------------------------------------- def test_autorevert_warms_up_then_holds_within_tolerance(): ar = AutoRevert(baseline=100.0, tolerance=0.05, window=3) @@ -92,6 +106,25 @@ def test_autorevert_fires_on_regression(): assert d.should_revert and d.relative_delta == pytest.approx(-0.1) +@pytest.mark.parametrize("baseline", [float("nan"), float("inf"), 0.0, -1.0, True]) +def test_autorevert_refuses_invalid_baseline(baseline): + with pytest.raises(ValueError, match="baseline"): + AutoRevert(baseline=baseline) + + +@pytest.mark.parametrize("tolerance", [float("nan"), float("inf"), -0.01, True]) +def test_autorevert_refuses_invalid_tolerance(tolerance): + with pytest.raises(ValueError, match="tolerance"): + AutoRevert(baseline=100.0, tolerance=tolerance) + + +@pytest.mark.parametrize("value", [float("nan"), float("inf"), float("-inf"), True]) +def test_autorevert_refuses_invalid_observation(value): + ar = AutoRevert(baseline=100.0) + with pytest.raises(ValueError, match="observation"): + ar.observe(value) + + # --------- gated rollout ------------------------------------------------------- def test_rollout_shadow_then_manual_promote(tmp_path: Path): log = AuditLog(tmp_path / "audit.jsonl") diff --git a/tests/test_scorer.py b/tests/test_scorer.py index 985b022..3eccdd4 100644 --- a/tests/test_scorer.py +++ b/tests/test_scorer.py @@ -1,4 +1,5 @@ import pandas as pd +import pytest from gitm.routing.scorer_v0 import score_dataframe, score_prospect @@ -26,16 +27,15 @@ def test_score_prospect_cold(): assert score == 7.5 def test_score_prospect_unknown_tier(): - score = score_prospect( - warmth=0.5, - signal_recency=0.5, - company_tier=99, - pain_acknowledged=0, - engagement_score=0.0, - prior_engagement=0 - ) - # Unknown tier should default to tier 3 (0.2) - assert score == 34.0 + with pytest.raises(ValueError, match="company_tier must be 1, 2, or 3"): + score_prospect( + warmth=0.5, + signal_recency=0.5, + company_tier=99, + pain_acknowledged=0, + engagement_score=0.0, + prior_engagement=0, + ) def test_score_dataframe_sorted(): df = pd.DataFrame({ diff --git a/tests/test_serve_attach.py b/tests/test_serve_attach.py index 6bddb56..680d0f0 100644 --- a/tests/test_serve_attach.py +++ b/tests/test_serve_attach.py @@ -13,6 +13,8 @@ import json import os +from dataclasses import replace +from types import SimpleNamespace import pytest @@ -20,6 +22,30 @@ from gitm.serve import discover from gitm.tracer import injection + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("duration_s", 0.0, "duration_s"), + ("duration_s", float("nan"), "duration_s"), + ("requests", -1, "requests"), + ("concurrency", 0, "concurrency"), + ("input_tokens", 0, "input_tokens"), + ("output_tokens", -1, "output_tokens"), + ("request_timeout", float("inf"), "request_timeout"), + ("metrics_interval", 0.0, "metrics_interval"), + ("port", 70000, "port"), + ], +) +def test_attach_options_refuse_invalid_window_inputs(field, value, message): + with pytest.raises(ValueError, match=message): + att.AttachOptions(**{field: value}) + + +def test_attach_options_accept_normal_observe_and_drive_windows(): + assert att.AttachOptions().mode == "observe" + assert att.AttachOptions(requests=1, concurrency=1).mode == "drive" + LIB = str(injection.lib_path()) @@ -221,6 +247,176 @@ def test_base_url_prefers_explicit_then_flag_then_cmdline(): ) +def test_server_metric_lines_surface_missing_token_and_tpot_values(): + from gitm.serve.metrics import ServerWindow + + lines = att._server_metric_lines( + ServerWindow(requests_finished=3, generation_tokens=None, ttft_mean_s=0.012) + ) + + assert "output-token count unavailable" in lines[0] + assert "TPOT mean unavailable" in lines[1] + assert "0 output tokens" not in "\n".join(lines) + + +def test_server_metric_lines_render_measured_values_without_caveats(): + from gitm.serve.metrics import ServerWindow + + lines = att._server_metric_lines( + ServerWindow( + requests_finished=3, + generation_tokens=24, + output_tokens_per_s=12, + ttft_mean_s=0.012, + tpot_mean_s=0.0035, + ) + ) + + assert lines == [ + " server: 3 requests, 24 output tokens, 12 tok/s", + " server TTFT mean 12 ms TPOT mean 3.5 ms", + ] + + +def test_predicted_graph_surfaces_resolved_warnings_and_bytes_fallback( + tmp_path, monkeypatch, capsys +): + from gitm.planner.context import peak_for_sku + from gitm.planner.moe_graph import spec_from_hf_config + from gitm.planner.roofline import BatchConfig, ShardingConfig + from gitm.serve import model_config as mc + + cfg = { + "model_type": "deepseek_v4", + "hidden_size": 64, + "num_hidden_layers": 1, + "num_attention_heads": 1, + "num_key_value_heads": 1, + "head_dim": 64, + "qk_rope_head_dim": 0, + "q_lora_rank": 64, + "o_lora_rank": 64, + "o_groups": 1, + "vocab_size": 128, + "n_routed_experts": 2, + "n_shared_experts": 0, + "num_experts_per_tok": 1, + "moe_intermediate_size": 32, + "index_n_heads": 1, + "index_head_dim": 8, + "index_topk": 8, + "sliding_window": 128, + "compress_ratios": [0], + "expert_dtype": "fp4", + "quantization_config": {"quant_method": "fp8"}, + "torch_dtype": "bfloat16", + } + spec = replace(spec_from_hf_config(cfg), expert_dtype="future_fp3") + resolved = mc.LiveSpec( + spec=spec, + sharding=ShardingConfig(), + batch=BatchConfig(batch=1, kv_cache_len=128), + source_path=tmp_path / "config.json", + model_ref="org/model", + warnings=["decode batch was not observed; using batch=1 single-sequence floor"], + ) + monkeypatch.setattr(mc, "live_moe_spec", lambda _target, **_kwargs: resolved) + + import gitm.planner.context as planner_context + + monkeypatch.setattr( + planner_context, + "build_planner_context", + lambda: SimpleNamespace( + peak=peak_for_sku("NVIDIA B200"), + sku="NVIDIA B200", + num_gpus=1, + num_gpus_is_fallback=True, + ), + ) + + att._emit_predicted_graph(discover.Target(pid=1, cmdline=[]), tmp_path) + + payload = json.loads((tmp_path / "predicted_moe_graph.json").read_text()) + assert payload["has_fallback_bytes"] is True + assert payload["resident_weight_bytes_per_rank"] > 0 + assert payload["resident_weight_bytes_is_lower_bound"] is False + assert payload["kv_bytes_per_token_per_sequence"] == 0.0 + assert payload["kv_fixed_bytes_per_sequence"] > 0 + assert payload["num_gpus_is_fallback"] is True + assert any("GPU count was unavailable" in warning for warning in payload["warnings"]) + assert any(node["bytes_are_fallback"] for node in payload["nodes"]) + assert payload["warnings"] + stdout = capsys.readouterr().out + assert "single-sequence floor" in stdout + assert "unknown-dtype bf16 fallback" in stdout + + +def test_predicted_graph_known_dtypes_leave_bytes_fallback_clean(tmp_path, monkeypatch): + from gitm.planner.context import peak_for_sku + from gitm.planner.moe_graph import spec_from_hf_config + from gitm.planner.roofline import BatchConfig, ShardingConfig + from gitm.serve import model_config as mc + + cfg = { + "model_type": "deepseek_v4", + "hidden_size": 64, + "num_hidden_layers": 1, + "num_attention_heads": 1, + "num_key_value_heads": 1, + "head_dim": 64, + "qk_rope_head_dim": 0, + "q_lora_rank": 64, + "o_lora_rank": 64, + "o_groups": 1, + "vocab_size": 128, + "n_routed_experts": 2, + "n_shared_experts": 0, + "num_experts_per_tok": 1, + "moe_intermediate_size": 32, + "index_n_heads": 1, + "index_head_dim": 8, + "index_topk": 8, + "sliding_window": 128, + "compress_ratios": [0], + "expert_dtype": "fp4", + "quantization_config": {"quant_method": "fp8"}, + "torch_dtype": "bfloat16", + } + resolved = mc.LiveSpec( + spec=spec_from_hf_config(cfg), + sharding=ShardingConfig(), + batch=BatchConfig(batch=1, kv_cache_len=128), + source_path=tmp_path / "config.json", + model_ref="org/model", + ) + monkeypatch.setattr(mc, "live_moe_spec", lambda _target, **_kwargs: resolved) + + import gitm.planner.context as planner_context + + monkeypatch.setattr( + planner_context, + "build_planner_context", + lambda: SimpleNamespace( + peak=peak_for_sku("NVIDIA B200"), + sku="NVIDIA B200", + num_gpus=1, + num_gpus_is_fallback=False, + ), + ) + + att._emit_predicted_graph(discover.Target(pid=1, cmdline=[]), tmp_path) + payload = json.loads((tmp_path / "predicted_moe_graph.json").read_text()) + assert payload["has_fallback_bytes"] is False + assert payload["resident_weight_bytes_per_rank"] > 0 + assert payload["resident_weight_bytes_is_lower_bound"] is False + assert payload["kv_bytes_per_token_per_sequence"] == 0.0 + assert payload["kv_fixed_bytes_per_sequence"] > 0 + assert payload["num_gpus_is_fallback"] is False + assert not any("GPU count was unavailable" in warning for warning in payload["warnings"]) + assert not any(node["bytes_are_fallback"] for node in payload["nodes"]) + + # --- preflight --------------------------------------------------------------- @@ -326,6 +522,19 @@ def test_empty_trace_and_idle_window_are_different_failures(tmp_path): assert _write(tmp_path / "c", [_FakeKernel()], had_traffic=True).status == "ok" +def test_zero_duration_kernel_records_do_not_make_capture_successful(tmp_path): + result = _write( + tmp_path / "zero-duration", + [_FakeKernel(start_ns=10, end_ns=10)], + had_traffic=True, + ) + + assert result.status == "no_kernels" + assert result.n_kernels == 0 + assert result.breakdown.n_invalid_duration == 1 + assert any("non-positive duration" in warning for warning in result.warnings) + + def test_both_paths_write_the_same_artifact_set(tmp_path): """A driven benchmark and a production observation of the same server are only comparable if they leave the same files behind.""" diff --git a/tests/test_serve_capture.py b/tests/test_serve_capture.py index bc656ae..62215c1 100644 --- a/tests/test_serve_capture.py +++ b/tests/test_serve_capture.py @@ -11,6 +11,7 @@ import json import threading +import warnings from http.server import BaseHTTPRequestHandler, HTTPServer import pytest @@ -94,6 +95,7 @@ def test_chunk_count_is_the_fallback_when_usage_is_absent(sse_server): rec = sc.one_request(base, "m", "p", max_tokens=4, ignore_eos=True, timeout_s=30) assert rec is not None and rec.n_output_tokens == 3 + assert rec.token_count_source == "chunks" def test_unreachable_server_returns_none_not_a_fake_record(): @@ -103,6 +105,34 @@ def test_unreachable_server_returns_none_not_a_fake_record(): assert sc.one_request("http://127.0.0.1:1", "m", "p", 4, True, 0.5) is None +def test_served_model_name_warns_when_endpoint_falls_back(monkeypatch): + def fail(*_args, **_kwargs): + raise OSError("connection refused") + + monkeypatch.setattr(sc.urllib.request, "urlopen", fail) + + with pytest.warns(RuntimeWarning, match="served-model discovery failed"): + assert sc.served_model_name("http://server", "configured-model") == "configured-model" + + +def test_served_model_name_uses_endpoint_without_warning(monkeypatch): + class Response: + def __enter__(self): + return self + + def __exit__(self, *_args): + return None + + def read(self): + return b'{"data": [{"id": "served-alias"}]}' + + monkeypatch.setattr(sc.urllib.request, "urlopen", lambda *_args, **_kwargs: Response()) + + with warnings.catch_warnings(record=True) as caught: + assert sc.served_model_name("http://server", "configured-model") == "served-alias" + assert not caught + + def test_records_feed_the_existing_summary(sse_server): from gitm.tracer.vllm_stats import summarize_requests @@ -336,3 +366,31 @@ def test_preflight_fails_closed_without_a_gpu(): start a server that cannot possibly work.""" dev = sc.Devices(indices=[], count=0, source="nvidia-smi unavailable") assert sc.check_gpus(2, dev)[0].status == "fail" + + +def test_driver_stack_does_not_pass_when_build_versions_are_unreadable(monkeypatch): + from gitm import cuda_env + + monkeypatch.setattr(cuda_env, "driver_cuda", lambda: (13, 0)) + monkeypatch.setattr(cuda_env, "stack_for", lambda _driver: object()) + monkeypatch.setattr(cuda_env, "check", lambda: []) + monkeypatch.setattr(cuda_env, "torch_cuda", lambda: None) + monkeypatch.setattr(cuda_env, "vllm_cuda_major", lambda: None) + + check = sc.check_driver_stack()[0] + + assert check.status == "warn" + assert "PyTorch CUDA build" in check.detail + assert "vLLM CUDA build" in check.detail + + +def test_driver_stack_pass_requires_observed_build_versions(monkeypatch): + from gitm import cuda_env + + monkeypatch.setattr(cuda_env, "driver_cuda", lambda: (13, 0)) + monkeypatch.setattr(cuda_env, "stack_for", lambda _driver: object()) + monkeypatch.setattr(cuda_env, "check", lambda: []) + monkeypatch.setattr(cuda_env, "torch_cuda", lambda: (13, 0)) + monkeypatch.setattr(cuda_env, "vllm_cuda_major", lambda: 13) + + assert sc.check_driver_stack()[0].status == "pass" diff --git a/tests/test_serve_metrics.py b/tests/test_serve_metrics.py index 6920c0d..348216c 100644 --- a/tests/test_serve_metrics.py +++ b/tests/test_serve_metrics.py @@ -62,6 +62,11 @@ def test_parse_ignores_comments_and_nan(): assert snap["vllm:num_requests_running"] == 3.0 +def test_parse_ignores_infinite_values(): + snap = metrics.parse_prometheus("vllm:num_requests_running +Inf\n") + assert "vllm:num_requests_running" not in snap + + def test_window_is_the_difference_not_the_lifetime(): w = metrics.window_from_snapshots(BEFORE, AFTER, window_s=10.0) @@ -124,6 +129,12 @@ def test_missing_window_length_leaves_throughput_unset(): assert w.output_tokens_per_s is None +def test_nonpositive_window_refuses_throughput_with_note(): + w = metrics.window_from_snapshots(BEFORE, AFTER, window_s=0.0) + assert w.output_tokens_per_s is None + assert any("finite and positive" in note for note in w.notes) + + def test_fetch_never_raises_on_a_dead_endpoint(): text = metrics.fetch_metrics("http://127.0.0.1:1", timeout=0.5) assert text.startswith("# unavailable") diff --git a/tests/test_serve_model_config.py b/tests/test_serve_model_config.py index 7b49c00..c1079a9 100644 --- a/tests/test_serve_model_config.py +++ b/tests/test_serve_model_config.py @@ -29,11 +29,22 @@ def _deepseek_cfg(**over) -> dict: "model_type": "deepseek_v4", "num_hidden_layers": 4, "hidden_size": 4096, + "num_attention_heads": 64, + "num_key_value_heads": 1, + "head_dim": 512, + "qk_rope_head_dim": 64, + "q_lora_rank": 1024, + "o_lora_rank": 1024, + "o_groups": 8, + "vocab_size": 129280, "n_routed_experts": 256, + "n_shared_experts": 1, "num_experts_per_tok": 6, "moe_intermediate_size": 2048, - # DeepSeek Sparse Attention markers — what routes this to the MoE graph. + "index_n_heads": 64, + "index_head_dim": 128, "index_topk": 512, + "sliding_window": 128, "compress_ratios": [0, 0, 4, 128], "quantization_config": {"quant_method": "fp8"}, "expert_dtype": "fp4", @@ -94,14 +105,21 @@ def test_deepseek_config_is_usable(): assert mc.validate_moe_config(_deepseek_cfg()) == [] -def test_mixtral_aliases_are_recognized_not_rejected(): +def test_sparse_candidate_with_partial_expert_shape_is_not_misread_as_dense(): + assert mc.is_sparse_moe_config({"n_routed_experts": 8}) + assert not mc.is_sparse_moe_config({"model_type": "llama"}) + + +def test_mixtral_aliases_are_recognized_but_unsupported_shape_is_refused(): mixtral = { "model_type": "mixtral", "num_local_experts": 8, "num_experts_per_tok": 2, "intermediate_size": 14336, } - assert mc.validate_moe_config(mixtral) == [] + missing = mc.validate_moe_config(mixtral) + assert any("compress_ratios" in item for item in missing) + assert any("hidden_size" in item for item in missing) norm = mc.normalize_moe_config(mixtral) assert norm["n_routed_experts"] == 8 assert norm["moe_intermediate_size"] == 14336 @@ -115,12 +133,34 @@ def test_missing_topk_is_refused_and_names_the_key(): assert any("experts per token" in m for m in missing) +def test_nonpositive_expert_shape_is_refused_instead_of_priced_as_dense(): + missing = mc.validate_moe_config(_deepseek_cfg(n_routed_experts=0)) + assert any("routed expert count" in item and "positive" in item for item in missing) + + +def test_short_compression_schedule_is_refused_instead_of_reusing_last_ratio(): + missing = mc.validate_moe_config(_deepseek_cfg(compress_ratios=[0, 4])) + assert any("compress_ratios" in item and "4 layers" in item for item in missing) + + def test_unpriceable_quant_method_is_refused_not_defaulted(): cfg = _deepseek_cfg(quantization_config={"quant_method": "awq"}) missing = mc.validate_moe_config(cfg) assert any("awq" in m for m in missing) +def test_final_spec_dtype_validation_covers_every_byte_contributor(): + from dataclasses import replace + + spec = mc.spec_from_hf_config(_deepseek_cfg()) + assert mc.validate_priceable_dtypes(spec) == [] + + bad = replace(spec, kv_dtype="future_kv3", act_dtype="future_act3") + missing = mc.validate_priceable_dtypes(bad) + assert any("kv_dtype='future_kv3'" in item for item in missing) + assert any("act_dtype='future_act3'" in item for item in missing) + + # --- serving overrides ------------------------------------------------------- @@ -167,44 +207,36 @@ def test_live_moe_spec_applies_config_and_serving_facts(tmp_path): assert r.spec.expert_dtype == "fp4" -def test_live_moe_spec_refuses_dsa_config_missing_a_dominant_term(tmp_path): +def test_live_moe_spec_refuses_unpredictable_config_with_named_keys(tmp_path): ckpt = tmp_path / "ckpt" - # Passes the architecture gate (index_topk + routed experts) but never - # declares the expert intermediate size, which the graph would silently - # default — so it is refused with that key named. - _write_config(ckpt, { - "model_type": "deepseek_v4", - "n_routed_experts": 256, - "num_experts_per_tok": 6, - "index_topk": 512, - }) + _write_config(ckpt, {"model_type": "dense", "hidden_size": 4096}) # no MoE fields r = mc.live_moe_spec(_target(["vllm", "serve", str(ckpt)]), environ={}) assert isinstance(r, mc.LiveSpecError) - assert any("expert intermediate size" in k for k in r.missing_keys) + assert r.missing_keys + assert "routed expert count" in r.render() -def test_live_moe_spec_refuses_dense_model_at_the_architecture_gate(tmp_path): +def test_unpriceable_command_line_dtype_is_refused_after_overrides(tmp_path): ckpt = tmp_path / "ckpt" - _write_config(ckpt, {"model_type": "dense", "hidden_size": 4096}) # no MoE fields - r = mc.live_moe_spec(_target(["vllm", "serve", str(ckpt)]), environ={}) + _write_config(ckpt, _deepseek_cfg()) + target = _target(["vllm", "serve", str(ckpt), "--kv-cache-dtype", "future_kv3"]) + + r = mc.live_moe_spec(target, environ={}) + assert isinstance(r, mc.LiveSpecError) - assert "not a DeepSeek-V4-class" in r.reason + assert any("kv_dtype='future_kv3'" in item for item in r.missing_keys) -def test_live_moe_spec_refuses_mixtral_style_moe(tmp_path): - # A real MoE, but standard attention — predict_moe_graph would mis-model its - # KV path, so the sidecar declines rather than emit a confident wrong floor. +def test_missing_expert_dtype_refuses_dominant_term_prediction(tmp_path): ckpt = tmp_path / "ckpt" - _write_config(ckpt, { - "model_type": "mixtral", - "num_local_experts": 8, - "num_experts_per_tok": 2, - "intermediate_size": 14336, - "sliding_window": 4096, - }) + cfg = _deepseek_cfg() + cfg.pop("expert_dtype") + _write_config(ckpt, cfg) + r = mc.live_moe_spec(_target(["vllm", "serve", str(ckpt)]), environ={}) + assert isinstance(r, mc.LiveSpecError) - assert "not a DeepSeek-V4-class" in r.reason + assert any("expert_dtype must be declared" in key for key in r.missing_keys) def test_live_moe_spec_refuses_when_no_config_found(tmp_path): diff --git a/tests/test_serving_latency.py b/tests/test_serving_latency.py index 89db662..9cf91ca 100644 --- a/tests/test_serving_latency.py +++ b/tests/test_serving_latency.py @@ -17,6 +17,7 @@ def _rec(arrival, first, finished, n_tokens): first_token_wall_s=first, finished_wall_s=finished, n_output_tokens=n_tokens, + token_count_source="engine", ) @@ -44,6 +45,20 @@ def test_missing_timestamps_yield_none_not_zero(): assert r.ttft_s is None and r.tpot_s is None +def test_nonmonotonic_timestamps_are_excluded_and_warned(): + record = _rec(10.0, 9.0, 8.0, 3) + + summary = summarize_requests([record]) + + assert record.ttft_s is None + assert record.tpot_s is None + assert summary.n_ttft == 0 + assert summary.n_tpot == 0 + assert summary.n_met_slo == 0 + assert summary.goodput_rps is None + assert any("non-monotonic" in warning for warning in summary.warnings) + + # --------------------------------------------------------------------------- # # summary: percentiles exclude unmeasurable requests # # --------------------------------------------------------------------------- # @@ -113,6 +128,52 @@ def test_single_token_request_not_penalised_for_absent_tpot(): assert s.n_met_slo == 1 +def test_chunk_count_tpot_is_flagged_and_excluded_from_goodput(): + record = RequestRecord( + arrival_wall_s=0.0, + first_token_wall_s=0.1, + finished_wall_s=1.1, + n_output_tokens=3, + token_count_source="chunks", + ) + + summary = summarize_requests([record], ttft_slo_s=1.0, tpot_slo_s=1.0) + + assert record.tpot_s is None + assert record.estimated_tpot_s == 0.5 + assert summary.n_tpot == 0 + assert summary.n_tpot_estimated == 1 + assert summary.n_met_slo == 0 + assert summary.goodput_rps == 0.0 + assert any("excluded" in warning for warning in summary.warnings) + + +def test_unknown_token_count_cannot_pass_tpot_slo_by_absence(): + record = RequestRecord( + arrival_wall_s=0.0, + first_token_wall_s=0.1, + finished_wall_s=1.0, + n_output_tokens=0, + ) + + summary = summarize_requests([record]) + + assert summary.n_met_slo == 0 + assert any("no authoritative" in warning for warning in summary.warnings) + + +def test_authoritative_zero_token_count_cannot_pass_slo(): + record = RequestRecord( + arrival_wall_s=0.0, + first_token_wall_s=0.1, + finished_wall_s=0.1, + n_output_tokens=0, + token_count_source="usage", + ) + + assert not record.meets_slo(ttft_slo_s=1.0, tpot_slo_s=1.0) + + # --------------------------------------------------------------------------- # # vLLM RequestOutput adaptation # # --------------------------------------------------------------------------- # @@ -123,6 +184,7 @@ def test_records_from_vllm_outputs(): ) (rec,) = request_records_from_outputs([out]) assert rec.n_output_tokens == 3 + assert rec.token_count_source == "engine" assert rec.ttft_s == 0.5 diff --git a/tests/test_smoke.py b/tests/test_smoke.py index a0f1ddc..11e6831 100644 --- a/tests/test_smoke.py +++ b/tests/test_smoke.py @@ -116,3 +116,28 @@ def test_doctor_returns_jsonable(): json.dumps(info) assert info["gitm_version"] assert "telemetry_backends" in info + + +def test_gpu_headroom_cli_surfaces_live_snapshot(monkeypatch, capsys): + from gitm.cli import main + + monkeypatch.setattr( + "gitm.optimizer.headroom_kernel_rank.live_gpu_headroom", + lambda: [{"gpu_index": 0, "util_pct": 42.0}], + ) + + assert main(["gpu-headroom"]) == 0 + assert json.loads(capsys.readouterr().out) == [ + {"gpu_index": 0, "util_pct": 42.0} + ] + + +def test_gpu_headroom_cli_fails_when_snapshot_is_unavailable(monkeypatch, capsys): + from gitm.cli import main + + monkeypatch.setattr( + "gitm.optimizer.headroom_kernel_rank.live_gpu_headroom", lambda: [] + ) + + assert main(["gpu-headroom"]) == 3 + assert json.loads(capsys.readouterr().out) == [] diff --git a/tests/test_telemetry_fallbacks.py b/tests/test_telemetry_fallbacks.py new file mode 100644 index 0000000..0af8381 --- /dev/null +++ b/tests/test_telemetry_fallbacks.py @@ -0,0 +1,191 @@ +from __future__ import annotations + +import threading +import time + +import pytest + +from gitm.telemetry.collector import Collector, CollectorConfig +from gitm.telemetry.schema import Sample + + +def test_discovery_surfaces_unexpected_vendor_failure_and_closes_partial_backend(monkeypatch): + import gitm.telemetry.backends.amd as amd + import gitm.telemetry.backends.nvidia as nvidia + from gitm.telemetry.backends.discover import discover_backends + + closed = [] + + class BrokenCount: + def device_count(self): + raise RuntimeError("NVML count failed") + + def close(self): + closed.append(True) + + monkeypatch.setattr(nvidia, "NvidiaBackend", BrokenCount) + monkeypatch.setattr( + amd, "AmdBackend", lambda: (_ for _ in ()).throw(ImportError("optional ROCm absent")) + ) + diagnostics = [] + + assert discover_backends(diagnostics=diagnostics) == [] + assert closed == [True] + assert any("nvidia" in d and "NVML count failed" in d for d in diagnostics) + assert not any("optional ROCm absent" in d for d in diagnostics) + + +def test_collector_carries_discovery_diagnostics(monkeypatch): + monkeypatch.setattr( + "gitm.telemetry.collector.discover_backends", + lambda *, diagnostics: diagnostics.append("nvidia discovery failed: denied") or [], + ) + + with pytest.warns(RuntimeWarning, match="nvidia discovery failed"): + collector = Collector(CollectorConfig()) + + assert any("nvidia discovery failed" in d for d in collector.diagnostics) + + +def test_doctor_reports_discovery_diagnostics_and_closes_backend(monkeypatch): + from gitm.doctor import doctor + + closed = [] + + class Backend: + vendor = "nvidia" + + def device_count(self): + return 1 + + def close(self): + closed.append(True) + + def discover(*, diagnostics): + diagnostics.append("amd discovery failed: broken runtime") + return [Backend()] + + monkeypatch.setattr("gitm.telemetry.backends.discover_backends", discover) + report = doctor() + + assert report["telemetry_diagnostics"] == ["amd discovery failed: broken runtime"] + assert report["telemetry_backends"] == [{"vendor": "nvidia", "device_count": 1}] + assert closed == [True] + + +def test_live_headroom_warns_discovery_diagnostic_and_closes_backend(monkeypatch): + from gitm.optimizer.headroom_kernel_rank import live_gpu_headroom + + closed = [] + + class Backend: + def device_count(self): + return 1 + + def sample(self, _index): + return Sample( + ts_ns=1, node="n", gpu_uuid="g", gpu_index=0, vendor="nvidia" + ) + + def close(self): + closed.append(True) + + def discover(*, diagnostics): + diagnostics.append("amd discovery failed: broken runtime") + return [Backend()] + + monkeypatch.setattr("gitm.telemetry.backends.discover_backends", discover) + with pytest.warns(RuntimeWarning, match="amd discovery failed"): + rows = live_gpu_headroom() + + assert len(rows) == 1 + assert closed == [True] + + +class _BrokenBackend: + def device_count(self): + return 1 + + def sample(self, _index, labels=None): + raise RuntimeError("nvml read failed") + + def close(self): + return None + + +def test_collector_surfaces_background_sample_failure_once(): + collector = Collector(CollectorConfig(interval_s=0.001, backends=[_BrokenBackend()])) + + with pytest.warns(RuntimeWarning, match="sample failed"): + collector.start() + time.sleep(0.02) + collector.stop() + + assert len([d for d in collector.diagnostics if "sample failed" in d]) == 1 + + +def test_collector_names_missing_backend_instead_of_looking_idle(): + with pytest.warns(RuntimeWarning, match="no live GPU telemetry backend"): + collector = Collector(CollectorConfig(backends=[])) + + assert collector.diagnostics + + +class _PartialBackend: + def device_count(self): + return 1 + + def sample(self, _index, labels=None): + return Sample( + ts_ns=1, + node="n", + gpu_uuid="g", + gpu_index=0, + vendor="nvidia", + diagnostics=["clock throttle reasons unavailable (NVMLError: denied)"], + ) + + def close(self): + return None + + +def test_collector_surfaces_partial_sample_field_failure(): + collector = Collector(CollectorConfig(interval_s=0.001, backends=[_PartialBackend()])) + + with pytest.warns(RuntimeWarning, match="clock throttle reasons unavailable"): + collector.start() + time.sleep(0.01) + collector.stop() + + assert any("clock throttle reasons unavailable" in d for d in collector.diagnostics) + + +class _BrokenClose: + def flush(self): + raise OSError("disk full") + + def close(self): + raise AssertionError("flush failure should propagate first") + + +def test_jsonl_sink_close_propagates_for_collector_diagnostic(): + from gitm.telemetry.sinks.jsonl import JsonlSink + + sink = JsonlSink.__new__(JsonlSink) + sink._lock = threading.Lock() + sink._fh = _BrokenClose() + + with pytest.raises(OSError, match="disk full"): + sink.close() + + +def test_otlp_sink_close_propagates_for_collector_diagnostic(): + from gitm.telemetry.sinks.otlp import OtlpSink + + sink = OtlpSink.__new__(OtlpSink) + sink._provider = type( + "BrokenProvider", (), {"shutdown": lambda self: (_ for _ in ()).throw(RuntimeError("export"))} + )() + + with pytest.raises(RuntimeError, match="export"): + sink.close() diff --git a/tests/test_timing.py b/tests/test_timing.py new file mode 100644 index 0000000..524014e --- /dev/null +++ b/tests/test_timing.py @@ -0,0 +1,22 @@ +from __future__ import annotations + +import math + +import pytest + +from gitm._timing import require_positive_duration, require_positive_work + + +@pytest.mark.parametrize("duration", [0.0, -1.0, math.inf, -math.inf, math.nan]) +def test_nonpositive_or_nonfinite_timing_is_refused(duration): + with pytest.raises(RuntimeError, match="timing unavailable"): + require_positive_duration(duration, context="test workload") + + +def test_positive_timing_is_preserved(): + assert require_positive_duration(0.125, context="test workload") == 0.125 + + +def test_empty_work_unit_is_refused(): + with pytest.raises(RuntimeError, match="work coverage unavailable"): + require_positive_work(0, context="test workload") diff --git a/tests/test_vllm_embodiment.py b/tests/test_vllm_embodiment.py index 6fa6a22..c6a671a 100644 --- a/tests/test_vllm_embodiment.py +++ b/tests/test_vllm_embodiment.py @@ -9,6 +9,7 @@ from __future__ import annotations from pathlib import Path +from types import SimpleNamespace import pytest @@ -80,6 +81,39 @@ def test_build_planner_context_sku_override(monkeypatch): assert pctx.gate.workload == "vllm-decode" assert pctx.gate.hardware == "NVIDIA A100-SXM4-80GB" assert pctx.gate.dtype is None # no engine attached + assert pctx.num_gpus_is_fallback is False + + +def test_build_planner_context_flags_unknown_gpu_count(monkeypatch): + import gitm.planner.context as context + + monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200") + monkeypatch.setattr(context, "_query_nvml", lambda: ("NVIDIA B200", None)) + + pctx = context.build_planner_context() + + assert pctx.num_gpus == 1 + assert pctx.num_gpus_is_fallback is True + + +def test_multi_gpu_count_does_not_fabricate_collective_or_interconnect(monkeypatch): + import gitm.planner.context as context + + monkeypatch.setattr(context, "_query_nvml", lambda: ("NVIDIA A100-SXM4-80GB", 4)) + + pctx = context.build_planner_context() + + assert pctx.num_gpus == 4 + assert pctx.gate.has_collective is False + assert pctx.gate.has_interconnect is None + + +@pytest.mark.parametrize("count", [0, -1]) +def test_build_planner_context_refuses_invalid_explicit_gpu_count(count): + from gitm.planner.context import build_planner_context + + with pytest.raises(ValueError, match="num_gpus must be positive"): + build_planner_context(num_gpus=count) def test_unknown_sku_yields_no_peak(monkeypatch): @@ -115,6 +149,7 @@ def test_hardware_spec_for_uses_detected_peak_not_default(): l4_peak = HardwarePeak(name="NVIDIA L4", peak_flops=121e12, peak_bw_bytes_s=300e9) hw = hardware_spec_for(l4_peak) assert hw.name == "NVIDIA L4" + assert not hw.is_fallback assert hw.peak_flops_fp16_per_s == 121e12 assert hw.peak_flops_bf16_per_s == 121e12 assert hw.peak_mem_bw_bytes_per_s == 300e9 @@ -124,7 +159,9 @@ def test_hardware_spec_for_falls_back_to_default_on_unknown_sku(): from gitm.planner.context import hardware_spec_for from gitm.planner.roofline import HardwareSpec - assert hardware_spec_for(None) == HardwareSpec() + hw = hardware_spec_for(None) + assert hw == HardwareSpec() + assert hw.is_fallback def test_predict_graph_on_l4_predicts_slower_than_default_a100(): @@ -318,7 +355,27 @@ class _LiveEngine: """Duck-typed vLLM engine with a restartable max_num_seqs lever.""" def __init__(self, max_num_seqs: int = 32): - self.model_config = _ModelConfig() + self.model_config = SimpleNamespace( + dtype="torch.bfloat16", + max_model_len=4096, + hf_config=SimpleNamespace( + model_type="llama", + hidden_size=64, + num_hidden_layers=2, + num_attention_heads=1, + num_key_value_heads=1, + head_dim=64, + intermediate_size=128, + vocab_size=128, + torch_dtype="bfloat16", + ), + ) + self.cache_config = SimpleNamespace(max_model_len=4096, cache_dtype="bf16") + self.parallel_config = SimpleNamespace( + tensor_parallel_size=1, + data_parallel_size=1, + enable_expert_parallel=False, + ) self.max_num_seqs = max_num_seqs # Decode throughput scales with batch width, so raising max_num_seqs is a # measurable win and is kept. Other structural knobs raise from restart @@ -351,6 +408,7 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None): monkeypatch.setattr(loop, "capture", fake_capture) monkeypatch.setattr(loop, "sync_device", lambda: None) + monkeypatch.setenv("GITM_GPU_SKU", "NVIDIA B200") engine = _LiveEngine() cfg = LoopConfig( @@ -384,8 +442,8 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None): assert summary["n_rolled_back"] >= 1 -def test_run_loop_no_engine_is_predict_only(tmp_path: Path, monkeypatch): - """Without an engine, candidates are unverified (no measured delta), never won.""" +def test_run_loop_no_engine_refuses_default_prediction(tmp_path: Path, monkeypatch): + """Without an engine, measurement survives but graph-based claims refuse.""" from contextlib import contextmanager import gitm.scheduler.loop as loop @@ -401,6 +459,7 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None): monkeypatch.setattr(loop, "sync_device", lambda: None) out = run_loop(LoopConfig(workload="vllm-decode", budget="5s", scratch=str(tmp_path))) - assert out["summary"]["status"] == "ok" - # No scheduler stats without an engine. - assert out["summary"]["scheduler_stats"] is None + assert out["summary"]["status"] == "prediction_refused" + assert out["summary"]["mode"] == "measurement" + assert out["summary"]["n_claims"] == 0 + assert "no live engine" in out["report_md"] diff --git a/tests/test_vllm_knobs_and_restart.py b/tests/test_vllm_knobs_and_restart.py index a4bbd44..a0e39fd 100644 --- a/tests/test_vllm_knobs_and_restart.py +++ b/tests/test_vllm_knobs_and_restart.py @@ -261,6 +261,25 @@ def test_restart_apply_rolls_back_to_original_engine_on_regression(): assert app.engine is e0 # original engine restored +@pytest.mark.parametrize("sample", [0.0, -1.0, float("nan"), float("inf")]) +def test_live_engine_ab_refuses_invalid_throughput_samples(sample): + engine = _TpsEngine(sample) + app = LiveEngineApplicator(engine, throughput_fn=_tps_of) + + result = apply_intervention(_spec("max_num_seqs", 64), app) + + assert not result.rolled_back + assert not result.applied + assert "finite and positive" in result.error + assert app.last_result is None + + +@pytest.mark.parametrize("reps", [0, -1, 1.5, True]) +def test_live_engine_ab_refuses_invalid_repetition_counts(reps): + with pytest.raises(ValueError, match="repetition count must be a positive integer"): + LiveEngineApplicator(_TpsEngine(100.0), throughput_fn=_tps_of, reps=reps) + + def test_serial_restart_releases_baseline_before_building_candidate(): class Engine(_TpsEngine): @@ -393,8 +412,19 @@ def test_causes_sorted_by_severity_desc(): # --------------------------------------------------------------------------- # # end-to-end: scheduler stats feed attribution + claim evidence (Task 2) # # --------------------------------------------------------------------------- # +class _DenseHFConfig: + def __init__(self): + self.hidden_size = 768 + self.num_hidden_layers = 12 + self.num_attention_heads = 12 + self.intermediate_size = 3072 + self.vocab_size = 50272 + self.torch_dtype = "bf16" + + class _ModelCfgBf16: dtype = "torch.bfloat16" + hf_config = _DenseHFConfig() class _LowOccScheduler: @@ -443,6 +473,7 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None): monkeypatch.setattr(loop, "capture", fake_capture) monkeypatch.setattr(loop, "sync_device", lambda: None) + monkeypatch.setenv("GITM_GPU_SKU", "A100-SXM4-80GB") engine = _FullEngine() # Non-expiring budget: max_num_seqs_dynamic ranks low and this asserts it's @@ -464,7 +495,7 @@ def fake_capture(out_path, *, workload_id="w", fingerprint="f", run_id=None): # Scheduler summary surfaced in the run summary (synchronous first sample). assert out["summary"]["scheduler_stats"] is not None -def test_report_kernel_time_residual_uses_weighted_total_and_clamps(): +def test_report_kernel_time_residual_preserves_raw_weighted_total(): from gitm.optimizer.monitor import KernelResidual, Residuals from gitm.scheduler.loop import _agg_kt_residual @@ -475,7 +506,9 @@ def test_report_kernel_time_residual_uses_weighted_total_and_clamps(): ] ) - assert _agg_kt_residual(res) == 1.0 + assert _agg_kt_residual(res) == pytest.approx( + (211e-6 - 10.01e-6) / 10.01e-6 + ) sane = Residuals( per_kernel=[ @@ -494,7 +527,7 @@ def test_ar_target_residual_uses_the_search_target_not_a_hardcoded_zero(): empty = AutoresearchRun(bottleneck_class="idle_stall", results=[], target=None) assert _ar_target_residual(empty) == 0.0 - # A real target -> its residual surfaces, clamped like every other residual. + # A real target -> its raw residual surfaces; display capping belongs to Claim. modest = AutoresearchRun( bottleneck_class="idle_stall", results=[], target=ResidualTarget(op="attn_score_value", residual=0.42, n_kernels=8), @@ -505,4 +538,4 @@ def test_ar_target_residual_uses_the_search_target_not_a_hardcoded_zero(): bottleneck_class="idle_stall", results=[], target=ResidualTarget(op="attn_score_value", residual=17.8, n_kernels=8), ) - assert _ar_target_residual(huge) == 1.0 + assert _ar_target_residual(huge) == 17.8 diff --git a/tests/test_vllm_stats_v1.py b/tests/test_vllm_stats_v1.py index 5089176..66ee856 100644 --- a/tests/test_vllm_stats_v1.py +++ b/tests/test_vllm_stats_v1.py @@ -48,3 +48,33 @@ def test_v1_engine_without_stats_is_none(): ) ) assert read_scheduler_stats(empty) is None + + +def test_v1_scheduler_read_failure_is_diagnostic(): + class BrokenScheduler: + def make_stats(self): + raise RuntimeError("v1 stats moved") + + engine = SimpleNamespace( + engine_core=SimpleNamespace(scheduler=BrokenScheduler()) + ) + + sample = read_scheduler_stats(engine) + + assert sample is not None + assert sample.num_running is None + assert any( + "make_stats failed" in note and "v1 stats moved" in note + for note in sample.diagnostics + ) + + +def test_v1_invalid_values_are_diagnostic_not_clamped(): + sample = read_scheduler_stats(_v1_engine(running=300, waiting=-1, cache=1.2)) + + assert sample is not None + assert sample.num_waiting is None + assert sample.gpu_cache_usage is None + assert sample.batch_occupancy is None + assert any("outside [0, 1]" in note for note in sample.diagnostics) + assert any("exceeds declared capacity" in note for note in sample.diagnostics) diff --git a/tests/test_vllm_stress.py b/tests/test_vllm_stress.py index adde9cf..7c1cca4 100644 --- a/tests/test_vllm_stress.py +++ b/tests/test_vllm_stress.py @@ -9,6 +9,8 @@ import threading +import pytest + from gitm.kernels.spec import InterventionSpec from gitm.optimizer.apply import LiveEngineApplicator, apply_intervention from gitm.optimizer.deviation import deviating_kernel_indices, deviation_summary, deviation_trace @@ -93,14 +95,17 @@ def get_num_unfinished_requests(self): raise RuntimeError("engine busy") -def test_sampler_swallows_engine_exceptions(): +def test_sampler_surfaces_engine_exceptions_without_crashing(): sampler = SchedulerStatsSampler(_RaisingEngine(), interval_s=0.002) - sampler.start() - for _ in range(500): - pass - sampler.stop() # must not raise + with pytest.warns(RuntimeWarning, match="scheduler telemetry degraded"): + sampler.start() + for _ in range(500): + pass + sampler.stop() # must not raise # A raising engine yields no usable samples, summary is the empty shape. - assert sampler.summary().n_samples == 0 + summary = sampler.summary() + assert summary.n_samples == 0 + assert summary.diagnostics def test_sampler_repeated_start_stop_is_safe(): @@ -131,6 +136,51 @@ class _Partial: assert s.num_running is None and s.batch_occupancy is None +def test_scheduler_sampler_surfaces_unrecognized_engine_shape(): + sampler = SchedulerStatsSampler(object(), interval_s=1.0) + + with pytest.warns(RuntimeWarning, match="no recognized scheduler fields"): + sampler.start() + sampler.stop() + + assert any( + "no recognized scheduler fields" in note + for note in sampler.summary().diagnostics + ) + + +def test_read_scheduler_stats_surfaces_field_probe_failures(): + class BlockManager: + num_total_gpu_blocks = 8 + + def get_num_free_gpu_blocks(self): + raise RuntimeError("cache probe broke") + + class Scheduler: + running = [object()] + block_manager = BlockManager() + + class Engine: + scheduler = Scheduler() + + def get_num_unfinished_requests(self): + raise RuntimeError("unfinished probe broke") + + sample = read_scheduler_stats(Engine()) + + assert sample is not None + assert sample.num_running == 1 + assert any("cache usage probe failed" in note for note in sample.diagnostics) + assert any("unfinished-request probe failed" in note for note in sample.diagnostics) + + +def test_reversed_scheduler_sample_timeline_is_refused(): + from gitm.tracer.vllm_stats import SchedulerSample + + with pytest.raises(ValueError, match="timestamps are not monotonic"): + summarize([SchedulerSample(t_ns=10), SchedulerSample(t_ns=5)]) + + def test_summarize_single_sample(): s = read_scheduler_stats( type("E", (), {"scheduler": [type("S", (), {"running": [0, 1], "waiting": [], "swapped": []})()], @@ -220,14 +270,15 @@ class _E: def test_measure_rolls_back_on_nonpositive_baseline(): - """An unmeasurable (zero) baseline must roll the candidate back, not keep it.""" + """An unmeasurable baseline refuses before apply, so no rollback is claimed.""" class _Idle: scheduler_config = _SchedCfg() - # Baseline probe returns 0 (idle engine), so measure() raises after apply. + # Baseline probe returns 0 (idle engine), so snapshot refuses before apply. app = LiveEngineApplicator(_Idle(), throughput_fn=lambda e: 0.0, restart_fn=lambda _e, _kv: _Idle()) res = apply_intervention(_spec("max_num_seqs", 256), app, min_keep_delta=0.0) - assert not res.applied and res.rolled_back + assert not res.applied and not res.rolled_back + assert "intervention not applied" in res.error def test_deviation_multistep_does_not_keep_everything(): diff --git a/tests/test_workload_bootstrap.py b/tests/test_workload_bootstrap.py index bb0bd38..e4fd155 100644 --- a/tests/test_workload_bootstrap.py +++ b/tests/test_workload_bootstrap.py @@ -20,7 +20,8 @@ def test_ensure_hft_data_generates_then_reuses(tmp_path: Path, monkeypatch): monkeypatch.setenv("GITM_BENCH_EVENTS", "2000") stage = tmp_path / "stage" - _ensure_hft_data(stage, 42) + with pytest.warns(RuntimeWarning, match="synthetic smoke coverage"): + _ensure_hft_data(stage, 42) seed_dir = stage / "hft_smoke_seed42" shards = sorted(seed_dir.glob("part-*.parquet")) assert shards, "smoke data should have been generated" @@ -133,6 +134,7 @@ def generate(self, prompts, params): "vllm", types.SimpleNamespace(LLM=FakeLLM, SamplingParams=FakeSamplingParams), ) + monkeypatch.setattr("gitm.cuda_env.require_compatible", lambda: None) monkeypatch.setattr(workloads, "sync_device", lambda: None) monkeypatch.setenv("GITM_VLLM_MODEL", "fake/model") monkeypatch.setenv("GITM_VLLM_PROMPTS", "2")