From 5d43386f1f4828ecf74eeba1988ea91407259474 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 26 Aug 2026 21:01:07 -0700 Subject: [PATCH 01/35] =?UTF-8?q?feat:=20model=20GLM-5.2=20(glm=5Fmoe=5Fds?= =?UTF-8?q?a)=20=E2=80=94=20MLA=20+=20DSA=20indexer=20+=20IndexShare?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fork the sparse-MoE predicted execution graph into a new glm_moe_dsa family specialised for zai-org/GLM-5.2 (GlmMoeDsaForCausalLM). Built from config.json and the checkpoint's safetensors index only — no traces. What the fork prices that the V4 graph could not: - MLA attention: one kv_lora_rank=512 latent shared across 64 query heads (KV traffic derived from the latent, not num_key_value_heads); no CSA/HCA/SWA compression schedule — every layer runs the same DSA attention. - IndexShare: only 21 of 78 layers compute the lightning indexer; the 57 'shared' layers reuse a neighbour's top-2048 selection and carry no indexer weights (proven from the weight map). Emitting indexer nodes on full layers only keeps the indexer at ~0.4% instead of a 4x overcount. - dense-then-sparse MLP schedule (first_k_dense_replace=3), bf16 throughout (no fp4 expert default leaking in from V4), plain dense o_proj. model_weight_bytes validates to +0.1% against the published 1.507 TB checkpoint. Wiring: new GlmMoeDsaModelSpec + predict_glm_graph + spec/detector in glm_graph.py; catalogue entry models/glm-5.2.yaml; all eight dispatch sites in model_catalogue.py and registry.py (glm_moe_dsa detected before sparse_moe, since both carry index_topk + n_routed_experts — model_type is the separator). Deliverables under docs/glm-5.2/: standup BRIEF.md (end-to-end, headroom, RunPod) and three JSON graph artifacts. Tests in tests/test_glm_graph.py pin IndexShare, MLA KV sharing, the dense/sparse split, bf16 precision, footprint, and detection order. Co-Authored-By: Claude Opus 4.8 --- docs/glm-5.2/BRIEF.md | 198 + .../glm-5.2_H200_b1_kv131072_tp1ep1.json | 9152 +++++++++++++ .../glm-5.2_H200_b1_kv4096_tp1ep1.json | 9152 +++++++++++++ .../glm-5.2_H200_b32_kv8192_tp16ep16.json | 10824 ++++++++++++++++ gitm/planner/glm_graph.py | 700 + gitm/planner/model_catalogue.py | 14 +- gitm/planner/models/glm-5.2.yaml | 188 + gitm/planner/registry.py | 30 +- tests/test_glm_graph.py | 182 + tests/test_moe_graph.py | 4 + 10 files changed, 30438 insertions(+), 6 deletions(-) create mode 100644 docs/glm-5.2/BRIEF.md create mode 100644 docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv131072_tp1ep1.json create mode 100644 docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv4096_tp1ep1.json create mode 100644 docs/glm-5.2/artifacts/glm-5.2_H200_b32_kv8192_tp16ep16.json create mode 100644 gitm/planner/glm_graph.py create mode 100644 gitm/planner/models/glm-5.2.yaml create mode 100644 tests/test_glm_graph.py diff --git a/docs/glm-5.2/BRIEF.md b/docs/glm-5.2/BRIEF.md new file mode 100644 index 0000000..461fc90 --- /dev/null +++ b/docs/glm-5.2/BRIEF.md @@ -0,0 +1,198 @@ +# GLM-5.2 — standup brief (execution-graph view) + +**What this is:** GLM-5.2 (`zai-org/GLM-5.2`, `GlmMoeDsaForCausalLM`) modelled end-to-end +in the GitM runtime/execution-graph stack, as a new **`glm_moe_dsa`** family forked from +the DeepSeek-V4 sparse-MoE graph. Everything below is derived **from `config.json` and the +checkpoint's own `model.safetensors.index.json`** — no traces, no observed performance. The +numbers are a roofline **floor at vendor peak**, i.e. a lower bound on time, not a target. + +Reproduce any figure here: +```bash +gitm plan glm-5.2 --gpu H200 --batch 1 --kv-len 4096 # op-share table +gitm plan glm-5.2 --gpu H200 --sweep 1,4,16,32,64,128,256 # batch crossover +gitm plan glm-5.2 --gpu H200 --batch 1 --kv-len 131072 --json > graph.json +``` + +--- + +## 1. What GLM-5.2 is, in one breath + +A **754B-parameter** (1.507 TB bf16, validated to **+0.1%** against the published checkpoint) +sparse-MoE decoder, **~40B active per token** (config-derived, using the repo's `active_params` +convention — which folds in the ~1B untied embed + lm_head, so ~39B is the real per-token +multiply). Three things make it its own family rather than "another DeepSeek": + +1. **MLA attention** — one compressed KV latent (`kv_lora_rank=512`) shared across all 64 + query heads. There is **no** per-layer compression schedule (no CSA/HCA, no sliding + window) — every layer runs the same attention. +2. **DeepSeek Sparse Attention (DSA)** — a lightning indexer scores the whole history and + keeps the top **`index_topk=2048`** positions for the attention core. +3. **IndexShare** — the mechanism this fork exists to price. Only **21 of 78 layers** + compute the index; the other **57 reuse** a neighbour's selection and physically carry + **no indexer weights**. + +## 2. Layer stack / block structure + +| | | +|---|---| +| Layers | **78** transformer + **1** MTP draft head | +| Hidden | 6144, vocab 154,880 (untied embed + lm_head) | +| MLP schedule | layers **0–2 dense** FFN (`intermediate=12288`); **3–77 MoE** (`first_k_dense_replace=3`) | +| MoE | 256 routed experts, **top-8**, **1 shared**, `moe_intermediate=2048`, sigmoid + `noaux_tc` routing, `routed_scaling=2.5` | +| Attention | MLA: `q_lora=2048`, `kv_lora=512`, per-head `qk=256` (nope 192 + rope 64), **`v_head=256`** | +| Indexer | 32 heads × 128 dim, top-2048, **IndexShare period 4** | +| MTP | 1 draft layer, `index_share_for_mtp_iteration=true` (reuses the main index) | +| Precision | **bf16 throughout** — no `quantization_config` in the release | + +Each block = `attn_q_a → attn_q_b → attn_kv_a → [attn_index_proj → attn_index_score on full +layers only] → attn_score_value → attn_qnorm_rope_insert → attn_out_proj → {dense FFN | moe_router → moe_shared → moe_routed}`. + +## 3. Attention pattern & IndexShare — the headline + +`indexer_types` is read verbatim from the checkpoint: layers **0,1,2 `full`**, then a strict +**period-4** cycle — one `full` layer that recomputes the top-2048 selection, then three +`shared` layers that reuse it (`index_topk_freq=4`, `index_skip_topk_offset=3`). + +**This is proven from the weight map, not inferred:** indexer tensors (`*.indexer.*`) exist on +exactly the 21 `full` layers and on **none** of the 57 `shared` layers. A shared layer that +recomputed the index would need those weights; it doesn't have them. So the fork emits indexer +nodes on 21 layers only. Pricing all 78 at full rate — the naive reading of `index_topk` — +would overstate the indexer's share ~4× and mis-rank it against the MoE term. + +**Consequence in the graph:** attention is *flat in context*. The core reads at most +`index_topk=2048` selected positions however long the context grows; only the indexer *scan* +grows, and IndexShare caps that to 21 layers. Measured in the graph, 4K → 128K context: + +| op | 4K ctx | 128K ctx | behaviour | +|---|---|---|---| +| `attn_score_value` (core) | 0.039 ms | 0.039 ms | **flat** — bounded by top-k | +| `attn_index_score` (scan) | 0.005 ms | 0.147 ms | grows with context, but only 21 layers, still <1% | + +## 4. MoE / routing structure + +75 sparse layers + the MTP head each run: a replicated **`moe_router`** (h→256), one always-on +**`moe_shared`** expert, and **`moe_routed`** — top-8 of 256. The routed term is the whole +decode story: **FLOPs scale with `positions × 8`**, but **weight traffic scales with the number +of _distinct_ experts the batch woke**, which saturates at 256 (`distinct_experts`). At batch 1 +a step fetches ~8 experts/layer; by batch ~256 it fetches essentially all of them for the same +per-token FLOPs — which is why decode is memory-bound at low batch and only turns compute-bound +at high batch. + +## 5. Expected kernel classes & roofline (the op-share table) + +**H200 (989 TFLOP/s bf16, 4.80 TB/s), batch 1, kv 4096, whole-model floor** — ridge 206 FLOP/byte, so **every node is memory-bound**: + +| op | share | bound | what it is | +|---|---:|---|---| +| `moe_routed` | **56.1%** | memory | routed-expert weight fetch (the union term) | +| `attn_out_proj` | **19.4%** | memory | output GEMM — wide because `v_head=256` → 16,384-wide input | +| `moe_shared` | 7.0% | memory | always-on shared expert | +| `attn_q_b` | 6.5% | memory | query up-projection (64×256 heads) | +| `attn_kv_b` | 2.8% | memory | KV up-projection from latent (unabsorbed MLA — see §7.3) | +| `attn_q_a` | 2.4% | memory | query down-projection (replicated) | +| `lm_head` | 2.3% | memory | vocab projection | +| everything else | <2% each | memory | kv_a, dense FFN, **indexer (0.4%)**, router, core, norms | + +Floor: **17.0 ms/step** at batch 1 on this shape. **This is a per-GPU rate, not an achievable +config — the model is 1.5 TB and does not fit one H200 (see §7).** Kernel classes the graph +expects: **memory-bound GEMV/GEMM** (all projections + experts), **grouped-GEMM** (routed +experts), **paged-attention core** over selected KV, **indexer score/top-k** (21 layers), **fused +qnorm+RoPE+cache-insert**, and — under sharding — **all-reduce** (TP) / **all-to-all** (EP). + +**Honest end-to-end throughput** (a shape that fits — 16×H200, TP16/EP16, batch 32, kv 8192): +**15.5 ms/step, ~2,060 tok/s**, collectives priced, still fully memory-bound. + +**Batch crossover** (`--sweep`, TP=1 per-GPU reference, kv 4096): 59 tok/s @ b1 → 782 tok/s @ +b256, where **341/830 nodes turn compute-bound** as the expert union saturates. (Per-GPU rates — +multiply by ranks, minus collective overhead, for a real deployment.) + +## 6. Algebraic roofline structure & assumptions + +Per op: `t = max(flops/peak_dtype, bytes/HBM_bw)`, peak resolved **per dtype** (bf16 here). +Load-bearing terms and the assumptions behind them: + +- **Expert traffic = a set-union, not a multiply:** `distinct(B) = 256·(1−(1−8/256)^B)`. Assumes + **uniform routing** — real skew touches *fewer* experts, so this over-predicts traffic (the + safe direction). +- **KV traffic uses the shared latent** `kv_lora_rank + qk_rope = 576` elems/token/layer — **not** + `num_key_value_heads × head_dim`. The config's `num_key_value_heads=64` is a red herring; a GQA + reading inflates decode KV ~57×. +- **Attention core is not divided by TP** — a single shared latent can't be split, so the cache is + replicated and every rank reads all of it. **TP buys no KV bandwidth on this architecture.** +- **`q_a` / `kv_a` are replicated** across TP ranks (they make the shared latent), so TP's speedup + on attention is strictly less than `tp`. +- **Collectives are bandwidth-only** (no latency floor) → optimistic at decode message sizes, + flagged `estimated`. **EP imbalance stays at 1.0** (perfect balance) — it is trace-calibrated by + design and, with the no-traces constraint, is declared rather than fitted. +- **Decode only.** Prefill (chunked, different indexer asymptotics) is unmodelled, as in every + family. + +## 7. Where the headroom likely is — and what to validate first + +Ranked by expected payoff, all **hypotheses from the graph** to be confirmed against a capture: + +1. **The model doesn't fit one box — the deployment shape is the first-order lever.** 1.507 TB + bf16 needs **≥11 H200 / ≥8 B200** for weights alone (realistically **16×H200 = 2 nodes** with + KV+activation headroom). **FP8 halves it to ~0.75 TB → fits 8×H200.** Validate first: does an + FP8 expert cast hold GLM-5.2's quality? Everything else is downstream of this. +2. **`moe_routed` (58%) is the memory-bound heart.** Levers: **EP with a good expert-placement / + EPLB** to cut per-rank distinct-expert traffic and, at serving batch, the **grouped-GEMM + backend** (DeepGEMM). Validate: measured per-rank expert traffic vs the `distinct_experts` + prediction, and the **real EP imbalance** (the one number we must calibrate from a trace). +3. **`attn_out_proj` (19%) + `attn_kv_b` (3%) hinge on one question: does the engine run + _absorbed_ MLA?** We model it **unabsorbed** — `kv_b` runs as its own GEMM and `o_proj` is + narrow (16384→6144). If vLLM absorbs MLA (the usual decode path), `kv_b` disappears and + `o_proj` **doubles** to 32768→6144. Same resident weights, but the #2 line moves by 2× in + either direction. Validate first: which path the serving engine takes — it re-ranks the whole + attention side. +4. **IndexShare is already a big win the graph shows is _cheap_ (0.4%).** The validate-first + question is the opposite of a lever: **confirm the runtime actually skips the scan on `shared` + layers** (and reuses across the group), because if it doesn't, a hidden 4× indexer cost is + sitting off our books. Also confirm `index_topk_freq` is purely spatial (layer-group) vs also + temporal (step-to-step) — extra temporal reuse only *reduces* cost. +5. **MTP acceptance** — we emit the draft as a shared (no-indexer) block per + `index_share_for_mtp_iteration`. Validate the accepted-token rate to see whether MTP pays off, + and that the draft indexer is genuinely shared. + +**First capture to take:** a single decode step at a shape that fits (16×H200, TP16/EP16, batch +16–32, kv 8K — or 8×B200, TP8/EP8), attributed per-op, and diff it against +`gitm plan glm-5.2 --gpu H200 --batch 32 --kv-len 8192 --tp 16 --ep 16 --json`. The three numbers +that matter most: **EP imbalance**, **`moe_routed` per-rank traffic**, and whether **`shared` +layers emit any indexer kernel at all**. + +## 8. How to run it on RunPod + +The model is too large for a single GPU, so this is a **multi-GPU / multi-node** job. Two paths: + +**A. Predict-only (no GPU needed — do this first, it's free and already works):** +```bash +pip install -e . # this repo +gitm plan glm-5.2 --gpu H200 --batch 32 --kv-len 8192 --tp 16 --ep 16 # 16×H200, fits +gitm plan glm-5.2 --gpu B200 --batch 32 --kv-len 8192 --tp 8 --ep 8 --json # 8×B200, fits +``` + +**B. Serve + capture on RunPod (to get the numbers §7 wants validated):** +1. Pick hardware by the footprint math above: **bf16 → 2× `8×H200` (or 8×B200)**; **fp8 → + 1× `8×H200`**. On RunPod choose an **8×H200 SXM** pod (or two, networked) with a **network + volume ≥ 2 TB** for the 282-shard checkpoint. +2. Serve with a DSA-aware engine (vLLM/SGLang build with `glm_moe_dsa` + MLA + DSA support), + e.g. `--tensor-parallel-size 8 --enable-expert-parallel`, `--kv-cache-dtype fp8` (serving + choice, not a model fact), `--max-model-len` to taste. +3. Attach the GitM collector to the live server and capture a bounded decode window: + ```bash + gitm capture serve # or: gitm capture attach (see PR #79) + ``` +4. Import the trace and diff observed-vs-predicted per op; residuals outside the efficiency band + are the leads §7 lists — **a residual here is a lead, not a defect.** + +> ⚠️ The engine flags in step 2 are the shape to aim for, not verified command lines — confirm +> the serving engine actually supports `glm_moe_dsa` before booking a multi-node pod. Start with +> path A (free) and the FP8 fit question, which gates everything else. + +--- + +### Artifacts in this directory +- `artifacts/glm-5.2_H200_b1_kv4096_tp1ep1.json` — canonical decode-step graph (830 nodes, per-GPU rate) +- `artifacts/glm-5.2_H200_b1_kv131072_tp1ep1.json` — long-context (indexer scan grows) +- `artifacts/glm-5.2_H200_b32_kv8192_tp16ep16.json` — a shape that fits: 16×H200, collectives priced +- Graph code: `gitm/planner/glm_graph.py` · catalogue: `gitm/planner/models/glm-5.2.yaml` · tests: `tests/test_glm_graph.py` diff --git a/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv131072_tp1ep1.json b/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv131072_tp1ep1.json new file mode 100644 index 0000000..7613b5e --- /dev/null +++ b/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv131072_tp1ep1.json @@ -0,0 +1,9152 @@ +{ + "model": "zai-org/GLM-5.2", + "family": "glm_moe_dsa", + "hardware": "H200", + "sharding": { + "tp": 1, + "ep": 1, + "dp": 1 + }, + "batch": { + "batch": 1, + "kv_cache_len": 131072, + "prefill_tokens": 0, + "prefill_context": 0, + "prefill_requests": 1 + }, + "total_pred_s": 0.017185272906666667, + "has_unpriced_collectives": false, + "has_fallback_peaks": false, + "nodes": [ + { + "op": "attn_q_a", + "layer": 0, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 0, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 0, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 0, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 0, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 0, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 0, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 0, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 0, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "mlp_gate_up", + "layer": 0, + "t_pred_s": 6.292736e-05, + "t_compute_s": 3.053487239635996e-07, + "t_memory_s": 6.292736e-05, + "bound": "memory", + "flops": 301989888.0, + "bytes": 302051328.0, + "estimated": false + }, + { + "op": "mlp_down", + "layer": 0, + "t_pred_s": 3.146496e-05, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 3.146496e-05, + "bound": "memory", + "flops": 150994944.0, + "bytes": 151031808.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 1, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 1, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 1, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 1, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 1, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 1, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 1, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 1, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 1, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "mlp_gate_up", + "layer": 1, + "t_pred_s": 6.292736e-05, + "t_compute_s": 3.053487239635996e-07, + "t_memory_s": 6.292736e-05, + "bound": "memory", + "flops": 301989888.0, + "bytes": 302051328.0, + "estimated": false + }, + { + "op": "mlp_down", + "layer": 1, + "t_pred_s": 3.146496e-05, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 3.146496e-05, + "bound": "memory", + "flops": 150994944.0, + "bytes": 151031808.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 2, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 2, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 2, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 2, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 2, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 2, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 2, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 2, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 2, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "mlp_gate_up", + "layer": 2, + "t_pred_s": 6.292736e-05, + "t_compute_s": 3.053487239635996e-07, + "t_memory_s": 6.292736e-05, + "bound": "memory", + "flops": 301989888.0, + "bytes": 302051328.0, + "estimated": false + }, + { + "op": "mlp_down", + "layer": 2, + "t_pred_s": 3.146496e-05, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 3.146496e-05, + "bound": "memory", + "flops": 150994944.0, + "bytes": 151031808.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 3, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 3, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 3, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 3, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 3, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 3, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 3, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 3, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 3, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 3, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 4, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 4, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 4, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 4, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 4, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 4, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 4, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 4, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 4, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 4, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 5, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 5, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 5, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 5, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 5, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 5, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 5, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 5, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 5, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 5, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 6, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 6, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 6, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 6, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 6, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 6, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 6, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 6, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 6, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 6, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 6, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 6, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 7, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 7, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 7, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 7, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 7, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 7, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 7, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 7, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 7, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 7, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 8, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 8, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 8, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 8, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 8, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 8, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 8, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 8, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 8, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 8, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 9, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 9, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 9, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 9, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 9, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 9, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 9, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 9, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 9, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 9, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 10, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 10, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 10, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 10, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 10, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 10, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 10, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 10, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 10, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 10, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 10, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 10, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 11, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 11, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 11, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 11, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 11, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 11, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 11, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 11, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 11, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 11, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 12, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 12, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 12, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 12, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 12, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 12, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 12, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 12, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 12, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 12, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 13, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 13, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 13, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 13, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 13, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 13, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 13, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 13, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 13, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 13, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 14, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 14, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 14, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 14, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 14, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 14, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 14, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 14, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 14, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 14, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 14, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 14, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 15, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 15, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 15, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 15, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 15, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 15, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 15, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 15, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 15, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 15, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 16, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 16, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 16, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 16, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 16, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 16, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 16, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 16, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 16, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 16, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 17, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 17, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 17, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 17, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 17, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 17, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 17, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 17, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 17, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 17, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 18, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 18, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 18, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 18, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 18, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 18, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 18, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 18, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 18, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 18, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 18, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 18, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 19, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 19, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 19, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 19, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 19, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 19, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 19, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 19, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 19, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 19, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 20, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 20, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 20, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 20, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 20, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 20, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 20, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 20, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 20, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 20, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 21, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 21, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 21, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 21, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 21, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 21, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 21, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 21, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 21, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 21, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 22, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 22, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 22, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 22, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 22, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 22, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 22, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 22, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 22, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 22, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 22, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 22, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 23, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 23, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 23, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 23, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 23, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 23, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 23, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 23, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 23, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 23, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 24, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 24, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 24, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 24, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 24, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 24, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 24, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 24, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 24, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 24, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 25, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 25, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 25, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 25, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 25, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 25, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 25, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 25, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 25, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 25, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 26, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 26, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 26, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 26, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 26, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 26, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 26, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 26, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 26, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 26, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 26, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 26, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 27, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 27, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 27, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 27, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 27, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 27, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 27, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 27, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 27, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 27, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 28, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 28, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 28, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 28, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 28, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 28, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 28, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 28, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 28, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 28, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 29, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 29, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 29, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 29, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 29, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 29, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 29, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 29, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 29, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 29, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 30, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 30, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 30, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 30, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 30, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 30, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 30, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 30, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 30, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 30, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 30, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 30, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 31, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 31, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 31, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 31, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 31, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 31, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 31, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 31, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 31, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 31, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 32, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 32, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 32, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 32, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 32, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 32, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 32, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 32, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 32, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 32, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 33, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 33, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 33, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 33, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 33, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 33, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 33, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 33, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 33, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 33, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 34, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 34, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 34, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 34, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 34, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 34, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 34, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 34, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 34, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 34, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 34, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 34, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 35, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 35, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 35, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 35, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 35, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 35, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 35, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 35, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 35, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 35, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 36, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 36, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 36, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 36, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 36, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 36, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 36, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 36, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 36, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 36, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 37, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 37, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 37, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 37, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 37, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 37, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 37, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 37, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 37, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 37, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 38, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 38, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 38, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 38, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 38, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 38, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 38, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 38, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 38, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 38, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 38, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 38, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 39, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 39, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 39, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 39, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 39, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 39, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 39, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 39, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 39, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 39, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 40, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 40, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 40, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 40, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 40, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 40, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 40, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 40, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 40, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 40, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 41, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 41, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 41, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 41, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 41, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 41, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 41, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 41, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 41, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 41, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 42, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 42, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 42, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 42, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 42, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 42, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 42, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 42, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 42, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 42, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 42, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 42, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 43, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 43, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 43, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 43, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 43, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 43, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 43, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 43, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 43, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 43, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 44, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 44, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 44, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 44, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 44, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 44, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 44, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 44, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 44, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 44, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 45, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 45, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 45, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 45, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 45, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 45, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 45, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 45, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 45, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 45, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 46, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 46, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 46, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 46, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 46, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 46, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 46, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 46, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 46, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 46, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 46, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 46, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 47, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 47, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 47, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 47, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 47, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 47, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 47, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 47, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 47, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 47, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 48, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 48, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 48, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 48, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 48, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 48, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 48, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 48, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 48, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 48, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 49, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 49, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 49, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 49, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 49, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 49, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 49, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 49, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 49, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 49, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 50, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 50, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 50, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 50, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 50, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 50, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 50, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 50, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 50, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 50, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 50, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 50, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 51, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 51, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 51, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 51, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 51, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 51, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 51, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 51, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 51, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 51, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 52, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 52, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 52, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 52, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 52, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 52, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 52, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 52, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 52, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 52, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 53, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 53, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 53, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 53, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 53, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 53, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 53, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 53, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 53, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 53, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 54, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 54, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 54, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 54, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 54, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 54, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 54, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 54, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 54, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 54, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 54, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 54, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 55, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 55, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 55, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 55, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 55, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 55, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 55, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 55, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 55, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 55, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 56, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 56, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 56, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 56, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 56, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 56, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 56, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 56, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 56, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 56, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 57, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 57, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 57, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 57, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 57, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 57, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 57, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 57, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 57, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 57, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 58, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 58, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 58, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 58, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 58, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 58, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 58, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 58, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 58, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 58, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 58, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 58, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 59, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 59, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 59, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 59, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 59, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 59, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 59, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 59, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 59, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 59, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 60, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 60, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 60, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 60, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 60, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 60, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 60, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 60, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 60, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 60, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 61, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 61, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 61, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 61, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 61, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 61, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 61, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 61, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 61, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 61, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 62, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 62, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 62, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 62, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 62, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 62, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 62, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 62, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 62, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 62, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 62, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 62, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 63, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 63, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 63, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 63, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 63, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 63, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 63, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 63, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 63, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 63, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 64, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 64, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 64, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 64, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 64, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 64, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 64, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 64, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 64, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 64, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 65, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 65, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 65, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 65, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 65, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 65, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 65, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 65, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 65, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 65, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 66, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 66, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 66, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 66, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 66, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 66, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 66, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 66, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 66, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 66, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 66, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 66, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 67, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 67, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 67, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 67, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 67, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 67, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 67, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 67, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 67, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 67, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 68, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 68, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 68, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 68, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 68, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 68, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 68, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 68, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 68, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 68, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 69, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 69, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 69, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 69, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 69, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 69, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 69, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 69, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 69, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 69, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 70, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 70, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 70, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 70, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 70, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 70, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 70, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 70, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 70, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 70, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 70, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 70, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 71, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 71, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 71, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 71, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 71, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 71, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 71, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 71, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 71, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 71, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 72, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 72, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 72, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 72, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 72, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 72, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 72, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 72, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 72, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 72, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 73, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 73, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 73, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 73, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 73, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 73, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 73, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 73, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 73, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 73, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 74, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 74, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 74, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 74, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 74, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 74, + "t_pred_s": 6.990506666666666e-06, + "t_compute_s": 1.0856843518705764e-06, + "t_memory_s": 6.990506666666666e-06, + "bound": "memory", + "flops": 1073741824.0, + "bytes": 33554432.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 74, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 74, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 74, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 74, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 74, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 74, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 75, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 75, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 75, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 75, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 75, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 75, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 75, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 75, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 75, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 75, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 76, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 76, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 76, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 76, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 76, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 76, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 76, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 76, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 76, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 76, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 77, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 77, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 77, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 77, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 77, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 77, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 77, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 77, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 77, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 77, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 78, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 78, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 78, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 78, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 78, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 78, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 78, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 78, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 78, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 78, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "lm_head", + "layer": null, + "t_pred_s": 0.00039655989333333336, + "t_compute_s": 1.9243331041456017e-06, + "t_memory_s": 0.00039655989333333336, + "bound": "memory", + "flops": 1903165440.0, + "bytes": 1903487488.0, + "estimated": false + } + ] +} diff --git a/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv4096_tp1ep1.json b/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv4096_tp1ep1.json new file mode 100644 index 0000000..727500e --- /dev/null +++ b/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv4096_tp1ep1.json @@ -0,0 +1,9152 @@ +{ + "model": "zai-org/GLM-5.2", + "family": "glm_moe_dsa", + "hardware": "H200", + "sharding": { + "tp": 1, + "ep": 1, + "dp": 1 + }, + "batch": { + "batch": 1, + "kv_cache_len": 4096, + "prefill_tokens": 0, + "prefill_context": 0, + "prefill_requests": 1 + }, + "total_pred_s": 0.017043059786666665, + "has_unpriced_collectives": false, + "has_fallback_peaks": false, + "nodes": [ + { + "op": "attn_q_a", + "layer": 0, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 0, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 0, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 0, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 0, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 0, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 0, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 0, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 0, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "mlp_gate_up", + "layer": 0, + "t_pred_s": 6.292736e-05, + "t_compute_s": 3.053487239635996e-07, + "t_memory_s": 6.292736e-05, + "bound": "memory", + "flops": 301989888.0, + "bytes": 302051328.0, + "estimated": false + }, + { + "op": "mlp_down", + "layer": 0, + "t_pred_s": 3.146496e-05, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 3.146496e-05, + "bound": "memory", + "flops": 150994944.0, + "bytes": 151031808.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 1, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 1, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 1, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 1, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 1, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 1, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 1, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 1, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 1, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "mlp_gate_up", + "layer": 1, + "t_pred_s": 6.292736e-05, + "t_compute_s": 3.053487239635996e-07, + "t_memory_s": 6.292736e-05, + "bound": "memory", + "flops": 301989888.0, + "bytes": 302051328.0, + "estimated": false + }, + { + "op": "mlp_down", + "layer": 1, + "t_pred_s": 3.146496e-05, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 3.146496e-05, + "bound": "memory", + "flops": 150994944.0, + "bytes": 151031808.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 2, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 2, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 2, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 2, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 2, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 2, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 2, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 2, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 2, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "mlp_gate_up", + "layer": 2, + "t_pred_s": 6.292736e-05, + "t_compute_s": 3.053487239635996e-07, + "t_memory_s": 6.292736e-05, + "bound": "memory", + "flops": 301989888.0, + "bytes": 302051328.0, + "estimated": false + }, + { + "op": "mlp_down", + "layer": 2, + "t_pred_s": 3.146496e-05, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 3.146496e-05, + "bound": "memory", + "flops": 150994944.0, + "bytes": 151031808.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 3, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 3, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 3, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 3, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 3, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 3, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 3, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 3, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 3, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 3, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 4, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 4, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 4, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 4, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 4, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 4, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 4, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 4, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 4, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 4, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 5, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 5, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 5, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 5, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 5, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 5, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 5, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 5, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 5, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 5, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 6, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 6, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 6, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 6, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 6, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 6, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 6, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 6, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 6, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 6, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 6, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 6, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 7, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 7, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 7, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 7, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 7, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 7, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 7, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 7, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 7, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 7, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 8, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 8, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 8, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 8, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 8, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 8, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 8, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 8, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 8, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 8, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 9, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 9, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 9, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 9, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 9, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 9, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 9, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 9, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 9, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 9, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 10, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 10, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 10, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 10, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 10, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 10, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 10, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 10, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 10, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 10, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 10, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 10, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 11, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 11, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 11, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 11, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 11, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 11, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 11, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 11, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 11, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 11, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 12, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 12, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 12, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 12, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 12, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 12, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 12, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 12, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 12, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 12, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 13, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 13, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 13, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 13, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 13, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 13, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 13, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 13, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 13, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 13, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 14, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 14, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 14, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 14, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 14, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 14, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 14, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 14, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 14, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 14, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 14, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 14, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 15, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 15, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 15, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 15, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 15, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 15, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 15, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 15, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 15, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 15, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 16, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 16, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 16, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 16, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 16, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 16, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 16, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 16, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 16, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 16, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 17, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 17, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 17, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 17, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 17, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 17, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 17, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 17, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 17, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 17, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 18, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 18, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 18, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 18, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 18, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 18, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 18, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 18, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 18, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 18, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 18, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 18, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 19, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 19, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 19, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 19, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 19, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 19, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 19, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 19, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 19, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 19, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 20, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 20, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 20, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 20, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 20, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 20, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 20, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 20, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 20, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 20, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 21, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 21, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 21, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 21, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 21, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 21, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 21, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 21, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 21, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 21, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 22, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 22, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 22, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 22, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 22, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 22, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 22, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 22, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 22, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 22, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 22, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 22, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 23, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 23, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 23, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 23, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 23, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 23, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 23, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 23, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 23, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 23, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 24, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 24, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 24, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 24, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 24, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 24, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 24, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 24, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 24, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 24, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 25, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 25, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 25, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 25, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 25, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 25, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 25, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 25, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 25, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 25, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 26, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 26, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 26, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 26, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 26, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 26, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 26, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 26, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 26, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 26, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 26, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 26, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 27, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 27, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 27, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 27, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 27, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 27, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 27, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 27, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 27, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 27, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 28, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 28, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 28, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 28, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 28, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 28, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 28, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 28, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 28, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 28, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 29, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 29, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 29, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 29, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 29, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 29, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 29, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 29, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 29, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 29, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 30, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 30, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 30, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 30, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 30, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 30, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 30, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 30, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 30, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 30, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 30, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 30, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 31, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 31, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 31, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 31, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 31, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 31, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 31, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 31, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 31, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 31, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 32, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 32, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 32, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 32, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 32, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 32, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 32, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 32, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 32, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 32, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 33, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 33, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 33, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 33, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 33, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 33, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 33, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 33, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 33, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 33, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 34, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 34, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 34, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 34, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 34, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 34, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 34, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 34, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 34, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 34, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 34, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 34, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 35, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 35, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 35, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 35, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 35, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 35, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 35, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 35, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 35, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 35, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 36, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 36, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 36, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 36, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 36, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 36, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 36, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 36, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 36, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 36, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 37, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 37, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 37, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 37, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 37, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 37, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 37, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 37, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 37, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 37, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 38, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 38, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 38, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 38, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 38, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 38, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 38, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 38, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 38, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 38, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 38, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 38, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 39, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 39, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 39, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 39, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 39, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 39, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 39, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 39, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 39, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 39, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 40, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 40, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 40, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 40, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 40, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 40, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 40, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 40, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 40, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 40, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 41, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 41, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 41, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 41, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 41, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 41, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 41, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 41, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 41, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 41, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 42, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 42, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 42, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 42, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 42, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 42, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 42, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 42, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 42, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 42, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 42, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 42, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 43, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 43, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 43, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 43, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 43, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 43, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 43, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 43, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 43, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 43, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 44, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 44, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 44, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 44, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 44, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 44, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 44, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 44, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 44, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 44, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 45, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 45, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 45, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 45, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 45, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 45, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 45, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 45, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 45, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 45, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 46, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 46, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 46, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 46, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 46, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 46, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 46, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 46, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 46, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 46, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 46, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 46, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 47, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 47, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 47, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 47, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 47, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 47, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 47, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 47, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 47, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 47, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 48, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 48, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 48, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 48, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 48, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 48, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 48, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 48, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 48, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 48, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 49, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 49, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 49, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 49, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 49, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 49, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 49, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 49, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 49, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 49, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 50, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 50, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 50, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 50, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 50, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 50, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 50, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 50, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 50, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 50, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 50, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 50, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 51, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 51, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 51, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 51, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 51, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 51, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 51, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 51, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 51, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 51, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 52, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 52, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 52, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 52, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 52, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 52, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 52, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 52, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 52, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 52, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 53, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 53, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 53, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 53, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 53, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 53, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 53, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 53, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 53, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 53, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 54, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 54, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 54, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 54, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 54, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 54, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 54, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 54, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 54, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 54, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 54, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 54, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 55, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 55, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 55, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 55, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 55, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 55, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 55, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 55, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 55, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 55, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 56, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 56, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 56, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 56, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 56, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 56, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 56, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 56, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 56, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 56, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 57, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 57, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 57, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 57, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 57, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 57, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 57, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 57, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 57, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 57, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 58, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 58, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 58, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 58, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 58, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 58, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 58, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 58, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 58, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 58, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 58, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 58, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 59, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 59, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 59, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 59, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 59, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 59, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 59, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 59, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 59, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 59, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 60, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 60, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 60, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 60, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 60, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 60, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 60, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 60, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 60, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 60, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 61, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 61, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 61, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 61, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 61, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 61, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 61, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 61, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 61, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 61, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 62, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 62, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 62, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 62, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 62, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 62, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 62, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 62, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 62, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 62, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 62, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 62, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 63, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 63, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 63, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 63, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 63, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 63, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 63, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 63, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 63, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 63, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 64, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 64, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 64, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 64, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 64, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 64, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 64, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 64, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 64, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 64, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 65, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 65, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 65, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 65, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 65, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 65, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 65, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 65, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 65, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 65, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 66, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 66, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 66, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 66, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 66, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 66, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 66, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 66, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 66, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 66, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 66, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 66, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 67, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 67, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 67, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 67, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 67, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 67, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 67, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 67, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 67, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 67, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 68, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 68, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 68, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 68, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 68, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 68, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 68, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 68, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 68, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 68, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 69, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 69, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 69, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 69, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 69, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 69, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 69, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 69, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 69, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 69, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 70, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 70, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 70, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 70, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 70, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 70, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 70, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 70, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 70, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 70, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 70, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 70, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 71, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 71, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 71, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 71, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 71, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 71, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 71, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 71, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 71, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 71, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 72, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 72, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 72, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 72, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 72, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 72, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 72, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 72, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 72, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 72, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 73, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 73, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 73, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 73, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 73, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 73, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 73, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 73, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 73, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 73, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 74, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 74, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 74, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 74, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 74, + "t_pred_s": 3.4978133333333335e-06, + "t_compute_s": 1.6963817997977756e-08, + "t_memory_s": 3.4978133333333335e-06, + "bound": "memory", + "flops": 16777216.0, + "bytes": 16789504.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 74, + "t_pred_s": 2.1845333333333332e-07, + "t_compute_s": 3.392763599595551e-08, + "t_memory_s": 2.1845333333333332e-07, + "bound": "memory", + "flops": 33554432.0, + "bytes": 1048576.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 74, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 74, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 74, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 74, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 74, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 74, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 75, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 75, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 75, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 75, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 75, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 75, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 75, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 75, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 75, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 75, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 76, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 76, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 76, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 76, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 76, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 76, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 76, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 76, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 76, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 76, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 77, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 77, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 77, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 77, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 77, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 77, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 77, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 77, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 77, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 77, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 78, + "t_pred_s": 5.246293333333333e-06, + "t_compute_s": 2.5445726996966633e-08, + "t_memory_s": 5.246293333333333e-06, + "bound": "memory", + "flops": 25165824.0, + "bytes": 25182208.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 78, + "t_pred_s": 1.3988693333333333e-05, + "t_compute_s": 6.785527199191102e-08, + "t_memory_s": 1.3988693333333333e-05, + "bound": "memory", + "flops": 67108864.0, + "bytes": 67145728.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 78, + "t_pred_s": 1.4776e-06, + "t_compute_s": 7.156610717896866e-09, + "t_memory_s": 1.4776e-06, + "bound": "memory", + "flops": 7077888.0, + "bytes": 7092480.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 78, + "t_pred_s": 6.128853333333334e-06, + "t_compute_s": 2.9686681496461072e-08, + "t_memory_s": 6.128853333333334e-06, + "bound": "memory", + "flops": 29360128.0, + "bytes": 29418496.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 78, + "t_pred_s": 4.9152e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 4.9152e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 2359296.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 78, + "t_pred_s": 1.76e-08, + "t_compute_s": 7.66835187057634e-11, + "t_memory_s": 1.76e-08, + "bound": "memory", + "flops": 75840.0, + "bytes": 84480.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 78, + "t_pred_s": 4.1952426666666666e-05, + "t_compute_s": 2.0356581597573307e-07, + "t_memory_s": 4.1952426666666666e-05, + "bound": "memory", + "flops": 201326592.0, + "bytes": 201371648.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 78, + "t_pred_s": 6.580266666666667e-07, + "t_compute_s": 3.180715874620829e-09, + "t_memory_s": 6.580266666666667e-07, + "bound": "memory", + "flops": 3145728.0, + "bytes": 3158528.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 78, + "t_pred_s": 1.5735466666666665e-05, + "t_compute_s": 7.63371809908999e-08, + "t_memory_s": 1.5735466666666665e-05, + "bound": "memory", + "flops": 75497472.0, + "bytes": 75530240.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 78, + "t_pred_s": 0.00012584789333333334, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 0.00012584789333333334, + "bound": "memory", + "flops": 603979776.0, + "bytes": 604069888.0, + "estimated": false + }, + { + "op": "lm_head", + "layer": null, + "t_pred_s": 0.00039655989333333336, + "t_compute_s": 1.9243331041456017e-06, + "t_memory_s": 0.00039655989333333336, + "bound": "memory", + "flops": 1903165440.0, + "bytes": 1903487488.0, + "estimated": false + } + ] +} diff --git a/docs/glm-5.2/artifacts/glm-5.2_H200_b32_kv8192_tp16ep16.json b/docs/glm-5.2/artifacts/glm-5.2_H200_b32_kv8192_tp16ep16.json new file mode 100644 index 0000000..f5e90bd --- /dev/null +++ b/docs/glm-5.2/artifacts/glm-5.2_H200_b32_kv8192_tp16ep16.json @@ -0,0 +1,10824 @@ +{ + "model": "zai-org/GLM-5.2", + "family": "glm_moe_dsa", + "hardware": "H200", + "sharding": { + "tp": 16, + "ep": 16, + "dp": 1 + }, + "batch": { + "batch": 32, + "kv_cache_len": 8192, + "prefill_tokens": 0, + "prefill_context": 0, + "prefill_requests": 1 + }, + "total_pred_s": 0.015507311864019567, + "has_unpriced_collectives": false, + "has_fallback_peaks": false, + "nodes": [ + { + "op": "attn_q_a", + "layer": 0, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 0, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 0, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 0, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 0, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 0, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 0, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 0, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 0, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "mlp_gate_up", + "layer": 0, + "t_pred_s": 4.03456e-06, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 4.03456e-06, + "bound": "memory", + "flops": 603979776.0, + "bytes": 19365888.0, + "estimated": false + }, + { + "op": "mlp_down", + "layer": 0, + "t_pred_s": 2.05824e-06, + "t_compute_s": 3.053487239635996e-07, + "t_memory_s": 2.05824e-06, + "bound": "memory", + "flops": 301989888.0, + "bytes": 9879552.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 1, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 1, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 1, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 1, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 1, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 1, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 1, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 1, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 1, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "mlp_gate_up", + "layer": 1, + "t_pred_s": 4.03456e-06, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 4.03456e-06, + "bound": "memory", + "flops": 603979776.0, + "bytes": 19365888.0, + "estimated": false + }, + { + "op": "mlp_down", + "layer": 1, + "t_pred_s": 2.05824e-06, + "t_compute_s": 3.053487239635996e-07, + "t_memory_s": 2.05824e-06, + "bound": "memory", + "flops": 301989888.0, + "bytes": 9879552.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 2, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 2, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 2, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 2, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 2, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 2, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 2, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 2, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 2, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "mlp_gate_up", + "layer": 2, + "t_pred_s": 4.03456e-06, + "t_compute_s": 6.106974479271992e-07, + "t_memory_s": 4.03456e-06, + "bound": "memory", + "flops": 603979776.0, + "bytes": 19365888.0, + "estimated": false + }, + { + "op": "mlp_down", + "layer": 2, + "t_pred_s": 2.05824e-06, + "t_compute_s": 3.053487239635996e-07, + "t_memory_s": 2.05824e-06, + "bound": "memory", + "flops": 301989888.0, + "bytes": 9879552.0, + "estimated": false + }, + { + "op": "attn_q_a", + "layer": 3, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 3, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 3, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 3, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 3, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 3, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 3, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 3, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 3, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 3, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 3, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 3, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 4, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 4, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 4, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 4, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 4, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 4, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 4, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 4, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 4, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 4, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 4, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 4, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 5, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 5, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 5, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 5, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 5, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 5, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 5, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 5, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 5, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 5, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 5, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 5, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 6, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 6, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 6, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 6, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 6, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 6, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 6, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 6, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 6, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 6, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 6, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 6, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 6, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 6, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 7, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 7, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 7, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 7, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 7, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 7, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 7, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 7, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 7, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 7, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 7, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 7, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 8, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 8, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 8, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 8, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 8, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 8, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 8, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 8, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 8, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 8, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 8, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 8, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 9, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 9, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 9, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 9, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 9, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 9, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 9, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 9, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 9, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 9, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 9, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 9, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 10, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 10, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 10, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 10, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 10, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 10, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 10, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 10, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 10, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 10, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 10, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 10, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 10, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 10, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 11, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 11, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 11, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 11, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 11, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 11, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 11, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 11, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 11, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 11, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 11, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 11, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 12, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 12, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 12, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 12, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 12, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 12, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 12, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 12, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 12, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 12, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 12, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 12, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 13, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 13, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 13, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 13, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 13, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 13, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 13, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 13, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 13, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 13, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 13, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 13, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 14, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 14, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 14, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 14, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 14, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 14, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 14, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 14, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 14, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 14, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 14, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 14, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 14, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 14, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 15, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 15, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 15, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 15, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 15, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 15, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 15, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 15, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 15, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 15, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 15, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 15, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 16, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 16, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 16, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 16, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 16, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 16, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 16, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 16, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 16, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 16, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 16, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 16, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 17, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 17, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 17, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 17, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 17, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 17, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 17, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 17, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 17, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 17, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 17, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 17, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 18, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 18, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 18, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 18, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 18, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 18, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 18, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 18, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 18, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 18, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 18, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 18, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 18, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 18, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 19, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 19, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 19, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 19, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 19, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 19, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 19, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 19, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 19, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 19, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 19, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 19, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 20, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 20, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 20, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 20, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 20, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 20, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 20, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 20, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 20, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 20, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 20, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 20, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 21, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 21, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 21, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 21, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 21, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 21, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 21, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 21, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 21, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 21, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 21, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 21, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 22, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 22, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 22, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 22, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 22, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 22, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 22, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 22, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 22, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 22, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 22, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 22, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 22, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 22, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 23, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 23, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 23, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 23, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 23, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 23, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 23, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 23, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 23, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 23, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 23, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 23, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 24, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 24, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 24, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 24, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 24, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 24, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 24, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 24, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 24, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 24, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 24, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 24, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 25, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 25, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 25, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 25, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 25, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 25, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 25, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 25, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 25, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 25, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 25, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 25, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 26, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 26, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 26, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 26, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 26, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 26, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 26, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 26, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 26, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 26, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 26, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 26, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 26, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 26, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 27, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 27, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 27, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 27, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 27, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 27, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 27, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 27, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 27, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 27, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 27, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 27, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 28, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 28, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 28, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 28, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 28, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 28, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 28, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 28, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 28, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 28, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 28, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 28, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 29, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 29, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 29, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 29, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 29, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 29, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 29, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 29, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 29, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 29, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 29, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 29, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 30, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 30, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 30, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 30, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 30, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 30, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 30, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 30, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 30, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 30, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 30, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 30, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 30, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 30, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 31, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 31, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 31, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 31, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 31, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 31, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 31, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 31, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 31, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 31, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 31, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 31, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 32, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 32, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 32, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 32, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 32, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 32, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 32, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 32, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 32, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 32, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 32, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 32, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 33, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 33, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 33, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 33, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 33, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 33, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 33, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 33, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 33, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 33, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 33, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 33, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 34, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 34, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 34, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 34, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 34, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 34, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 34, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 34, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 34, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 34, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 34, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 34, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 34, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 34, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 35, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 35, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 35, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 35, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 35, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 35, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 35, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 35, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 35, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 35, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 35, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 35, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 36, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 36, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 36, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 36, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 36, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 36, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 36, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 36, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 36, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 36, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 36, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 36, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 37, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 37, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 37, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 37, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 37, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 37, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 37, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 37, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 37, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 37, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 37, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 37, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 38, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 38, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 38, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 38, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 38, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 38, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 38, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 38, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 38, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 38, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 38, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 38, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 38, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 38, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 39, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 39, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 39, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 39, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 39, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 39, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 39, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 39, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 39, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 39, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 39, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 39, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 40, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 40, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 40, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 40, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 40, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 40, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 40, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 40, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 40, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 40, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 40, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 40, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 41, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 41, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 41, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 41, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 41, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 41, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 41, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 41, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 41, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 41, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 41, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 41, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 42, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 42, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 42, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 42, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 42, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 42, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 42, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 42, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 42, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 42, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 42, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 42, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 42, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 42, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 43, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 43, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 43, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 43, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 43, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 43, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 43, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 43, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 43, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 43, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 43, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 43, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 44, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 44, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 44, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 44, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 44, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 44, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 44, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 44, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 44, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 44, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 44, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 44, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 45, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 45, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 45, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 45, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 45, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 45, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 45, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 45, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 45, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 45, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 45, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 45, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 46, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 46, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 46, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 46, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 46, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 46, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 46, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 46, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 46, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 46, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 46, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 46, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 46, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 46, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 47, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 47, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 47, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 47, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 47, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 47, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 47, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 47, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 47, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 47, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 47, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 47, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 48, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 48, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 48, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 48, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 48, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 48, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 48, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 48, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 48, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 48, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 48, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 48, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 49, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 49, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 49, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 49, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 49, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 49, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 49, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 49, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 49, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 49, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 49, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 49, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 50, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 50, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 50, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 50, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 50, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 50, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 50, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 50, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 50, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 50, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 50, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 50, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 50, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 50, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 51, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 51, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 51, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 51, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 51, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 51, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 51, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 51, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 51, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 51, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 51, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 51, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 52, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 52, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 52, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 52, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 52, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 52, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 52, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 52, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 52, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 52, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 52, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 52, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 53, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 53, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 53, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 53, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 53, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 53, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 53, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 53, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 53, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 53, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 53, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 53, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 54, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 54, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 54, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 54, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 54, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 54, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 54, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 54, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 54, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 54, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 54, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 54, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 54, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 54, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 55, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 55, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 55, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 55, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 55, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 55, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 55, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 55, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 55, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 55, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 55, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 55, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 56, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 56, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 56, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 56, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 56, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 56, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 56, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 56, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 56, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 56, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 56, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 56, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 57, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 57, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 57, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 57, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 57, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 57, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 57, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 57, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 57, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 57, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 57, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 57, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 58, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 58, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 58, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 58, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 58, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 58, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 58, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 58, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 58, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 58, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 58, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 58, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 58, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 58, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 59, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 59, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 59, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 59, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 59, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 59, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 59, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 59, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 59, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 59, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 59, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 59, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 60, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 60, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 60, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 60, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 60, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 60, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 60, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 60, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 60, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 60, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 60, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 60, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 61, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 61, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 61, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 61, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 61, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 61, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 61, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 61, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 61, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 61, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 61, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 61, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 62, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 62, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 62, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 62, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 62, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 62, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 62, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 62, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 62, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 62, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 62, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 62, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 62, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 62, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 63, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 63, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 63, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 63, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 63, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 63, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 63, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 63, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 63, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 63, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 63, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 63, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 64, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 64, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 64, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 64, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 64, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 64, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 64, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 64, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 64, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 64, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 64, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 64, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 65, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 65, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 65, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 65, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 65, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 65, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 65, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 65, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 65, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 65, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 65, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 65, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 66, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 66, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 66, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 66, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 66, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 66, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 66, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 66, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 66, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 66, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 66, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 66, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 66, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 66, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 67, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 67, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 67, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 67, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 67, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 67, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 67, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 67, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 67, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 67, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 67, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 67, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 68, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 68, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 68, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 68, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 68, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 68, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 68, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 68, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 68, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 68, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 68, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 68, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 69, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 69, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 69, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 69, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 69, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 69, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 69, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 69, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 69, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 69, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 69, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 69, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 70, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 70, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 70, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 70, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 70, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 70, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 70, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 70, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 70, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 70, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 70, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 70, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 70, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 70, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 71, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 71, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 71, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 71, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 71, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 71, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 71, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 71, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 71, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 71, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 71, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 71, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 72, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 72, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 72, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 72, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 72, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 72, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 72, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 72, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 72, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 72, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 72, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 72, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 73, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 73, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 73, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 73, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 73, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 73, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 73, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 73, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 73, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 73, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 73, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 73, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 74, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 74, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 74, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 74, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_index_proj", + "layer": 74, + "t_pred_s": 3.5771733333333333e-06, + "t_compute_s": 5.428421759352882e-07, + "t_memory_s": 3.5771733333333333e-06, + "bound": "memory", + "flops": 536870912.0, + "bytes": 17170432.0, + "estimated": false + }, + { + "op": "attn_index_score", + "layer": 74, + "t_pred_s": 1.3981013333333332e-05, + "t_compute_s": 2.1713687037411527e-06, + "t_memory_s": 1.3981013333333332e-05, + "bound": "memory", + "flops": 2147483648.0, + "bytes": 67108864.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 74, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 74, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 74, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 74, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 74, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 74, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 74, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 74, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 75, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 75, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 75, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 75, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 75, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 75, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 75, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 75, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 75, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 75, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 75, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 75, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 76, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 76, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 76, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 76, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 76, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 76, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 76, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 76, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 76, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 76, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 76, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 76, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 77, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 77, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 77, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 77, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 77, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 77, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 77, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 77, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 77, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 77, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 77, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 77, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "attn_q_a", + "layer": 78, + "t_pred_s": 5.352106666666667e-06, + "t_compute_s": 8.142632639029323e-07, + "t_memory_s": 5.352106666666667e-06, + "bound": "memory", + "flops": 805306368.0, + "bytes": 25690112.0, + "estimated": false + }, + { + "op": "attn_q_b", + "layer": 78, + "t_pred_s": 9.147733333333334e-07, + "t_compute_s": 1.3571054398382204e-07, + "t_memory_s": 9.147733333333334e-07, + "bound": "memory", + "flops": 134217728.0, + "bytes": 4390912.0, + "estimated": false + }, + { + "op": "attn_kv_a", + "layer": 78, + "t_pred_s": 1.57184e-06, + "t_compute_s": 2.290115429726997e-07, + "t_memory_s": 1.57184e-06, + "bound": "memory", + "flops": 226492416.0, + "bytes": 7544832.0, + "estimated": false + }, + { + "op": "attn_kv_b", + "layer": 78, + "t_pred_s": 4.1301333333333334e-07, + "t_compute_s": 5.9373362992922145e-08, + "t_memory_s": 4.1301333333333334e-07, + "bound": "memory", + "flops": 58720256.0, + "bytes": 1982464.0, + "estimated": false + }, + { + "op": "attn_score_value", + "layer": 78, + "t_pred_s": 1.572864e-05, + "t_compute_s": 2.714210879676441e-07, + "t_memory_s": 1.572864e-05, + "bound": "memory", + "flops": 268435456.0, + "bytes": 75497472.0, + "estimated": false + }, + { + "op": "attn_qnorm_rope_insert", + "layer": 78, + "t_pred_s": 5.12e-08, + "t_compute_s": 2.1743174924165824e-10, + "t_memory_s": 5.12e-08, + "bound": "memory", + "flops": 215040.0, + "bytes": 245760.0, + "estimated": false + }, + { + "op": "attn_out_proj", + "layer": 78, + "t_pred_s": 2.7170133333333333e-06, + "t_compute_s": 4.0713163195146613e-07, + "t_memory_s": 2.7170133333333333e-06, + "bound": "memory", + "flops": 402653184.0, + "bytes": 13041664.0, + "estimated": false + }, + { + "op": "moe_router", + "layer": 78, + "t_pred_s": 7.406933333333333e-07, + "t_compute_s": 1.0178290798786653e-07, + "t_memory_s": 7.406933333333333e-07, + "bound": "memory", + "flops": 100663296.0, + "bytes": 3555328.0, + "estimated": false + }, + { + "op": "moe_shared", + "layer": 78, + "t_pred_s": 1.1502933333333334e-06, + "t_compute_s": 1.526743619817998e-07, + "t_memory_s": 1.1502933333333334e-06, + "bound": "memory", + "flops": 150994944.0, + "bytes": 5521408.0, + "estimated": false + }, + { + "op": "moe_routed", + "layer": 78, + "t_pred_s": 0.00016073518978973114, + "t_compute_s": 1.2213948958543984e-06, + "t_memory_s": 0.00016073518978973114, + "bound": "memory", + "flops": 1207959552.0, + "bytes": 771528910.9907094, + "estimated": false + }, + { + "op": "moe_all_to_all", + "layer": 78, + "t_pred_s": 6.5536e-06, + "t_compute_s": 0.0, + "t_memory_s": 6.5536e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 5898240.0, + "estimated": true + }, + { + "op": "tp_all_reduce", + "layer": 78, + "t_pred_s": 1.6384e-06, + "t_compute_s": 0.0, + "t_memory_s": 1.6384e-06, + "bound": "memory", + "flops": 0.0, + "bytes": 1474560.0, + "estimated": true + }, + { + "op": "lm_head", + "layer": null, + "t_pred_s": 2.4991786666666668e-05, + "t_compute_s": 3.8486662082912035e-06, + "t_memory_s": 2.4991786666666668e-05, + "bound": "memory", + "flops": 3806330880.0, + "bytes": 119960576.0, + "estimated": false + } + ] +} diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py new file mode 100644 index 0000000..6fd8761 --- /dev/null +++ b/gitm/planner/glm_graph.py @@ -0,0 +1,700 @@ +"""Predicted execution graph for a GLM-5.2-class (``glm_moe_dsa``) decode step. + +A fork of :mod:`gitm.planner.moe_graph`, specialised for ZhipuAI's +``GlmMoeDsaForCausalLM``. Both families are sparse-MoE with a lightning indexer, +but the attention differs in kind, not degree, so a shared spec would carry a +field for each that is dead on the other — the exact "a default silently +activates" hazard the roofline module warns about. The two graphs therefore live +apart, and share only the canonical op names (so residuals stay comparable) and +the :func:`~gitm.planner.roofline.distinct_experts` union term (one owner, no +drift). + +What GLM-5.2 is, read from ``config.json`` and the checkpoint's own tensor index +(``model.safetensors.index.json``), never from a trace: + +**Attention is MLA + DeepSeek Sparse Attention, with no per-layer compression.** +Unlike DeepSeek-V4's CSA/HCA schedule (``compress_ratios``), every GLM layer runs +the *same* attention: a compressed KV latent (``kv_lora_rank=512``, shared across +all 64 query heads) plus a lightning indexer that scores the whole history and +keeps ``index_topk=2048`` positions for the core. There is no ``m``/``m'`` split, +no HCA, no sliding window. So the V4 fields that encode that schedule are simply +absent here. + +**IndexShare — the mechanism this fork exists to price.** The checkpoint declares +``indexer_types`` per layer: ``full`` or ``shared``. A ``full`` layer computes its +own indexer (projection + score) and selects the top-k; the next three ``shared`` +layers *reuse that selection* and run no indexer at all. This is not an inference: +the ``shared`` layers physically **carry no indexer tensors** in the weight map. +Only 21 of 78 layers run the indexer; 57 skip it. ``index_topk_freq=4`` is the +period of that grouping (one full + three shared), and ``index_skip_topk_offset`` +its offset. Pricing every layer's indexer at full rate — as a naive reading of +``index_topk`` would — overstates the indexer's share of the step roughly 4x and +mis-ranks it against the MoE weight traffic that actually dominates decode. + +**The MLP schedule is dense-then-sparse.** ``first_k_dense_replace=3``: the first +three layers run a conventional dense FFN (``intermediate_size=12288``) with no +router and no experts; the remaining 75 are MoE (256 routed experts, top-8, one +shared, ``moe_intermediate_size=2048``). Modelling the dense layers as MoE would +invent a router GEMM and expert traffic that the weight map shows are not there. + +**Precision is bf16 throughout.** The published checkpoint carries no +``quantization_config`` — the on-disk size (1.507 TB) matches an all-bf16 read. +Any fp8 the KV cache or experts pick up is a *serving* decision, not a model fact, +so — unlike the V4 reader, which defaults to fp4 experts — this graph prices bf16 +and records quantisation as a deployment lever in provenance. Defaulting to fp4 +here would deflate the dominant expert term ~3.8x and manufacture headroom that +the checkpoint does not have. + +Known limits, stated rather than hidden: + +* **Decode only.** As with every other graph family; the chunked-prefill indexer + path has different asymptotics. +* **Uniform routing.** :func:`~gitm.planner.roofline.distinct_experts` assumes a + balanced router; real skew touches fewer distinct experts, moving *less* traffic + than predicted — the conservative direction. +* **Expert-parallel skew is calibrated, not predicted.** + :attr:`ShardingConfig.ep_imbalance` stays at 1.0 until a trace measures it. +* **Collectives are bandwidth-only** (no latency floor), flagged ``estimated``. +""" + +from __future__ import annotations + +from dataclasses import dataclass, replace +from typing import Any + +from gitm.planner.graph import Graph, PredictedNode +from gitm.planner.roofline import ( + BatchConfig, + HardwareSpec, + ShardingConfig, + distinct_experts, + roofline, + weight_bytes, +) + +FULL_INDEXER = "full" +SHARED_INDEXER = "shared" +DENSE_MLP = "dense" +SPARSE_MLP = "sparse" + + +@dataclass(frozen=True) +class GlmMoeDsaModelSpec: + """Model shape for a GLM-5.2-class (``glm_moe_dsa``) decode step. + + **The defaults below are a small reference shape, not any real checkpoint** — + deliberately far too small to be mistaken for GLM-5.2. Real checkpoints live in + ``gitm/planner/models/*.yaml`` (``family: glm_moe_dsa``) or are read from a + config by :func:`spec_from_hf_config`. The comments cite GLM-5.2 as the worked + example precisely where its values differ from these defaults in ways that + matter. + """ + + name: str = "glm-moe-dsa-reference" + hidden: int = 512 + n_layers: int = 6 + vocab: int = 2048 + + # ── MLA attention ──────────────────────────────────────────────────────── + n_heads: int = 8 + #: Query down-projection rank. The query goes ``hidden -> q_lora_rank -> + #: n_heads * q_head_dim``; the middle rank is a real, replicated matrix. + q_lora_rank: int = 256 + #: Compressed KV latent width. The cache holds **one** latent per token per + #: layer, shared across every query head — deriving KV traffic from + #: ``num_key_value_heads * head_dim`` instead is the classic MLA error and + #: overstates it by ``n_heads`` (64x on GLM-5.2). + kv_lora_rank: int = 128 + #: Per-head query/key width carrying no rotary embedding. + qk_nope_head_dim: int = 96 + #: Per-head query/key width carrying rotary embedding. The decoupled RoPE key + #: is shared (one per token, MQA-style) alongside the latent in the cache. + qk_rope_head_dim: int = 32 + #: Per-head value width. On GLM-5.2 this (256) differs from ``qk_nope`` (192), + #: so the score and the value read use different per-head widths. + v_head_dim: int = 128 + + # ── DeepSeek Sparse Attention indexer ──────────────────────────────────── + index_n_heads: int = 16 + index_head_dim: int = 64 + #: Positions the indexer keeps for the attention core. The core read is + #: bounded by this once history exceeds it, so attention stops growing with + #: context while the *indexer scan* keeps growing — a different node, a + #: different bound. + index_topk: int = 512 + #: Period of the IndexShare grouping: one ``full`` layer that computes the + #: selection, then ``index_topk_freq - 1`` ``shared`` layers that reuse it. + index_topk_freq: int = 4 + #: Per-layer ``full`` | ``shared``, straight from the checkpoint. Authoritative + #: when present — the ``shared`` layers carry no indexer weights, so this is + #: read, not a rule guessed from the frequency. Empty falls back to + #: :attr:`index_topk_freq` (first layer of each group is ``full``). + indexer_types: tuple[str, ...] = () + + # ── mixture of experts ─────────────────────────────────────────────────── + n_routed_experts: int = 8 + n_shared_experts: int = 1 + num_experts_per_tok: int = 2 + moe_intermediate_size: int = 256 + #: Dense-FFN width for the leading ``first_k_dense_replace`` layers. + intermediate_size: int = 768 + #: Leading layers that run a dense FFN instead of the mixture (GLM + #: ``first_k_dense_replace``). Those layers carry no router and no experts. + first_k_dense_replace: int = 1 + #: Per-layer ``dense`` | ``sparse`` when the checkpoint declares it; overrides + #: :attr:`first_k_dense_replace` if both are given. + mlp_layer_types: tuple[str, ...] = () + #: Scales routed-expert outputs (GLM ``routed_scaling_factor``). Numerics only — + #: no effect on the FLOP/byte roofline, carried for completeness. + routed_scaling_factor: float = 1.0 + + # ── multi-token prediction ─────────────────────────────────────────────── + num_nextn_predict_layers: int = 0 + #: The MTP layer reuses the main model's index rather than recomputing it + #: (GLM ``index_share_for_mtp_iteration``). The tensor exists but the iteration + #: shares — carried as a headroom lever, not banked into the floor. + index_share_for_mtp_iteration: bool = True + + # ── precision (bf16 checkpoint; quantisation is a serving decision) ─────── + weight_dtype: str = "bf16" + expert_dtype: str = "bf16" + kv_dtype: str = "bf16" + act_dtype: str = "bf16" + + # ── derived shapes / schedule ──────────────────────────────────────────── + + @property + def q_head_dim(self) -> int: + """Per-head query/key width: the nope part plus the RoPE part.""" + return self.qk_nope_head_dim + self.qk_rope_head_dim + + @property + def kv_entry_dim(self) -> int: + """Elements one cached KV entry occupies: the latent plus the shared RoPE key. + + One per token per layer, shared across every query head — the whole point + of MLA. Not multiplied by ``n_heads`` or ``num_kv_heads``. + """ + return self.kv_lora_rank + self.qk_rope_head_dim + + def mlp_kind(self, layer: int) -> str: + """``"dense"`` | ``"sparse"`` for ``layer``'s FFN.""" + if self.mlp_layer_types and layer < len(self.mlp_layer_types): + return self.mlp_layer_types[layer] + return DENSE_MLP if layer < self.first_k_dense_replace else SPARSE_MLP + + def is_sparse_mlp(self, layer: int) -> bool: + return self.mlp_kind(layer) == SPARSE_MLP + + def indexer_kind(self, layer: int) -> str: + """``"full"`` | ``"shared"`` — whether ``layer`` computes its own index. + + Prefers the checkpoint's explicit ``indexer_types``. The frequency fallback + makes the first layer of each ``index_topk_freq``-sized group ``full`` and + the rest ``shared``, which is how GLM-5.2 is laid out past its dense prefix. + """ + if self.indexer_types and layer < len(self.indexer_types): + return self.indexer_types[layer] + n = max(1, self.index_topk_freq) + return FULL_INDEXER if layer % n == 0 else SHARED_INDEXER + + def is_full_indexer(self, layer: int) -> bool: + return self.indexer_kind(layer) == FULL_INDEXER + + @property + def n_full_indexer_layers(self) -> int: + return sum(1 for i in range(self.n_layers) if self.is_full_indexer(i)) + + @property + def n_sparse_mlp_layers(self) -> int: + return sum(1 for i in range(self.n_layers) if self.is_sparse_mlp(i)) + + @property + def top_k(self) -> int: + return min(self.num_experts_per_tok, self.n_routed_experts) + + +def kv_entry_bytes(spec: GlmMoeDsaModelSpec) -> float: + """Bytes one cached KV entry occupies. + + The latent plus the decoupled RoPE key. DeepSeek-family checkpoints keep the + RoPE dimensions in bf16 while the latent may be quantised; on a pure-bf16 + checkpoint both are 2 bytes and this reduces to ``kv_entry_dim * 2``. The split + is kept so an fp8-KV *serving* config prices the two halves correctly. + """ + rope = spec.qk_rope_head_dim * weight_bytes("bf16") + latent = spec.kv_lora_rank * weight_bytes(spec.kv_dtype) + return latent + rope + + +def effective_kv_tokens(spec: GlmMoeDsaModelSpec, kv_len: int) -> int: + """KV positions the attention *core* reads — bounded by the indexer's top-k. + + Every GLM layer runs DSA, so the core reads at most ``index_topk`` selected + positions regardless of how long the context grows. This is why attention is + flat in context on this architecture and the term that still grows is the + indexer scan, not the core. + """ + if kv_len <= 0: + return 0 + return min(kv_len, spec.index_topk) + + +def index_candidates(spec: GlmMoeDsaModelSpec, kv_len: int) -> int: + """Positions the indexer must score — the whole (uncompressed) history. + + GLM does not compress before selecting, so the indexer scores every past + token. This is the only term in the attention path that grows with context, + and it is paid on ``full`` layers alone. + """ + return max(0, kv_len) + + +def model_weight_bytes( + spec: GlmMoeDsaModelSpec, sharding: ShardingConfig | None = None +) -> float: + """Resident weight bytes on one rank. + + Decides whether a deployment shape fits at all, which the timing graph cannot. + Experts dominate overwhelmingly: 75 sparse layers x 256 experts x three + ``hidden x moe_intermediate`` matrices is the great majority of the checkpoint, + and it is what makes a ~753B model activate ~25B per token. + + Validated against ground truth: GLM-5.2 predicts within a few percent of the + published 1.507 TB checkpoint (``model.safetensors.index.json`` ``total_size``), + the residual being norms, biases and the MTP head this rolls in coarsely. + """ + sh = sharding or ShardingConfig() + tp = max(1, sh.tp) + es = max(1, sh.expert_shards) + ew = weight_bytes(spec.expert_dtype) + ww = weight_bytes(spec.weight_dtype) + h = spec.hidden + inter = spec.moe_intermediate_size + + n_sparse = spec.n_sparse_mlp_layers + spec.num_nextn_predict_layers + n_dense = spec.n_layers - spec.n_sparse_mlp_layers + n_full_idx = spec.n_full_indexer_layers + spec.num_nextn_predict_layers + n_attn = spec.n_layers + spec.num_nextn_predict_layers + + experts = n_sparse * spec.n_routed_experts * 3 * h * inter * ew / es + shared_exp = n_sparse * spec.n_shared_experts * 3 * h * inter * ew / tp + router = n_sparse * h * spec.n_routed_experts * ww # replicated + dense_ffn = n_dense * 3 * h * spec.intermediate_size * ww / tp + + # MLA projections, per attention layer. q_a and kv_a are replicated (they + # produce the shared latent, which has nothing to split); q_b, o_proj and the + # per-head value up-projection shard on heads. + attn_per_layer = ( + h * spec.q_lora_rank # q_a, replicated + + spec.q_lora_rank * spec.n_heads * spec.q_head_dim / tp # q_b + + h * spec.kv_entry_dim # kv_a (latent + rope key), replicated + + spec.kv_lora_rank * spec.n_heads * (spec.qk_nope_head_dim + spec.v_head_dim) / tp # kv_b + + spec.n_heads * spec.v_head_dim * h / tp # o_proj + ) + # Indexer weights live only on ``full`` layers (proven: ``shared`` layers carry + # none). Replicated across ranks, as vLLM builds the indexer ReplicatedLinear. + indexer_per_full = spec.q_lora_rank * spec.index_n_heads * spec.index_head_dim + + embed = 2.0 * spec.vocab * h / tp # untied input embedding + lm_head + return ( + experts + + shared_exp + + router + + (n_attn * attn_per_layer + n_full_idx * indexer_per_full + dense_ffn + embed) + * ww + ) + + +def kv_bytes_per_token(spec: GlmMoeDsaModelSpec) -> float: + """KV bytes each additional token of context costs, across the whole model. + + One MLA latent (plus the indexer key on ``full`` layers) per token per layer. + Flat across layers — there is no compression schedule to sum over — but the + indexer key is only cached where an indexer runs. + """ + kw = weight_bytes(spec.kv_dtype) + latent = spec.n_layers * kv_entry_bytes(spec) + index_keys = spec.n_full_indexer_layers * spec.index_head_dim * kw + return latent + index_keys + + +def _linear(rows: float, k: int, n: int, act_b: float, w_b: float) -> tuple[float, float]: + """(flops, bytes) for a ``(rows, k) @ (k, n)`` projection. + + Bytes count the activation in, the weights, and the activation out. At decode + ``rows`` is small and the weight term dominates — which is why weight dtype, + not activation dtype, sets the floor for every projection here. + """ + return 2.0 * rows * k * n, act_b * rows * k + w_b * k * n + act_b * rows * n + + +def _emit_layer( + g: Graph, + spec: GlmMoeDsaModelSpec, + hw: HardwareSpec, + layer: int, + *, + positions: float, + sequences: int, + kv_len: int, + sh: ShardingConfig, + prefix: str = "", + force_full_indexer: bool | None = None, +) -> None: + """Append one transformer layer's predicted nodes to ``g``. + + ``positions`` is how many sequence positions this layer computes; ``sequences`` + is how many distinct KV caches those positions read from. They differ under + speculative decoding, and the difference is why MTP helps a memory-bound + decode: eight draft positions verified in one step read the KV cache *once*, so + cache traffic amortises while FLOPs do not. + """ + h = spec.hidden + aw = weight_bytes(spec.act_dtype) + ww = weight_bytes(spec.weight_dtype) + ew = weight_bytes(spec.expert_dtype) + wd, ed = spec.weight_dtype, spec.expert_dtype + tp = max(1, sh.tp) + es = max(1, sh.expert_shards) + + def add( + op: str, flops: float, byts: float, dtype: str, + *, estimated: bool = False, serial_launches: int = 0, + ) -> None: + name = f"{prefix}{op}" + g.nodes.append( + PredictedNode( + name, layer, + roofline( + name, flops, byts, hw, dtype, + estimated=estimated, serial_launches=serial_launches, + ), + ) + ) + + # ── MLA attention: low-rank query, compressed KV latent ────────────────── + # q_a and kv_a are replicated across TP ranks: they produce the shared latent, + # which has nothing to split when there is one KV latent. Every rank pays them + # in full, so TP's speedup on attention is strictly less than ``tp``. + f, b = _linear(positions, h, spec.q_lora_rank, aw, ww) + add("attn_q_a", f, b, wd) + + f, b = _linear(positions, spec.q_lora_rank, spec.n_heads * spec.q_head_dim // tp, aw, ww) + add("attn_q_b", f, b, wd) + + # The compressed latent plus the decoupled RoPE key, and the cache write for + # the positions just computed. One projection (no CSA/HCA overlap here). + f, b = _linear(positions, h, spec.kv_entry_dim, aw, ww) + add("attn_kv_a", f, b + positions * kv_entry_bytes(spec), wd) + + # KV up-projection: reconstruct per-head K_nope and V from the cached latent + # (W^UK, W^UV). Modelled *unabsorbed* — it runs as its own GEMM and the output + # projection stays narrow (n_heads * v_head_dim). A serving engine that absorbs + # MLA folds W^UK into the query and W^UV into attn_out_proj instead, dropping + # this node and doubling attn_out_proj's input width; that is a serving-path + # variant, flagged in the catalogue provenance. Kept as a node because the + # weight physically exists in the checkpoint and model_weight_bytes counts it. + f, b = _linear( + positions, spec.kv_lora_rank, + spec.n_heads * (spec.qk_nope_head_dim + spec.v_head_dim) // tp, aw, ww, + ) + add("attn_kv_b", f, b, wd) + + # ── indexer: only ``full`` layers run one ──────────────────────────────── + # ``shared`` layers reuse the group's selection and carry no indexer weights, + # so they emit no indexer node. Emitting one would put a kernel in the graph + # that never ran and inflate the indexer's share ~4x. + runs_indexer = spec.is_full_indexer(layer) if force_full_indexer is None else force_full_indexer + cand = index_candidates(spec, kv_len) + if runs_indexer and cand > 0: + # The indexer query comes off the same latent as the attention query, so + # only the up-projection is charged. Not divided by ``tp`` — vLLM builds + # the indexer as ReplicatedLinear, so every rank runs the whole thing. + f, b = _linear( + positions, spec.q_lora_rank, spec.index_n_heads * spec.index_head_dim, aw, ww + ) + add("attn_index_proj", f, b, wd) + + add( + "attn_index_score", + 2.0 * positions * spec.index_n_heads * spec.index_head_dim * cand, + # Index keys live in the cache: read once per sequence, not per + # position, and replicated across ranks alongside the KV latent. + sequences * cand * spec.index_head_dim * weight_bytes(spec.kv_dtype), + spec.kv_dtype, + ) + + # ── attention core over the selected positions ────────────────────────── + t_eff = effective_kv_tokens(spec, kv_len) + heads = max(1, spec.n_heads // tp) + qk = 2.0 * positions * heads * spec.q_head_dim * t_eff + pv = 2.0 * positions * heads * spec.v_head_dim * t_eff + add( + "attn_score_value", + qk + pv, + # One latent, shared by every query head, so *not* multiplied by n_heads. + # Deliberately not divided by ``tp`` either: a single shared latent cannot + # be split, so the cache is replicated and every rank reads all of it — + # tensor parallelism buys no KV bandwidth on this architecture. + sequences * t_eff * kv_entry_bytes(spec), + wd, + ) + + # RMSNorm on every query head and the single KV latent, plus partial RoPE on + # the last ``qk_rope_head_dim`` dims and the cache insert — one fused kernel. + normed = positions * (heads * spec.q_head_dim + spec.kv_entry_dim) + roped = positions * (heads + 1) * spec.qk_rope_head_dim + add( + "attn_qnorm_rope_insert", + 3.0 * normed + 6.0 * roped, + 2.0 * normed * aw + 2.0 * roped * aw, + spec.act_dtype, + ) + + # Output projection: plain dense from the per-head value space back to hidden. + # No o_lora/o_groups here (unlike V4), so no ``estimated`` grouping guess. + f, b = _linear(positions, spec.n_heads * spec.v_head_dim // tp, h, aw, ww) + add("attn_out_proj", f, b, wd) + + # ── FFN: dense on the leading layers, mixture on the rest ──────────────── + if not spec.is_sparse_mlp(layer): + # Dense FFN (first_k_dense_replace). gate+up then down over the wide + # intermediate. Canonical dense-graph names so residuals stay comparable. + inter = spec.intermediate_size + f_gu, b_gu = _linear(positions, h, 2 * inter // tp, aw, ww) + add("mlp_gate_up", f_gu, b_gu, wd) + f_d, b_d = _linear(positions, inter // tp, h, aw, ww) + add("mlp_down", f_d, b_d, wd) + return + + # Router is replicated: every rank scores every expert to know what to keep. + f, b = _linear(positions, h, spec.n_routed_experts, aw, ww) + add("moe_router", f, b, wd) + + inter = spec.moe_intermediate_size + per_expert_weights = 3.0 * h * inter + per_position_flops = 6.0 * h * inter # 2 * (gate + up + down) * h * inter + + if spec.n_shared_experts > 0: + add( + "moe_shared", + per_position_flops * positions * spec.n_shared_experts / tp, + per_expert_weights * spec.n_shared_experts * ew / tp + + aw * (positions * h * 2 + positions * inter * 2 * spec.n_shared_experts / tp), + ed, + ) + + # The saturating set-union: FLOPs scale with positions x top_k, weight traffic + # with how many *distinct* experts the batch woke — shared with the dense-MoE + # roofline so there is one owner for the term. + distinct = distinct_experts( + int(positions), spec.n_routed_experts, spec.num_experts_per_tok + ) + skew = sh.ep_imbalance if sh.ep > 1 else 1.0 + add( + "moe_routed", + per_position_flops * positions * spec.num_experts_per_tok * skew / es, + per_expert_weights * distinct * ew * skew / es + + aw * (positions * h * 2 + positions * inter * 2 * spec.num_experts_per_tok / es), + ed, + ) + + # ── cross-rank traffic ────────────────────────────────────────────────── + if tp > 1 or sh.ep > 1: + link = replace(hw, peak_mem_bw_bytes_per_s=hw.interconnect_bw_bytes_per_s) + + def add_link(op: str, byts: float) -> None: + name = f"{prefix}{op}" + g.nodes.append( + PredictedNode( + name, layer, + roofline(name, 0.0, byts, link, spec.act_dtype, estimated=True), + ) + ) + + if sh.ep > 1: + off_rank = (sh.ep - 1) / sh.ep + add_link( + "moe_all_to_all", + 2.0 * positions * spec.num_experts_per_tok * h * aw * off_rank, + ) + if tp > 1: + add_link("tp_all_reduce", 2.0 * (2.0 * (tp - 1) / tp) * positions * h * aw) + + +def predict_glm_graph( + model: GlmMoeDsaModelSpec | None = None, + hw: HardwareSpec | None = None, + batch: BatchConfig | None = None, + sharding: ShardingConfig | None = None, +) -> Graph: + """Emit a predicted execution graph for one GLM-5.2-class decode step, per rank. + + The main stack runs over every position in the step (the verified token plus + any speculative drafts); the MTP head then runs over one position per sequence + to propose the next draft. With ``sharding`` left at its default the graph is + whole-model; given a real sharding it predicts what *one rank* does. + """ + spec = model or GlmMoeDsaModelSpec() + hw = hw or HardwareSpec() + batch = batch or BatchConfig() + sh = sharding or ShardingConfig() + + # Refuse a sharding the model cannot take. Head counts floor-divide throughout, + # so ``tp > n_heads`` silently prices the whole attention path at zero work. + if spec.n_heads % max(1, sh.tp) != 0: + raise ValueError( + f"tensor-parallel size {sh.tp} does not divide {spec.n_heads} attention " + "heads — every head-sharded op would floor to zero work" + ) + if spec.qk_rope_head_dim > spec.q_head_dim: + raise ValueError( + f"qk_rope_head_dim ({spec.qk_rope_head_dim}) exceeds the query head width " + f"({spec.q_head_dim}) — the RoPE slice cannot exceed the head it slices" + ) + if spec.n_layers <= 0: + raise ValueError("n_layers must be positive — an empty model predicts nothing") + + g = Graph(model=spec, hw=hw, batch=batch, sharding=sh) # type: ignore[arg-type] + positions = batch.positions_per_step + sequences = batch.batch + kv_len = batch.kv_cache_len + + for layer in range(spec.n_layers): + _emit_layer( + g, spec, hw, layer, + positions=positions, sequences=sequences, kv_len=kv_len, sh=sh, + ) + + # Multi-token prediction head. ``index_share_for_mtp_iteration`` says the MTP + # iteration reuses the main model's index rather than recomputing it, so the + # draft layer is emitted as a ``shared`` (no indexer) block — the tensor exists + # but the iteration shares it, and banking the projection+scan into the floor + # would over-predict a node the runtime skips. + for i in range(spec.num_nextn_predict_layers): + _emit_layer( + g, spec, hw, spec.n_layers + i, + positions=sequences, sequences=sequences, kv_len=kv_len, sh=sh, + force_full_indexer=not spec.index_share_for_mtp_iteration, + ) + + aw = weight_bytes(spec.act_dtype) + ww = weight_bytes(spec.weight_dtype) + f, b = _linear(positions, spec.hidden, spec.vocab // max(1, sh.tp), aw, ww) + g.nodes.append( + PredictedNode("lm_head", None, roofline("lm_head", f, b, hw, spec.weight_dtype)) + ) + return g + + +def is_glm_moe_dsa_config(cfg: dict[str, Any]) -> bool: + """True for the GLM-5.2-class checkpoints this module models. + + The discriminator is unambiguous and cheap: the checkpoint declares + ``model_type == "glm_moe_dsa"`` (equivalently ``GlmMoeDsaForCausalLM`` in + ``architectures``). Both this family and DeepSeek-V4 carry ``index_topk`` and + ``n_routed_experts``, so a structural test would collide — the model_type is + the clean separator, and this check must run *before* ``is_sparse_moe_config`` + in :func:`gitm.planner.registry.detect_family`. + """ + if str(cfg.get("model_type", "")).lower() == "glm_moe_dsa": + return True + archs = cfg.get("architectures") or [] + return any("glmmoedsa" in str(a).lower() for a in archs) + + +def spec_from_hf_config( + cfg: dict[str, Any], *, name: str | None = None +) -> GlmMoeDsaModelSpec: + """Build a :class:`GlmMoeDsaModelSpec` from a HuggingFace ``config.json``. + + Reads the checkpoint's declared shape so the graph cannot drift from the model. + The ``indexer_types`` and ``mlp_layer_types`` arrays are read verbatim — they + are the IndexShare and dense/sparse schedules, and a rule guessed from the + frequencies would misplace layers while producing a plausible total. + """ + + def _int(key: str, default: int) -> int: + try: + return int(cfg.get(key, default)) + except (TypeError, ValueError): + return default + + def _req(key: str) -> int: + v = cfg.get(key) + if v is None: + raise ValueError( + f"config declares no {key!r}; refusing to substitute a default, " + "which would predict a model this checkpoint is not" + ) + try: + return int(v) + except (TypeError, ValueError): + raise ValueError(f"config field {key!r} is not an integer: {v!r}") from None + + n_layers = _req("num_hidden_layers") + + def _types(key: str) -> tuple[str, ...]: + v = cfg.get(key) + if not isinstance(v, list | tuple): + return () + # Validate the *raw* length before any slicing: GLM-5.2's schedules are + # exactly num_hidden_layers long, and an over-long array is a sign the + # config is not what the caller thinks — worth an error, not a silent + # truncation that leaves the trailing layers on the last entry's kind. + if len(v) != n_layers: + raise ValueError( + f"{key} has {len(v)} entries for {n_layers} layers — the " + "schedule must cover the model exactly" + ) + return tuple(str(t) for t in v) + + dtype = str(cfg.get("dtype") or cfg.get("torch_dtype") or "bf16").lower() + if dtype.startswith("bfloat"): + dtype = "bf16" + elif dtype.startswith("float16") or dtype == "half": + dtype = "fp16" + elif dtype.startswith("float32"): + dtype = "fp32" + + # Quantisation, if the checkpoint declares any. Absent on the base GLM-5.2 + # release — an all-bf16 read matches the on-disk size — so this defaults to the + # model dtype rather than to fp4/fp8, which would manufacture headroom. + q = cfg.get("quantization_config") or {} + weight_dtype = str(q.get("quant_method", dtype)).lower() if q else dtype + expert_dtype = str(cfg.get("expert_dtype", weight_dtype)).lower() + + return GlmMoeDsaModelSpec( + name=name or str(cfg.get("_name_or_path") or cfg.get("model_type") or "glm-moe-dsa"), + hidden=_req("hidden_size"), + n_layers=n_layers, + vocab=_req("vocab_size"), + n_heads=_req("num_attention_heads"), + q_lora_rank=_req("q_lora_rank"), + kv_lora_rank=_req("kv_lora_rank"), + qk_nope_head_dim=_int("qk_nope_head_dim", _int("head_dim", 128)), + qk_rope_head_dim=_int("qk_rope_head_dim", 64), + v_head_dim=_int("v_head_dim", _int("head_dim", 128)), + index_n_heads=_int("index_n_heads", 32), + index_head_dim=_int("index_head_dim", 128), + index_topk=_int("index_topk", 2048), + index_topk_freq=_int("index_topk_freq", 4), + indexer_types=_types("indexer_types"), + n_routed_experts=_req("n_routed_experts"), + n_shared_experts=_int("n_shared_experts", 1), + num_experts_per_tok=_req("num_experts_per_tok"), + moe_intermediate_size=_req("moe_intermediate_size"), + intermediate_size=_int("intermediate_size", 12288), + first_k_dense_replace=_int("first_k_dense_replace", 0), + mlp_layer_types=_types("mlp_layer_types"), + routed_scaling_factor=float(cfg.get("routed_scaling_factor", 1.0) or 1.0), + num_nextn_predict_layers=_int("num_nextn_predict_layers", 0), + index_share_for_mtp_iteration=bool(cfg.get("index_share_for_mtp_iteration", True)), + weight_dtype=weight_dtype, + expert_dtype=expert_dtype, + # The config declares no cache dtype; bf16 is the model fact. A served + # deployment may pick fp8 — that is a serving decision, set at deploy time. + kv_dtype=dtype, + act_dtype=dtype, + ) diff --git a/gitm/planner/model_catalogue.py b/gitm/planner/model_catalogue.py index d414397..b3cf7e7 100644 --- a/gitm/planner/model_catalogue.py +++ b/gitm/planner/model_catalogue.py @@ -9,7 +9,7 @@ CATALOGUE_DIR = Path(__file__).resolve().parent / "models" #: Families a catalogue entry may declare, and the spec each one builds. -_FAMILIES = ("hybrid", "sparse_moe") +_FAMILIES = ("hybrid", "sparse_moe", "glm_moe_dsa") def available() -> list[str]: @@ -102,6 +102,8 @@ def load_spec(name_or_path: str | Path): if family == "hybrid": from gitm.planner.hybrid_graph import HybridMoEModelSpec as cls + elif family == "glm_moe_dsa": + from gitm.planner.glm_graph import GlmMoeDsaModelSpec as cls # type: ignore[assignment] else: from gitm.planner.roofline import SparseMoEModelSpec as cls # type: ignore[assignment] @@ -110,7 +112,10 @@ def load_spec(name_or_path: str | Path): raw["layer_types"] = _expand_layer_types( raw["layer_types"], int(raw.get("n_layers", 0)) ) - for key in ("compress_ratios", "dspark_layer_ids"): + # Per-layer schedule lists that must reach the frozen dataclass as tuples. A + # list would make the spec unhashable; a dropped tuple-coercion here is how a + # schedule silently arrives as the wrong type. + for key in ("compress_ratios", "dspark_layer_ids", "indexer_types", "mlp_layer_types"): if key in raw and isinstance(raw[key], list): raw[key] = tuple(raw[key]) @@ -142,6 +147,11 @@ def predict( return predict_hybrid_graph(spec, hw, batch, sharding, **kwargs), family + if family == "glm_moe_dsa": + from gitm.planner.glm_graph import predict_glm_graph + + return predict_glm_graph(spec, hw, batch, sharding, **kwargs), family + from gitm.planner.moe_graph import predict_moe_graph return predict_moe_graph(spec, hw, batch, sharding, **kwargs), family diff --git a/gitm/planner/models/glm-5.2.yaml b/gitm/planner/models/glm-5.2.yaml new file mode 100644 index 0000000..b89e7c5 --- /dev/null +++ b/gitm/planner/models/glm-5.2.yaml @@ -0,0 +1,188 @@ +# zai-org/GLM-5.2 — GlmMoeDsaForCausalLM (MLA + DeepSeek Sparse Attention + MoE). +# Source: https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json +# Read from config.json and model.safetensors.index.json only — no traces. + +name: zai-org/GLM-5.2 +family: glm_moe_dsa +description: > + 78 layers, hidden 6144, ~754B total / ~40B active per token. MLA attention + (kv_lora_rank 512, one latent shared across 64 query heads) with a DeepSeek + Sparse Attention lightning indexer selecting top-2048. IndexShare: only 21 of + 78 layers compute the index; the other 57 reuse a neighbour's selection and + carry no indexer weights. First 3 layers dense, remaining 75 MoE (256 experts, + top-8, one shared). One MTP draft head. bf16 throughout. + +spec: + hidden: 6144 + n_layers: 78 + vocab: 154880 + + # MLA attention. head_dim (192) is the nope width; q_head_dim widens to + # 192+64=256, and v_head_dim (256) differs from it — score and value reads use + # different per-head widths. num_key_value_heads is 64 in the config but is a + # red herring: the cache holds one kv_lora_rank latent per token, shared across + # all 64 query heads, so KV traffic derives from kv_lora_rank, never 64*head_dim. + n_heads: 64 + q_lora_rank: 2048 + kv_lora_rank: 512 + qk_nope_head_dim: 192 + qk_rope_head_dim: 64 + v_head_dim: 256 + + # DeepSeek Sparse Attention indexer. index_topk_freq 4 is the IndexShare period: + # one 'full' layer computes the top-2048 selection, the next three reuse it. + index_n_heads: 32 + index_head_dim: 128 + index_topk: 2048 + index_topk_freq: 4 + # Read verbatim from the checkpoint (period-4 past the dense prefix). The 'shared' + # layers physically carry no indexer weights — this schedule is proven, not fitted. + indexer_types: + - full + - full + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + + # Mixture of experts. moe_intermediate_size 2048 per expert; the first 3 layers + # are dense FFN (intermediate_size 12288) with no router or experts. + n_routed_experts: 256 + n_shared_experts: 1 + num_experts_per_tok: 8 + moe_intermediate_size: 2048 + intermediate_size: 12288 + first_k_dense_replace: 3 + routed_scaling_factor: 2.5 + + # Multi-token prediction. index_share_for_mtp_iteration: the draft reuses the + # main model's index rather than recomputing it. + num_nextn_predict_layers: 1 + index_share_for_mtp_iteration: true + + # bf16 checkpoint — no quantization_config published. fp8/fp4 is a serving + # decision, applied at deploy time, not a model fact. + weight_dtype: bf16 + expert_dtype: bf16 + kv_dtype: bf16 + act_dtype: bf16 + +provenance: + verified: + - claim: IndexShare is read from the weight map, not inferred + detail: > + model.safetensors.index.json carries indexer tensors (*.indexer.*) on + exactly the 21 layers whose indexer_types is 'full' (0,1,2,6,10,...,74) + and on none of the 57 'shared' layers. A shared layer that recomputed the + index would need those weights; it does not have them. + - claim: dense/sparse MLP split matches the tensor map + detail: > + Layers 0-2 carry no expert tensors (dense FFN); layers 3-77 and the MTP + layer carry experts.* and shared_experts.*. Matches first_k_dense_replace 3. + - claim: predicted weight bytes match the published checkpoint + detail: > + model.safetensors.index.json total_size is 1,506,659,919,872 B (1.507 TB) + across 282 shards. model_weight_bytes at bf16 lands within a few percent; + the residual is norms/biases and the coarse MTP roll-in. + + estimated: + - field: index_share_for_mtp_iteration handling + value: MTP emitted as a shared (no-indexer) block + detail: > + The MTP layer carries indexer weights in the map, but the config sets + index_share_for_mtp_iteration=true, so the iteration reuses the main + model's index. Emitting the draft as a full-indexer block would over- + predict a scan the runtime skips; the alternative reading is a headroom + lever to check once a trace exists. + - field: MLA absorption (attn_kv_b vs attn_out_proj width) + value: unabsorbed — kv_b runs as its own GEMM, o_proj narrow + detail: > + The graph models MLA unabsorbed: attn_kv_b reconstructs per-head K/V from + the latent and attn_out_proj is n_heads*v_head_dim -> hidden (16384->6144). + A serving engine that absorbs MLA (the common vLLM decode path) folds W^UK + into the query and W^UV into the output, dropping attn_kv_b and doubling + attn_out_proj's input width to n_heads*kv_lora_rank (32768->6144). Same + resident weights either way (validated footprint holds), but the two + readings move attn_out_proj by 2x — which is exactly headroom item #3 to + settle against a capture: does the engine run absorbed MLA? + + unmodelled: + - Prefill. Decode only, as with every other graph family. Chunked prefill runs + a different indexer path (score-and-select over the whole prompt) with + different asymptotics. + - index_topk_freq as a *temporal* reuse across decode steps. Modelled here as + the spatial layer-group period (which the weight map proves); any additional + step-to-step reuse would only reduce indexer cost further. + - Serving quantisation (fp8 KV cache, fp8/fp4 experts). Priced at bf16 — the + published checkpoint's precision. A quantised deploy is a separate spec. diff --git a/gitm/planner/registry.py b/gitm/planner/registry.py index df10472..ef3c9c6 100644 --- a/gitm/planner/registry.py +++ b/gitm/planner/registry.py @@ -10,12 +10,20 @@ def detect_family(cfg: dict[str, Any]) -> str: - """``"hybrid"`` | ``"sparse_moe"`` | ``"dense"`` for a HuggingFace config.""" + """``"hybrid"`` | ``"glm_moe_dsa"`` | ``"sparse_moe"`` | ``"dense"`` for a config.""" + from gitm.planner.glm_graph import is_glm_moe_dsa_config from gitm.planner.hybrid_graph import is_hybrid_moe_config from gitm.planner.moe_graph import is_sparse_moe_config + # The hybrid guard reads ``num_experts``; GLM and V4 both spell it + # ``n_routed_experts``, so they fall through it. GLM must be tested *before* + # sparse_moe: both carry ``index_topk`` + ``n_routed_experts``, so the + # structural sparse-MoE test would claim GLM first — the model_type check is + # the clean separator and has to win. if is_hybrid_moe_config(cfg): return "hybrid" + if is_glm_moe_dsa_config(cfg): + return "glm_moe_dsa" if is_sparse_moe_config(cfg): return "sparse_moe" return "dense" @@ -28,6 +36,10 @@ def spec_from_hf_config(cfg: dict[str, Any], *, name: str | None = None): from gitm.planner.hybrid_graph import spec_from_hf_config as _hybrid return _hybrid(cfg, name=name) + if family == "glm_moe_dsa": + from gitm.planner.glm_graph import spec_from_hf_config as _glm + + return _glm(cfg, name=name) if family == "sparse_moe": from gitm.planner.moe_graph import spec_from_hf_config as _sparse @@ -61,15 +73,21 @@ def predict_for_config( from gitm.planner.hybrid_graph import spec_from_hf_config as _hybrid return predict_hybrid_graph(_hybrid(cfg, name=name), hw, batch, sharding), family + if family == "glm_moe_dsa": + from gitm.planner.glm_graph import predict_glm_graph + from gitm.planner.glm_graph import spec_from_hf_config as _glm + + return predict_glm_graph(_glm(cfg, name=name), hw, batch, sharding), family if family == "sparse_moe": from gitm.planner.moe_graph import predict_moe_graph from gitm.planner.moe_graph import spec_from_hf_config as _sparse return predict_moe_graph(_sparse(cfg, name=name), hw, batch, sharding), family raise NotImplementedError( - f"{name or 'this checkpoint'} is neither a hybrid linear-attention MoE nor a " - "DeepSeek-V4-class sparse-MoE checkpoint. The dense graph models it, but has " - "no config reader — construct a ModelSpec and call predict_graph directly." + f"{name or 'this checkpoint'} is neither a hybrid linear-attention MoE, a " + "GLM-5.2-class glm_moe_dsa, nor a DeepSeek-V4-class sparse-MoE checkpoint. The " + "dense graph models it, but has no config reader — construct a ModelSpec and " + "call predict_graph directly." ) @@ -160,6 +178,10 @@ def _predict(spec, family: str, hw, batch, sharding): from gitm.planner.hybrid_graph import predict_hybrid_graph return predict_hybrid_graph(spec, hw, batch, sharding) + if family == "glm_moe_dsa": + from gitm.planner.glm_graph import predict_glm_graph + + return predict_glm_graph(spec, hw, batch, sharding) from gitm.planner.moe_graph import predict_moe_graph return predict_moe_graph(spec, hw, batch, sharding) diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py new file mode 100644 index 0000000..3addfe2 --- /dev/null +++ b/tests/test_glm_graph.py @@ -0,0 +1,182 @@ +"""The GLM-5.2 (``glm_moe_dsa``) decode graph, pinned against the checkpoint. + +Every assertion guards a term that separates GLM-5.2 from the DeepSeek-V4 sparse +family it was forked from, or a wiring seam a plausible-but-wrong graph would slip +through: + +* IndexShare — only ``full`` layers emit indexer nodes; ``shared`` layers reuse + the selection and carry no indexer weights, +* MLA KV traffic scales with the shared latent, never ``n_heads``, +* the dense prefix runs an FFN, not a mixture, +* precision is bf16 (no fp4 experts leaking in from the V4 defaults), +* the predicted footprint matches the published 1.507 TB checkpoint, +* ``detect_family`` routes ``glm_moe_dsa`` before the structural sparse-MoE test. +""" + +from __future__ import annotations + +import pytest + +from gitm.planner.glm_graph import ( + GlmMoeDsaModelSpec, + is_glm_moe_dsa_config, + kv_entry_bytes, + model_weight_bytes, + predict_glm_graph, + spec_from_hf_config, +) +from gitm.planner.model_catalogue import available, load_spec, predict +from gitm.planner.registry import detect_family +from gitm.planner.roofline import BatchConfig, ShardingConfig + +# GLM-5.2's shape, trimmed to the keys the planner reads. Arrays are the real +# checkpoint's schedules (period-4 IndexShare past a 3-layer dense prefix). +GLM_CONFIG = { + "model_type": "glm_moe_dsa", + "architectures": ["GlmMoeDsaForCausalLM"], + "hidden_size": 6144, + "num_hidden_layers": 78, + "num_attention_heads": 64, + "num_key_value_heads": 64, + "head_dim": 192, + "q_lora_rank": 2048, + "kv_lora_rank": 512, + "qk_nope_head_dim": 192, + "qk_rope_head_dim": 64, + "v_head_dim": 256, + "index_n_heads": 32, + "index_head_dim": 128, + "index_topk": 2048, + "index_topk_freq": 4, + "indexer_types": ( + ["full", "full", "full"] + + ["shared", "shared", "shared", "full"] * 18 + + ["shared", "shared", "shared"] + ), + "n_routed_experts": 256, + "n_shared_experts": 1, + "num_experts_per_tok": 8, + "moe_intermediate_size": 2048, + "intermediate_size": 12288, + "first_k_dense_replace": 3, + "mlp_layer_types": ["dense", "dense", "dense"] + ["sparse"] * 75, + "routed_scaling_factor": 2.5, + "num_nextn_predict_layers": 1, + "index_share_for_mtp_iteration": True, + "dtype": "bfloat16", + "vocab_size": 154880, +} + + +def _spec() -> GlmMoeDsaModelSpec: + return spec_from_hf_config(GLM_CONFIG, name="GLM-5.2") + + +def _ops(g, op: str) -> list: + return [n for n in g.nodes if n.op == op] + + +def test_config_reader_reads_schedules_verbatim(): + spec = _spec() + assert spec.n_layers == 78 + assert spec.n_full_indexer_layers == 21 # 3 dense-prefix + 18 period-4 + assert spec.n_sparse_mlp_layers == 75 + # The dense prefix is dense; a mid-stack layer is sparse. + assert not spec.is_sparse_mlp(0) and spec.is_sparse_mlp(40) + # Read verbatim, not derived: layer 2 is 'full' though 2 % 4 != 0. + assert spec.is_full_indexer(2) + assert not spec.is_full_indexer(3) + + +def test_indexshare_shared_layers_emit_no_indexer(): + """The mechanism the fork exists to price: 57 of 78 layers skip the indexer.""" + spec = _spec() + g = predict_glm_graph(spec, batch=BatchConfig(batch=1, kv_cache_len=4096)) + n_proj = len(_ops(g, "attn_index_proj")) + n_score = len(_ops(g, "attn_index_score")) + # One per full-indexer layer, and the MTP head shares (no indexer node). + assert n_proj == n_score == spec.n_full_indexer_layers == 21 + # Every layer still runs the attention core over the selected positions. + assert len(_ops(g, "attn_score_value")) == spec.n_layers + spec.num_nextn_predict_layers + + +def test_mla_kv_traffic_uses_shared_latent_not_heads(): + """The classic MLA error: charging KV as ``n_heads * head_dim``. + + The cache holds one latent per token, shared across all 64 query heads, so the + per-entry byte count is ``kv_lora_rank + qk_rope`` — independent of ``n_heads``. + """ + spec = _spec() + entry = kv_entry_bytes(spec) + assert entry == (512 + 64) * 2 # bf16 latent + bf16 rope key + # A per-head (GQA-style) K+V reading would be tens of times larger. + gqa_wrong = spec.n_heads * (spec.q_head_dim + spec.v_head_dim) * 2 + assert gqa_wrong > 25 * entry + + +def test_dense_prefix_runs_ffn_not_mixture(): + spec = _spec() + g = predict_glm_graph(spec, batch=BatchConfig(batch=1, kv_cache_len=4096)) + # Exactly the 3 dense layers carry an mlp_gate_up/down; the router runs on the + # 75 sparse layers plus the sparse MTP head. + assert len(_ops(g, "mlp_gate_up")) == 3 + assert len(_ops(g, "moe_router")) == spec.n_sparse_mlp_layers + spec.num_nextn_predict_layers + + +def test_precision_is_bf16_no_fp4_leak(): + """Forked from a fp4-expert default; a leak would deflate the dominant term.""" + spec = _spec() + assert spec.weight_dtype == "bf16" + assert spec.expert_dtype == "bf16" + assert spec.kv_dtype == "bf16" + + +def test_footprint_matches_published_checkpoint(): + """Predicted weight bytes within a few percent of the 1.507 TB on disk.""" + published = 1_506_659_919_872 # model.safetensors.index.json total_size + wb = model_weight_bytes(_spec()) + assert abs(wb / published - 1.0) < 0.03 + + +def test_attention_core_flat_indexer_scan_grows_with_context(): + """DSA: the core is bounded by top-k; only the indexer scan grows.""" + spec = _spec() + short = predict_glm_graph(spec, batch=BatchConfig(batch=1, kv_cache_len=4096)) + long = predict_glm_graph(spec, batch=BatchConfig(batch=1, kv_cache_len=131072)) + + def total(g, op): + return sum(n.prediction.t_pred_s for n in g.nodes if n.op == op) + + # Core read is capped at index_topk (2048) — unchanged from 4K to 128K. + assert total(long, "attn_score_value") == pytest.approx( + total(short, "attn_score_value"), rel=1e-9 + ) + # The scan scores the whole history, so it grows. + assert total(long, "attn_index_score") > 10 * total(short, "attn_index_score") + + +def test_detect_family_routes_glm_before_sparse_moe(): + """Both families carry index_topk + n_routed_experts; model_type must win.""" + assert detect_family(GLM_CONFIG) == "glm_moe_dsa" + assert is_glm_moe_dsa_config(GLM_CONFIG) + + +def test_catalogue_entry_loads_and_predicts(): + assert "glm-5.2" in available() + spec = load_spec("glm-5.2") + assert spec.n_layers == 78 and spec.n_full_indexer_layers == 21 + g, family = predict("glm-5.2", batch=BatchConfig(batch=1, kv_cache_len=4096)) + assert family == "glm_moe_dsa" + assert g.total_pred_s > 0 + + +def test_tp_must_divide_heads(): + spec = _spec() + with pytest.raises(ValueError, match="does not divide"): + predict_glm_graph(spec, sharding=ShardingConfig(tp=7)) + + +def test_missing_required_field_refuses_default(): + broken = {k: v for k, v in GLM_CONFIG.items() if k != "kv_lora_rank"} + with pytest.raises(ValueError, match="kv_lora_rank"): + spec_from_hf_config(broken) diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py index 1b33325..c922623 100644 --- a/tests/test_moe_graph.py +++ b/tests/test_moe_graph.py @@ -1154,6 +1154,10 @@ def test_defaults_do_not_match_any_catalogued_checkpoint(): """Stronger than a size bound: no field-by-field match with a real entry.""" d = HybridMoEModelSpec() for entry in available(): + # Only hybrid entries are HybridMoEModelSpecs; other families (glm_moe_dsa, + # sparse_moe) have their own reference-default tests and their own fields. + if load_entry(entry).get("family") != "hybrid": + continue hybrid_spec = load_spec(entry) assert (hybrid_spec.hidden, hybrid_spec.n_layers, hybrid_spec.num_experts) != ( d.hidden, d.n_layers, d.num_experts From 937e5b4ba8426e13ad662df844ea41d8bc23f3cd Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 13:02:39 -0700 Subject: [PATCH 02/35] wip: GLM-5.2 per-op precision, prefill phase, launch bound, MTP chain --- gitm/planner/glm_graph.py | 614 ++++++++++++++++++++++++++++++-------- 1 file changed, 497 insertions(+), 117 deletions(-) diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 6fd8761..e2e97cb 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -37,24 +37,47 @@ shared, ``moe_intermediate_size=2048``). Modelling the dense layers as MoE would invent a router GEMM and expert traffic that the weight map shows are not there. -**Precision is bf16 throughout.** The published checkpoint carries no -``quantization_config`` — the on-disk size (1.507 TB) matches an all-bf16 read. -Any fp8 the KV cache or experts pick up is a *serving* decision, not a model fact, -so — unlike the V4 reader, which defaults to fp4 experts — this graph prices bf16 -and records quantisation as a deployment lever in provenance. Defaulting to fp4 -here would deflate the dominant expert term ~3.8x and manufacture headroom that -the checkpoint does not have. +**Three precisions in one block, and which op runs in which is read from the +checkpoint.** ``zai-org/GLM-5.2`` is bf16 (1.507 TB on disk, no +``quantization_config``); ``zai-org/GLM-5.2-FP8`` is the vendor's *recommended* +deployment (753.33 GB, e4m3, 128x128 block-scaled) and its +``quantization_config.modules_to_not_convert`` names what stays wide. On the FP8 +checkpoint the backbone GEMMs — ``q_a``/``q_b``/``kv_a``/``kv_b``, **``o_proj``**, +the dense FFN, the shared expert and all 256 routed experts — are fp8, while +``lm_head``, ``embed_tokens``, the MTP ``eh_proj``, every norm and — the outlier +worth naming — the **lightning indexer's projections** are bf16. The router is +fp32 on both checkpoints (``moe_router_dtype: "float32"``, a field of the *base* +config, so it is a model fact and not a quantisation choice). + +That is three widths inside one attention block, which one ``weight_dtype`` per +spec cannot say. :attr:`GlmMoeDsaModelSpec.op_dtype_overrides` says it per op, and +it is load-bearing in both directions: pricing ``lm_head`` at 1 byte/weight +understates 154,880 x 6,144 of real traffic, and pricing the indexer at fp8 +understates the one node whose cost grows with context. Note the inversion against +the fp8-backbone models this planner has seen before — here ``o_proj`` is *inside* +the quantised set and the *indexer* is outside it. Known limits, stated rather than hidden: -* **Decode only.** As with every other graph family; the chunked-prefill indexer - path has different asymptotics. +* **Prefill is modelled, and it is not decode with a bigger M.** DSA makes the two + phases disagree about what ``index_topk`` buys: at decode the core reads at most + ``index_topk`` cached entries, so attention is flat in context; at prefill every + query in the chunk selects a *different* top-k, so their union is the whole + history and the core streams the entire cache once per request. Top-k bounds + prefill FLOPs, not prefill bytes. See :func:`core_qk_pairs` / + :func:`core_read_entries`. * **Uniform routing.** :func:`~gitm.planner.roofline.distinct_experts` assumes a balanced router; real skew touches fewer distinct experts, moving *less* traffic than predicted — the conservative direction. * **Expert-parallel skew is calibrated, not predicted.** :attr:`ShardingConfig.ep_imbalance` stays at 1.0 until a trace measures it. -* **Collectives are bandwidth-only** (no latency floor), flagged ``estimated``. +* **Collectives are bandwidth-plus-launch**: a ring latency floor of one launch + each, which is what a 262 kB decode all-reduce is actually bounded by. Still + flagged ``estimated``, and still reported unpriced when the SKU carries no + interconnect bandwidth. +* **The MTP economics are a prediction, not a measurement.** Acceptance rate is a + serving observable; the graph prices the *cost* of D drafts and a 1+D verify and + leaves the payoff to :attr:`BatchConfig.tokens_per_step`. """ from __future__ import annotations @@ -155,11 +178,23 @@ class GlmMoeDsaModelSpec: #: shares — carried as a headroom lever, not banked into the floor. index_share_for_mtp_iteration: bool = True - # ── precision (bf16 checkpoint; quantisation is a serving decision) ─────── + # ── precision ──────────────────────────────────────────────────────────── weight_dtype: str = "bf16" expert_dtype: str = "bf16" kv_dtype: str = "bf16" act_dtype: str = "bf16" + #: Per-op precision, for the ops that do not run in :attr:`weight_dtype`. + #: ``(op_name, dtype)`` pairs — a tuple rather than a mapping so the spec + #: stays hashable, matching how the per-layer schedules are carried. + #: + #: One dtype per model is a fiction on this checkpoint. GLM-5.2-FP8 quantises + #: the backbone GEMMs and the experts to e4m3 but leaves ``lm_head``, the MTP + #: ``eh_proj`` and the **lightning indexer** in bf16 + #: (``quantization_config.modules_to_not_convert``), and the router is fp32 on + #: every variant (``moe_router_dtype``). Without this the indexer — the one + #: attention node whose cost grows with context — is priced at half its real + #: weight traffic and against a peak it never sees. + op_dtype_overrides: tuple[tuple[str, str], ...] = () # ── derived shapes / schedule ──────────────────────────────────────────── @@ -177,6 +212,18 @@ def kv_entry_dim(self) -> int: """ return self.kv_lora_rank + self.qk_rope_head_dim + def dtype_for(self, op: str, default: str) -> str: + """Precision ``op`` runs in — an override if the checkpoint declares one. + + ``default`` is the op's family dtype (:attr:`weight_dtype` for + projections, :attr:`expert_dtype` for the mixture), so an entry is needed + only where the checkpoint departs from it. + """ + for name, dtype in self.op_dtype_overrides: + if name == op: + return dtype + return default + def mlp_kind(self, layer: int) -> str: """``"dense"`` | ``"sparse"`` for ``layer``'s FFN.""" if self.mlp_layer_types and layer < len(self.mlp_layer_types): @@ -250,6 +297,81 @@ def index_candidates(spec: GlmMoeDsaModelSpec, kv_len: int) -> int: return max(0, kv_len) +def _capped_prefix_sum(context: int, tokens: int, cap: int) -> float: + """``sum(min(context + i, cap) for i in 1..tokens)`` in closed form. + + How many positions a causal chunk of ``tokens`` queries actually attends to + when each query is capped at ``cap`` selected keys. Written out rather than + looped because a prefill chunk is 8,192 queries and this is called per layer. + """ + if tokens <= 0: + return 0.0 + c, n, k = float(context), float(tokens), float(cap) + # Queries whose own history is still under the cap attend to all of it. + uncapped = max(0.0, min(n, k - c)) + total = uncapped * (2.0 * c + uncapped + 1.0) / 2.0 + return total + (n - uncapped) * k + + +def core_qk_pairs(spec: GlmMoeDsaModelSpec, batch: BatchConfig) -> float: + """Query-key products the attention *core* evaluates this step. + + Deliberately **not** :attr:`BatchConfig.attention_qk_pairs`, which is the + dense causal count. DSA hands the core at most ``index_topk`` selected + positions per query, so past 2,048 tokens of context the core is linear in + context where a dense model is quadratic — the whole point of the + architecture, and the term a mechanical copy of another family's prefill path + would get wrong by ``kv_len / index_topk`` (64x at 128K). + """ + k = spec.index_topk + pairs = batch.positions_per_step * float(min(batch.kv_cache_len, k)) + if batch.is_prefill: + pairs += _capped_prefix_sum(batch.prefill_context, batch.prefill_tokens, k) + return pairs + + +def core_read_entries(spec: GlmMoeDsaModelSpec, batch: BatchConfig) -> float: + """Cached KV entries the core actually reads — the phases disagree here. + + **Decode:** one query row per sequence, reading its own top-``index_topk`` + selection. ``sequences x min(kv_len, index_topk)`` — flat in context. + + **Prefill:** every query in the chunk selects a *different* top-k, and their + union over a few thousand queries is the whole history. The kernel therefore + streams the entire cache once per request: top-k bounds prefill *FLOPs*, not + prefill *bytes*. Charging ``P x index_topk`` here instead would understate + long-context prefill traffic by the ratio of context to 2,048, which is the + single most inviting error on this architecture. + """ + entries = batch.batch * float(min(batch.kv_cache_len, spec.index_topk)) + if batch.is_prefill: + entries += batch.prefill_requests * float( + batch.prefill_context + batch.prefill_tokens + ) + return entries + + +def index_scan_pairs(batch: BatchConfig) -> float: + """Query-key products the *indexer* scores — the uncompressed history. + + GLM does not compress before selecting, so every past token is scored. This + is the dense causal count, and it is the term that carries the quadratic at + prefill and the growth-in-context at decode. Paid on ``full`` layers only, + which is what IndexShare is worth. + """ + return batch.attention_qk_pairs + + +def index_scan_entries(batch: BatchConfig) -> float: + """Cached index keys read per step: the whole history, once per sequence.""" + entries = float(batch.batch * batch.kv_cache_len) + if batch.is_prefill: + entries += batch.prefill_requests * float( + batch.prefill_context + batch.prefill_tokens + ) + return entries + + def model_weight_bytes( spec: GlmMoeDsaModelSpec, sharding: ShardingConfig | None = None ) -> float: @@ -267,8 +389,12 @@ def model_weight_bytes( sh = sharding or ShardingConfig() tp = max(1, sh.tp) es = max(1, sh.expert_shards) - ew = weight_bytes(spec.expert_dtype) + ew = weight_bytes(spec.dtype_for("moe_routed", spec.expert_dtype)) + sw = weight_bytes(spec.dtype_for("moe_shared", spec.expert_dtype)) ww = weight_bytes(spec.weight_dtype) + rw = weight_bytes(spec.dtype_for("moe_router", spec.weight_dtype)) + iw = weight_bytes(spec.dtype_for("attn_index_proj", spec.weight_dtype)) + lw = weight_bytes(spec.dtype_for("lm_head", spec.weight_dtype)) h = spec.hidden inter = spec.moe_intermediate_size @@ -278,8 +404,8 @@ def model_weight_bytes( n_attn = spec.n_layers + spec.num_nextn_predict_layers experts = n_sparse * spec.n_routed_experts * 3 * h * inter * ew / es - shared_exp = n_sparse * spec.n_shared_experts * 3 * h * inter * ew / tp - router = n_sparse * h * spec.n_routed_experts * ww # replicated + shared_exp = n_sparse * spec.n_shared_experts * 3 * h * inter * sw / tp + router = n_sparse * h * spec.n_routed_experts * rw # replicated dense_ffn = n_dense * 3 * h * spec.intermediate_size * ww / tp # MLA projections, per attention layer. q_a and kv_a are replicated (they @@ -294,15 +420,26 @@ def model_weight_bytes( ) # Indexer weights live only on ``full`` layers (proven: ``shared`` layers carry # none). Replicated across ranks, as vLLM builds the indexer ReplicatedLinear. - indexer_per_full = spec.q_lora_rank * spec.index_n_heads * spec.index_head_dim + # ``wq_b`` from the query latent, ``wk`` from hidden, and the per-head gate. + indexer_per_full = ( + spec.q_lora_rank * spec.index_n_heads * spec.index_head_dim + + h * spec.index_head_dim + + h * spec.index_n_heads + ) + + # Untied input embedding and vocabulary projection. Both stay wide on the FP8 + # checkpoint (``embed_tokens`` and ``lm_head`` are in ``modules_to_not_convert``), + # so they are priced at their own width rather than the backbone's — 1.9 GB of + # the resident footprint that an fp8 read would halve on paper and not on disk. + embed = 2.0 * spec.vocab * h * lw / tp - embed = 2.0 * spec.vocab * h / tp # untied input embedding + lm_head return ( experts + shared_exp + router - + (n_attn * attn_per_layer + n_full_idx * indexer_per_full + dense_ffn + embed) - * ww + + n_full_idx * indexer_per_full * iw + + embed + + (n_attn * attn_per_layer + dense_ffn) * ww ) @@ -335,58 +472,86 @@ def _emit_layer( hw: HardwareSpec, layer: int, *, - positions: float, - sequences: int, - kv_len: int, + batch: BatchConfig, sh: ShardingConfig, prefix: str = "", force_full_indexer: bool | None = None, ) -> None: """Append one transformer layer's predicted nodes to ``g``. - ``positions`` is how many sequence positions this layer computes; ``sequences`` - is how many distinct KV caches those positions read from. They differ under - speculative decoding, and the difference is why MTP helps a memory-bound - decode: eight draft positions verified in one step read the KV cache *once*, so - cache traffic amortises while FLOPs do not. + ``batch`` is the phase this layer runs in, already adjusted by the caller: a + decode step, a chunked-prefill step, or a draft stage (prefill stripped). The + node *set* is identical across all three — what changes is the class of four + of them, which is why phase is a parameter here rather than a second emitter. + + Three row counts, and conflating any two is a real error: + + ``rows`` + Sequence positions this layer computes — decode positions (one per + sequence per speculative slot) plus whatever prefill chunk rides along. + Every projection and every FFN scales with this. + ``batch.batch`` + Distinct KV caches those rows read from. Cache traffic is charged per + *sequence*, not per row, which is why verifying 1+D drafted positions in + one step reads the cache once — and is what makes MTP pay at all on a + memory-bound step. + ``batch.prefill_requests`` + Prompts the prefill tokens belong to — the denominator for anything read + once per request rather than once per token. """ h = spec.hidden aw = weight_bytes(spec.act_dtype) - ww = weight_bytes(spec.weight_dtype) - ew = weight_bytes(spec.expert_dtype) wd, ed = spec.weight_dtype, spec.expert_dtype tp = max(1, sh.tp) es = max(1, sh.expert_shards) + rows = float(batch.positions_per_step + batch.prefill_tokens) + def add( op: str, flops: float, byts: float, dtype: str, - *, estimated: bool = False, serial_launches: int = 0, + *, estimated: bool = False, serial_launches: int = 1, ) -> None: + """Emit one node, at the precision the checkpoint says the op runs in. + + ``serial_launches`` defaults to 1 because every node here *is* one + dependent kernel launch: it consumes the previous node's output, so its + wall time cannot fall below the launch overhead however few bytes it + moves. On a decode step that floor is what the small pointwise and + routing nodes are actually bounded by, and omitting it does not make the + prediction slightly optimistic — it makes a whole bound label absent. + """ name = f"{prefix}{op}" g.nodes.append( PredictedNode( name, layer, roofline( - name, flops, byts, hw, dtype, + name, flops, byts, hw, spec.dtype_for(op, dtype), estimated=estimated, serial_launches=serial_launches, ), ) ) + def w_bytes(op: str, default: str) -> float: + """Bytes per stored weight for ``op``, after any precision override.""" + return weight_bytes(spec.dtype_for(op, default)) + # ── MLA attention: low-rank query, compressed KV latent ────────────────── # q_a and kv_a are replicated across TP ranks: they produce the shared latent, # which has nothing to split when there is one KV latent. Every rank pays them # in full, so TP's speedup on attention is strictly less than ``tp``. - f, b = _linear(positions, h, spec.q_lora_rank, aw, ww) + f, b = _linear(rows, h, spec.q_lora_rank, aw, w_bytes("attn_q_a", wd)) add("attn_q_a", f, b, wd) - f, b = _linear(positions, spec.q_lora_rank, spec.n_heads * spec.q_head_dim // tp, aw, ww) + f, b = _linear( + rows, spec.q_lora_rank, spec.n_heads * spec.q_head_dim // tp, aw, + w_bytes("attn_q_b", wd), + ) add("attn_q_b", f, b, wd) # The compressed latent plus the decoupled RoPE key, and the cache write for # the positions just computed. One projection (no CSA/HCA overlap here). - f, b = _linear(positions, h, spec.kv_entry_dim, aw, ww) - add("attn_kv_a", f, b + positions * kv_entry_bytes(spec), wd) + f, b = _linear(rows, h, spec.kv_entry_dim, aw, w_bytes("attn_kv_a", wd)) + add("attn_kv_a", f, b + rows * kv_entry_bytes(spec), wd) # KV up-projection: reconstruct per-head K_nope and V from the cached latent # (W^UK, W^UV). Modelled *unabsorbed* — it runs as its own GEMM and the output @@ -396,8 +561,9 @@ def add( # variant, flagged in the catalogue provenance. Kept as a node because the # weight physically exists in the checkpoint and model_weight_bytes counts it. f, b = _linear( - positions, spec.kv_lora_rank, - spec.n_heads * (spec.qk_nope_head_dim + spec.v_head_dim) // tp, aw, ww, + rows, spec.kv_lora_rank, + spec.n_heads * (spec.qk_nope_head_dim + spec.v_head_dim) // tp, aw, + w_bytes("attn_kv_b", wd), ) add("attn_kv_b", f, b, wd) @@ -405,31 +571,50 @@ def add( # ``shared`` layers reuse the group's selection and carry no indexer weights, # so they emit no indexer node. Emitting one would put a kernel in the graph # that never ran and inflate the indexer's share ~4x. - runs_indexer = spec.is_full_indexer(layer) if force_full_indexer is None else force_full_indexer - cand = index_candidates(spec, kv_len) - if runs_indexer and cand > 0: + runs_indexer = ( + spec.is_full_indexer(layer) if force_full_indexer is None else force_full_indexer + ) + scan_pairs = index_scan_pairs(batch) + if runs_indexer and scan_pairs > 0: # The indexer query comes off the same latent as the attention query, so - # only the up-projection is charged. Not divided by ``tp`` — vLLM builds - # the indexer as ReplicatedLinear, so every rank runs the whole thing. - f, b = _linear( - positions, spec.q_lora_rank, spec.index_n_heads * spec.index_head_dim, aw, ww + # only the up-projection is charged (``wq_b``), plus the per-token key + # (``wk`` — one key per token, MQA-style, not one per index head) and the + # per-head gate (``weights_proj``). Not divided by ``tp``: vLLM builds the + # indexer as ReplicatedLinear, so every rank runs the whole thing. bf16 on + # the FP8 checkpoint — the indexer is named in ``modules_to_not_convert``. + idx_w = w_bytes("attn_index_proj", wd) + f_q, b_q = _linear( + rows, spec.q_lora_rank, spec.index_n_heads * spec.index_head_dim, aw, idx_w + ) + f_k, b_k = _linear(rows, h, spec.index_head_dim, aw, idx_w) + f_g, b_g = _linear(rows, h, spec.index_n_heads, aw, idx_w) + add( + "attn_index_proj", + f_q + f_k + f_g, + b_q + b_k + b_g + rows * spec.index_head_dim * weight_bytes(spec.kv_dtype), + wd, ) - add("attn_index_proj", f, b, wd) add( "attn_index_score", - 2.0 * positions * spec.index_n_heads * spec.index_head_dim * cand, - # Index keys live in the cache: read once per sequence, not per - # position, and replicated across ranks alongside the KV latent. - sequences * cand * spec.index_head_dim * weight_bytes(spec.kv_dtype), + 2.0 * scan_pairs * spec.index_head_dim, + # Index keys live in the cache: read once per sequence (or per + # prefilling request), not per position, and replicated across ranks + # alongside the KV latent. + index_scan_entries(batch) * spec.index_head_dim + * weight_bytes(spec.kv_dtype), spec.kv_dtype, ) # ── attention core over the selected positions ────────────────────────── - t_eff = effective_kv_tokens(spec, kv_len) + # FLOPs follow the *selected* pairs (top-k bounded); bytes follow what the + # kernel must stream, which at prefill is the whole cache and at decode is one + # top-k window per sequence. The two do not move together on this + # architecture, which is why they are separate helpers. heads = max(1, spec.n_heads // tp) - qk = 2.0 * positions * heads * spec.q_head_dim * t_eff - pv = 2.0 * positions * heads * spec.v_head_dim * t_eff + pairs = core_qk_pairs(spec, batch) + qk = 2.0 * pairs * heads * spec.q_head_dim + pv = 2.0 * pairs * heads * spec.v_head_dim add( "attn_score_value", qk + pv, @@ -437,14 +622,14 @@ def add( # Deliberately not divided by ``tp`` either: a single shared latent cannot # be split, so the cache is replicated and every rank reads all of it — # tensor parallelism buys no KV bandwidth on this architecture. - sequences * t_eff * kv_entry_bytes(spec), + core_read_entries(spec, batch) * kv_entry_bytes(spec), wd, ) # RMSNorm on every query head and the single KV latent, plus partial RoPE on # the last ``qk_rope_head_dim`` dims and the cache insert — one fused kernel. - normed = positions * (heads * spec.q_head_dim + spec.kv_entry_dim) - roped = positions * (heads + 1) * spec.qk_rope_head_dim + normed = rows * (heads * spec.q_head_dim + spec.kv_entry_dim) + roped = rows * (heads + 1) * spec.qk_rope_head_dim add( "attn_qnorm_rope_insert", 3.0 * normed + 6.0 * roped, @@ -453,74 +638,155 @@ def add( ) # Output projection: plain dense from the per-head value space back to hidden. - # No o_lora/o_groups here (unlike V4), so no ``estimated`` grouping guess. - f, b = _linear(positions, spec.n_heads * spec.v_head_dim // tp, h, aw, ww) + # No o_lora/o_groups here (unlike V4), so no ``estimated`` grouping guess. On + # the FP8 checkpoint this one *is* quantised — absent from + # ``modules_to_not_convert`` — the opposite of the fp8-backbone checkpoints + # that keep o_proj wide. + f, b = _linear( + rows, spec.n_heads * spec.v_head_dim // tp, h, aw, w_bytes("attn_out_proj", wd) + ) add("attn_out_proj", f, b, wd) + _emit_collective(g, spec, hw, layer, "tp_all_reduce_attn", rows, sh, prefix) + # ── FFN: dense on the leading layers, mixture on the rest ──────────────── if not spec.is_sparse_mlp(layer): # Dense FFN (first_k_dense_replace). gate+up then down over the wide # intermediate. Canonical dense-graph names so residuals stay comparable. inter = spec.intermediate_size - f_gu, b_gu = _linear(positions, h, 2 * inter // tp, aw, ww) + f_gu, b_gu = _linear(rows, h, 2 * inter // tp, aw, w_bytes("mlp_gate_up", wd)) add("mlp_gate_up", f_gu, b_gu, wd) - f_d, b_d = _linear(positions, inter // tp, h, aw, ww) + f_d, b_d = _linear(rows, inter // tp, h, aw, w_bytes("mlp_down", wd)) add("mlp_down", f_d, b_d, wd) + _emit_collective(g, spec, hw, layer, "tp_all_reduce_mlp", rows, sh, prefix) return # Router is replicated: every rank scores every expert to know what to keep. - f, b = _linear(positions, h, spec.n_routed_experts, aw, ww) + # fp32 on every GLM-5.2 variant (``moe_router_dtype``) — a model fact, not a + # quantisation choice, and the reason this op carries its own dtype. + f, b = _linear(rows, h, spec.n_routed_experts, aw, w_bytes("moe_router", wd)) add("moe_router", f, b, wd) + # Sigmoid scoring, the noaux_tc bias correction, top-8 and the renorm. Almost + # no bytes and almost no arithmetic — but it is the **only data-dependent + # shape in the step**, so it is the node that decides whether the step can be + # CUDA-graph captured at all. Kept as its own node for that reason alone. + add( + "moe_topk", + 3.0 * rows * spec.n_routed_experts, + 2.0 * rows * spec.n_routed_experts * aw, + spec.act_dtype, + ) + inter = spec.moe_intermediate_size per_expert_weights = 3.0 * h * inter per_position_flops = 6.0 * h * inter # 2 * (gate + up + down) * h * inter + ew = w_bytes("moe_routed", ed) if spec.n_shared_experts > 0: + sw = w_bytes("moe_shared", ed) add( "moe_shared", - per_position_flops * positions * spec.n_shared_experts / tp, - per_expert_weights * spec.n_shared_experts * ew / tp - + aw * (positions * h * 2 + positions * inter * 2 * spec.n_shared_experts / tp), + per_position_flops * rows * spec.n_shared_experts / tp, + per_expert_weights * spec.n_shared_experts * sw / tp + + aw * (rows * h * 2 + rows * inter * 2 * spec.n_shared_experts / tp), ed, ) - # The saturating set-union: FLOPs scale with positions x top_k, weight traffic + # Gather rows into expert-major order and scatter the results back. Zero + # arithmetic in the gather, and at decode a rounding error — but the expanded + # tensor is ``rows x top_k`` wide, so at an 8,192-token prefill chunk this pair + # moves hundreds of megabytes per layer for essentially no FLOPs. A graph that + # folds it into the expert GEMM cannot show that, and it is a real and + # separately addressable share of prefill traffic. + expanded = rows * spec.top_k + add("moe_permute", 0.0, aw * (rows * h + expanded * h) / es, spec.act_dtype) + + # The saturating set-union: FLOPs scale with rows x top_k, weight traffic # with how many *distinct* experts the batch woke — shared with the dense-MoE - # roofline so there is one owner for the term. + # roofline so there is one owner for the term. The argument is *rows*, not + # sequences: under speculative decoding and under prefill every extra row is + # another draw on the expert bank, which is exactly why a 1+D verify costs + # more than a decode without doing more work per token. distinct = distinct_experts( - int(positions), spec.n_routed_experts, spec.num_experts_per_tok + int(rows), spec.n_routed_experts, spec.num_experts_per_tok ) skew = sh.ep_imbalance if sh.ep > 1 else 1.0 add( "moe_routed", - per_position_flops * positions * spec.num_experts_per_tok * skew / es, + per_position_flops * rows * spec.num_experts_per_tok * skew / es, per_expert_weights * distinct * ew * skew / es - + aw * (positions * h * 2 + positions * inter * 2 * spec.num_experts_per_tok / es), + + aw * (rows * inter * 2 * spec.num_experts_per_tok / es), ed, ) - # ── cross-rank traffic ────────────────────────────────────────────────── - if tp > 1 or sh.ep > 1: - link = replace(hw, peak_mem_bw_bytes_per_s=hw.interconnect_bw_bytes_per_s) + # Weighted scatter-add back to ``rows x hidden``, including the + # routed_scaling_factor multiply. + add( + "moe_combine", + 2.0 * expanded * h, + aw * (expanded * h + rows * h) / es, + spec.act_dtype, + ) + + _emit_collective(g, spec, hw, layer, "tp_all_reduce_mlp", rows, sh, prefix) - def add_link(op: str, byts: float) -> None: - name = f"{prefix}{op}" - g.nodes.append( - PredictedNode( - name, layer, - roofline(name, 0.0, byts, link, spec.act_dtype, estimated=True), - ) - ) - if sh.ep > 1: - off_rank = (sh.ep - 1) / sh.ep - add_link( - "moe_all_to_all", - 2.0 * positions * spec.num_experts_per_tok * h * aw * off_rank, +def _emit_collective( + g: Graph, + spec: GlmMoeDsaModelSpec, + hw: HardwareSpec, + layer: int, + op: str, + rows: float, + sh: ShardingConfig, + prefix: str, +) -> None: + """Emit the cross-rank traffic that closes one sub-block. + + **Two per layer, not one.** A tensor-parallel layer all-reduces after + ``o_proj`` and again after the FFN combine; folding them into a single node + with double the payload gets the bytes right and the *count* wrong — and at + decode payloads a collective is bounded by its ring latency rather than by its + bytes, so the count is the cost. Under expert parallelism the MoE half + additionally dispatches and combines across expert ranks. + + ``serial_launches`` is withheld when the SKU carries no interconnect + bandwidth, so an unpriced collective still predicts zero time and + :attr:`Graph.has_unpriced_collectives` keeps reporting it. A latency floor + applied there would quietly convert "we cannot price this" into "it costs two + microseconds". + """ + tp = max(1, sh.tp) + if tp <= 1 and sh.ep <= 1: + return + aw = weight_bytes(spec.act_dtype) + link = replace(hw, peak_mem_bw_bytes_per_s=hw.interconnect_bw_bytes_per_s) + priced = hw.interconnect_bw_bytes_per_s > 0 + + def add_link(name_op: str, byts: float) -> None: + name = f"{prefix}{name_op}" + g.nodes.append( + PredictedNode( + name, layer, + roofline( + name, 0.0, byts, link, spec.act_dtype, + estimated=True, serial_launches=1 if priced else 0, + ), + # Collectives are the one region this graph expects off the + # compute stream; the stream-concurrency invariant reads this. + expected_stream_id=1, ) - if tp > 1: - add_link("tp_all_reduce", 2.0 * (2.0 * (tp - 1) / tp) * positions * h * aw) + ) + + if sh.ep > 1 and op == "tp_all_reduce_mlp": + off_rank = (sh.ep - 1) / sh.ep + add_link( + "moe_all_to_all", + 2.0 * rows * spec.num_experts_per_tok * spec.hidden * aw * off_rank, + ) + if tp > 1: + add_link(op, (2.0 * (tp - 1) / tp) * rows * spec.hidden * aw) def predict_glm_graph( @@ -529,12 +795,30 @@ def predict_glm_graph( batch: BatchConfig | None = None, sharding: ShardingConfig | None = None, ) -> Graph: - """Emit a predicted execution graph for one GLM-5.2-class decode step, per rank. - - The main stack runs over every position in the step (the verified token plus - any speculative drafts); the MTP head then runs over one position per sequence - to propose the next draft. With ``sharding`` left at its default the graph is - whole-model; given a real sharding it predicts what *one rank* does. + """Emit a predicted execution graph for one GLM-5.2-class engine step, per rank. + + One step, three passes, and they are not three graphs: + + **The backbone** runs over every position in the step — decode positions plus + any prefill chunk riding along under chunked prefill. With speculative + decoding on, "decode positions" is ``batch x (1 + D)``: **verify is not a new + graph, it is this one with the row dimension multiplied by 1+D**. + + **The draft chain** is the one genuinely new subgraph. GLM-5.2 has a single + MTP module (``num_nextn_predict_layers: 1``) invoked ``D`` times serially, + EAGLE-style — stage *k* cannot start until stage *k-1*'s token id exists — and + each stage runs its own vocabulary projection. Emitting one draft block and + one ``lm_head`` for a ``D``-deep chain understates it by ``D``, and the + ``lm_head`` term is not small: it reads the whole untied vocabulary matrix + per stage regardless of how few rows ride on it. + + **The epilogue** projects only the rows that need logits — + :attr:`BatchConfig.logits_rows`, which is one row per prefilling *request* + plus every decode position. Charging every prefill token would overstate a + 154,880-wide projection by the chunk size. + + With ``sharding`` left at its default the graph is whole-model; given a real + sharding it predicts what *one rank* does. """ spec = model or GlmMoeDsaModelSpec() hw = hw or HardwareSpec() @@ -557,34 +841,71 @@ def predict_glm_graph( raise ValueError("n_layers must be positive — an empty model predicts nothing") g = Graph(model=spec, hw=hw, batch=batch, sharding=sh) # type: ignore[arg-type] - positions = batch.positions_per_step - sequences = batch.batch - kv_len = batch.kv_cache_len for layer in range(spec.n_layers): - _emit_layer( - g, spec, hw, layer, - positions=positions, sequences=sequences, kv_len=kv_len, sh=sh, - ) + _emit_layer(g, spec, hw, layer, batch=batch, sh=sh) - # Multi-token prediction head. ``index_share_for_mtp_iteration`` says the MTP - # iteration reuses the main model's index rather than recomputing it, so the - # draft layer is emitted as a ``shared`` (no indexer) block — the tensor exists - # but the iteration shares it, and banking the projection+scan into the floor - # would over-predict a node the runtime skips. - for i in range(spec.num_nextn_predict_layers): - _emit_layer( - g, spec, hw, spec.n_layers + i, - positions=sequences, sequences=sequences, kv_len=kv_len, sh=sh, - force_full_indexer=not spec.index_share_for_mtp_iteration, + aw = weight_bytes(spec.act_dtype) + lm_w = weight_bytes(spec.dtype_for("lm_head", spec.weight_dtype)) + lm_dtype = spec.dtype_for("lm_head", spec.weight_dtype) + + def add_lm_head(rows: float, layer: int | None) -> None: + f, b = _linear(rows, spec.hidden, spec.vocab // max(1, sh.tp), aw, lm_w) + g.nodes.append( + PredictedNode( + "lm_head", layer, + roofline("lm_head", f, b, hw, lm_dtype, serial_launches=1), + ) ) - aw = weight_bytes(spec.act_dtype) - ww = weight_bytes(spec.weight_dtype) - f, b = _linear(positions, spec.hidden, spec.vocab // max(1, sh.tp), aw, ww) - g.nodes.append( - PredictedNode("lm_head", None, roofline("lm_head", f, b, hw, spec.weight_dtype)) - ) + add_lm_head(batch.logits_rows, None) + + # ── the draft chain ────────────────────────────────────────────────────── + # The MTP module is invoked once per drafted token, serially. Each stage sees + # one row per sequence (it proposes for the sequence, not for the verify rows) + # and carries no prefill: a draft head proposes continuations, it does not + # ingest a prompt. + # + # ``index_share_for_mtp_iteration`` says the iteration reuses the main model's + # selection rather than recomputing it, and the weight map agrees — the MTP + # block carries **no** ``self_attn.indexer.*`` tensors, exactly like the 57 + # ``shared`` layers. So the draft is emitted as a shared block; banking the + # projection and the scan into the floor would price a scan the runtime skips. + # + # What the draft is *not* is a smaller copy of the model. The MTP block carries + # a full ``mlp.experts.*`` bank in the checkpoint, so every draft stage draws on + # a 256-expert mixture — the draft's cost is dominated by expert weight traffic + # it pays ``D`` times over, not by its arithmetic. + if spec.num_nextn_predict_layers > 0: + draft_batch = replace(batch, prefill_tokens=0, speculative_tokens=0) + stages = max(1, batch.speculative_tokens) + for stage in range(stages): + _emit_layer( + g, spec, hw, spec.n_layers + stage, + batch=draft_batch, sh=sh, + force_full_indexer=not spec.index_share_for_mtp_iteration, + ) + # ``eh_proj``: the [2*hidden, hidden] fusion of the previous hidden + # state with the embedding of the token just drafted. bf16 on the FP8 + # checkpoint (named in ``modules_to_not_convert``), and replicated per + # rank unless the engine shards it. + eh_w = weight_bytes(spec.dtype_for("mtp_eh_proj", spec.weight_dtype)) + f, b = _linear(draft_batch.batch, 2 * spec.hidden, spec.hidden, aw, eh_w) + g.nodes.append( + PredictedNode( + "mtp_eh_proj", spec.n_layers + stage, + roofline( + "mtp_eh_proj", f, b, hw, + spec.dtype_for("mtp_eh_proj", spec.weight_dtype), + serial_launches=1, + ), + ) + ) + # There is no ``mtp.*.lm_head`` in the checkpoint — the draft shares the + # backbone's, which means it re-reads the same vocabulary weights and + # gets no cheaper for being a draft. + add_lm_head(draft_batch.batch, spec.n_layers + stage) + return g @@ -604,6 +925,60 @@ def is_glm_moe_dsa_config(cfg: dict[str, Any]) -> bool: return any("glmmoedsa" in str(a).lower() for a in archs) +#: Which graph op each ``modules_to_not_convert`` entry belongs to. Substring +#: match against the tensor name, first hit wins. Norms and biases are omitted +#: deliberately — they are not nodes in this graph, so a precision for them would +#: have nothing to price. +_UNQUANTISED_OPS: tuple[tuple[str, str], ...] = ( + ("lm_head", "lm_head"), + ("embed_tokens", "lm_head"), # untied, but priced together in the epilogue + ("eh_proj", "mtp_eh_proj"), + ("indexer", "attn_index_proj"), + ("indexers_proj", "attn_index_proj"), + ("mlp.gate", "moe_router"), +) + + +def _op_dtype_overrides( + cfg: dict[str, Any], q: dict[str, Any], weight_dtype: str, model_dtype: str +) -> tuple[tuple[str, str], ...]: + """Per-op precision, read from the checkpoint rather than assumed. + + Two independent sources, and they answer different questions: + + ``quantization_config.modules_to_not_convert`` + *What the quantiser skipped.* Those tensors stay at the model dtype while + the backbone drops to fp8 — on GLM-5.2-FP8 that is ``lm_head``, + ``embed_tokens``, the MTP ``eh_proj`` and, the one worth naming, the + **lightning indexer**. Pricing the indexer at fp8 would halve the weight + traffic of the one attention node whose cost grows with context. + + ``moe_router_dtype`` + *What the model computes in regardless.* fp32 on every GLM-5.2 variant, + including the unquantised one, so it is emitted whether or not a + quantisation config exists. + + Returns an empty tuple when the checkpoint says nothing — an unquantised model + needs no overrides beyond the router, and inventing entries would put a + precision in the graph that the checkpoint never declared. + """ + found: dict[str, str] = {} + + skipped = q.get("modules_to_not_convert") or q.get("ignored_layers") or [] + if isinstance(skipped, (list, tuple)) and weight_dtype != model_dtype: + for tensor in skipped: + name = str(tensor).lower() + for needle, op in _UNQUANTISED_OPS: + if needle in name: + found.setdefault(op, model_dtype) + + router = str(cfg.get("moe_router_dtype") or "").lower() + if router.startswith("float32") or router == "fp32": + found["moe_router"] = "fp32" + + return tuple(sorted(found.items())) + + def spec_from_hf_config( cfg: dict[str, Any], *, name: str | None = None ) -> GlmMoeDsaModelSpec: @@ -658,12 +1033,16 @@ def _types(key: str) -> tuple[str, ...]: elif dtype.startswith("float32"): dtype = "fp32" - # Quantisation, if the checkpoint declares any. Absent on the base GLM-5.2 - # release — an all-bf16 read matches the on-disk size — so this defaults to the - # model dtype rather than to fp4/fp8, which would manufacture headroom. + # Quantisation, if the checkpoint declares any. The base GLM-5.2 release + # carries none — an all-bf16 read matches its 1.507 TB on disk — while + # GLM-5.2-FP8 declares e4m3 with 128x128 weight blocks. Read rather than + # assumed in either direction: defaulting to fp8 would manufacture headroom on + # the bf16 checkpoint, and defaulting to bf16 would double every weight term on + # the one the vendor actually recommends deploying. q = cfg.get("quantization_config") or {} weight_dtype = str(q.get("quant_method", dtype)).lower() if q else dtype expert_dtype = str(cfg.get("expert_dtype", weight_dtype)).lower() + overrides = _op_dtype_overrides(cfg, q, weight_dtype, dtype) return GlmMoeDsaModelSpec( name=name or str(cfg.get("_name_or_path") or cfg.get("model_type") or "glm-moe-dsa"), @@ -693,6 +1072,7 @@ def _types(key: str) -> tuple[str, ...]: index_share_for_mtp_iteration=bool(cfg.get("index_share_for_mtp_iteration", True)), weight_dtype=weight_dtype, expert_dtype=expert_dtype, + op_dtype_overrides=overrides, # The config declares no cache dtype; bf16 is the model fact. A served # deployment may pick fp8 — that is a serving decision, set at deploy time. kv_dtype=dtype, From 9c119271057acaa540c45d04e99fcf051ae5da43 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 13:08:29 -0700 Subject: [PATCH 03/35] GLM-5.2: FP8 catalogue entry, fp32 SKU peaks, launch-aware plan table, tests --- .../glm-5.2_H200_b1_kv131072_tp1ep1.json | 9152 ------------- .../glm-5.2_H200_b1_kv4096_tp1ep1.json | 9152 ------------- .../glm-5.2_H200_b32_kv8192_tp16ep16.json | 10824 ---------------- gitm/planner/context.py | 44 +- gitm/planner/model_catalogue.py | 9 + gitm/planner/models/glm-5.2-fp8.yaml | 306 + gitm/planner/models/glm-5.2.yaml | 29 +- gitm/planner/registry.py | 44 +- tests/test_glm_graph.py | 226 +- 9 files changed, 633 insertions(+), 29153 deletions(-) delete mode 100644 docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv131072_tp1ep1.json delete mode 100644 docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv4096_tp1ep1.json delete mode 100644 docs/glm-5.2/artifacts/glm-5.2_H200_b32_kv8192_tp16ep16.json create mode 100644 gitm/planner/models/glm-5.2-fp8.yaml diff --git a/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv131072_tp1ep1.json b/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv131072_tp1ep1.json deleted file mode 100644 index 7613b5e..0000000 --- a/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv131072_tp1ep1.json +++ /dev/null @@ -1,9152 +0,0 @@ -{ - "model": "zai-org/GLM-5.2", - "family": "glm_moe_dsa", - "hardware": "H200", - "sharding": { - "tp": 1, - "ep": 1, - "dp": 1 - }, - "batch": { - "batch": 1, - "kv_cache_len": 131072, - "prefill_tokens": 0, - "prefill_context": 0, - "prefill_requests": 1 - }, - "total_pred_s": 0.017185272906666667, - "has_unpriced_collectives": false, - "has_fallback_peaks": false, - "nodes": [ - { - "op": "attn_q_a", - "layer": 0, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 0, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 0, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 0, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 0, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 0, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 0, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 0, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 0, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "mlp_gate_up", - "layer": 0, - "t_pred_s": 6.292736e-05, - "t_compute_s": 3.053487239635996e-07, - "t_memory_s": 6.292736e-05, - "bound": "memory", - "flops": 301989888.0, - "bytes": 302051328.0, - "estimated": false - }, - { - "op": "mlp_down", - "layer": 0, - "t_pred_s": 3.146496e-05, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 3.146496e-05, - "bound": "memory", - "flops": 150994944.0, - "bytes": 151031808.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 1, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 1, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 1, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 1, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 1, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 1, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 1, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 1, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 1, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "mlp_gate_up", - "layer": 1, - "t_pred_s": 6.292736e-05, - "t_compute_s": 3.053487239635996e-07, - "t_memory_s": 6.292736e-05, - "bound": "memory", - "flops": 301989888.0, - "bytes": 302051328.0, - "estimated": false - }, - { - "op": "mlp_down", - "layer": 1, - "t_pred_s": 3.146496e-05, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 3.146496e-05, - "bound": "memory", - "flops": 150994944.0, - "bytes": 151031808.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 2, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 2, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 2, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 2, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 2, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 2, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 2, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 2, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 2, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "mlp_gate_up", - "layer": 2, - "t_pred_s": 6.292736e-05, - "t_compute_s": 3.053487239635996e-07, - "t_memory_s": 6.292736e-05, - "bound": "memory", - "flops": 301989888.0, - "bytes": 302051328.0, - "estimated": false - }, - { - "op": "mlp_down", - "layer": 2, - "t_pred_s": 3.146496e-05, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 3.146496e-05, - "bound": "memory", - "flops": 150994944.0, - "bytes": 151031808.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 3, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 3, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 3, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 3, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 3, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 3, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 3, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 3, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 3, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 3, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 4, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 4, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 4, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 4, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 4, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 4, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 4, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 4, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 4, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 4, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 5, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 5, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 5, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 5, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 5, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 5, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 5, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 5, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 5, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 5, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 6, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 6, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 6, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 6, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 6, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 6, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 6, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 6, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 6, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 6, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 6, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 6, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 7, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 7, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 7, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 7, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 7, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 7, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 7, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 7, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 7, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 7, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 8, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 8, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 8, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 8, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 8, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 8, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 8, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 8, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 8, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 8, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 9, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 9, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 9, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 9, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 9, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 9, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 9, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 9, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 9, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 9, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 10, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 10, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 10, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 10, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 10, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 10, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 10, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 10, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 10, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 10, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 10, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 10, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 11, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 11, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 11, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 11, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 11, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 11, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 11, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 11, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 11, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 11, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 12, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 12, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 12, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 12, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 12, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 12, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 12, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 12, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 12, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 12, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 13, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 13, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 13, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 13, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 13, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 13, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 13, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 13, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 13, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 13, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 14, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 14, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 14, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 14, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 14, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 14, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 14, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 14, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 14, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 14, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 14, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 14, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 15, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 15, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 15, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 15, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 15, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 15, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 15, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 15, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 15, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 15, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 16, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 16, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 16, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 16, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 16, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 16, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 16, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 16, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 16, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 16, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 17, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 17, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 17, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 17, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 17, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 17, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 17, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 17, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 17, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 17, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 18, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 18, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 18, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 18, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 18, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 18, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 18, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 18, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 18, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 18, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 18, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 18, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 19, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 19, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 19, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 19, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 19, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 19, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 19, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 19, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 19, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 19, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 20, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 20, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 20, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 20, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 20, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 20, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 20, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 20, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 20, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 20, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 21, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 21, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 21, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 21, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 21, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 21, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 21, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 21, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 21, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 21, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 22, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 22, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 22, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 22, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 22, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 22, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 22, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 22, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 22, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 22, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 22, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 22, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 23, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 23, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 23, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 23, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 23, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 23, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 23, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 23, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 23, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 23, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 24, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 24, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 24, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 24, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 24, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 24, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 24, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 24, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 24, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 24, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 25, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 25, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 25, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 25, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 25, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 25, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 25, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 25, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 25, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 25, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 26, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 26, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 26, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 26, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 26, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 26, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 26, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 26, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 26, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 26, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 26, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 26, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 27, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 27, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 27, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 27, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 27, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 27, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 27, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 27, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 27, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 27, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 28, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 28, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 28, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 28, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 28, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 28, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 28, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 28, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 28, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 28, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 29, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 29, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 29, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 29, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 29, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 29, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 29, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 29, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 29, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 29, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 30, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 30, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 30, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 30, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 30, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 30, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 30, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 30, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 30, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 30, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 30, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 30, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 31, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 31, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 31, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 31, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 31, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 31, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 31, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 31, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 31, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 31, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 32, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 32, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 32, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 32, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 32, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 32, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 32, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 32, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 32, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 32, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 33, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 33, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 33, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 33, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 33, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 33, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 33, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 33, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 33, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 33, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 34, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 34, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 34, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 34, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 34, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 34, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 34, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 34, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 34, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 34, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 34, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 34, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 35, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 35, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 35, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 35, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 35, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 35, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 35, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 35, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 35, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 35, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 36, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 36, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 36, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 36, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 36, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 36, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 36, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 36, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 36, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 36, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 37, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 37, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 37, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 37, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 37, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 37, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 37, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 37, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 37, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 37, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 38, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 38, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 38, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 38, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 38, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 38, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 38, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 38, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 38, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 38, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 38, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 38, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 39, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 39, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 39, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 39, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 39, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 39, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 39, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 39, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 39, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 39, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 40, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 40, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 40, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 40, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 40, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 40, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 40, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 40, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 40, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 40, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 41, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 41, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 41, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 41, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 41, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 41, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 41, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 41, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 41, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 41, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 42, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 42, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 42, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 42, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 42, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 42, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 42, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 42, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 42, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 42, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 42, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 42, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 43, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 43, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 43, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 43, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 43, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 43, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 43, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 43, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 43, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 43, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 44, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 44, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 44, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 44, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 44, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 44, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 44, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 44, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 44, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 44, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 45, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 45, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 45, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 45, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 45, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 45, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 45, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 45, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 45, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 45, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 46, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 46, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 46, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 46, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 46, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 46, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 46, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 46, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 46, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 46, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 46, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 46, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 47, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 47, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 47, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 47, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 47, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 47, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 47, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 47, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 47, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 47, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 48, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 48, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 48, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 48, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 48, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 48, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 48, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 48, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 48, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 48, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 49, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 49, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 49, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 49, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 49, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 49, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 49, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 49, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 49, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 49, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 50, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 50, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 50, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 50, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 50, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 50, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 50, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 50, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 50, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 50, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 50, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 50, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 51, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 51, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 51, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 51, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 51, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 51, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 51, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 51, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 51, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 51, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 52, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 52, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 52, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 52, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 52, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 52, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 52, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 52, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 52, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 52, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 53, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 53, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 53, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 53, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 53, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 53, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 53, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 53, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 53, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 53, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 54, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 54, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 54, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 54, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 54, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 54, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 54, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 54, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 54, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 54, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 54, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 54, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 55, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 55, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 55, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 55, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 55, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 55, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 55, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 55, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 55, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 55, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 56, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 56, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 56, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 56, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 56, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 56, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 56, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 56, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 56, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 56, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 57, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 57, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 57, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 57, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 57, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 57, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 57, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 57, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 57, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 57, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 58, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 58, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 58, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 58, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 58, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 58, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 58, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 58, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 58, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 58, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 58, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 58, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 59, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 59, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 59, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 59, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 59, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 59, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 59, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 59, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 59, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 59, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 60, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 60, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 60, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 60, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 60, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 60, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 60, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 60, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 60, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 60, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 61, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 61, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 61, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 61, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 61, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 61, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 61, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 61, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 61, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 61, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 62, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 62, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 62, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 62, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 62, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 62, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 62, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 62, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 62, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 62, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 62, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 62, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 63, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 63, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 63, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 63, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 63, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 63, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 63, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 63, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 63, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 63, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 64, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 64, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 64, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 64, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 64, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 64, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 64, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 64, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 64, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 64, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 65, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 65, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 65, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 65, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 65, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 65, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 65, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 65, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 65, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 65, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 66, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 66, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 66, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 66, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 66, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 66, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 66, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 66, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 66, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 66, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 66, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 66, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 67, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 67, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 67, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 67, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 67, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 67, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 67, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 67, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 67, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 67, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 68, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 68, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 68, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 68, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 68, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 68, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 68, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 68, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 68, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 68, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 69, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 69, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 69, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 69, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 69, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 69, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 69, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 69, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 69, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 69, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 70, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 70, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 70, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 70, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 70, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 70, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 70, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 70, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 70, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 70, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 70, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 70, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 71, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 71, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 71, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 71, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 71, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 71, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 71, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 71, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 71, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 71, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 72, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 72, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 72, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 72, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 72, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 72, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 72, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 72, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 72, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 72, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 73, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 73, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 73, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 73, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 73, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 73, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 73, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 73, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 73, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 73, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 74, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 74, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 74, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 74, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 74, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 74, - "t_pred_s": 6.990506666666666e-06, - "t_compute_s": 1.0856843518705764e-06, - "t_memory_s": 6.990506666666666e-06, - "bound": "memory", - "flops": 1073741824.0, - "bytes": 33554432.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 74, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 74, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 74, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 74, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 74, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 74, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 75, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 75, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 75, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 75, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 75, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 75, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 75, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 75, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 75, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 75, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 76, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 76, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 76, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 76, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 76, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 76, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 76, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 76, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 76, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 76, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 77, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 77, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 77, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 77, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 77, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 77, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 77, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 77, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 77, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 77, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 78, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 78, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 78, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 78, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 78, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 78, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 78, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 78, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 78, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 78, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "lm_head", - "layer": null, - "t_pred_s": 0.00039655989333333336, - "t_compute_s": 1.9243331041456017e-06, - "t_memory_s": 0.00039655989333333336, - "bound": "memory", - "flops": 1903165440.0, - "bytes": 1903487488.0, - "estimated": false - } - ] -} diff --git a/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv4096_tp1ep1.json b/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv4096_tp1ep1.json deleted file mode 100644 index 727500e..0000000 --- a/docs/glm-5.2/artifacts/glm-5.2_H200_b1_kv4096_tp1ep1.json +++ /dev/null @@ -1,9152 +0,0 @@ -{ - "model": "zai-org/GLM-5.2", - "family": "glm_moe_dsa", - "hardware": "H200", - "sharding": { - "tp": 1, - "ep": 1, - "dp": 1 - }, - "batch": { - "batch": 1, - "kv_cache_len": 4096, - "prefill_tokens": 0, - "prefill_context": 0, - "prefill_requests": 1 - }, - "total_pred_s": 0.017043059786666665, - "has_unpriced_collectives": false, - "has_fallback_peaks": false, - "nodes": [ - { - "op": "attn_q_a", - "layer": 0, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 0, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 0, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 0, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 0, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 0, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 0, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 0, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 0, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "mlp_gate_up", - "layer": 0, - "t_pred_s": 6.292736e-05, - "t_compute_s": 3.053487239635996e-07, - "t_memory_s": 6.292736e-05, - "bound": "memory", - "flops": 301989888.0, - "bytes": 302051328.0, - "estimated": false - }, - { - "op": "mlp_down", - "layer": 0, - "t_pred_s": 3.146496e-05, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 3.146496e-05, - "bound": "memory", - "flops": 150994944.0, - "bytes": 151031808.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 1, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 1, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 1, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 1, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 1, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 1, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 1, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 1, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 1, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "mlp_gate_up", - "layer": 1, - "t_pred_s": 6.292736e-05, - "t_compute_s": 3.053487239635996e-07, - "t_memory_s": 6.292736e-05, - "bound": "memory", - "flops": 301989888.0, - "bytes": 302051328.0, - "estimated": false - }, - { - "op": "mlp_down", - "layer": 1, - "t_pred_s": 3.146496e-05, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 3.146496e-05, - "bound": "memory", - "flops": 150994944.0, - "bytes": 151031808.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 2, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 2, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 2, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 2, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 2, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 2, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 2, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 2, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 2, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "mlp_gate_up", - "layer": 2, - "t_pred_s": 6.292736e-05, - "t_compute_s": 3.053487239635996e-07, - "t_memory_s": 6.292736e-05, - "bound": "memory", - "flops": 301989888.0, - "bytes": 302051328.0, - "estimated": false - }, - { - "op": "mlp_down", - "layer": 2, - "t_pred_s": 3.146496e-05, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 3.146496e-05, - "bound": "memory", - "flops": 150994944.0, - "bytes": 151031808.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 3, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 3, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 3, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 3, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 3, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 3, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 3, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 3, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 3, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 3, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 4, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 4, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 4, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 4, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 4, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 4, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 4, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 4, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 4, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 4, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 5, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 5, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 5, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 5, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 5, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 5, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 5, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 5, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 5, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 5, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 6, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 6, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 6, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 6, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 6, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 6, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 6, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 6, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 6, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 6, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 6, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 6, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 7, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 7, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 7, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 7, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 7, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 7, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 7, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 7, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 7, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 7, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 8, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 8, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 8, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 8, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 8, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 8, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 8, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 8, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 8, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 8, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 9, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 9, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 9, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 9, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 9, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 9, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 9, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 9, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 9, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 9, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 10, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 10, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 10, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 10, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 10, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 10, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 10, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 10, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 10, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 10, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 10, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 10, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 11, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 11, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 11, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 11, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 11, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 11, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 11, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 11, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 11, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 11, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 12, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 12, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 12, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 12, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 12, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 12, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 12, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 12, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 12, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 12, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 13, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 13, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 13, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 13, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 13, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 13, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 13, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 13, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 13, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 13, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 14, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 14, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 14, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 14, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 14, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 14, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 14, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 14, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 14, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 14, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 14, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 14, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 15, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 15, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 15, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 15, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 15, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 15, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 15, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 15, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 15, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 15, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 16, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 16, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 16, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 16, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 16, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 16, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 16, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 16, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 16, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 16, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 17, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 17, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 17, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 17, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 17, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 17, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 17, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 17, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 17, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 17, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 18, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 18, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 18, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 18, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 18, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 18, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 18, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 18, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 18, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 18, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 18, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 18, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 19, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 19, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 19, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 19, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 19, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 19, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 19, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 19, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 19, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 19, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 20, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 20, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 20, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 20, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 20, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 20, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 20, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 20, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 20, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 20, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 21, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 21, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 21, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 21, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 21, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 21, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 21, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 21, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 21, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 21, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 22, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 22, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 22, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 22, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 22, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 22, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 22, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 22, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 22, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 22, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 22, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 22, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 23, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 23, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 23, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 23, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 23, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 23, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 23, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 23, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 23, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 23, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 24, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 24, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 24, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 24, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 24, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 24, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 24, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 24, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 24, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 24, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 25, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 25, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 25, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 25, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 25, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 25, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 25, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 25, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 25, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 25, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 26, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 26, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 26, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 26, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 26, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 26, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 26, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 26, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 26, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 26, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 26, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 26, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 27, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 27, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 27, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 27, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 27, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 27, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 27, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 27, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 27, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 27, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 28, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 28, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 28, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 28, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 28, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 28, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 28, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 28, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 28, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 28, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 29, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 29, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 29, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 29, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 29, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 29, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 29, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 29, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 29, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 29, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 30, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 30, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 30, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 30, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 30, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 30, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 30, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 30, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 30, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 30, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 30, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 30, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 31, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 31, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 31, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 31, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 31, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 31, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 31, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 31, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 31, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 31, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 32, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 32, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 32, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 32, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 32, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 32, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 32, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 32, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 32, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 32, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 33, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 33, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 33, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 33, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 33, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 33, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 33, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 33, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 33, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 33, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 34, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 34, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 34, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 34, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 34, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 34, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 34, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 34, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 34, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 34, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 34, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 34, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 35, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 35, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 35, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 35, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 35, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 35, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 35, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 35, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 35, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 35, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 36, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 36, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 36, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 36, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 36, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 36, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 36, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 36, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 36, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 36, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 37, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 37, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 37, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 37, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 37, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 37, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 37, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 37, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 37, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 37, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 38, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 38, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 38, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 38, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 38, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 38, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 38, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 38, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 38, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 38, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 38, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 38, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 39, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 39, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 39, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 39, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 39, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 39, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 39, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 39, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 39, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 39, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 40, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 40, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 40, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 40, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 40, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 40, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 40, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 40, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 40, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 40, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 41, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 41, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 41, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 41, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 41, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 41, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 41, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 41, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 41, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 41, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 42, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 42, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 42, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 42, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 42, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 42, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 42, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 42, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 42, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 42, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 42, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 42, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 43, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 43, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 43, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 43, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 43, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 43, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 43, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 43, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 43, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 43, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 44, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 44, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 44, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 44, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 44, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 44, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 44, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 44, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 44, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 44, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 45, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 45, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 45, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 45, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 45, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 45, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 45, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 45, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 45, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 45, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 46, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 46, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 46, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 46, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 46, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 46, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 46, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 46, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 46, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 46, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 46, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 46, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 47, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 47, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 47, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 47, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 47, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 47, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 47, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 47, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 47, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 47, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 48, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 48, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 48, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 48, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 48, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 48, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 48, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 48, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 48, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 48, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 49, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 49, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 49, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 49, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 49, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 49, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 49, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 49, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 49, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 49, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 50, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 50, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 50, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 50, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 50, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 50, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 50, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 50, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 50, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 50, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 50, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 50, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 51, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 51, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 51, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 51, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 51, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 51, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 51, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 51, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 51, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 51, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 52, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 52, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 52, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 52, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 52, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 52, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 52, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 52, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 52, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 52, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 53, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 53, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 53, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 53, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 53, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 53, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 53, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 53, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 53, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 53, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 54, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 54, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 54, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 54, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 54, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 54, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 54, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 54, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 54, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 54, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 54, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 54, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 55, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 55, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 55, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 55, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 55, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 55, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 55, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 55, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 55, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 55, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 56, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 56, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 56, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 56, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 56, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 56, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 56, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 56, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 56, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 56, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 57, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 57, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 57, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 57, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 57, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 57, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 57, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 57, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 57, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 57, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 58, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 58, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 58, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 58, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 58, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 58, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 58, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 58, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 58, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 58, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 58, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 58, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 59, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 59, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 59, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 59, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 59, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 59, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 59, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 59, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 59, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 59, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 60, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 60, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 60, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 60, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 60, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 60, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 60, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 60, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 60, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 60, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 61, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 61, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 61, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 61, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 61, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 61, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 61, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 61, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 61, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 61, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 62, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 62, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 62, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 62, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 62, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 62, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 62, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 62, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 62, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 62, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 62, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 62, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 63, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 63, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 63, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 63, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 63, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 63, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 63, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 63, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 63, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 63, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 64, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 64, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 64, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 64, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 64, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 64, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 64, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 64, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 64, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 64, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 65, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 65, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 65, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 65, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 65, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 65, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 65, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 65, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 65, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 65, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 66, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 66, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 66, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 66, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 66, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 66, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 66, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 66, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 66, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 66, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 66, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 66, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 67, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 67, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 67, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 67, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 67, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 67, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 67, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 67, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 67, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 67, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 68, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 68, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 68, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 68, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 68, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 68, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 68, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 68, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 68, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 68, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 69, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 69, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 69, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 69, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 69, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 69, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 69, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 69, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 69, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 69, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 70, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 70, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 70, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 70, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 70, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 70, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 70, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 70, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 70, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 70, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 70, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 70, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 71, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 71, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 71, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 71, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 71, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 71, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 71, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 71, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 71, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 71, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 72, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 72, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 72, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 72, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 72, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 72, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 72, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 72, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 72, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 72, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 73, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 73, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 73, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 73, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 73, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 73, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 73, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 73, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 73, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 73, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 74, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 74, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 74, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 74, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 74, - "t_pred_s": 3.4978133333333335e-06, - "t_compute_s": 1.6963817997977756e-08, - "t_memory_s": 3.4978133333333335e-06, - "bound": "memory", - "flops": 16777216.0, - "bytes": 16789504.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 74, - "t_pred_s": 2.1845333333333332e-07, - "t_compute_s": 3.392763599595551e-08, - "t_memory_s": 2.1845333333333332e-07, - "bound": "memory", - "flops": 33554432.0, - "bytes": 1048576.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 74, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 74, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 74, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 74, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 74, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 74, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 75, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 75, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 75, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 75, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 75, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 75, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 75, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 75, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 75, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 75, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 76, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 76, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 76, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 76, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 76, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 76, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 76, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 76, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 76, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 76, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 77, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 77, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 77, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 77, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 77, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 77, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 77, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 77, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 77, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 77, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 78, - "t_pred_s": 5.246293333333333e-06, - "t_compute_s": 2.5445726996966633e-08, - "t_memory_s": 5.246293333333333e-06, - "bound": "memory", - "flops": 25165824.0, - "bytes": 25182208.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 78, - "t_pred_s": 1.3988693333333333e-05, - "t_compute_s": 6.785527199191102e-08, - "t_memory_s": 1.3988693333333333e-05, - "bound": "memory", - "flops": 67108864.0, - "bytes": 67145728.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 78, - "t_pred_s": 1.4776e-06, - "t_compute_s": 7.156610717896866e-09, - "t_memory_s": 1.4776e-06, - "bound": "memory", - "flops": 7077888.0, - "bytes": 7092480.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 78, - "t_pred_s": 6.128853333333334e-06, - "t_compute_s": 2.9686681496461072e-08, - "t_memory_s": 6.128853333333334e-06, - "bound": "memory", - "flops": 29360128.0, - "bytes": 29418496.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 78, - "t_pred_s": 4.9152e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 4.9152e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 2359296.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 78, - "t_pred_s": 1.76e-08, - "t_compute_s": 7.66835187057634e-11, - "t_memory_s": 1.76e-08, - "bound": "memory", - "flops": 75840.0, - "bytes": 84480.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 78, - "t_pred_s": 4.1952426666666666e-05, - "t_compute_s": 2.0356581597573307e-07, - "t_memory_s": 4.1952426666666666e-05, - "bound": "memory", - "flops": 201326592.0, - "bytes": 201371648.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 78, - "t_pred_s": 6.580266666666667e-07, - "t_compute_s": 3.180715874620829e-09, - "t_memory_s": 6.580266666666667e-07, - "bound": "memory", - "flops": 3145728.0, - "bytes": 3158528.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 78, - "t_pred_s": 1.5735466666666665e-05, - "t_compute_s": 7.63371809908999e-08, - "t_memory_s": 1.5735466666666665e-05, - "bound": "memory", - "flops": 75497472.0, - "bytes": 75530240.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 78, - "t_pred_s": 0.00012584789333333334, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 0.00012584789333333334, - "bound": "memory", - "flops": 603979776.0, - "bytes": 604069888.0, - "estimated": false - }, - { - "op": "lm_head", - "layer": null, - "t_pred_s": 0.00039655989333333336, - "t_compute_s": 1.9243331041456017e-06, - "t_memory_s": 0.00039655989333333336, - "bound": "memory", - "flops": 1903165440.0, - "bytes": 1903487488.0, - "estimated": false - } - ] -} diff --git a/docs/glm-5.2/artifacts/glm-5.2_H200_b32_kv8192_tp16ep16.json b/docs/glm-5.2/artifacts/glm-5.2_H200_b32_kv8192_tp16ep16.json deleted file mode 100644 index f5e90bd..0000000 --- a/docs/glm-5.2/artifacts/glm-5.2_H200_b32_kv8192_tp16ep16.json +++ /dev/null @@ -1,10824 +0,0 @@ -{ - "model": "zai-org/GLM-5.2", - "family": "glm_moe_dsa", - "hardware": "H200", - "sharding": { - "tp": 16, - "ep": 16, - "dp": 1 - }, - "batch": { - "batch": 32, - "kv_cache_len": 8192, - "prefill_tokens": 0, - "prefill_context": 0, - "prefill_requests": 1 - }, - "total_pred_s": 0.015507311864019567, - "has_unpriced_collectives": false, - "has_fallback_peaks": false, - "nodes": [ - { - "op": "attn_q_a", - "layer": 0, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 0, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 0, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 0, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 0, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 0, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 0, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 0, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 0, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "mlp_gate_up", - "layer": 0, - "t_pred_s": 4.03456e-06, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 4.03456e-06, - "bound": "memory", - "flops": 603979776.0, - "bytes": 19365888.0, - "estimated": false - }, - { - "op": "mlp_down", - "layer": 0, - "t_pred_s": 2.05824e-06, - "t_compute_s": 3.053487239635996e-07, - "t_memory_s": 2.05824e-06, - "bound": "memory", - "flops": 301989888.0, - "bytes": 9879552.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 1, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 1, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 1, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 1, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 1, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 1, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 1, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 1, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 1, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "mlp_gate_up", - "layer": 1, - "t_pred_s": 4.03456e-06, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 4.03456e-06, - "bound": "memory", - "flops": 603979776.0, - "bytes": 19365888.0, - "estimated": false - }, - { - "op": "mlp_down", - "layer": 1, - "t_pred_s": 2.05824e-06, - "t_compute_s": 3.053487239635996e-07, - "t_memory_s": 2.05824e-06, - "bound": "memory", - "flops": 301989888.0, - "bytes": 9879552.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 2, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 2, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 2, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 2, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 2, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 2, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 2, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 2, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 2, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "mlp_gate_up", - "layer": 2, - "t_pred_s": 4.03456e-06, - "t_compute_s": 6.106974479271992e-07, - "t_memory_s": 4.03456e-06, - "bound": "memory", - "flops": 603979776.0, - "bytes": 19365888.0, - "estimated": false - }, - { - "op": "mlp_down", - "layer": 2, - "t_pred_s": 2.05824e-06, - "t_compute_s": 3.053487239635996e-07, - "t_memory_s": 2.05824e-06, - "bound": "memory", - "flops": 301989888.0, - "bytes": 9879552.0, - "estimated": false - }, - { - "op": "attn_q_a", - "layer": 3, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 3, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 3, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 3, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 3, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 3, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 3, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 3, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 3, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 3, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 3, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 3, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 4, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 4, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 4, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 4, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 4, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 4, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 4, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 4, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 4, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 4, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 4, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 4, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 5, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 5, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 5, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 5, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 5, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 5, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 5, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 5, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 5, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 5, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 5, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 5, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 6, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 6, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 6, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 6, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 6, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 6, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 6, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 6, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 6, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 6, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 6, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 6, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 6, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 6, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 7, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 7, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 7, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 7, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 7, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 7, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 7, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 7, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 7, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 7, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 7, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 7, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 8, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 8, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 8, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 8, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 8, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 8, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 8, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 8, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 8, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 8, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 8, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 8, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 9, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 9, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 9, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 9, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 9, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 9, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 9, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 9, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 9, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 9, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 9, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 9, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 10, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 10, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 10, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 10, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 10, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 10, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 10, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 10, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 10, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 10, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 10, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 10, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 10, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 10, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 11, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 11, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 11, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 11, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 11, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 11, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 11, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 11, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 11, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 11, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 11, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 11, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 12, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 12, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 12, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 12, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 12, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 12, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 12, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 12, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 12, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 12, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 12, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 12, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 13, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 13, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 13, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 13, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 13, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 13, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 13, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 13, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 13, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 13, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 13, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 13, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 14, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 14, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 14, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 14, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 14, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 14, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 14, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 14, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 14, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 14, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 14, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 14, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 14, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 14, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 15, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 15, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 15, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 15, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 15, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 15, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 15, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 15, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 15, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 15, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 15, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 15, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 16, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 16, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 16, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 16, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 16, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 16, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 16, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 16, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 16, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 16, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 16, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 16, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 17, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 17, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 17, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 17, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 17, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 17, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 17, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 17, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 17, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 17, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 17, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 17, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 18, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 18, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 18, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 18, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 18, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 18, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 18, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 18, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 18, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 18, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 18, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 18, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 18, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 18, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 19, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 19, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 19, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 19, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 19, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 19, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 19, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 19, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 19, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 19, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 19, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 19, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 20, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 20, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 20, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 20, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 20, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 20, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 20, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 20, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 20, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 20, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 20, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 20, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 21, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 21, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 21, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 21, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 21, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 21, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 21, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 21, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 21, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 21, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 21, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 21, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 22, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 22, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 22, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 22, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 22, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 22, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 22, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 22, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 22, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 22, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 22, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 22, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 22, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 22, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 23, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 23, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 23, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 23, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 23, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 23, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 23, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 23, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 23, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 23, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 23, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 23, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 24, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 24, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 24, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 24, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 24, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 24, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 24, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 24, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 24, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 24, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 24, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 24, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 25, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 25, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 25, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 25, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 25, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 25, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 25, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 25, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 25, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 25, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 25, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 25, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 26, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 26, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 26, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 26, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 26, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 26, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 26, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 26, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 26, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 26, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 26, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 26, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 26, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 26, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 27, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 27, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 27, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 27, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 27, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 27, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 27, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 27, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 27, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 27, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 27, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 27, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 28, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 28, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 28, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 28, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 28, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 28, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 28, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 28, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 28, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 28, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 28, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 28, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 29, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 29, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 29, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 29, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 29, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 29, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 29, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 29, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 29, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 29, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 29, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 29, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 30, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 30, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 30, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 30, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 30, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 30, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 30, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 30, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 30, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 30, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 30, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 30, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 30, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 30, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 31, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 31, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 31, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 31, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 31, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 31, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 31, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 31, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 31, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 31, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 31, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 31, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 32, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 32, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 32, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 32, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 32, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 32, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 32, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 32, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 32, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 32, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 32, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 32, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 33, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 33, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 33, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 33, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 33, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 33, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 33, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 33, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 33, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 33, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 33, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 33, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 34, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 34, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 34, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 34, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 34, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 34, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 34, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 34, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 34, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 34, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 34, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 34, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 34, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 34, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 35, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 35, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 35, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 35, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 35, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 35, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 35, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 35, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 35, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 35, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 35, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 35, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 36, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 36, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 36, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 36, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 36, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 36, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 36, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 36, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 36, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 36, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 36, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 36, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 37, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 37, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 37, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 37, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 37, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 37, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 37, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 37, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 37, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 37, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 37, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 37, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 38, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 38, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 38, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 38, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 38, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 38, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 38, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 38, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 38, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 38, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 38, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 38, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 38, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 38, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 39, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 39, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 39, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 39, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 39, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 39, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 39, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 39, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 39, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 39, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 39, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 39, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 40, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 40, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 40, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 40, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 40, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 40, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 40, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 40, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 40, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 40, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 40, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 40, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 41, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 41, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 41, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 41, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 41, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 41, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 41, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 41, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 41, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 41, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 41, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 41, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 42, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 42, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 42, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 42, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 42, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 42, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 42, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 42, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 42, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 42, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 42, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 42, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 42, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 42, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 43, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 43, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 43, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 43, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 43, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 43, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 43, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 43, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 43, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 43, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 43, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 43, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 44, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 44, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 44, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 44, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 44, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 44, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 44, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 44, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 44, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 44, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 44, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 44, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 45, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 45, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 45, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 45, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 45, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 45, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 45, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 45, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 45, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 45, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 45, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 45, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 46, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 46, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 46, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 46, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 46, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 46, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 46, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 46, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 46, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 46, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 46, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 46, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 46, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 46, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 47, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 47, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 47, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 47, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 47, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 47, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 47, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 47, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 47, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 47, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 47, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 47, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 48, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 48, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 48, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 48, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 48, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 48, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 48, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 48, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 48, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 48, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 48, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 48, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 49, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 49, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 49, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 49, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 49, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 49, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 49, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 49, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 49, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 49, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 49, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 49, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 50, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 50, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 50, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 50, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 50, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 50, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 50, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 50, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 50, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 50, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 50, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 50, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 50, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 50, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 51, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 51, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 51, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 51, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 51, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 51, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 51, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 51, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 51, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 51, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 51, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 51, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 52, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 52, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 52, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 52, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 52, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 52, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 52, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 52, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 52, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 52, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 52, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 52, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 53, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 53, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 53, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 53, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 53, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 53, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 53, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 53, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 53, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 53, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 53, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 53, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 54, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 54, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 54, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 54, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 54, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 54, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 54, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 54, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 54, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 54, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 54, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 54, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 54, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 54, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 55, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 55, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 55, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 55, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 55, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 55, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 55, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 55, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 55, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 55, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 55, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 55, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 56, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 56, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 56, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 56, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 56, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 56, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 56, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 56, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 56, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 56, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 56, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 56, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 57, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 57, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 57, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 57, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 57, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 57, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 57, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 57, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 57, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 57, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 57, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 57, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 58, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 58, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 58, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 58, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 58, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 58, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 58, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 58, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 58, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 58, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 58, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 58, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 58, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 58, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 59, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 59, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 59, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 59, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 59, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 59, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 59, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 59, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 59, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 59, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 59, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 59, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 60, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 60, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 60, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 60, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 60, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 60, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 60, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 60, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 60, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 60, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 60, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 60, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 61, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 61, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 61, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 61, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 61, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 61, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 61, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 61, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 61, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 61, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 61, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 61, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 62, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 62, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 62, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 62, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 62, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 62, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 62, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 62, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 62, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 62, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 62, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 62, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 62, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 62, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 63, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 63, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 63, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 63, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 63, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 63, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 63, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 63, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 63, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 63, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 63, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 63, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 64, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 64, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 64, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 64, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 64, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 64, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 64, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 64, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 64, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 64, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 64, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 64, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 65, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 65, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 65, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 65, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 65, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 65, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 65, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 65, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 65, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 65, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 65, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 65, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 66, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 66, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 66, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 66, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 66, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 66, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 66, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 66, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 66, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 66, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 66, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 66, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 66, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 66, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 67, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 67, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 67, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 67, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 67, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 67, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 67, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 67, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 67, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 67, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 67, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 67, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 68, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 68, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 68, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 68, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 68, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 68, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 68, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 68, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 68, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 68, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 68, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 68, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 69, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 69, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 69, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 69, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 69, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 69, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 69, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 69, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 69, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 69, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 69, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 69, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 70, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 70, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 70, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 70, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 70, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 70, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 70, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 70, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 70, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 70, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 70, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 70, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 70, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 70, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 71, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 71, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 71, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 71, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 71, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 71, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 71, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 71, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 71, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 71, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 71, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 71, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 72, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 72, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 72, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 72, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 72, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 72, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 72, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 72, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 72, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 72, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 72, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 72, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 73, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 73, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 73, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 73, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 73, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 73, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 73, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 73, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 73, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 73, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 73, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 73, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 74, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 74, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 74, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 74, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_index_proj", - "layer": 74, - "t_pred_s": 3.5771733333333333e-06, - "t_compute_s": 5.428421759352882e-07, - "t_memory_s": 3.5771733333333333e-06, - "bound": "memory", - "flops": 536870912.0, - "bytes": 17170432.0, - "estimated": false - }, - { - "op": "attn_index_score", - "layer": 74, - "t_pred_s": 1.3981013333333332e-05, - "t_compute_s": 2.1713687037411527e-06, - "t_memory_s": 1.3981013333333332e-05, - "bound": "memory", - "flops": 2147483648.0, - "bytes": 67108864.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 74, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 74, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 74, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 74, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 74, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 74, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 74, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 74, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 75, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 75, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 75, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 75, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 75, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 75, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 75, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 75, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 75, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 75, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 75, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 75, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 76, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 76, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 76, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 76, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 76, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 76, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 76, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 76, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 76, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 76, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 76, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 76, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 77, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 77, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 77, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 77, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 77, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 77, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 77, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 77, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 77, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 77, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 77, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 77, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "attn_q_a", - "layer": 78, - "t_pred_s": 5.352106666666667e-06, - "t_compute_s": 8.142632639029323e-07, - "t_memory_s": 5.352106666666667e-06, - "bound": "memory", - "flops": 805306368.0, - "bytes": 25690112.0, - "estimated": false - }, - { - "op": "attn_q_b", - "layer": 78, - "t_pred_s": 9.147733333333334e-07, - "t_compute_s": 1.3571054398382204e-07, - "t_memory_s": 9.147733333333334e-07, - "bound": "memory", - "flops": 134217728.0, - "bytes": 4390912.0, - "estimated": false - }, - { - "op": "attn_kv_a", - "layer": 78, - "t_pred_s": 1.57184e-06, - "t_compute_s": 2.290115429726997e-07, - "t_memory_s": 1.57184e-06, - "bound": "memory", - "flops": 226492416.0, - "bytes": 7544832.0, - "estimated": false - }, - { - "op": "attn_kv_b", - "layer": 78, - "t_pred_s": 4.1301333333333334e-07, - "t_compute_s": 5.9373362992922145e-08, - "t_memory_s": 4.1301333333333334e-07, - "bound": "memory", - "flops": 58720256.0, - "bytes": 1982464.0, - "estimated": false - }, - { - "op": "attn_score_value", - "layer": 78, - "t_pred_s": 1.572864e-05, - "t_compute_s": 2.714210879676441e-07, - "t_memory_s": 1.572864e-05, - "bound": "memory", - "flops": 268435456.0, - "bytes": 75497472.0, - "estimated": false - }, - { - "op": "attn_qnorm_rope_insert", - "layer": 78, - "t_pred_s": 5.12e-08, - "t_compute_s": 2.1743174924165824e-10, - "t_memory_s": 5.12e-08, - "bound": "memory", - "flops": 215040.0, - "bytes": 245760.0, - "estimated": false - }, - { - "op": "attn_out_proj", - "layer": 78, - "t_pred_s": 2.7170133333333333e-06, - "t_compute_s": 4.0713163195146613e-07, - "t_memory_s": 2.7170133333333333e-06, - "bound": "memory", - "flops": 402653184.0, - "bytes": 13041664.0, - "estimated": false - }, - { - "op": "moe_router", - "layer": 78, - "t_pred_s": 7.406933333333333e-07, - "t_compute_s": 1.0178290798786653e-07, - "t_memory_s": 7.406933333333333e-07, - "bound": "memory", - "flops": 100663296.0, - "bytes": 3555328.0, - "estimated": false - }, - { - "op": "moe_shared", - "layer": 78, - "t_pred_s": 1.1502933333333334e-06, - "t_compute_s": 1.526743619817998e-07, - "t_memory_s": 1.1502933333333334e-06, - "bound": "memory", - "flops": 150994944.0, - "bytes": 5521408.0, - "estimated": false - }, - { - "op": "moe_routed", - "layer": 78, - "t_pred_s": 0.00016073518978973114, - "t_compute_s": 1.2213948958543984e-06, - "t_memory_s": 0.00016073518978973114, - "bound": "memory", - "flops": 1207959552.0, - "bytes": 771528910.9907094, - "estimated": false - }, - { - "op": "moe_all_to_all", - "layer": 78, - "t_pred_s": 6.5536e-06, - "t_compute_s": 0.0, - "t_memory_s": 6.5536e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 5898240.0, - "estimated": true - }, - { - "op": "tp_all_reduce", - "layer": 78, - "t_pred_s": 1.6384e-06, - "t_compute_s": 0.0, - "t_memory_s": 1.6384e-06, - "bound": "memory", - "flops": 0.0, - "bytes": 1474560.0, - "estimated": true - }, - { - "op": "lm_head", - "layer": null, - "t_pred_s": 2.4991786666666668e-05, - "t_compute_s": 3.8486662082912035e-06, - "t_memory_s": 2.4991786666666668e-05, - "bound": "memory", - "flops": 3806330880.0, - "bytes": 119960576.0, - "estimated": false - } - ] -} diff --git a/gitm/planner/context.py b/gitm/planner/context.py index 70fd7df..8bcdd33 100644 --- a/gitm/planner/context.py +++ b/gitm/planner/context.py @@ -71,6 +71,42 @@ } +# CUDA-core FP32 peaks (FLOP/s), same substring keys. Not a tensor-core rate: +# an fp32 op in a graph is there because the *model* asked for fp32 — a MoE router +# under ``moe_router_dtype: "float32"``, a softmax accumulation — and those run on +# the FP32 pipe, not on the tensor cores. Vendor "TF32 tensor core" figures are an +# order of magnitude higher and pricing a router against one would make the node +# disappear from the table. +# +# A SKU absent here keeps the dataclass default (an A100's 19.5 TF/s), which is +# low for anything newer and so under-reports rather than over-reports headroom — +# but on an H200 it is 3.4x low, which is enough to move a small fp32 node's bound +# label, so the SKUs this planner actually targets are listed. +_FP32_PEAKS: dict[str, float] = { + "GB300": 80e12, + "B300": 80e12, + "GB200": 80e12, + "B200": 80e12, + "H100": 67e12, + "H200": 67e12, + "A100": 19.5e12, + "L40": 90e12, + "L4": 30e12, + "T4": 8.1e12, + "V100": 15.7e12, +} + + +def fp32_peak_for_sku(sku: str | None) -> float: + """CUDA-core FP32 peak for a SKU (substring match), else ``0.0``.""" + if not sku: + return 0.0 + for key, peak in _FP32_PEAKS.items(): + if key.lower() in sku.lower(): + return peak + return 0.0 + + # Per-GPU bidirectional NVLink bandwidth (bytes/s), same substring keys. Used to # price the collectives a sharded graph emits. A SKU absent here leaves the spec # at 0.0, which makes the sharded planner report collectives as unpriced instead @@ -162,16 +198,20 @@ def hardware_spec_for(peak: HardwarePeak | None) -> HardwareSpec: ``peak_flops`` covers fp16/bf16; fp8/fp4 come from ``_QUANT_PEAKS`` when the SKU has them, and stay ``0.0`` otherwise so ``resolve_peak`` can fall back and mark the prediction rather than pricing an fp4 GEMM at the bf16 rate. - fp32 peak isn't in the catalogue and stays at the dataclass default since - nothing currently predicts fp32 kernels. + The fp32 peak comes from ``_FP32_PEAKS`` — the CUDA-core rate, since a model + that declares an fp32 op (a MoE router under ``moe_router_dtype``) runs it on + that pipe. A SKU without an entry keeps the dataclass default. """ if peak is None: return HardwareSpec() quant = quant_peaks_for_sku(peak.name) + fp32 = fp32_peak_for_sku(peak.name) + defaults = HardwareSpec() return HardwareSpec( name=peak.name, peak_flops_fp16_per_s=peak.peak_flops, peak_flops_bf16_per_s=peak.peak_flops, + peak_flops_fp32_per_s=fp32 or defaults.peak_flops_fp32_per_s, peak_flops_fp8_per_s=quant.get("fp8", 0.0), peak_flops_fp4_per_s=quant.get("fp4", 0.0), peak_mem_bw_bytes_per_s=peak.peak_bw_bytes_s, diff --git a/gitm/planner/model_catalogue.py b/gitm/planner/model_catalogue.py index b3cf7e7..3b272b3 100644 --- a/gitm/planner/model_catalogue.py +++ b/gitm/planner/model_catalogue.py @@ -118,6 +118,15 @@ def load_spec(name_or_path: str | Path): for key in ("compress_ratios", "dspark_layer_ids", "indexer_types", "mlp_layer_types"): if key in raw and isinstance(raw[key], list): raw[key] = tuple(raw[key]) + # Nested one level: ``op_dtype_overrides`` is a list of ``[op, dtype]`` pairs + # in YAML and must reach the frozen spec as a tuple of tuples. Coercing only + # the outer list would leave inner lists inside a frozen dataclass — hashable + # in appearance, not in fact. + if isinstance(raw.get("op_dtype_overrides"), list): + raw["op_dtype_overrides"] = tuple( + tuple(pair) if isinstance(pair, list) else pair + for pair in raw["op_dtype_overrides"] + ) unknown = set(raw) - known if unknown: diff --git a/gitm/planner/models/glm-5.2-fp8.yaml b/gitm/planner/models/glm-5.2-fp8.yaml new file mode 100644 index 0000000..225c6ca --- /dev/null +++ b/gitm/planner/models/glm-5.2-fp8.yaml @@ -0,0 +1,306 @@ +# zai-org/GLM-5.2-FP8 — the vendor's recommended deployment shape. +# Source: https://huggingface.co/zai-org/GLM-5.2-FP8/blob/main/config.json +# https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json +# Read from config.json and model.safetensors.index.json only — no traces. +# +# Identical architecture to `glm-5.2`; the difference is precision, and precision +# is the difference between a model that fits one 8xH200 node and one that does +# not. 753.33 GB on disk against the bf16 release's 1.507 TB. + +name: zai-org/GLM-5.2-FP8 +family: glm_moe_dsa +description: > + GLM-5.2 quantised to FP8 e4m3 with 128x128 weight blocks — 78 layers, hidden + 6144, 256 routed experts top-8, MLA + DeepSeek Sparse Attention with + IndexShare (21 of 78 layers compute the index). The backbone GEMMs, the dense + FFN, the shared expert and all routed experts are fp8; lm_head, embed_tokens, + the MTP eh_proj and the lightning indexer stay bf16; the router is fp32. This + is the entry to plan against — the vendor recommends it and it is what fits + 8xH200. + +spec: + hidden: 6144 + n_layers: 78 + vocab: 154880 + + # MLA attention. head_dim (192) is the nope width; q_head_dim widens to + # 192+64=256, and v_head_dim (256) differs from it — score and value reads use + # different per-head widths. num_key_value_heads is 64 in the config but is a + # red herring: the cache holds one kv_lora_rank latent per token, shared across + # all 64 query heads, so KV traffic derives from kv_lora_rank, never 64*head_dim. + n_heads: 64 + q_lora_rank: 2048 + kv_lora_rank: 512 + qk_nope_head_dim: 192 + qk_rope_head_dim: 64 + v_head_dim: 256 + + # DeepSeek Sparse Attention indexer. index_topk_freq 4 is the IndexShare period: + # one 'full' layer computes the top-2048 selection, the next three reuse it. + index_n_heads: 32 + index_head_dim: 128 + index_topk: 2048 + index_topk_freq: 4 + # Read verbatim from the checkpoint (period-4 past the 3-layer 'full' prefix). + # The 'shared' layers physically carry no indexer weights — proven, not fitted. + indexer_types: + - full + - full + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + - full + - shared + - shared + - shared + + # Mixture of experts. moe_intermediate_size 2048 per expert; the first 3 layers + # are dense FFN (intermediate_size 12288) with no router or experts. + n_routed_experts: 256 + n_shared_experts: 1 + num_experts_per_tok: 8 + moe_intermediate_size: 2048 + intermediate_size: 12288 + first_k_dense_replace: 3 + routed_scaling_factor: 2.5 + mlp_layer_types: + - dense + - dense + - dense + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + - sparse + + # Multi-token prediction. One MTP module, invoked once per drafted token; the + # vendor recipe runs it 5 deep (--speculative-config.num_speculative_tokens 5). + # index_share_for_mtp_iteration: the draft reuses the main model's selection, + # and the weight map agrees — the MTP block carries no indexer tensors. + num_nextn_predict_layers: 1 + index_share_for_mtp_iteration: true + + # FP8 e4m3, 128x128 block-scaled, dynamic activation scaling. + weight_dtype: fp8 + expert_dtype: fp8 + act_dtype: bf16 + # A serving choice, not a model fact — but it is the one the vendor recipe + # makes (`--kv-cache-dtype fp8`), and it halves the latent half of every cached + # entry. The decoupled RoPE key stays bf16 either way; kv_entry_bytes prices + # the two halves separately for exactly this reason. + kv_dtype: fp8 + + # The ops the quantiser skipped, from quantization_config.modules_to_not_convert, + # plus the router's own dtype from the base config. Without these the indexer — + # the one attention node whose cost grows with context — is priced at half its + # real weight traffic, and lm_head at half of 154,880 x 6,144. + op_dtype_overrides: + - [attn_index_proj, bf16] + - [lm_head, bf16] + - [mtp_eh_proj, bf16] + - [moe_router, fp32] + +provenance: + verified: + - claim: predicted weight bytes match the published FP8 checkpoint + detail: > + model.safetensors totals 753,329,940,480 B (753.33 GB) across 141 shards. + model_weight_bytes predicts 755.9 GB — +0.34%, and the same spec at bf16 + predicts 1.5079 TB against the bf16 release's 1,506,659,919,872 B (+0.08%). + Two checkpoints at two precisions agreeing to under half a percent is a + stronger check on the shape arithmetic than either one alone. + - claim: the unquantised op list is read, not assumed + detail: > + quantization_config.modules_to_not_convert names lm_head, embed_tokens, + eh_proj, enorm/hnorm, every layernorm, mlp.gate (+ e_score_correction_bias) + and the indexer's projections and k_norm. o_proj is absent from that list — + it IS quantised here, the opposite of the fp8-backbone checkpoints that + keep the output projection wide. + - claim: the router is fp32 on every variant + detail: > + moe_router_dtype "float32" is a field of the base GLM-5.2 config, not of + the quantisation config, so it is a model fact and applies to the bf16 + release too. + - claim: IndexShare is read from the weight map + detail: > + Indexer tensors (*.indexer.wq_b / .wk / .weights_proj / .k_norm) exist on + exactly the 21 layers whose indexer_types is 'full' and on none of the 57 + 'shared' layers — nor on the MTP block, which is what + index_share_for_mtp_iteration=true predicts. + + estimated: + - field: kv_dtype + value: fp8 + detail: > + A deployment choice, taken from the vendor's own vLLM recipe + (`--kv-cache-dtype fp8`), not from the checkpoint. Set it back to bf16 to + price the conservative cache; on this architecture that moves the latent + half of a 576-element entry, not the RoPE half. + - field: MLA absorption (attn_kv_b vs attn_out_proj width) + value: unabsorbed — kv_b runs as its own GEMM, o_proj narrow + detail: > + Modelled unabsorbed: attn_kv_b reconstructs per-head K/V from the latent + and attn_out_proj is n_heads*v_head_dim -> hidden (16384->6144). An engine + that absorbs MLA folds W^UK into the query and W^UV into the output, + dropping attn_kv_b and doubling attn_out_proj's input width to + n_heads*kv_lora_rank (32768->6144). Same resident weights either way; the + two readings move attn_out_proj by 2x, so it re-ranks the attention side. + - field: expert-parallel imbalance + value: 1.0 (perfect balance) + detail: > + Trace-calibrated by design. Declared rather than fitted while the + no-traces constraint holds; real skew moves less weight traffic than + predicted (fewer distinct experts) but lengthens the grouped-GEMM tail. + + unmodelled: + - Absorbed-MLA decode. See the estimated entry above; it is the one open + question that re-ranks the attention side by 2x in either direction. + - Acceptance rate under MTP. The graph prices D drafts and a 1+D verify; what + fraction is kept is a serving observable, carried in BatchConfig rather than + predicted here. + - index_topk_freq as a *temporal* reuse across decode steps. Modelled as the + spatial layer-group period, which the weight map proves; any additional + step-to-step reuse would only reduce indexer cost further. + - Expert-capacity padding. If the grouped-GEMM backend pads to a fixed + capacity to keep static shapes, every step reads all 256 experts and the + distinct_experts union term is an underestimate at low batch — the fork to + settle with one D2H count. diff --git a/gitm/planner/models/glm-5.2.yaml b/gitm/planner/models/glm-5.2.yaml index b89e7c5..dcc5632 100644 --- a/gitm/planner/models/glm-5.2.yaml +++ b/gitm/planner/models/glm-5.2.yaml @@ -126,18 +126,31 @@ spec: first_k_dense_replace: 3 routed_scaling_factor: 2.5 - # Multi-token prediction. index_share_for_mtp_iteration: the draft reuses the - # main model's index rather than recomputing it. + # Multi-token prediction. One MTP module, invoked once per drafted token + # (EAGLE-style); the vendor recipe runs it 5 deep. index_share_for_mtp_iteration: + # the draft reuses the main model's selection, and the weight map agrees — the + # MTP block carries no indexer tensors, exactly like the 57 'shared' layers. num_nextn_predict_layers: 1 index_share_for_mtp_iteration: true - # bf16 checkpoint — no quantization_config published. fp8/fp4 is a serving - # decision, applied at deploy time, not a model fact. + # The bf16 release: no quantization_config, and 1.507 TB on disk matches an + # all-bf16 read. This is the model fact. It is NOT the deployment shape — the + # vendor ships and recommends zai-org/GLM-5.2-FP8 (753.33 GB), which is the + # `glm-5.2-fp8` entry beside this one and the one to plan against. Keep this + # entry for what it answers: what the unquantised model costs, and therefore + # what quantisation is actually worth. weight_dtype: bf16 expert_dtype: bf16 kv_dtype: bf16 act_dtype: bf16 + # fp32 even here. moe_router_dtype is a field of the base config, not of any + # quantisation config, so the router's precision is a property of the model and + # not of the deployment — the one op that does not follow weight_dtype on an + # otherwise uniformly-bf16 checkpoint. + op_dtype_overrides: + - [moe_router, fp32] + provenance: verified: - claim: IndexShare is read from the weight map, not inferred @@ -178,11 +191,9 @@ provenance: settle against a capture: does the engine run absorbed MLA? unmodelled: - - Prefill. Decode only, as with every other graph family. Chunked prefill runs - a different indexer path (score-and-select over the whole prompt) with - different asymptotics. - index_topk_freq as a *temporal* reuse across decode steps. Modelled here as the spatial layer-group period (which the weight map proves); any additional step-to-step reuse would only reduce indexer cost further. - - Serving quantisation (fp8 KV cache, fp8/fp4 experts). Priced at bf16 — the - published checkpoint's precision. A quantised deploy is a separate spec. + - Absorbed-MLA decode. See the estimated entry above. + - Acceptance rate under MTP. The graph prices D drafts and a 1+D verify; the + fraction kept is a serving observable, not a config-derivable one. diff --git a/gitm/planner/registry.py b/gitm/planner/registry.py index ef3c9c6..3c40b88 100644 --- a/gitm/planner/registry.py +++ b/gitm/planner/registry.py @@ -2,6 +2,7 @@ import argparse import json +from collections import Counter from pathlib import Path from typing import Any @@ -120,6 +121,9 @@ def add_plan_arguments(ap: argparse.ArgumentParser) -> argparse.ArgumentParser: help="Context already cached before this chunk (0 for a first chunk).") ap.add_argument("--prefill-requests", type=int, default=1, help="How many prompts those tokens belong to — sets lm_head rows.") + ap.add_argument("--spec-tokens", type=int, default=0, + help="Speculative (MTP) draft tokens per step. Adds a D-deep " + "draft chain and makes the backbone a 1+D-row verify.") ap.add_argument("--tp", type=int, default=1, help="Tensor-parallel size.") ap.add_argument("--ep", type=int, default=1, help="Expert-parallel size.") ap.add_argument("--dp", type=int, default=1, help="Data-parallel size.") @@ -189,6 +193,7 @@ def _predict(spec, family: str, hw, batch, sharding): def _render_table(g, hw: HardwareSpec, spec, family: str, note: str) -> str: agg: dict[str, list[float]] = {} + bounds: dict[str, Counter[str]] = {} for n in g.nodes: p = n.prediction a = agg.setdefault(n.op, [0, 0.0, 0.0, 0.0, 0.0, 0.0]) @@ -198,10 +203,20 @@ def _render_table(g, hw: HardwareSpec, spec, family: str, note: str) -> str: a[3] += p.t_memory_s a[4] += p.flops a[5] += p.bytes + bounds.setdefault(n.op, Counter())[p.bound] += 1 total = g.total_pred_s - ridge = (hw.peak_flops_bf16_per_s / hw.peak_mem_bw_bytes_per_s - if hw.peak_mem_bw_bytes_per_s else 0.0) + # One ridge per *dtype in the graph*, not one for the model. A checkpoint that + # runs fp8 GEMMs against a bf16 ridge is measured against a ceiling half its + # own: on an H200 that is 412 FLOP/byte answering to 206, which moves the + # bound label of every op sitting between them. + dtypes = {n.prediction.peak_dtype: n.prediction.peak_flops_per_s for n in g.nodes} + ridges = { + d: peak / hw.peak_mem_bw_bytes_per_s + for d, peak in sorted(dtypes.items()) + if peak > 0 and hw.peak_mem_bw_bytes_per_s + } + ridge_line = " · ".join(f"{d} {r:.0f}" for d, r in ridges.items()) or "unpriced" out = [ f"model {getattr(spec, 'name', '?')} [{family}]", @@ -209,20 +224,29 @@ def _render_table(g, hw: HardwareSpec, spec, family: str, note: str) -> str: f"hardware {hw.name} " f"{hw.peak_flops_bf16_per_s / 1e12:.0f} TFLOP/s bf16, " f"{hw.peak_mem_bw_bytes_per_s / 1e12:.2f} TB/s", - f"ridge {ridge:.0f} FLOP/byte — a node below this is memory-bound", + f"ridge {ridge_line} FLOP/byte — a node below its own dtype's " + "ridge is memory-bound", "", f" {'op':24s} {'xN':>4s} {'t_pred':>9s} {'share':>7s} " f"{'t_comp':>9s} {'t_mem':>9s} {'AI':>7s} bound", ] for op, (n, tp, tc, tm, fl, by) in sorted(agg.items(), key=lambda kv: -kv[1][1]): ai = fl / by if by else 0.0 - bound = "compute" if tc >= tm else "memory" + # The node's own label, not a recomputed compute-vs-memory one. The + # roofline has three bounds and the third — ``launch`` — is the whole + # low-batch story on a sparse model: hundreds of pointwise and routing + # kernels whose wall time is the launch, not the bytes. Recomputing here + # silently relabelled every one of them as memory-bound. + bound = bounds[op].most_common(1)[0][0] + if len(bounds[op]) > 1: + bound = f"{bound}*" # the op's instances do not agree out.append( f" {op:24s} {int(n):4d} {tp * 1e3:8.3f}m {tp / total:6.1%} " f"{tc * 1e3:8.3f}m {tm * 1e3:8.3f}m {ai:7.1f} {bound}" ) n_compute = sum(1 for n in g.nodes if n.prediction.bound == "compute") + n_launch = sum(1 for n in g.nodes if n.prediction.bound == "launch") out += [ "", f" floor {total * 1e3:.3f} ms/step " + ( @@ -231,8 +255,12 @@ def _render_table(g, hw: HardwareSpec, spec, family: str, note: str) -> str: if g.batch.is_prefill else f"{g.batch.batch / total:,.0f} tok/s at batch {g.batch.batch}" ), - f" {len(g.nodes)} nodes, {n_compute} compute-bound", + f" {len(g.nodes)} nodes, {n_compute} compute-bound, " + f"{n_launch} launch-bound", ] + if any(b for b in bounds.values() if len(b) > 1): + out.append(" * this op's instances do not share a bound — the label is " + "the majority one") if g.has_unpriced_collectives: out.append(" ! collectives unpriced — this SKU has no interconnect bandwidth " "in the catalogue") @@ -298,7 +326,9 @@ def main(argv: list[str] | None = None) -> int: f"kv_len={args.kv_len}, TP={args.tp} EP={args.ep}") print(f" {'batch':>7s} {'ms/step':>10s} {'tok/s':>12s} {'compute-bound':>14s}") for b in sizes: - g = _predict(spec, family, hw, BatchConfig(batch=b, kv_cache_len=args.kv_len), + g = _predict(spec, family, hw, + BatchConfig(batch=b, kv_cache_len=args.kv_len, + speculative_tokens=args.spec_tokens), sharding) cb = sum(1 for n in g.nodes if n.prediction.bound == "compute") print(f" {b:7d} {g.total_pred_s * 1e3:9.3f} " @@ -307,6 +337,7 @@ def main(argv: list[str] | None = None) -> int: batch = BatchConfig( batch=args.batch, kv_cache_len=args.kv_len, + speculative_tokens=args.spec_tokens, prefill_tokens=args.prefill_tokens, prefill_context=args.prefill_context, prefill_requests=args.prefill_requests, ) @@ -323,6 +354,7 @@ def main(argv: list[str] | None = None) -> int: "hardware": hw.name, "sharding": {"tp": args.tp, "ep": args.ep, "dp": args.dp}, "batch": {"batch": args.batch, "kv_cache_len": args.kv_len, + "speculative_tokens": args.spec_tokens, "prefill_tokens": args.prefill_tokens, "prefill_context": args.prefill_context, "prefill_requests": args.prefill_requests}, diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index 3addfe2..4d75b58 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -8,8 +8,13 @@ the selection and carry no indexer weights, * MLA KV traffic scales with the shared latent, never ``n_heads``, * the dense prefix runs an FFN, not a mixture, -* precision is bf16 (no fp4 experts leaking in from the V4 defaults), -* the predicted footprint matches the published 1.507 TB checkpoint, +* precision is read per op, not per model — bf16 by default, and on the FP8 + checkpoint fp8 everywhere the quantiser went and bf16 where it did not, +* the predicted footprint matches *both* published checkpoints, 1.507 TB bf16 and + 753.33 GB fp8, +* prefill and decode disagree about what ``index_topk`` buys — it bounds the core + in both phases but bounds the *bytes* in only one, +* the MTP chain is D stages deep with D vocabulary projections, not one of each, * ``detect_family`` routes ``glm_moe_dsa`` before the structural sparse-MoE test. """ @@ -17,8 +22,11 @@ import pytest +from dataclasses import replace + from gitm.planner.glm_graph import ( GlmMoeDsaModelSpec, + core_read_entries, is_glm_moe_dsa_config, kv_entry_bytes, model_weight_bytes, @@ -61,6 +69,7 @@ "first_k_dense_replace": 3, "mlp_layer_types": ["dense", "dense", "dense"] + ["sparse"] * 75, "routed_scaling_factor": 2.5, + "moe_router_dtype": "float32", "num_nextn_predict_layers": 1, "index_share_for_mtp_iteration": True, "dtype": "bfloat16", @@ -129,6 +138,176 @@ def test_precision_is_bf16_no_fp4_leak(): assert spec.weight_dtype == "bf16" assert spec.expert_dtype == "bf16" assert spec.kv_dtype == "bf16" + # fp32 even on the unquantised checkpoint: moe_router_dtype is a base-config + # field, so the router's precision is a model fact, not a deployment one. + assert spec.dtype_for("moe_router", spec.weight_dtype) == "fp32" + + +def test_fp8_checkpoint_reads_what_the_quantiser_skipped(): + """One dtype per model is a fiction here; the checkpoint says which ops differ. + + ``modules_to_not_convert`` is the authority, and the interesting entries are + the ones that invert the usual fp8-backbone layout: ``o_proj`` is quantised + (absent from the list) while the *indexer* is not. + """ + cfg = dict(GLM_CONFIG) + cfg["quantization_config"] = { + "quant_method": "fp8", + "fmt": "e4m3", + "weight_block_size": [128, 128], + "modules_to_not_convert": [ + "model.layers.0.input_layernorm", + "model.layers.47.mlp.gate.e_score_correction_bias", + "model.layers.74.self_attn.indexer.k_norm", + "model.layers.78.eh_proj", + "lm_head", + ], + } + spec = spec_from_hf_config(cfg, name="GLM-5.2-FP8") + assert spec.weight_dtype == "fp8" and spec.expert_dtype == "fp8" + # Quantised: the backbone GEMMs and the experts. + for op in ("attn_q_b", "attn_kv_b", "attn_out_proj", "moe_routed"): + assert spec.dtype_for(op, spec.weight_dtype) == "fp8", op + # Skipped: the vocabulary projection, the MTP fusion, and the indexer. + for op in ("lm_head", "mtp_eh_proj", "attn_index_proj"): + assert spec.dtype_for(op, spec.weight_dtype) == "bf16", op + assert spec.dtype_for("moe_router", spec.weight_dtype) == "fp32" + + +def test_fp8_footprint_matches_published_checkpoint(): + """The same shape arithmetic, checked against a second published precision. + + Two checkpoints agreeing to under half a percent is a stronger check than + either alone: an error in the shape would have to be precision-proportional + to survive both. + """ + published_fp8 = 753_329_940_480 # 141 shards, zai-org/GLM-5.2-FP8 + spec = replace( + _spec(), weight_dtype="fp8", expert_dtype="fp8", + op_dtype_overrides=( + ("attn_index_proj", "bf16"), ("lm_head", "bf16"), + ("mtp_eh_proj", "bf16"), ("moe_router", "fp32"), + ), + ) + assert abs(model_weight_bytes(spec) / published_fp8 - 1.0) < 0.01 + # And the overrides are load-bearing, not decorative: pricing lm_head and the + # indexer at fp8 loses ~1 GB of real resident weight. + naive = replace(spec, op_dtype_overrides=()) + assert model_weight_bytes(spec) - model_weight_bytes(naive) > 1e9 + + +def test_prefill_core_streams_the_cache_that_decode_only_samples(): + """``index_topk`` bounds the core's FLOPs in both phases — its bytes in one. + + At decode a sequence reads its own top-2048 selection. At prefill every query + in the chunk selects a different top-2048 and their union is the whole + history, so the kernel streams the entire cache. A prefill path copied from a + dense family would charge ``P x index_topk`` here and understate long-context + prefill traffic by the ratio of context to 2,048. + """ + spec = _spec() + ctx = 65536 + dec = BatchConfig(batch=1, kv_cache_len=ctx) + pre = BatchConfig(batch=1, kv_cache_len=ctx, prefill_tokens=4096, + prefill_context=ctx, prefill_requests=1) + assert core_read_entries(spec, dec) == spec.index_topk + # Prefill adds the whole history once per request, not another top-k window. + assert core_read_entries(spec, pre) == spec.index_topk + ctx + 4096 + + +def test_prefill_scales_projections_by_chunk_but_not_the_epilogue(): + """Rows and logits rows are different numbers, and lm_head follows the second.""" + spec = _spec() + g = predict_glm_graph( + spec, + batch=BatchConfig(batch=1, kv_cache_len=4096, prefill_tokens=8192, + prefill_requests=2), + ) + dec = predict_glm_graph(spec, batch=BatchConfig(batch=1, kv_cache_len=4096)) + + def flops(graph, op, layer=None): + return sum( + n.prediction.flops for n in graph.nodes + if n.op == op and (layer is None or n.layer == layer) + ) + + # A backbone projection scales with every row in the step: 1 decode position + # plus the 8,192-token chunk riding along with it. Read off one layer — the + # draft stage in the same graph runs at one row and no prefill, which is the + # point of keeping the two row counts apart. + assert flops(g, "attn_q_a", layer=3) == pytest.approx( + 8193 * flops(dec, "attn_q_a", layer=3) + ) + # The vocabulary projection scales with rows that need logits: 1 per + # prefilling request plus the decode position, so 3 — not 8193. Charging the + # chunk here is the largest single error available on this path. + epi_pre = [n for n in g.nodes if n.op == "lm_head" and n.layer is None] + epi_dec = [n for n in dec.nodes if n.op == "lm_head" and n.layer is None] + assert epi_pre[0].prediction.flops == pytest.approx( + 3 * epi_dec[0].prediction.flops, rel=1e-6 + ) + + +def test_mtp_chain_is_d_deep_with_its_own_vocab_projection(): + """Verify is the backbone at 1+D rows; the draft is D serial stages. + + ``num_nextn_predict_layers`` is 1 — one *module*, invoked once per drafted + token. Emitting one draft block and one lm_head for a D-deep chain understates + the draft by D, and the vocabulary projection is the majority of its bytes. + """ + spec = _spec() + d = 5 # the vendor recipe's --speculative-config.num_speculative_tokens + g = predict_glm_graph( + spec, batch=BatchConfig(batch=8, kv_cache_len=8192, speculative_tokens=d) + ) + # One epilogue projection for the verify pass, plus one per draft stage. + assert len(_ops(g, "lm_head")) == 1 + d + assert len(_ops(g, "mtp_eh_proj")) == d + # The backbone still runs once — at 1+D rows, not 1+D times. + assert len(_ops(g, "attn_q_a")) == spec.n_layers + d + + # The draft carries no indexer: index_share_for_mtp_iteration is true and the + # MTP block has no indexer tensors in the weight map. + assert len(_ops(g, "attn_index_proj")) == spec.n_full_indexer_layers + + # And the draft is not a small copy of the model: its expert bank is a full + # 256-expert mixture, so the chain's cost is weight traffic paid D times. + assert len(_ops(g, "moe_routed")) == spec.n_sparse_mlp_layers + d + + +def test_two_collectives_per_layer_not_one(): + """A TP layer all-reduces after o_proj and again after the FFN combine. + + Folding them into one node with double the payload gets the bytes right and + the count wrong — and at decode payloads a collective is bounded by its ring + latency, so the count is the cost. + """ + spec = _spec() + g = predict_glm_graph( + spec, batch=BatchConfig(batch=1, kv_cache_len=4096), + sharding=ShardingConfig(tp=8), + ) + n_blocks = spec.n_layers + spec.num_nextn_predict_layers + assert len(_ops(g, "tp_all_reduce_attn")) == n_blocks + assert len(_ops(g, "tp_all_reduce_mlp")) == n_blocks + + +def test_unpriced_collectives_stay_visible_under_the_launch_floor(): + """A launch floor must not quietly price a collective the SKU cannot price. + + ``has_unpriced_collectives`` detects a node that moves bytes in zero time. If + every collective carried a 2 us launch cost, an SKU with no interconnect + bandwidth would report a priced graph and credit a sharded deployment with a + nearly-free all-reduce. + """ + from gitm.planner.roofline import HardwareSpec + + g = predict_glm_graph( + _spec(), HardwareSpec(interconnect_bw_bytes_per_s=0.0), + batch=BatchConfig(batch=1, kv_cache_len=4096), + sharding=ShardingConfig(tp=8), + ) + assert g.has_unpriced_collectives def test_footprint_matches_published_checkpoint(): @@ -151,8 +330,20 @@ def total(g, op): assert total(long, "attn_score_value") == pytest.approx( total(short, "attn_score_value"), rel=1e-9 ) - # The scan scores the whole history, so it grows. - assert total(long, "attn_index_score") > 10 * total(short, "attn_index_score") + # The scan scores the whole history, so its *work* grows with context — 32x + # from 4K to 128K. Asserted on bytes rather than on time: at 4K the scan moves + # a megabyte across 21 layers and is bounded by its kernel launches, not by + # its bytes, so predicted time there is a launch floor and cannot grow 32x. + # That floor is a real property of the node, not an artefact to assert around. + def total_bytes(g, op): + return sum(n.prediction.bytes for n in g.nodes if n.op == op) + + assert total_bytes(long, "attn_index_score") == pytest.approx( + 32 * total_bytes(short, "attn_index_score") + ) + assert total(long, "attn_index_score") > 5 * total(short, "attn_index_score") + short_scan = [n for n in short.nodes if n.op == "attn_index_score"] + assert all(n.prediction.bound == "launch" for n in short_scan) def test_detect_family_routes_glm_before_sparse_moe(): @@ -161,15 +352,34 @@ def test_detect_family_routes_glm_before_sparse_moe(): assert is_glm_moe_dsa_config(GLM_CONFIG) -def test_catalogue_entry_loads_and_predicts(): - assert "glm-5.2" in available() - spec = load_spec("glm-5.2") +@pytest.mark.parametrize("entry", ["glm-5.2", "glm-5.2-fp8"]) +def test_catalogue_entry_loads_and_predicts(entry): + assert entry in available() + spec = load_spec(entry) assert spec.n_layers == 78 and spec.n_full_indexer_layers == 21 - g, family = predict("glm-5.2", batch=BatchConfig(batch=1, kv_cache_len=4096)) + # Frozen and hashable: a schedule or an override that arrived as a list would + # look fine until something tried to key on the spec. + assert isinstance(spec.op_dtype_overrides, tuple) + assert hash(spec) + g, family = predict(entry, batch=BatchConfig(batch=1, kv_cache_len=4096)) assert family == "glm_moe_dsa" assert g.total_pred_s > 0 +def test_fp8_entry_is_the_deployable_one(): + """Precision is what decides whether the model fits a node, so it is checked. + + bf16 needs ~11 H200s for weights alone; fp8 fits 8 with room for KV. The two + entries exist to make that comparison, so a drift in either dtype is a real + regression. + """ + bf16, fp8 = load_spec("glm-5.2"), load_spec("glm-5.2-fp8") + assert bf16.weight_dtype == "bf16" and fp8.weight_dtype == "fp8" + per_gpu_h200 = 141e9 + assert model_weight_bytes(bf16) / per_gpu_h200 > 8 + assert model_weight_bytes(fp8) / per_gpu_h200 < 8 + + def test_tp_must_divide_heads(): spec = _spec() with pytest.raises(ValueError, match="does not divide"): From e81fe439fc1651af9381ee078bdaed4755fc670d Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 13:24:28 -0700 Subject: [PATCH 04/35] GLM-5.2: design note, prefill/MTP graph, per-op precision Rework the glm_moe_dsa runtime to the structure of the MiMo-V2.5 design note, for GLM-5.2's own architecture rather than by transcription. Graph: - op_dtype_overrides, read from GLM-5.2-FP8's modules_to_not_convert and moe_router_dtype. Three precisions in one block: fp8 backbone + experts, bf16 lm_head / eh_proj / *indexer*, fp32 router. The layout inverts the usual fp8 pattern (o_proj is quantised, the indexer is not), and the indexer is the node that owns 54% of a 1M-context step. - Prefill, with DSA asymptotics. index_topk bounds the core's FLOPs in both phases and its bytes in neither: at prefill each query selects a different top-k and their union is the whole cache. Four helpers rather than reusing BatchConfig.attention_qk_pairs, which is the dense causal count. - serial_launches on every node, so the launch bound exists. At B=1 it is 63% of the predicted floor. - Two collectives per layer, not one folded node -- at 688 kB the count is the cost. EP all-to-all now gated on the layer having experts; the three dense layers were being charged one. - MTP as a D-stage chain with an lm_head per stage, driven by --spec-tokens. GLM's draft block carries a full 256-expert bank, so the chain is 5.2% of the step, not the 1-2% a dense draft head costs. - Indexer wk + weights_proj in both the graph and the footprint. Planner: - _FP32_PEAKS: the router is the first fp32 op the planner has seen, and the A100 default was 3.4x low on an H200. - gitm plan keeps the node's own bound (854 launch-bound nodes printed as memory-bound) and prints one ridge per dtype present. Catalogue: glm-5.2-fp8 (the vendor's recommended shape, 753.33 GB, +0.34% against the published checkpoint); glm-5.2 keeps the bf16 model fact (1.507 TB, +0.08%) and gains the fp32 router. Docs: BRIEF.md -> DESIGN-NOTE.md, restructured to the note's eight sections plus appendix. Deletes 29k lines of committed node-dump JSON that goes stale on every graph change; the commands at the top regenerate any of it. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/BRIEF.md | 198 ------- docs/glm-5.2/DESIGN-NOTE.md | 1102 +++++++++++++++++++++++++++++++++++ gitm/planner/glm_graph.py | 15 +- tests/test_glm_graph.py | 23 +- 4 files changed, 1136 insertions(+), 202 deletions(-) delete mode 100644 docs/glm-5.2/BRIEF.md create mode 100644 docs/glm-5.2/DESIGN-NOTE.md diff --git a/docs/glm-5.2/BRIEF.md b/docs/glm-5.2/BRIEF.md deleted file mode 100644 index 461fc90..0000000 --- a/docs/glm-5.2/BRIEF.md +++ /dev/null @@ -1,198 +0,0 @@ -# GLM-5.2 — standup brief (execution-graph view) - -**What this is:** GLM-5.2 (`zai-org/GLM-5.2`, `GlmMoeDsaForCausalLM`) modelled end-to-end -in the GitM runtime/execution-graph stack, as a new **`glm_moe_dsa`** family forked from -the DeepSeek-V4 sparse-MoE graph. Everything below is derived **from `config.json` and the -checkpoint's own `model.safetensors.index.json`** — no traces, no observed performance. The -numbers are a roofline **floor at vendor peak**, i.e. a lower bound on time, not a target. - -Reproduce any figure here: -```bash -gitm plan glm-5.2 --gpu H200 --batch 1 --kv-len 4096 # op-share table -gitm plan glm-5.2 --gpu H200 --sweep 1,4,16,32,64,128,256 # batch crossover -gitm plan glm-5.2 --gpu H200 --batch 1 --kv-len 131072 --json > graph.json -``` - ---- - -## 1. What GLM-5.2 is, in one breath - -A **754B-parameter** (1.507 TB bf16, validated to **+0.1%** against the published checkpoint) -sparse-MoE decoder, **~40B active per token** (config-derived, using the repo's `active_params` -convention — which folds in the ~1B untied embed + lm_head, so ~39B is the real per-token -multiply). Three things make it its own family rather than "another DeepSeek": - -1. **MLA attention** — one compressed KV latent (`kv_lora_rank=512`) shared across all 64 - query heads. There is **no** per-layer compression schedule (no CSA/HCA, no sliding - window) — every layer runs the same attention. -2. **DeepSeek Sparse Attention (DSA)** — a lightning indexer scores the whole history and - keeps the top **`index_topk=2048`** positions for the attention core. -3. **IndexShare** — the mechanism this fork exists to price. Only **21 of 78 layers** - compute the index; the other **57 reuse** a neighbour's selection and physically carry - **no indexer weights**. - -## 2. Layer stack / block structure - -| | | -|---|---| -| Layers | **78** transformer + **1** MTP draft head | -| Hidden | 6144, vocab 154,880 (untied embed + lm_head) | -| MLP schedule | layers **0–2 dense** FFN (`intermediate=12288`); **3–77 MoE** (`first_k_dense_replace=3`) | -| MoE | 256 routed experts, **top-8**, **1 shared**, `moe_intermediate=2048`, sigmoid + `noaux_tc` routing, `routed_scaling=2.5` | -| Attention | MLA: `q_lora=2048`, `kv_lora=512`, per-head `qk=256` (nope 192 + rope 64), **`v_head=256`** | -| Indexer | 32 heads × 128 dim, top-2048, **IndexShare period 4** | -| MTP | 1 draft layer, `index_share_for_mtp_iteration=true` (reuses the main index) | -| Precision | **bf16 throughout** — no `quantization_config` in the release | - -Each block = `attn_q_a → attn_q_b → attn_kv_a → [attn_index_proj → attn_index_score on full -layers only] → attn_score_value → attn_qnorm_rope_insert → attn_out_proj → {dense FFN | moe_router → moe_shared → moe_routed}`. - -## 3. Attention pattern & IndexShare — the headline - -`indexer_types` is read verbatim from the checkpoint: layers **0,1,2 `full`**, then a strict -**period-4** cycle — one `full` layer that recomputes the top-2048 selection, then three -`shared` layers that reuse it (`index_topk_freq=4`, `index_skip_topk_offset=3`). - -**This is proven from the weight map, not inferred:** indexer tensors (`*.indexer.*`) exist on -exactly the 21 `full` layers and on **none** of the 57 `shared` layers. A shared layer that -recomputed the index would need those weights; it doesn't have them. So the fork emits indexer -nodes on 21 layers only. Pricing all 78 at full rate — the naive reading of `index_topk` — -would overstate the indexer's share ~4× and mis-rank it against the MoE term. - -**Consequence in the graph:** attention is *flat in context*. The core reads at most -`index_topk=2048` selected positions however long the context grows; only the indexer *scan* -grows, and IndexShare caps that to 21 layers. Measured in the graph, 4K → 128K context: - -| op | 4K ctx | 128K ctx | behaviour | -|---|---|---|---| -| `attn_score_value` (core) | 0.039 ms | 0.039 ms | **flat** — bounded by top-k | -| `attn_index_score` (scan) | 0.005 ms | 0.147 ms | grows with context, but only 21 layers, still <1% | - -## 4. MoE / routing structure - -75 sparse layers + the MTP head each run: a replicated **`moe_router`** (h→256), one always-on -**`moe_shared`** expert, and **`moe_routed`** — top-8 of 256. The routed term is the whole -decode story: **FLOPs scale with `positions × 8`**, but **weight traffic scales with the number -of _distinct_ experts the batch woke**, which saturates at 256 (`distinct_experts`). At batch 1 -a step fetches ~8 experts/layer; by batch ~256 it fetches essentially all of them for the same -per-token FLOPs — which is why decode is memory-bound at low batch and only turns compute-bound -at high batch. - -## 5. Expected kernel classes & roofline (the op-share table) - -**H200 (989 TFLOP/s bf16, 4.80 TB/s), batch 1, kv 4096, whole-model floor** — ridge 206 FLOP/byte, so **every node is memory-bound**: - -| op | share | bound | what it is | -|---|---:|---|---| -| `moe_routed` | **56.1%** | memory | routed-expert weight fetch (the union term) | -| `attn_out_proj` | **19.4%** | memory | output GEMM — wide because `v_head=256` → 16,384-wide input | -| `moe_shared` | 7.0% | memory | always-on shared expert | -| `attn_q_b` | 6.5% | memory | query up-projection (64×256 heads) | -| `attn_kv_b` | 2.8% | memory | KV up-projection from latent (unabsorbed MLA — see §7.3) | -| `attn_q_a` | 2.4% | memory | query down-projection (replicated) | -| `lm_head` | 2.3% | memory | vocab projection | -| everything else | <2% each | memory | kv_a, dense FFN, **indexer (0.4%)**, router, core, norms | - -Floor: **17.0 ms/step** at batch 1 on this shape. **This is a per-GPU rate, not an achievable -config — the model is 1.5 TB and does not fit one H200 (see §7).** Kernel classes the graph -expects: **memory-bound GEMV/GEMM** (all projections + experts), **grouped-GEMM** (routed -experts), **paged-attention core** over selected KV, **indexer score/top-k** (21 layers), **fused -qnorm+RoPE+cache-insert**, and — under sharding — **all-reduce** (TP) / **all-to-all** (EP). - -**Honest end-to-end throughput** (a shape that fits — 16×H200, TP16/EP16, batch 32, kv 8192): -**15.5 ms/step, ~2,060 tok/s**, collectives priced, still fully memory-bound. - -**Batch crossover** (`--sweep`, TP=1 per-GPU reference, kv 4096): 59 tok/s @ b1 → 782 tok/s @ -b256, where **341/830 nodes turn compute-bound** as the expert union saturates. (Per-GPU rates — -multiply by ranks, minus collective overhead, for a real deployment.) - -## 6. Algebraic roofline structure & assumptions - -Per op: `t = max(flops/peak_dtype, bytes/HBM_bw)`, peak resolved **per dtype** (bf16 here). -Load-bearing terms and the assumptions behind them: - -- **Expert traffic = a set-union, not a multiply:** `distinct(B) = 256·(1−(1−8/256)^B)`. Assumes - **uniform routing** — real skew touches *fewer* experts, so this over-predicts traffic (the - safe direction). -- **KV traffic uses the shared latent** `kv_lora_rank + qk_rope = 576` elems/token/layer — **not** - `num_key_value_heads × head_dim`. The config's `num_key_value_heads=64` is a red herring; a GQA - reading inflates decode KV ~57×. -- **Attention core is not divided by TP** — a single shared latent can't be split, so the cache is - replicated and every rank reads all of it. **TP buys no KV bandwidth on this architecture.** -- **`q_a` / `kv_a` are replicated** across TP ranks (they make the shared latent), so TP's speedup - on attention is strictly less than `tp`. -- **Collectives are bandwidth-only** (no latency floor) → optimistic at decode message sizes, - flagged `estimated`. **EP imbalance stays at 1.0** (perfect balance) — it is trace-calibrated by - design and, with the no-traces constraint, is declared rather than fitted. -- **Decode only.** Prefill (chunked, different indexer asymptotics) is unmodelled, as in every - family. - -## 7. Where the headroom likely is — and what to validate first - -Ranked by expected payoff, all **hypotheses from the graph** to be confirmed against a capture: - -1. **The model doesn't fit one box — the deployment shape is the first-order lever.** 1.507 TB - bf16 needs **≥11 H200 / ≥8 B200** for weights alone (realistically **16×H200 = 2 nodes** with - KV+activation headroom). **FP8 halves it to ~0.75 TB → fits 8×H200.** Validate first: does an - FP8 expert cast hold GLM-5.2's quality? Everything else is downstream of this. -2. **`moe_routed` (58%) is the memory-bound heart.** Levers: **EP with a good expert-placement / - EPLB** to cut per-rank distinct-expert traffic and, at serving batch, the **grouped-GEMM - backend** (DeepGEMM). Validate: measured per-rank expert traffic vs the `distinct_experts` - prediction, and the **real EP imbalance** (the one number we must calibrate from a trace). -3. **`attn_out_proj` (19%) + `attn_kv_b` (3%) hinge on one question: does the engine run - _absorbed_ MLA?** We model it **unabsorbed** — `kv_b` runs as its own GEMM and `o_proj` is - narrow (16384→6144). If vLLM absorbs MLA (the usual decode path), `kv_b` disappears and - `o_proj` **doubles** to 32768→6144. Same resident weights, but the #2 line moves by 2× in - either direction. Validate first: which path the serving engine takes — it re-ranks the whole - attention side. -4. **IndexShare is already a big win the graph shows is _cheap_ (0.4%).** The validate-first - question is the opposite of a lever: **confirm the runtime actually skips the scan on `shared` - layers** (and reuses across the group), because if it doesn't, a hidden 4× indexer cost is - sitting off our books. Also confirm `index_topk_freq` is purely spatial (layer-group) vs also - temporal (step-to-step) — extra temporal reuse only *reduces* cost. -5. **MTP acceptance** — we emit the draft as a shared (no-indexer) block per - `index_share_for_mtp_iteration`. Validate the accepted-token rate to see whether MTP pays off, - and that the draft indexer is genuinely shared. - -**First capture to take:** a single decode step at a shape that fits (16×H200, TP16/EP16, batch -16–32, kv 8K — or 8×B200, TP8/EP8), attributed per-op, and diff it against -`gitm plan glm-5.2 --gpu H200 --batch 32 --kv-len 8192 --tp 16 --ep 16 --json`. The three numbers -that matter most: **EP imbalance**, **`moe_routed` per-rank traffic**, and whether **`shared` -layers emit any indexer kernel at all**. - -## 8. How to run it on RunPod - -The model is too large for a single GPU, so this is a **multi-GPU / multi-node** job. Two paths: - -**A. Predict-only (no GPU needed — do this first, it's free and already works):** -```bash -pip install -e . # this repo -gitm plan glm-5.2 --gpu H200 --batch 32 --kv-len 8192 --tp 16 --ep 16 # 16×H200, fits -gitm plan glm-5.2 --gpu B200 --batch 32 --kv-len 8192 --tp 8 --ep 8 --json # 8×B200, fits -``` - -**B. Serve + capture on RunPod (to get the numbers §7 wants validated):** -1. Pick hardware by the footprint math above: **bf16 → 2× `8×H200` (or 8×B200)**; **fp8 → - 1× `8×H200`**. On RunPod choose an **8×H200 SXM** pod (or two, networked) with a **network - volume ≥ 2 TB** for the 282-shard checkpoint. -2. Serve with a DSA-aware engine (vLLM/SGLang build with `glm_moe_dsa` + MLA + DSA support), - e.g. `--tensor-parallel-size 8 --enable-expert-parallel`, `--kv-cache-dtype fp8` (serving - choice, not a model fact), `--max-model-len` to taste. -3. Attach the GitM collector to the live server and capture a bounded decode window: - ```bash - gitm capture serve # or: gitm capture attach (see PR #79) - ``` -4. Import the trace and diff observed-vs-predicted per op; residuals outside the efficiency band - are the leads §7 lists — **a residual here is a lead, not a defect.** - -> ⚠️ The engine flags in step 2 are the shape to aim for, not verified command lines — confirm -> the serving engine actually supports `glm_moe_dsa` before booking a multi-node pod. Start with -> path A (free) and the FP8 fit question, which gates everything else. - ---- - -### Artifacts in this directory -- `artifacts/glm-5.2_H200_b1_kv4096_tp1ep1.json` — canonical decode-step graph (830 nodes, per-GPU rate) -- `artifacts/glm-5.2_H200_b1_kv131072_tp1ep1.json` — long-context (indexer scan grows) -- `artifacts/glm-5.2_H200_b32_kv8192_tp16ep16.json` — a shape that fits: 16×H200, collectives priced -- Graph code: `gitm/planner/glm_graph.py` · catalogue: `gitm/planner/models/glm-5.2.yaml` · tests: `tests/test_glm_graph.py` diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md new file mode 100644 index 0000000..e8f27ad --- /dev/null +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -0,0 +1,1102 @@ +# GLM-5.2 — Working Design Note + +**Predicted execution model for Z.ai GLM-5.2 on 8×H200 SXM, TP8 / EP8, FP8** + +Built from the model repos' own files — `config.json` and +`model.safetensors.index.json` for both `zai-org/GLM-5.2` (bf16) and +`zai-org/GLM-5.2-FP8`, plus the vendor's published vLLM recipe for the deployment +shape. **No traces.** Every number is a roofline floor at vendor peak: a lower +bound on time, not a target. + +Reproduce any figure here: + +```bash +gitm plan glm-5.2-fp8 --gpu H200 --batch 32 --kv-len 8192 --tp 8 --ep 8 +gitm plan glm-5.2-fp8 --gpu H200 --batch 32 --kv-len 8192 --tp 8 --ep 8 --spec-tokens 5 +gitm plan glm-5.2-fp8 --gpu H200 --prefill-tokens 8192 --tp 8 --ep 8 +gitm plan glm-5.2-fp8 --gpu H200 --sweep 1,4,16,32,64,128,256 --kv-len 8192 --tp 8 --ep 8 +gitm plan glm-5.2 --gpu H200 --batch 32 --kv-len 8192 --tp 8 --ep 8 # what bf16 costs +``` + +## Hardware assumption: 8×H200 SXM, NVLink, TP8 / EP8, FP8 weights and KV + +Unlike a hardware assumption inferred from a checkpoint, **this one is the +vendor's own** — `recipes.vllm.ai/zai-org/GLM-5.2` publishes it verbatim. If +production differs, *graph topology does not change*; only these constants and +some bound labels do. Regions whose label would flip are marked ⚑ throughout. + +| Constant | Value used | Note | +| -------------------- | --------------------------------- | ------------------------------------------------------------- | +| FP8 e4m3 tensor peak | **1,979 TFLOP/s** | datasheet says 3,958 **"with sparsity"** — halved, see A2 | +| BF16 tensor peak | **989.5 TFLOP/s** | same halving | +| FP32 CUDA-core peak | **67 TFLOP/s** | the router runs here — see G4, and it is 3.4× the old default | +| HBM3e | **4.8 TB/s** | as published | +| NVLink | **900 GB/s** per GPU | the catalogue's bidirectional convention | +| Memory | 141 GB × 8 = 1,128 GB | one node | +| Kernel launch | ~2 µs graph-replay / ~5 µs eager | ⚑ the crossover hinge — see §5 rank 3 | + +``` +FP8 ridge = 1,979e12 / 4.8e12 = 412 FLOP/byte +BF16 ridge = 989.5e12 / 4.8e12 = 206 FLOP/byte +FP32 ridge = 67e12 / 4.8e12 = 14 FLOP/byte +``` + +**You need all three.** The backbone GEMMs and the experts are FP8; `lm_head`, +`embed_tokens`, the MTP `eh_proj` and — the one worth naming — the **lightning +indexer** are BF16; the router is FP32. §1's precision table shows why, and G5 is +the planner change that made it representable. + +--- + +## 1. Layer-by-layer architecture map + +### Headline structure + +| Property | Value | +| --------------------------- | ------------------------------------------------------------------------ | +| Layers | **78** transformer + **1** MTP draft module | +| Attention | **MLA + DeepSeek Sparse Attention on every layer** — no schedule at all | +| KV latent | `kv_lora_rank=512` + `qk_rope_head_dim=64` = **576 elems/token/layer** | +| Indexer | 32 heads × 128 dim, keeps **`index_topk=2048`** positions for the core | +| **IndexShare** | **21 of 78 layers** compute the index; **57 reuse** a neighbour's | +| Dense MLP | **3** — layers 0, 1, 2 (`first_k_dense_replace: 3`), `intermediate=12288` | +| MoE layers | **75** — layers 3–77, plus the MTP module | +| Experts / top-k | 256 routed, top-8, **1 shared**, `moe_intermediate_size=2048` | +| Routing | sigmoid scoring, `noaux_tc`, `routed_scaling_factor=2.5`, **fp32 router** | +| hidden_size | 6144 | +| Q heads / q_lora / kv_lora | 64 / 2048 / 512 | +| qk_nope / qk_rope / v_head | **192 / 64 / 256** — the value width differs from the score width | +| RoPE | θ=8e6, `rope_interleave`, `indexer_rope_interleave` | +| MTP | 1 module, `index_share_for_mtp_iteration: true`, **carries a full MoE** | +| Vocab | 154,880, untied `lm_head` | +| Max context | 1,048,576 | +| Total / active params | ~754 B / ~39 B | + +``` +L: 0 1 2 3 6 10 14 ... 74 77 + F F F ···F ···F ···F ···F ···F ··· + │ │ │ F = full indexer (recomputes top-2048) + └──┴──┴─ the only DENSE MLP layers · = shared indexer (reuses, no weights) + everything from 3 up: MoE 256e / top-8 / 1 shared +``` + +**A clean 3-full prefix then a strict period-4 cycle** — +`indexer_types` is read verbatim, and the two schedules do *not* line up: the +dense prefix is 3 layers, the `full` prefix is also 3 layers, and then the +IndexShare period is 4. No single modulo rule reproduces either (§7, G3). + +### Semantics read from the checkpoint, not guessed + +`config.json`: + +- `indexer_types[i] == "full"` → the layer **computes** its own top-2048 selection +- `indexer_types[i] == "shared"` → the layer **reuses** the group's selection +- `mlp_layer_types[i]` → `"dense"` | `"sparse"`, and it agrees with + `first_k_dense_replace: 3` +- `moe_router_dtype: "float32"` → the router is fp32 **on every variant**, because + this is a field of the base config and not of any quantisation config + +**This is proven from the weight map, not inferred.** Indexer tensors +(`*.indexer.wq_b`, `.wk`, `.weights_proj`, `.k_norm`) exist on exactly the 21 +`full` layers, on **none** of the 57 `shared` layers, and on **none** of the MTP +module. A shared layer that recomputed the index would need those weights; it does +not have them. Pricing all 78 at full rate — the naive reading of `index_topk` — +overstates the indexer ~3.7× and mis-ranks it against the MoE term. + +Attention shapes, per token per layer: + +| Quantity | Shape | Purpose | +| ------------------------ | -------------------- | -------------------------------------------------- | +| Q latent (`q_a`) | 2048 | replicated low-rank query | +| Q per-head (`q_b`) | 64 × 256 = 16,384 | 192 nope + 64 rope | +| **KV cache entry** | **512 + 64 = 576** | **one latent for all 64 heads** — the MLA point | +| `kv_b` output | 64 × (192+256) = 28,672 | reconstructed K_nope and V | +| Attention output | 64 × 256 = 16,384 | one 256-d result per Q head | +| After `o_proj` | 6,144 | back to `d_model` | +| Index key (`full` only) | 128 | cached alongside the latent | + +`num_key_value_heads: 64` is in the config and is a **red herring**. A GQA reading +of the cache gives `64 × 448 = 28,672` elements per token per layer against the +real 576 — a **50× overstatement of the single quantity decode is bound by**. + +### The 79-row table, collapsed to four archetypes + +The 78 layers plus the draft module are exactly four shapes. Everything not listed +is byte-for-byte identical between them. + +| Archetype | Count | Attn | Indexer | MLP | KV elems/tok | Collectives per layer | +| --------- | ----- | ---- | -------- | ------------- | ------------ | --------------------- | +| `Ld,f` | **3** | MLA+DSA | **full** | dense 12288 | 576 + 128 | 2× all-reduce | +| `Ls,f` | **18**| MLA+DSA | **full** | MoE 256×2048 | 576 + 128 | 2× all-reduce (+2× a2a under EP) | +| `Ls,sh` | **57**| MLA+DSA | shared | MoE 256×2048 | 576 | 2× all-reduce (+2× a2a under EP) | +| `Lmtp` | **1** | MLA+DSA | shared | MoE 256×2048 | 576 | 2× all-reduce, ×D stages | + +Note what is *absent*, because the absences are the design: **no sliding window, +no compression schedule, no attention-type alternation, no vision encoder, no +audio encoder.** GLM-5.2 is a text-only decoder in which every layer runs the same +attention. All of its structural variation is in two schedules — dense/sparse MLP +and full/shared indexer — and one of those has no cost consequence at all past +layer 2. + +### Verification — both checkpoints agree + +- **bf16:** predicted **1,508.1 GB** against `total_size` **1,506,659,919,872 B** + (282 shards) — **+0.08 %**. +- **fp8:** predicted **755.9 GB** against **753,329,940,480 B** (141 shards) — + **+0.34 %**. + +Two published checkpoints at two precisions agreeing to under half a percent is a +stronger check than either alone: an error in the shape arithmetic would have to +be precision-proportional to survive both. A 2:4-sparsity-compressed checkpoint +would be roughly half the fp8 size. It is not — which is the evidence behind A2. + +### KV cache — the number that drives decode + +``` +per layer per token, elements: + every layer kv_lora_rank 512 + qk_rope 64 = 576 ← one latent, all 64 heads + full-indexer layers, additionally + 128 ← the cached index key + +whole model, bytes per token of context (fp8 latent, bf16 rope key + index key): + 78 × (512·1 + 64·2) + 21 × 128·1 = 52,618 B/token + bf16 throughout: 95,232 B/token +``` + +| Context | fp8 KV | bf16 KV | +| ------- | ------- | ------- | +| 8,192 | 0.43 GB | 0.78 GB | +| 131,072 | 6.90 GB | 12.48 GB | +| 1,048,576 | **55.2 GB** | **99.9 GB** | + +**And it is replicated, not sharded.** One shared latent cannot be split across +tensor-parallel ranks, so every rank holds and reads the whole cache: **TP buys no +KV bandwidth on this architecture.** At the advertised 1M context that is 55 GB per +rank on top of a 96 GB per-rank weight share — which is the real reason the vendor +recipe reaches for B200s and `--max-num-seqs 32` when it wants full context. + +### FP8 — what is and is not quantized + +| Component | Precision | Evidence | +| -------------------------------------------------- | ------------------------------- | ------------------------------------------------- | +| `q_a`/`q_b`/`kv_a`/`kv_b`, **`o_proj`**, dense FFN | **FP8 e4m3**, 128×128 block | absent from `modules_to_not_convert` | +| routed experts, shared expert | **FP8 e4m3**, 128×128 block | absent from `modules_to_not_convert` | +| `lm_head`, `embed_tokens` | **BF16** | named in `modules_to_not_convert` | +| **lightning indexer** (`indexers_proj`, `k_norm`) | **BF16** | named in `modules_to_not_convert` | +| MTP `eh_proj`, `enorm`, `hnorm` | **BF16** | named in `modules_to_not_convert` | +| MoE router (`mlp.gate` + `e_score_correction_bias`) | **FP32** | `moe_router_dtype: "float32"`, base config | +| all norms | **BF16** | named in `modules_to_not_convert` | + +**Three precisions inside one attention block**, and the layout **inverts** the +familiar fp8-backbone pattern: here `o_proj` is *inside* the quantised set and the +*indexer* is outside it. Pricing the indexer at fp8 halves the weight traffic of +the one attention node whose cost grows with context — and at 1M context that node +is 54 % of the step (§4.1). This is why the planner grew +`op_dtype_overrides` (§7, G5). + +--- + +## 2. Per-phase execution diagrams + +### 2.1 Prefill — a chunk of P tokens against C cached + +```mermaid +%%{init: {'theme':'neutral'}}%% +flowchart TD + T["input_ids"] --> EMB["embed_tokens gather — BF16"] + EMB --> L0["layers 0-2 — MLA+DSA + DENSE FFN"] + L0 --> LB["layers 3-77 — MLA+DSA + MoE 256/top-8"] + LB --> FN["final RMSNorm
LAST TOKEN OF EACH PROMPT ONLY"] + FN --> LM["lm_head GEMM BF16
1.9 GB of weights for 1 row per request"] + + subgraph LB + direction TB + N1["input_layernorm"] --> QA["q_a fp8 → q_a_layernorm"] + QA --> QB["q_b fp8 — M=P, COMPUTE-BOUND"] + N1 --> KA["kv_a fp8 → latent[576] → CACHE WRITE"] + KA --> KB["kv_b fp8 — reconstruct K_nope,V"] + QA --> IX{"full indexer layer?"} + IX -- "21 layers" --> IP["indexer wq_b/wk/weights_proj — BF16"] + IP --> IS["index_score over the WHOLE history
O(P·C + P²/2) — the quadratic lives HERE"] + IS --> TK2["top-2048 per query"] + IX -- "57 layers" --> RE["reuse the group's selection
NO KERNEL AT ALL"] + TK2 --> ATT + RE --> ATT + KB --> ATT{"attention core over ≤2048 selected keys
FLOPs capped · BYTES ARE NOT"} + ATT --> OP["o_proj fp8"] + OP --> AR1{{"all_reduce #1 — 174 MB, BANDWIDTH-bound"}} + AR1 --> N2["post_attention_layernorm"] + N2 --> G["router GEMM — FP32, 26 GF/layer"] + G --> SIG["sigmoid + e_score_correction_bias"] + SIG --> TK["top-8 of 256 + renorm
DATA-DEPENDENT SHAPE"] + TK --> A2A{{"EP dispatch all-to-all
695 MB/layer — the largest single term"}} + A2A --> PERM["permute/gather — 8P rows"] + PERM --> EG["grouped GEMM gate+up fp8
ALL 256 experts hit"] + EG --> SW["SiLU × up"] --> ED["grouped GEMM down fp8"] + ED --> COMB["scatter-add × routed_scaling 2.5"] + COMB --> A2B{{"EP combine all-to-all"}} + A2B --> AR2{{"all_reduce #2"}} + end +``` + +**The structural claim of prefill:** with 256 experts and top-8, a chunk of P +tokens issues `8P` token→expert assignments. Once `8P ≫ 256` — P above a few +hundred — every expert receives at least one token, so **every layer reads its +entire expert bank**, 1.26 GB per rank per layer at EP8, **constant in P**. That is +95.7 GB per pass, 29 % of all prefill traffic. But it is not the top line: **the EP +all-to-all is**, at 105.7 GB and 49 % of predicted prefill time. Three quarters of +prefill cost is the MoE path, and under expert parallelism most of that is *wire*, +not DRAM. + +**And DSA inverts the usual prefill story.** In a dense model prefill attention is +the `O(P²)` term. Here the *core* is capped at 2,048 selected keys per query, so it +is linear in context past 2,048 — and the quadratic has moved into the **indexer +scan**, which IndexShare then pays on only 21 of 78 layers. `index_topk` bounds the +core's FLOPs in both phases; it does **not** bound its bytes at prefill (§2.2). + +### 2.2 Decode — steady state, B sequences, one token each + +Identical node set to prefill. Four nodes change **kind**: + +| operator | prefill class | decode class | why the class itself changes | +| ------------------- | ------------------------------------- | ----------------------------------------------------- | -------------------------------------------------------------------- | +| attention core | tiled over query blocks, causal | **paged decode attention** over a top-k block table | one query row against a gathered selection; no tiling over queries | +| indexer scan | `O(P·C + P²/2)`, **compute-bound** | `O(B·C)`, **memory-bound** — streams the whole key set | the query count collapses from P to B; the key set does not | +| **attention bytes** | whole cache, once **per request** | **top-2048 window, per sequence** | prefill queries' selections union to everything; one query's does not | +| every GEMM | M = P ≈ 8192, compute-bound | M = B, **weight-streaming** | AI falls ~1,400 → ~60; same kernel name, different regime | + +Plus one epilogue change: **in prefill only the last position of each prompt runs +`lm_head`. At decode every row is a last position**, so it reads 1.9 GB of bf16 +vocabulary weights *every step* rather than once per request. + +``` + [hidden BF16 B×6144] + │ + RMSNorm ─▶ q_a fp8 (REPLICATED, 2048) ─▶ q_a_layernorm ─▶ q_b fp8 (64×256) + │ + kv_a fp8 ─▶ latent[512] + rope key[64] ─▶ KV-cache APPEND (576 elems/seq) + │ + ┌────┴─────────────────────────────────────┐ + │ full-indexer layer (21 of 78)? │ + │ yes → wq_b/wk/gate BF16, then │ ← the ONLY term that grows with S. + │ score the WHOLE history │ 0.9 % of the step at 8K, + │ B×C×128 bytes, top-2048 │ 13.0 % at 128K, 54.4 % at 1M + │ no → reuse. NO KERNEL. (57 layers) │ + └────┬─────────────────────────────────────┘ + │ + attention core over ≤2048 selected entries ·· 42 MB/layer — FLAT IN CONTEXT + │ and NOT divided by TP + o_proj fp8 (16384→6144) + │ + all_reduce #1 ······························ 688 kB · LATENCY-bound [stream: unresolved] + │ + RMSNorm ─▶ router GEMM FP32 ─▶ sigmoid+bias ─▶ top-8 of 256 + │ + ├─▶ per-expert histogram ─▶ ❓ D2H sync ❓ · S1 · 76/token if real · conf LOW + │ ← the ONLY data-dependent shape + │ in the graph. Blocks CUDA-graph + │ capture. §5 rank 2. + EP dispatch a2a ─▶ permute ─▶ grouped GEMM fp8 ×2 ─▶ SiLU ─▶ grouped GEMM fp8 + │ 163/256 experts woken at B=32 · 771 MB/layer · 86 % of DRAM + scatter-add ×2.5 ─▶ EP combine a2a ─▶ all_reduce #2 + ▼ + … ×78 layers, then: + final RMSNorm (ALL B rows) ─▶ lm_head 1.9 GB BF16 ─▶ sample ─▶ D2H ─▶ scheduler gap +``` + +### 2.3 Encoders — there are none, and the absence is worth stating + +GLM-5.2 is a **text-only** decoder. There is no vision tower, no audio encoder, no +patch embedding, no multimodal scatter into `inputs_embeds`. The design template +this note follows devotes two sections to encoder cost; here they collapse to a +single fact, and it changes three things downstream: + +1. **There is no encoder→backbone seam**, so the one unavoidable serial dependency + that dominates a multimodal prefill does not exist. Prefill starts at + `embed_tokens`. +2. **Prompt length is the only input-side variable.** No image or video token + count feeds P, so the flip-variable index (§4.4) is one row shorter than a + multimodal model's and the remaining rows are correspondingly better + constrained. +3. **No second model is hiding off-checkpoint.** A multimodal note has to carry an + open question for an external audio codec it cannot see. Here the checkpoint is + the whole model, so every FLOP in the trace should map to a node in §3 — which + makes an unexplained kernel block a much stronger signal than it would be + elsewhere (§6.4 row 6). + +The GLM family does ship vision variants (GLM-4.5V and successors). **They are a +different checkpoint with a different `model_type` and this graph does not model +them** — `is_glm_moe_dsa_config` would decline them rather than price a tower it +never read. + +### 2.4 MTP-on decode — draft and verify + +```mermaid +%%{init: {'theme':'neutral'}}%% +flowchart LR + subgraph V["VANILLA DECODE"] + direction TB + v1["1 row per seq"] --> v2["78 layers + 1 MTP
1,294 nodes"] --> v3["lm_head"] --> v4["sample"] + v4 --> v5(("1 token")) + end + subgraph D["DRAFT — 5 SERIAL stages, 90 nodes"] + direction TB + d0["h at last accepted pos
+ its token"] --> eh["eh_proj [12288→6144] BF16"] + eh --> d1["MTP block
MLA+DSA (shared index) + FULL MoE"] + d1 --> dl1["lm_head 1.9 GB"] --> ds1["argmax → D2H"] --> d2["stage 2 …"] + d2 --> dt(("5 draft tokens")) + end + subgraph W["VERIFY = decode at 1+D rows"] + direction TB + w1["6 rows per seq"] --> w2["THE SAME 78 layers
THE SAME 1,276 nodes"] + w2 --> w3["lm_head, 6 rows"] --> w4["compare vs draft"] + w4 --> w5(("1..6 accepted")) + end + v5 -.->|replaced by| dt + dt --> w1 + w5 -->|"h of last accepted"| d0 +``` + +**Verify is not a new graph.** It is the decode graph with the row dimension +multiplied by `1+D`. The only genuinely new subgraph is the draft chain — 90 nodes +against the backbone's 1,276. + +**The dependency point:** the loop is strictly serial and now has `D+1` sampling +points instead of one. Nothing in stage `k` can start before stage `k-1`'s token id +exists. **Those five gaps are architectural** (§6.1) and no scheduling closes them. + +**And here GLM departs sharply from the MTP designs this template was written +against.** Per the weight map, the MTP module is `enorm` + `hnorm` + `eh_proj` +`[12288, 6144]` BF16 + one MLA+DSA attention block + **a full `mlp.experts.*` bank +of 256 experts**. It carries no indexer (consistent with +`index_share_for_mtp_iteration: true`) and no `lm_head` of its own — it shares the +backbone's. So the intuition that "the draft is a small dense copy of the big +model" is **wrong here**: the draft is one *full* MoE layer, and it draws on the +expert bank once per stage. §3.3 puts a number on it. + +--- + +## 3. Predicted execution graph + +Detailed enough to put a trace next to. The 79 blocks collapse to **four +archetypes** exactly (§1) plus a prologue, an epilogue and, under MTP, a 90-node +draft chain. All figures at **B=32, S=8192, TP8/EP8, FP8 weights and KV, per +rank** unless stated. + +**The per-node tables live in Appendix A.** What stays here is the part that is +argued rather than looked up. + +### 3.1 Prologue and epilogue + +| id | operator | kernel class | shape | FLOPs | bytes | stream | conf | +| --- | ----------------- | ------------------------- | ------------------------------ | -------- | ------------- | ------- | ------ | +| D0 | `embed_tokens` | gather (index_select) | `[32] → [32,6144]` | 0 F | 393 kB | compute | high | +| E0 | final norm | fused RMSNorm | `[32,6144]` **every sequence** | 590 kF | 786 kB | compute | high | +| E1 | `lm_head` | GEMM (BF16), tall-skinny | `[32,6144] × [6144,19360]` | 7.61 GF | **239.5 MB**/rank | memory | high | +| E2 | sample + D2H | argmax/top-p + host copy | `[32,154880] → [32]` | — | small | compute | high | + +`lm_head` is TP-sharded on the vocabulary, so per rank it is 239.5 MB of the 1.903 GB +whole-model matrix. It is 0.6 % of a decode step and **0.02 %** of a prefill one — +the epilogue is nearly free in prefill and is not in decode. + +### 3.2 What prefill changes + +**Same node ids, same order — `M` becomes `P` instead of `B`, and four nodes change +in kind rather than degree** (§2.2's table). Concretely, at P = 8,192 in one chunk +against P = 1 decode row per sequence: + +- Every projection crosses into **compute-bound**: `q_a`/`o_proj` run AI 1,403, + `q_b` 963, `kv_a` 488 — all far past the fp8 ridge of 412. +- **The indexer scan flips from memory to compute.** At decode it streams keys for + one query per sequence (AI 2.0); at prefill it scores 8,192 queries against the + same keys (AI 473). +- **The attention core's bytes and FLOPs stop moving together.** FLOPs are capped + at 2,048 keys per query either way. Bytes are not: at decode one sequence reads + its own 2,048-entry window; at prefill 8,192 queries each select a *different* + 2,048 and their union is the whole history, so the kernel streams the entire + cache once per request. **`index_topk` bounds prefill FLOPs, not prefill bytes** — + and a prefill path copied mechanically from a dense family would charge + `P × index_topk` here and understate long-context prefill traffic by C/2048. +- **The collectives change character completely.** `all_reduce` carries 688 kB at + decode (latency-bound) against **174 MB each, 27.5 GB/pass** at prefill. The EP + all-to-all goes from 5.5 MB/layer to **1.39 GB/layer** — 105.7 GB per pass, and + the single largest line in prefill at 50 % of predicted time. +- **The epilogue runs the opposite way:** at prefill `E0/E1` process the last token + of each prompt only, so `lm_head` reads 239.5 MB of weights to produce one row per + request; at decode every row is a last position. + +| Prefill, P=8192, C=0, TP8/EP8, per rank | value | +| --- | --- | +| predicted floor | **240.4 ms** for the chunk (34.1 k tok/s) | +| bytes moved | **331.0 GB** | +| FLOPs | **112.8 TF** | +| whole-pass AI | **341** — below the fp8 ridge of 412, so **memory-bound overall** | +| facets | compute 666 n / 83.5 ms / 34.7 % · memory 539 n / 156.7 ms / 65.2 % · launch 89 n / 0.2 ms | + +> ⚠ **The prefill rows overturn the dense-model intuition, and the reversal is +> worth keeping visible.** Dense intuition says prefill is compute-bound. It is +> **not**, for two structural reasons: 256 experts × top-8 means the whole bank is +> read per layer regardless of P, and expert parallelism turns the MoE dispatch +> into a wire-bound all-to-all that no amount of arithmetic hides. +> `confidence: high for the arithmetic, medium for the conclusion` — the soft link +> is "essentially all 256 experts are hit", which assumes routing is not +> pathologically concentrated. `e_score_correction_bias` exists precisely to spread +> load, so concentration is unlikely, but it is an assumption (A5). + +**Chunk size is a multiplier on the whole MoE term.** The same 8,192 tokens: + +| chunking | bytes | floor | +| ------------------- | ----------- | ---------- | +| 1 × 8,192 | **331 GB** | 240 ms | +| 2 × 4,096 | 427 GB | 256 ms | +| 8 × 1,024 | 1,000 GB | 372 ms | +| 64 × 128 | **6,249 GB**| **1,505 ms** | + +**18.9× the bytes for identical FLOPs.** The expert bank is read per *chunk*, not per +token — so a small `--max-num-batched-tokens`, chosen to protect decode latency, +is paid for here at a rate nothing in a per-token cost model shows. + +### 3.3 MTP — the whole-step economics + +At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): + +| pass | nodes | bytes | floor | +| ---------------------- | --------- | ------------ | ------------- | +| vanilla decode (D=0) | 1,294 | 68.22 GB | 15.887 ms | +| — of which the draft | 18 | 1.26 GB | 0.285 ms | +| draft chain, D=5 | 90 | **6.31 GB** | 1.423 ms | +| verify, 192 rows | 1,276 | 104.2 GB | 25.756 ms | +| **MTP step total** | **1,366** | **110.5 GB** | **27.179 ms** | + +**Cost ratio 1.71× for up to 6 tokens.** Where the extra 42.3 GB goes: + +- **The verify pass, +36.0 GB.** Almost all of it is one line: expert weights go + from 163 distinct experts at 32 rows to 256 at 192 rows — **the union saturates**, + so 6× the rows costs 1.57× the expert bytes. KV read does **not** move at all + (0.43 GB either way, because it is read per *sequence*, not per row), and verify + `lm_head` does not move either (239.5 MB regardless of rows). +- **The draft chain, +5.05 GB.** And this is where GLM differs from a dense-draft + model: **the draft is 5.2 % of the MTP step, not 1–2 %**, because each of its 5 + stages draws on a full 256-expert bank. Its cost is linear in D with no + saturation to help it — 32 rows wakes ~163 experts every stage, five times over. + +**So ~85 % of the price of speculation is the MoE expert bank** — charged because +more rows and more stages touch more experts, not because more work is done per +token. + +**Break-even.** The step costs 1.71×; it produces up to 6 tokens instead of 1, so +acceptance α must exceed `(1.71 − 1)/5 = 0.142` to pay. Predicted throughput: + +| α | 0.0 | 0.5 | 0.7 | 0.9 | +| ---- | ----- | ----- | ----- | ----- | +| tok/s| 1,177 | 4,121 | 5,298 | **6,476** | + +against 2,014 tok/s with MTP off. The vendor claims the GLM-5.2 MTP layer raises +accepted length by up to 20 % over its predecessor, which puts the operating point +well past break-even — but **α is a serving observable and this graph does not +predict it.** It prices the cost and leaves the payoff to a measurement (§6.2, C4). + +⚑ **All of this assumes the step is memory-bound.** At B ≤ 8 it is not (§4.1), and +in the launch regime the draft's 90 extra launches are pure cost against a step +that was never moving bytes. The sign of the MTP decision flips with batch. + +### 3.4 Predicted synchronization points + +| # | Where | Kind | conf | Trace signature if real | +| ------ | --------------------------- | ----------------------------------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------ | +| **S1** | after top-k (`moe_topk`) | host readback of the expert histogram to size the grouped GEMM | **low** | **76 D2H per decoded token** — fatal for graph capture | +| S2 | around each all-reduce | stream-to-stream event wait | medium | 158 event pairs/step; at 688 kB **the gap *is* the cost** | +| S3 | around each EP all-to-all | dispatch/combine barrier | medium | 76 more, and **none on the 3 dense layers**; an imbalanced rank stalls every other one | +| S4 | sampling / detokenisation | D2H of sampled ids every step | **high** | one D2H + host round-trip per step; unavoidable, but its *placement* decides overlap | +| S5 | scheduler / block manager | host-side work between steps | medium | a CPU-shaped gap between steps, growing with batch churn | +| S6 | after each draft `argmax` | D2H of drafted ids, ×D per step | medium | 5 extra D2H + host round-trips; at B=1 can exceed the draft's own compute | +| S7 | verify → accept/reject | **host-visible, variable-length** result | **high** it exists | a small kernel + a D2H whose *value* decides how far the sequence advanced | +| S8 | KV rollback | discard rejected rows | **low** on mechanism | pointer rewind (free) or real memmove (not free) — the trace tells you which | +| S9 | indexer selection handoff | the `full` layer's top-k must be visible to its 3 `shared` layers | **low** | if it round-trips the host, IndexShare costs a sync it should not — **21 per step** | + +**S5 is the decode-specific one worth chasing.** Decode runs ~1,294 kernels in +~16 ms and then hands control back to a Python scheduler. If the scheduler takes +longer than the step, the GPU idles and no kernel-level work matters. + +**S7 is the one that breaks CUDA graphs.** MTP adds a per-step, host-visible, +data-dependent sequence length. A stack that captures the decode step needs two +captured shapes plus a padded accept path, or no capture at all. + +**S9 is GLM-specific and cheap to check.** IndexShare's whole value is that 57 +layers run no indexer. If the selection is passed device-side (a tensor handed +down the stack) it is free; if it is materialised through the host it costs 21 +syncs a step to save 57 kernels. + +--- + +## 4. Execution-bound / roofline hypotheses + +Five labels: **compute · memory-bandwidth · communication · launch/sync/latency · +mixed or shape-dependent.** Every row names the precision its dominant kernels run +in, the peak it is bounded against, and the variable that flips it. Labels are +**against peak**; a realistic achievable fraction is never used to move a row +across a bound boundary. + +**Two notations, on purpose.** §4.1 is a **node** table — decode is where cost +concentrates, so the useful question is *which nodes own the step*. §4.2 is a +**region** table — for prefill and MTP the useful question is *what would flip this +label*, which needs a flip-variable column and not a cost ranking. + +### 4.1 Decode as a node table — B=32, S=8192, TP8/EP8, FP8, per rank + +``` +ridge 412 F/B (fp8) · 206 (bf16) · 14 (fp32) · launch floor 2.0 µs (graph-replay) + + node kernel class bytes AI xN Σ ms bound share + ────────────────────── ─────────────────────── ──────── ──── ─── ───── ────── ──────────────────────── + moe_routed grouped GEMM fp8 ×3 771.1 MB 3.1 76 12.208 memory ██████████████████████ 76.8% + attn_score_value paged decode attention 42.0 MB 12.8 79 0.690 memory █ 4.3% + moe_all_to_all collective (EP a2a) 5.5 MB — 76 0.465 comm █ 2.9% + attn_q_a GEMM fp8, REPLICATED 13.1 MB 61.4 79 0.216 memory ▏ 1.4% + attn_out_proj GEMM fp8, tall-skinny 13.1 MB 61.4 79 0.216 memory ▏ 1.4% + attn_q_b GEMM fp8, tall-skinny 4.5 MB 60.2 79 0.158 launch ▏ 1.0% + attn_kv_a GEMM fp8 + cache append 4.0 MB 56.8 79 0.158 launch ▏ 1.0% + attn_kv_b GEMM fp8 (unabsorbed) 2.1 MB 56.0 79 0.158 launch ▏ 1.0% + attn_qnorm_rope_insert fused norm+RoPE+insert 0.4 MB 0.9 79 0.158 launch ▏ 1.0% + tp_all_reduce_attn collective NCCL ring 0.7 MB — 79 0.158 launch ▏ 1.0% + tp_all_reduce_mlp collective NCCL ring 0.7 MB — 79 0.158 launch ▏ 1.0% + moe_router GEMM **fp32**, replicated 6.7 MB 15.0 76 0.152 launch ▏ 1.0% + moe_topk sigmoid+bias+top-8 0.03 MB 0.8 76 0.152 launch ▏ 1.0% ← the only data-dependent shape + moe_shared grouped GEMM fp8 5.5 MB 54.5 76 0.152 launch ▏ 1.0% + attn_index_score index scan + top-k 33.6 MB 2.0 21 0.147 memory ▏ 0.9% ← the only term that grows with S + lm_head GEMM bf16, tall-skinny 238.0 MB 32.0 2 0.100 memory ▏ 0.6% + ────────────────────── ─────────────────────── ──────── ──── ─── ───── ────── ──────────────────────── + 1,294 nodes 15.887 ms/step · 2,014 tok/s @ B=32 + moe_routed alone 12.208 ms = 76.8% of the step + + facets + memory 440 nodes 14.179 ms 89.2% + launch 854 nodes 1.708 ms 10.8% ← 66% of all nodes, an ninth of the time + compute 0 nodes 0.000 ms 0.0% ← the entire roofline claim, one row +``` + +Two rows are kept **despite** being small: `moe_topk` because it is the only +data-dependent shape in the graph and therefore the thing that blocks CUDA-graph +capture, and `attn_index_score` because "the only term that grows with S" is the +architecture's whole payoff — and because it does not stay small: + +| context S | `attn_index_score` | share of step | step floor | +| --------- | ------------------ | ------------- | ---------- | +| 8,192 | 0.147 ms | 0.9 % | 15.887 ms | +| 131,072 | 2.349 ms | 13.0 % | 18.090 ms | +| **1,048,576** | **18.795 ms** | **54.4 %** | 34.536 ms | + +**At the model's advertised context the indexer scan is the largest node in the +step**, and it is the node IndexShare already cut by 3.7×. Everything anyone says +about GLM-5.2 being "flat in context" is true of the attention *core* and false of +the step. + +**The batch story, and where the labels flip:** + +| B | floor | tok/s | launch nodes | launch time | compute nodes | +| --- | ---------- | ----- | ------------ | ----------- | ------------- | +| 1 | 3.303 ms | 303 | 1,033 | 2.066 ms = **63 %** | 0 | +| 4 | 4.990 ms | 802 | 1,033 | 2.066 ms = 41 % | 0 | +| 16 | 10.637 ms | 1,504 | 854 | 1.708 ms = 16 % | 0 | +| 32 | 15.887 ms | 2,014 | 854 | 1.708 ms = 11 % | 0 | +| 64 | 21.713 ms | 2,948 | 778 | 1.556 ms = 7 % | 76 | +| 128 | 26.948 ms | 4,750 | 620 | 1.240 ms = 5 % | 76 | +| 256 | 33.583 ms | 7,623 | 544 | 1.088 ms = 4 % | 79 | + +**Below B≈16 the step is a launch-bound step wearing a memory-bound model's +clothes.** At B=1, 63 % of the predicted floor is 1,033 kernel launches at 2 µs — +and that figure already assumes CUDA-graph replay. At the eager 5 µs it is 5.17 ms +of launches against a 1.24 ms memory term, and the whole low-batch analysis changes +sign (A4, §5 rank 3). + +### 4.2 Prefill and MTP — regions and what flips them + +Prefill at **P = 8,192, C = 0**; MTP at **D = 5, B = 32, S = 8,192**. All per rank +at TP8/EP8, FP8. + +| Phase | Region | Bound | Why (point at a number) | Precision / peak | Flip variable | +|---|---|---|---|---|---| +| **Pre** | **EP dispatch/combine all-to-all** | **comm — BANDWIDTH** | 1.39 GB/layer × 76 = **105.7 GB**, 117.4 ms = **48.9 % of the step** | BF16 payload | ⚑ **fp8 dispatch halves it**; EP degree; TP-only removes the node and doubles the bank | +| **Pre** | **MoE expert grouped GEMMs** | compute (AI 485) | 1.26 GB/layer **constant in P**, 95.7 GB/pass = 29 % of traffic | **FP8 block-scaled** | **chunk size** (18.9× across 1→64 chunks); imbalance | +| **Pre** | **router GEMM** | **compute** | 26 GF/layer at **fp32's 67 TF/s** → 28.8 ms = **12.0 %** | ⚠ **FP32** — see Q3 | ⚑ whether the engine runs the GEMM in fp32 or only accumulates there | +| **Pre** | `all_reduce` ×158 | **comm — BANDWIDTH** | 174 MB each, 27.5 GB/pass = 30.5 ms | BF16 payload | P; below P≈256 flips to latency | +| **Pre** | projections (`q_a`,`o_proj`,`q_b`,`kv_a`,`kv_b`) | **compute** | AI 1,403 / 963 / 488 vs the fp8 ridge 412 | FP8 e4m3 | prompt length — below P≈512 memory-bound | +| **Pre** | indexer scan ×21 | **compute** | AI 473; `O(P·C + P²/2)` — **the quadratic lives here, not in the core** | **BF16** (unquantised) | P **and** C; at C=131k it is 3.4 ms more | +| **Pre** | attention core ×79 | compute, **linear in C** | FLOPs capped at 2,048 keys/query; **bytes are the whole cache once per request** | FP8 KV | ⚑ **`index_topk`**; C. Not P² | +| **Pre** | permute / combine | memory | 8.5 GB/layer-pass on the `8P`-row expanded tensor, near-zero FLOPs | BF16 activations | top-k; **expert imbalance** | +| **Pre** | *everything else* — embed gather, norms, `lm_head`, `moe_topk` | memory or launch | each under 1.5 %; `lm_head` reads 239.5 MB for **one row per request** | BF16; FP32 top-k | none of them flips | +| **Pre** | **whole prefill pass** | **memory** | **AI 341 vs ridge 412**; 65 % memory / 35 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | +| **MTP** | verify expert GEMMs | **memory** | 256 distinct experts at 192 rows vs 163 at 32 — **+57 %/layer, ~85 % of MTP's cost** | FP8 block-scaled | `B(1+D)` vs E=256; **D**; imbalance | +| **MTP** | draft expert GEMMs ×D | **memory** | the MTP block carries a **full 256-expert bank**; 5 stages × 163 experts, **linear in D, no saturation** | FP8 block-scaled | **D**; batch | +| **MTP** | draft `lm_head` ×5 | memory | 239.5 MB × 5 = **1.20 GB = 19 % of the draft's bytes** | **BF16** | sharded sampling; draft vocab | +| **MTP** | draft `eh_proj` ×5 | memory | `[12288,6144]` BF16, **replicated per rank** | **BF16** — in `modules_to_not_convert` | whether it is TP-sharded | +| **MTP** | verify attention | **memory, unchanged** | 0.43 GB — read **per sequence, not per row**; 1+D rows share one block table | FP8 KV | seq length; explicitly *not* D | +| **MTP** | accept/reject + KV rollback | launch + **host sync** | tiny tensors, but a data-dependent host-visible seq length (S7) | n/a | pointer rewind vs memmove | +| **MTP** | **whole MTP step** | ⚑ **memory above B≈16, launch below — and the two regimes disagree about whether MTP helps** | **1.71×** cost for ≤6 tokens at B=32; break-even α = **0.142** | mixed | **graph capture**; batch; α; D | + +**Hardware sensitivity:** nothing flips between H200 SXM and H20 on the compute +rows — but H20's much lower FP8 peak moves every prefill projection further into +compute-bound, and its bandwidth moves the decode floor directly. + +### 4.3 Same kernel, opposite label — the contradictions worth naming + +| Kernel | Prefill | Decode | Why the same kernel flips | +| --------------------- | ------------------------------------ | --------------------------------------------- | ---------------------------------------------------------------- | +| expert grouped GEMM | **compute** (AI 485) | **memory** (AI 3.1) | M = P vs M = B. **156× apart in AI**, same weights | +| indexer scan | **compute** (AI 473) | **memory** (AI 2.0) | the query count collapses; the key set does not | +| attention core | compute, **whole cache** in bytes | **launch/memory**, one 2,048-window | 8,192 queries' selections union to everything; one query's do not | +| `all_reduce` ×158 | **comm-bandwidth** (30.5 ms wire) | **comm-latency** (0.16 ms floor, 0.8 µs wire) | payload 174 MB vs 688 kB | +| EP all-to-all | **the top line** (48.9 %) | 3.0 % | payload scales with rows; the ring latency does not | +| `lm_head` | memory, 1 row per **request** | memory, **every row every step** | the epilogue is free in prefill and is not in decode | +| `moe_router` (fp32) | **compute** (12.0 % of prefill) | launch (1.0 %) | 26 GF/layer against 67 TF/s only matters when P is large | + +### 4.4 Flip-variable index + +| Flip variable | Rows it flips | Direction and magnitude | +| ------------- | ------------- | ----------------------- | +| **Decode batch B** | every decode GEMM, expert hit-rate, all collectives, the whole-step label (launch below B≈16, memory above), the sign of the MTP decision | expert bytes **sub-linear** in B: 8 experts at B=1, 163 at B=32, 252 at B=128. 303 → 7,623 tok/s across 1→256 | +| **Sequence length S** | the indexer scan, and **only** the indexer scan | 0.3 % of the step at 8K → 13.0 % at 128K → **54.4 % at 1M**. The attention core does not move at all | +| **Prompt length P** | every prefill projection (memory→compute above P≈512), the router, the indexer's quadratic, all-reduce (latency→bandwidth above P≈256) | whole-pass AI 341 at P=8k | +| ⚑ **Chunked prefill / chunk size** | prefill expert GEMMs, permute, and through them the whole prefill label | 8,192 tokens in 1 chunk = **331 GB**; in 64 chunks of 128 = **6,249 GB** for identical FLOPs | +| ⚑ **CUDA-graph capture** | the whole decode step, the whole MTP step, every launch row | at B=1 it is 63 % of the floor; at 5 µs eager it is 81 %. Decides whether MTP is a 3× win or a net loss | +| **EP vs TP** | every collective row, every MoE memory row | under EP8 the a2a is **49 % of prefill** but the per-rank bank is **8× smaller**. Since the bank is ~86 % of decode DRAM, this is a **trade, not a cost** | +| ⚑ **Precision (fp8 vs bf16)** | every weight term, and whether the model fits at all | 1.84× on the decode floor (15.887 ms vs 29.227 ms) and **10.7 → 5.4 H200s** for weights | +| ⚑ **KV dtype** | the attention core, and the 1M-context footprint | 55.2 GB vs 99.9 GB per rank at 1M — the difference between fitting and not | +| ⚑ **Absorbed vs unabsorbed MLA** | `attn_kv_b` + `attn_out_proj` | drops one node and doubles the other's input width: ±2× on 2.4 % of decode, more at prefill | +| **`index_topk`** | the attention core's FLOPs in both phases; its bytes in neither | 2,048 → 4,096 doubles core FLOPs and changes no byte term | +| **Draft depth D, acceptance α** | verify expert GEMMs, whole MTP step | 1.71× cost at D=5; break-even α = 0.142; 1,177 → 6,476 tok/s across α | +| **Expert imbalance** | prefill + decode expert GEMMs, permute, the a2a, grouped-GEMM tail | skew *reduces* bytes while *increasing* tail latency and stalling every other EP rank | + +--- + +## 5. Ranked headroom hypotheses + +Ranked by expected recoverable time × confidence. **All are hypotheses from the +graph**, to be confirmed against a capture. + +| Rank | Region | Prediction | Why | Evidence to inspect | What would prove it wrong | +|---|---|---|---|---|---| +| **1** | **EP all-to-all at prefill** | **≥49 % of prefill time is wire, and roughly half of it is recoverable** | 105.7 GB/pass in BF16. An fp8 dispatch halves the payload outright; overlapping dispatch with the shared-expert GEMM hides more | NCCL kernel duration vs payload at prefill (§6.4 row 5); whether dispatch is bf16 | duration ≪ payload/900 GB/s → the engine already fuses or compresses it | +| **2** | **Grouped-GEMM group sizing (the fork)** | either **76 D2H per token** (no graph capture possible) **or** fixed-capacity padding (**all 256 experts read every step**) | The only data-dependent shape in the graph is `moe_topk`. A stack does one or the other — see §5.2 | `cuda_api_sum` D2H count per decode step | 0 D2H **and** MoE bytes that track `distinct_experts(B)` → a device-side path, nothing to recover | +| **3** | ⚑ **CUDA-graph capture at low batch** | **63 % of the B=1 floor is launch overhead** (81 % at eager 5 µs) | 1,033 nodes × 2 µs = 2.07 ms against a 1.24 ms memory term | `cuda_api_sum` launch count with `--cuda-graph-trace=node`; expect **1** `cudaGraphLaunch` | already captured → this rank is worth nothing, and rank 2's fork is already resolved to "padded" | +| **4** | **The indexer scan at long context** | at 1M it is **54 % of the step**, and IndexShare's 3.7× is already banked | 90.2 GB/step of index keys at S=1M. Levers: fp8 index keys (2×), temporal reuse across steps, a smaller `index_topk_freq` group | `dram__bytes_read.sum` on the scan vs 90.2 GB; whether keys are fp8 | keys already fp8 and no temporal reuse available → architectural | +| **5** | **`moe_routed` — the memory-bound heart** | 76.8 % of decode; **EPLB placement and the grouped-GEMM backend are the levers** | Weight traffic scales with *distinct* experts woken, not with FLOPs. Good placement cuts per-rank distinct traffic; DeepGEMM cuts the constant | measured per-rank expert traffic vs `distinct_experts`; **the real EP imbalance** | measured traffic already at the union prediction with balance ≈1.0 → the bank is the bank | +| **6** | ⚑ **fp32 router at prefill** | **12.0 % of prefill** rests on the reading that the router *GEMM* runs in fp32 | 26 GF/layer at 67 TF/s. If only the softmax/top-k accumulates in fp32 and the GEMM is bf16, this row shrinks ~15× | the engine's router implementation; `cuda_gpu_kern_sum` dtype of the gate GEMM | the GEMM is genuinely fp32 → architectural, and the row stays | +| **7** | **Decode collective placement** | 158 all-reduces + 76 all-to-alls per step on the compute stream, **latency-bound**, with idle SMs around them | 688 kB payloads: the gap *is* the cost. Nothing prevents other layers' work overlapping | NCCL ranges and stream ids on the timeline (S2/S3) | already on a separate stream with overlap → nothing to recover | +| **8** | **Chunk size** | a decode-latency-protecting `--max-num-batched-tokens` can cost **18.9× the prefill bytes** | the expert bank is read per chunk | total prefill MoE DRAM ÷ 95.7 GB — the quotient **is** the chunk count | quotient ≈ 1 → prefill is already unchunked | +| **9** | **`attn_q_a` / `attn_kv_a` replication** | 2.8 % of decode is paid **in full on every rank** and TP does not reduce it | they produce the shared latent, which has nothing to split | per-rank duration of `q_a` vs `q_b` under TP8 | already sharded via DP-attention → the graph is wrong here, not the engine | +| **10** | **IndexShare selection handoff (S9)** | if the top-k round-trips the host, **21 syncs/step** to save 57 kernels | the selection must reach three downstream layers | D2H count attributable to the indexer region | 0 → device-side, and IndexShare is pure win | + +### 5.1 Gate check — every row maps to a GitM-observable category + +| GitM category | Rows | +| -------------------------------- | --------- | +| launch gaps | 3 | +| needless syncs | 2, 10 | +| serialised work | 1, 7 | +| stream underuse / overlap misses | 1, 7 | +| collective placement | 1, 7 | +| dispatch/combine cost | 1, 5 | +| routing imbalance | 5 | +| phase transitions | 8 | +| precision selection | 4, 6 | + +No row sits outside the list. + +### 5.2 Ranks 2 and 3 are the same fork, not two independent bets + +Either group sizes are resolved on the **host** (rank 2) *or* the kernel pads to +**fixed capacity** to keep static shapes (rank 3). A stack does one or the other: + +- host-resolved sizes → exact shapes, no padding, **but a sync per layer and no graph** +- fixed capacity → no sync, graph-capturable, **but all 256 experts read every step** + +**You cannot pay both, and you cannot escape both without a device-side grouped +GEMM.** Counting D2H per step is the single cheapest measurement in this document. + +### 5.3 Deliberately excluded — architectural, not recoverable + +These will look alarming on a timeline and are **not** actionable: + +- **The five MTP draft gaps.** Stage `k` consumes stage `k-1`'s sampled id. A + producer→consumer edge; no scheduling closes it. +- **The accept/reject host readback (S7).** Genuinely data-dependent and genuinely + host-visible. +- **The sampling D2H (S4).** One host round-trip per step is the floor for any + autoregressive decoder. +- **The KV cache being replicated across TP ranks.** One shared MLA latent cannot + be split. That is the architecture doing what it was designed to do. +- **The indexer scan growing with context.** IndexShare already cut it 3.7×; the + remainder is the cost of selecting from an uncompressed history. + +--- + +## 6. Validation plan + +Assume the Nsight Systems / CUPTI trace arrives tomorrow. + +### 6.1 The classification rule — *unexpected ≠ recoverable* + +```mermaid +%%{init: {'theme':'neutral'}}%% +flowchart TD + Q1{"Is there a producer→consumer edge
across this gap?"} + Q1 -- yes --> A1["ARCHITECTURAL
the §2/§3 graphs exist precisely
to answer this without guessing"] + Q1 -- no --> Q2{"Does the gap scale with something
the deployment controls?
batch · chunk size · graph capture
KV dtype · D · EP degree · stream"} + Q2 -- yes --> R1["RECOVERABLE
name the knob AND the expected delta"] + Q2 -- no --> Q3{"Would the gap survive a perfect
implementation of the same model?"} + Q3 -- yes --> A2["ARCHITECTURAL
it is the model, not the stack"] + Q3 -- no --> R2["RECOVERABLE"] + + classDef arch fill:#5a2a2a,stroke:#c88,color:#fff + classDef rec fill:#24543a,stroke:#7c9,color:#fff + class A1,A2 arch + class R1,R2 rec +``` + +Two worked examples, because the rule is easy to agree with and hard to apply: + +- **The draft chain shows five gaps with no kernel spanning them.** Q1: *yes* — + stage `k` consumes stage `k-1`'s token id. **Architectural.** +- **The 234 decode collectives sit on the compute stream with idle gaps around + them.** Q1: *no* — the output feeds the next layer, but nothing prevents *other* + layers' work overlapping. Q2: *yes* — stream assignment. **Recoverable** (rank 7). + +**The trap runs in both directions.** The accept/reject readback will look alarming +and is architectural. The 1,294 kernel launches are entirely *expected* from §3 and +are the largest recoverable item at low batch. **Neither surprise nor familiarity +is evidence.** + +### 6.2 Capture plan — request these before anyone opens a timeline + +| # | Capture | Why | What dies without it | +|---|---|---|---| +| **C1** | Decode, **B ∈ {1, 8, 32, 128}**, S fixed at 8k | the coupon-collector curve, the launch/memory crossover, collective latency share | ranks 2, 3, 5, 7 — the entire low-batch story | +| **C2** | Decode, **S ∈ {8k, 131k, 1M}**, B fixed at 32 | separates the indexer scan from everything else | the whole §4.1 context table; rank 4 | +| **C3** | Prefill, **P ∈ {512, 8192}** × **chunked / unchunked** | the chunking multiplier and the AI curve | ranks 1, 6, 8 | +| **C4** | **MTP on and off** at identical B and S, **with the engine's acceptance metric** | isolates draft cost from verify cost, and α is the only number here that cannot be predicted | all of §2.4/§3.3 | +| **C5** | **TP8-only vs TP8/EP8** at the same B, S | the a2a-vs-bank trade | rank 1, and the EP recommendation | +| **C6** | **The engine's launch arguments and version, as text** | chunk size, graph capture, KV dtype, D, TP/EP, whether MLA is absorbed, router dtype | roughly half of every table in §4–§5 | + +**C6 is not a trace and is worth more than most of the traces.** + +``` +nsys profile --trace=cuda,nvtx,osrt,cublas --cuda-graph-trace=node ... +``` + +**`--cuda-graph-trace=node` is load-bearing.** Without it a captured graph appears +as **one** timeline blob and the kernel count — the thing rank 3 turns on — is +unobservable. + +### 6.3 Instrument map — three tools, three questions + +| Question shape | Instrument | What it gives | +| --- | --- | --- | +| *Where are the gaps, syncs and serialisations?* | **Nsight Systems** (`nsys`) | timeline, CUDA API calls, launch counts, `cudaMemcpyAsync` D2H, NCCL ranges, stream assignment, CPU scheduler time | +| *How many bytes did that kernel actually move?* | **Nsight Compute** (`ncu`) | per-kernel DRAM read/write, L2 traffic, achieved bandwidth, tensor-pipe activity | +| *What happened across the whole run, cheaply?* | **CUPTI activity records** | kernel/memcpy/NCCL counts and durations without `ncu`'s serialising replay — and GitM's own `spec_decode` bucket already separates the MTP scaffolding kernels from ordinary sampling | + +Counters: `dram__bytes_read.sum`, `dram__bytes_write.sum`, `lts__t_bytes.sum` (L2 — +the escape hatch for the expert-bank claim), `gpu__time_duration.sum`, tensor-pipe +active %. **Exact names vary by architecture and `ncu` version.** + +**Two cautions.** `ncu` serialises kernels and destroys exactly the overlap +information ranks 1 and 7 depend on — **profile bytes with `ncu`, overlap with +`nsys`.** And DRAM counters measure traffic that missed L2; a low reading is +ambiguous between "did not read it" and "read it from cache". + +### 6.4 Trace triage — what to measure, in order + +**Both branches are written before the data arrives.** That is the entire point. +Key: **R:** recoverable → the rank it feeds · **A:** architectural, do not chase · +**F:** whole-model falsifier. + +| # | Measure | Scope | Expected | Deviation → meaning | +|---|---|---|---|---| +| **0** | **Launch args, as text** — not a measurement | C6 | chunk size, graph capture, KV dtype, D, TP/EP, absorbed MLA | Resolves or reframes **ranks 1, 2, 3, 6, 8 before a timeline is opened** | +| **1** | `cuda_api_sum` — **D2H count per decode step** | C1 | **0** in the MoE region | **R:** 76/token → host-resolved group sizes, no graph capture → **rank 2**. **R:** 0 but MoE bytes flat in B → padded capacity → **rank 3** instead (§5.2). **A:** none | +| **2** | `cuda_api_sum` — **launches per step**, needs `--cuda-graph-trace=node` | C1 | **1** `cudaGraphLaunch` | **R:** ~1,294 individual launches + CPU gaps below B≈16 → **rank 3**. **F1:** count wildly off 1,294 → the lowering in §3 is wrong by an order of magnitude | +| **3** | `dram__bytes_read.sum` — **MoE region** | C1, C3 | 1.26 GB/layer/rank = **95.7 GB/pass**; **≈86 %** of decode | **R:** prefill total ÷ 95.7 GB > 1 → chunked re-read, and the quotient **is** the chunk count → **rank 8**. **R:** flat in B → padded capacity → **rank 3**. **F2:** much lower → L2 residency, and the central claim of both phases is wrong | +| **4** | `dram__bytes_read.sum` — **indexer region**, swept in S | C2 | 33.6 MB/layer at 8K → **90.2 GB/step at 1M**, on **21 layers only** | **R:** keys read on 78 layers → IndexShare is not being honoured, a hidden 3.7× → **rank 4**. **R:** 2× the prediction → keys are bf16 where fp8 would do. **A:** growth on 21 layers is the architecture | +| **5** | **NCCL kernel duration vs payload** | C1, C3, C5 | prefill **∝ payload** (~900 GB/s, 117 ms a2a); decode **flat, ~2 µs × 234** | **R:** prefill a2a at bf16 payload → fp8 dispatch → **rank 1**. **R:** decode collectives on the compute stream with idle SMs → **rank 7**. **A:** the ring latency floor | +| **6** | **Kernel-name coverage** — every kernel maps to a §3 node | C1 | **complete** | **A/F:** an unmapped kernel block is not headroom, it is a node this graph does not have — and unlike a multimodal model (§2.3) there is no external encoder to explain it away, so it is a **model-validity failure** | +| **7** | `cuda_gpu_kern_sum` — **attention core duration vs S** | C2 | **flat** from 8K to 1M | **R:** grows with S → `index_topk` is not being applied and the core is reading the whole cache. **A:** flat — that is DSA working | +| **8** | **Tensor-pipe active %** | C1, C3 | **near-idle at every decode batch**; prefill well below peak with DRAM and NVLink busy | **R:** pipe busy at low batch → something does far more FLOPs than the graph predicts. **A:** near-idle at decode — that is what decode *is* | +| **9** | **CPU thread sampling, between steps** | C1 | no gap between step *N* and *N*+1 | **R:** CPU-shaped inter-step gap with the scheduler hot → scheduler-bound (S5). **A:** the single sampling D2H | +| **10** | **`spec_decode` bucket counts + acceptance** | C4 | **D = 5** draft stages, 5 extra `lm_head`-shaped GEMMs, verify KV **flat** as D rises | **R:** KV scales with 1+D → rows treated as sequences; use a multi-query kernel. **R:** one `lm_head` for five stages → the draft samples on a sharded vocab already. **A:** the five serial gaps | +| **11** | **Router GEMM dtype** | C3 | fp32 if the config is honoured | **R:** bf16 GEMM with fp32 accumulate → **rank 6 evaporates and §4.2's 12.0 % row shrinks 15×.** **A:** genuinely fp32 → it is the model | + +**Rows 0–3 are the thirty-minute version.** + +--- + +## 7. GitM planner gaps — and what this branch changed + +Read against `GitM-Labs/runtime` @ `main`. §7.0 is what the planner already got +right; §7.1 is what GLM-5.2 broke; §7.2 is the code that now exists. + +### 7.0 What the planner already gets right + +Listed first because several findings here turn out to be things GitM already +models, and proposing them as gaps would waste the pilot's time. + +| Already in the IR | Where | The step that independently derived it | +| --- | --- | --- | +| `positions` vs `sequences` — multi-row verify reads the cache **once per sequence** | `_emit_layer` docstring | §3.3's central MTP result | +| Coupon-collector distinct-expert traffic: FLOPs ∝ `positions·top_k`, bytes ∝ *distinct* experts | `roofline.distinct_experts` | the decode sub-linearity finding, and MTP's break-even | +| EP vs TP as "a collective trade, not a memory trade", with `ep_imbalance` **calibrated from a trace, not predicted** | `roofline.ShardingConfig` | §4.4 | +| A three-way bound — compute / memory / **launch** — via `serial_launches` | `roofline.roofline` | §4.1's launch facet | +| FP8 block-scale overhead: `weight_bytes("fp8") = 1.000244` | `roofline` | the constants section | +| Self-reported model debt: `has_fallback_peaks`, `has_unpriced_collectives` | `graph.Graph` | — | +| Per-checkpoint `provenance: verified / estimated / unmodelled` | `models/*.yaml` | mirrors this note's source/confidence columns | +| Explicit per-layer schedules preferred over a modulo rule | `glm_graph.indexer_kind` | §1 — and GLM has *two* schedules that do not align | +| Prefill as `rows = positions + prefill_tokens` with `logits_rows` for the epilogue | `hybrid_graph` | §3.2 | + +**This is a planner built by someone who has been wrong about these before.** The +gaps below are narrower because of it. + +### 7.1 The gaps GLM-5.2 exposed + +| # | What needs representing | Why the abstraction broke | The extension | Shipped? | +|---|---|---|---|---| +| **G1** | **Three precisions in one block**: fp8 backbone + experts, **bf16 indexer / `lm_head` / `eh_proj`**, **fp32 router** | `GlmMoeDsaModelSpec` carried one `weight_dtype`, and every `add(...)` passed it. No way to say "this op runs at a different width". The indexer was priced at fp8 — **half its real weight traffic on the node that owns 54 % of a 1M-context step** | `op_dtype_overrides: tuple[tuple[str, str], ...]`, consulted by `add()` and by `model_weight_bytes` before the family default. Read from `quantization_config.modules_to_not_convert` + `moe_router_dtype`, never assumed | **yes** | +| **G2** | **Prefill, with DSA asymptotics that invert a dense model's** | The family was decode-only. The obvious fix — copy `hybrid_graph`'s prefill path — produces a **confidently wrong** graph: `BatchConfig.attention_qk_pairs` is the *dense causal* count, which over-charges the DSA core by `C/index_topk` (64× at 128K) and, worse, under-charges its **bytes**, because at prefill the queries' selections union to the whole cache | `core_qk_pairs` / `core_read_entries` / `index_scan_pairs` / `index_scan_entries` — four helpers rather than one, because FLOPs and bytes stop moving together on this architecture | **yes** | +| **G3** | **Two per-layer schedules that do not align** — dense/sparse (3 + 75) and full/shared indexer (3 + period-4) | Already handled via `mlp_layer_types` / `indexer_types`, read verbatim. Worth recording as a *near*-gap: a modulo rule fitted to either one alone misplaces layers while producing an entirely plausible total | none needed | n/a | +| **G4** | **An fp32 peak for a modern SKU** | `hardware_spec_for` left `peak_flops_fp32_per_s` at the A100 default (19.5 TF/s) with the comment *"nothing currently predicts fp32 kernels"*. GLM's router does. On an H200 that default is **3.4× low**, enough to move the router's bound label | `_FP32_PEAKS` keyed by the same SKU substrings, CUDA-core rates (H200 67 TF/s), wired through `hardware_spec_for` | **yes** | +| **G5** | **`gitm plan` dropping the launch bound and mispricing the ridge** | `_render_table` recomputed `bound` as compute-vs-memory, **discarding `"launch"` entirely**, and divided the ridge by `peak_flops_bf16_per_s` regardless of op dtype. So **854 launch-bound nodes printed as memory-bound**, against ridge 206 where fp8 answers to 412 | use the node's own `bound`; print one ridge per dtype present in the graph; add a launch-bound count and a `*` marker where an op's instances disagree | **yes** | +| **G6** | **Two collectives per layer, not one** | `_emit_layer` folded the post-attention and post-FFN all-reduces into one node with double the payload. Bytes right, **count wrong** — and at 688 kB a decode collective is bounded by its ring latency, so the count *is* the cost | `_emit_collective` called at both sub-block boundaries, emitting `tp_all_reduce_attn` and `tp_all_reduce_mlp` separately, with the EP all-to-all on the MoE half only | **yes** | +| **G7** | **An MTP chain D stages deep, each with its own vocabulary projection** | The graph emitted **one** draft block and **one** `lm_head` for what the vendor recipe runs **five** deep. `lm_head` is 19 % of the draft's bytes, so a D-deep chain was understated by ~5× on its largest term | a stage loop in `predict_glm_graph` driven by `BatchConfig.speculative_tokens`, with `mtp_eh_proj` and an `lm_head` per stage; `--spec-tokens` on the CLI | **yes** | +| **G8** | **Which adjacent nodes may overlap and which may not** | `Graph.total_pred_s` is a sum, not a DAG. GLM puts both kinds of serialisation in one step — the draft chain is genuinely serial, the 158 collectives are not — and **both appear as the same positive residual today**. §6.1 is unanswerable without telling them apart | *Not shipped.* It is a cross-family IR change and it is already sequenced on the roadmap. What this note adds is the requirement, and `expected_stream_id=1` on collectives so the stream-concurrency invariant has something to read | **no — deliberately** | + +### 7.2 The one that needed more than a table row + +**G2 must not ship as a copy of another family's prefill path.** Aliasing +`BatchConfig.attention_qk_pairs` into the DSA core is a two-line change that +produces a complete, plausible graph — and it would be wrong in *both* directions +at once: the core's FLOPs over-charged by the ratio of context to `index_topk`, +and its bytes under-charged by the same ratio, because the two errors come from the +same false premise (that a query's selection is the whole cache, or that the +chunk's selection is one query's). The two mistakes partly cancel in the total, +which is exactly what makes them survivable. **A prefill path that is wrong in a +self-cancelling way is worse than no prefill path**, and it is the version that +will get suggested — hence four helpers with four docstrings rather than one alias. + +### 7.3 What this branch actually changed + +``` +gitm/planner/glm_graph.py op_dtype_overrides + dtype_for; four DSA phase helpers; + rows = positions + prefill_tokens throughout; + serial_launches on every node; moe_topk / moe_permute / + moe_combine as their own nodes; _emit_collective ×2 per + layer; the D-stage draft chain with per-stage lm_head; + indexer wk + weights_proj in both the graph and the + footprint; the quantisation-map reader +gitm/planner/context.py _FP32_PEAKS + fp32_peak_for_sku, wired to hardware_spec_for +gitm/planner/registry.py node-owned bound labels, per-dtype ridges, launch count, + --spec-tokens +gitm/planner/model_catalogue.py nested tuple coercion for op_dtype_overrides +gitm/planner/models/ + glm-5.2.yaml the bf16 model fact + the fp32 router + glm-5.2-fp8.yaml NEW — the vendor's recommended deployment +tests/test_glm_graph.py 20 tests: the fp8 footprint, the precision map, the + prefill/decode byte inversion, the D-deep chain, two + collectives, and that a launch floor does not hide an + unpriced collective +docs/glm-5.2/artifacts/*.json DELETED — 29k lines of node dumps that go stale on + every graph change; the commands at the top of this + note regenerate any of them +``` + +--- + +## 8. Open questions and assumptions + +### 8.1 Open questions, ranked by what they change + +| # | Question | What it changes | How to resolve | +|---|---|---|---| +| **Q1** | Does the engine run **absorbed** MLA at decode? | drops `attn_kv_b` and **doubles** `attn_out_proj`'s input width (16384→32768). ±2× on the #4 and #8 lines | serving image / C6 | +| **Q2** | Is the decode step **CUDA-graph captured**? | at B=1 the difference between 1.24 ms and 3.30 ms per step, and it decides whether MTP is a 3× win or a net loss | engine config + `--cuda-graph-trace=node` | +| **Q3** | Is the **router GEMM** fp32, or only its accumulation? | **12.0 % of prefill.** At bf16 the row shrinks ~15× | the engine's MoE gate implementation | +| **Q4** | Is the grouped GEMM **device-sized or host-sized**? | decides **rank 2 *or* rank 3** — the fork in §5.2 | trace D2H count, or the kernel source | +| **Q5** | Is the **EP dispatch** bf16 or fp8? | **half of 105.7 GB** at prefill — the largest single recoverable number here | serving image / C6 | +| **Q6** | Is the **IndexShare selection** passed device-side? | 21 syncs/step if not (S9) | trace D2H attribution | +| **Q7** | Chunked prefill on, at what chunk size? | **up to 18.9× on prefill bytes** | engine launch args (C6) | +| **Q8** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is 0.142; the range 0.5→0.9 is 4,121→6,476 tok/s | engine metrics (C4) — **not predictable from a config** | +| **Q9** | Are the **index keys** cached in fp8 or bf16? | 2× on 90.2 GB/step at 1M context | C6 / trace | +| **Q10** | Is `eh_proj` **TP-sharded** in the MTP block? | 151 MB → 19 MB per rank per draft stage | serving image | +| **Q11** | Does the prefill attention kernel read the selected KV **once per request** or once per query tile? | up to 64× on the prefill core's bytes — the graph takes the optimistic floor | `dram__bytes_read.sum` on the prefill core | + +### 8.2 Assumptions in force + +| # | Assumption | Status | What would falsify it | +|---|---|---|---| +| **A1** | 8×H200 SXM, NVLink, TP8/EP8, fp8 weights and KV | **From the vendor's own published recipe**, not inferred. If production differs, only the constants section redoes | procurement; C6 | +| **A2** | **Dense FP8 peak — halving the datasheet's 3,958 to 1,979** | An inference, held with high confidence. Every tensor-core row is footnoted "with sparsity"; GLM-5.2-FP8 declares no sparsity, and 753.33 GB observed against 755.9 GB predicted **dense** confirms full density — a 2:4 checkpoint would be ~half that. Rooflining against the sparse peak would make every region look 2× more memory-bound than it is | a sparsity flag in the serving image, or a sparse GEMM path in the trace | +| **A3** | Collectives are priced bandwidth-plus-one-launch, on an unresolved stream | `estimated=True` throughout; the stream assignment is a guess about a stack nobody has opened, and **rank 7 depends entirely on it** | the trace | +| **A4** | ~2 µs kernel launch (CUDA-graph replay) | Eager is nearer 5 µs. **The factor of 2.5 moves rank 3 from 63 % to 81 % of the B=1 floor** and moves the launch/memory crossover batch | calibrate from launch-to-launch gaps | +| **A5** | Routing is not pathologically concentrated — `distinct_experts` assumes a uniform router | Real skew touches *fewer* experts, so this over-predicts traffic — the conservative direction. `e_score_correction_bias` exists precisely to spread load | expert-GEMM DRAM read well below 1.26 GB/layer | +| **A6** | The serving path uses a grouped GEMM, not a per-expert loop | Architecture rule; the reference implementation is the *semantics*, not the execution | a per-expert kernel launch pattern in the trace | +| **A7** | 158 collectives per step (2 per layer × 79) | TP convention, now modelled explicitly (G6) | NCCL kernel count per step | +| **A8** | `ep_imbalance = 1.0` | **Declared, not fitted** — it is trace-calibrated by design and there are no traces | any measured skew | +| **A9** | The prefill attention core streams the selected cache **once per request** | An optimistic floor (Q11). A tiled kernel re-reads per query block | `dram__bytes_read.sum` on the prefill core | +| **A10** | The exact kernel names, everywhere | `confidence: none` throughout. The *class* is justified; the implementation is not knowable without the serving image | — | + +--- + +## 9. How to run it + +**A. Predict-only (no GPU, free, works now):** + +```bash +pip install -e . +gitm plan glm-5.2-fp8 --gpu H200 --batch 32 --kv-len 8192 --tp 8 --ep 8 +gitm plan glm-5.2-fp8 --gpu B200 --batch 32 --kv-len 131072 --tp 8 --ep 8 --json +gitm plan --list +``` + +**B. Serve + capture (to get the numbers §6 wants validated).** The footprint math +decides the hardware: **fp8 → one 8×H200 node** (755.9 GB of weights against +1,128 GB, leaving ~370 GB for KV and activations); **bf16 → two nodes** (1,508 GB, +10.7 H200s for weights alone). Full 1M context wants B200/B300 for the extra HBM. + +1. On RunPod take an **8×H200 SXM** pod with a **network volume ≥ 1 TB** for the + 141-shard fp8 checkpoint. +2. Serve with the vendor's own recipe — reproduced here verbatim because §4's + constants assume it: + ```bash + vllm serve zai-org/GLM-5.2-FP8 \ + --kv-cache-dtype fp8 \ + --tensor-parallel-size 8 \ + --speculative-config.method mtp \ + --speculative-config.num_speculative_tokens 5 \ + --tool-call-parser glm47 --reasoning-parser glm45 --enable-auto-tool-choice + ``` + Add `--enable-expert-parallel` for the EP8 shape §4 prices; **without it the + graph's `moe_all_to_all` rows should not appear at all**, and rank 1 does not + exist. That difference is capture C5. +3. Attach the GitM collector and capture a bounded decode window: + ```bash + gitm capture serve # or: gitm capture attach + ``` +4. Import and diff observed-vs-predicted per op. **A residual here is a lead, not a + defect.** + +> ⚠ Take path A first. It is free, and it answers the two questions that gate +> everything else — does the fp8 shape fit (yes, 755.9 GB on 1,128 GB), and is the +> step launch-bound at your batch (yes, below B≈16). + +--- + +## Appendix A — Predicted node tables + +Trace-day reference for §3. **B=32, S=8192, TP8/EP8, FP8 weights and KV, per +rank.** The 79 blocks are exactly `Ld,f` ×3 + `Ls,f` ×18 + `Ls,sh` ×57 + `Lmtp` ×1; +A.2–A.4 are stated as **deltas** from A.1, because everything not listed is +byte-for-byte identical. + +**Two columns are omitted rather than repeated.** Every node runs on the compute +stream except the collectives (`expected_stream_id=1`). Confidence is **high** +throughout — these rows are read from `config.json` and the tensor index — except +the collectives (**medium**, a TP/EP convention) and the S1 histogram readback +(**low**, a hypothesis about a stack nobody has opened). + +### A.1 — Archetype `Ls,sh`, 57 layers (shared indexer + MoE) + +Read straight off the graph, not retyped: every row is a `PredictedNode` at this +shape. **Norms are not nodes here** — `input_layernorm` and +`post_attention_layernorm` are folded into the projection that consumes them and +into `attn_qnorm_rope_insert`, which is one fused kernel in every serving path this +targets. That is a modelling choice, and it is the reason a trace will show ~2 +more small kernels per layer than this table has rows. + +| id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | +|---|---|---|---|---|---|---|---| +| .1 | `attn_q_a` | GEMM, **replicated** | 805.3 MF | 13.110 MB | FP8 | 2.73 | memory | +| .2 | `attn_q_b` | GEMM, head-sharded | 268.4 MF | 4.457 MB | FP8 | 2.00 | launch | +| .3 | `attn_kv_a` | GEMM + cache append | 226.5 MF | 3.990 MB | FP8 | 2.00 | launch | +| .4 | `attn_kv_b` | GEMM, head-sharded (**unabsorbed**) | 117.4 MF | 2.098 MB | FP8 | 2.00 | launch | +| .5 | `attn_score_value` | paged decode attn over ≤2048 entries | 536.9 MF | 41.951 MB | FP8 KV | 8.74 | memory | +| .6 | `attn_qnorm_rope_insert` | fused norm + partial RoPE + insert | 0.4 MF | 0.410 MB | BF16 | 2.00 | launch | +| .7 | `attn_out_proj` | GEMM, tall-skinny (16384→6144) | 805.3 MF | 13.110 MB | FP8 | 2.73 | memory | +| .8 | `tp_all_reduce_attn` | NCCL ring | — | 0.688 MB | BF16 | 2.00 | launch | +| .9 | `moe_router` | GEMM, **replicated** | 100.7 MF | 6.701 MB | **FP32** | 2.00 | launch | +| .10 | `moe_topk` | sigmoid + bias + top-8 + renorm | 0.02 MF | 0.033 MB | BF16 | 2.00 | launch ← **the only data-dependent shape** | +| .11 | `moe_shared` | grouped GEMM ×3, always on | 302.0 MF | 5.539 MB | FP8 | 2.00 | launch | +| .12 | `moe_permute` | gather into expert-major order | 0 F | 0.442 MB | BF16 | 2.00 | launch | +| .13 | `moe_routed` | grouped GEMM ×3, **163 distinct of 256** | 2,415.9 MF | **771.062 MB** | FP8 | **160.64** | **memory** | +| .14 | `moe_combine` | scatter-add × `routed_scaling 2.5` | 3.1 MF | 0.442 MB | BF16 | 2.00 | launch | +| .15 | `moe_all_to_all` | EP dispatch + combine | — | 5.505 MB | BF16 | 6.12 | comm | +| .16 | `tp_all_reduce_mlp` | NCCL ring | — | 0.688 MB | BF16 | 2.00 | launch | + +**Σ per layer: 5.6 GF, 870.2 MB, 0.203 ms.** `.13` alone is **89 % of the layer's +bytes and 79 % of its time**, and it is the same 771 MB whether the layer is one of +57 or one of 75 — which is why every ranked hypothesis in §5 that is not about +launches is about this row. + +### A.2 — Archetype `Ls,f`, 18 layers (full indexer + MoE) + +**Delta from A.1: two nodes inserted after `.4`.** Everything else is identical. + +| id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | +|---|---|---|---|---|---|---|---| +| .4a | `attn_index_proj` | GEMM ×3 — `wq_b` (2048→4096), `wk` (6144→128), `weights_proj` (6144→32), **replicated** | 599.8 MF | 19.937 MB | **BF16** | 4.15 | memory | +| .4b | `attn_index_score` | index scan over the whole history + top-2048 | 67.1 MF | 33.563 MB | FP8 KV | 6.99 | memory | + +`.4a` is BF16 because the indexer is named in `modules_to_not_convert` — at FP8 it +would price at 10.0 MB, and this is the layer type whose cost grows with context. + +**`.4b` is the only node in the model that grows with S**, and it does not stay +small: + +| context S | `.4b` bytes/layer | `.4b` time/layer | Σ over 21 layers | share of step | +| --------- | ----------------- | ---------------- | ---------------- | ------------- | +| 8,192 | 33.6 MB | 6.99 µs | 0.147 ms | 0.9 % | +| 131,072 | 537.0 MB | 0.112 ms | 2.349 ms | 13.0 % | +| 1,048,576 | **4,296.0 MB** | **0.895 ms** | **18.795 ms** | **54.4 %** | + +**Σ per layer at S=8192: 6.2 GF, 923.7 MB, 0.214 ms** — 5 % more than `Ls,sh`, and +that 5 % is the whole price of IndexShare's 21-of-78 schedule at short context. + +### A.3 — Archetype `Ld,f`, 3 layers (full indexer + DENSE FFN) + +**Delta from A.2: `.9`–`.15` replaced by two nodes.** No router, no top-k, no +expert bank — and therefore **no data-dependent shape and no expert-parallel +traffic** on these three layers. They are the only blocks in the model a CUDA graph +could capture unconditionally. + +| id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | +|---|---|---|---|---|---|---|---| +| .9′ | `mlp_gate_up` | GEMM (6144→3072/rank) | 1,208.0 MF | 19.469 MB | FP8 | 4.06 | memory | +| .10′ | `mlp_down` | GEMM (1536/rank→6144) | 604.0 MF | 9.931 MB | FP8 | 2.07 | memory | + +**Σ per layer: 5.2 GF, 168.9 MB, 0.050 ms** — **a quarter the time of a MoE layer +at a fifth the bytes.** Three of 78 layers are 0.9 % of the step. + +### A.4 — MTP draft chain, per stage (×5 at D=5) + +Rows are `[B,·]` = `[32,·]`, **not** the verify pass's `[192,·]`: the draft proposes +for the sequence, not for the verify rows. + +| id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | +|---|---|---|---|---|---|---|---| +| M.1 | `mtp_eh_proj` | GEMM `[12288→6144]`, **replicated** | 4,831.8 MF | **152.175 MB** | **BF16** | 31.70 | memory | +| M.2–M.17 | the whole `Ls,sh` block (A.1) | as A.1 | 5.6 GF | 870.2 MB | mixed | 203 | memory | +| M.18 | `lm_head` | GEMM, vocab-sharded `[6144→19360]` | 7,612.7 MF | **239.528 MB** | **BF16** | 49.90 | memory | +| M.19 | `argmax` + D2H | sampling + host round-trip | — | small | BF16 | — | **sync (S6)** | + +**Σ per stage: 18.0 GF, 1,261.9 MB, 0.285 ms — ×5 = 6.31 GB, 1.423 ms.** + +Three things in that table are the whole §3.3 argument: + +- **M.2–M.17 is a full MoE block.** The MTP module carries its own 256-expert + `mlp.experts.*` bank in the checkpoint, so 69 % of a draft stage's bytes are + expert weights it re-reads every stage. There is no saturation to help: 32 rows + wakes ~163 experts, five times over. +- **M.1 and M.18 are BF16**, both named in `modules_to_not_convert`, and together + they are **31 %** of the stage. `eh_proj` is replicated per rank (Q10); the + vocabulary projection is shared with the backbone and gets no cheaper for being + a draft. +- **What is absent:** no `attn_index_proj`, no `attn_index_score`. The MTP block + has no indexer tensors in the weight map, which is + `index_share_for_mtp_iteration: true` made visible — and it means the draft + inherits the top-2048 selection the backbone already paid for. diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index e2e97cb..2f228ea 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -729,7 +729,10 @@ def w_bytes(op: str, default: str) -> float: spec.act_dtype, ) - _emit_collective(g, spec, hw, layer, "tp_all_reduce_mlp", rows, sh, prefix) + _emit_collective( + g, spec, hw, layer, "tp_all_reduce_mlp", rows, sh, prefix, + dispatches_experts=True, + ) def _emit_collective( @@ -741,6 +744,8 @@ def _emit_collective( rows: float, sh: ShardingConfig, prefix: str, + *, + dispatches_experts: bool = False, ) -> None: """Emit the cross-rank traffic that closes one sub-block. @@ -751,6 +756,12 @@ def _emit_collective( bytes, so the count is the cost. Under expert parallelism the MoE half additionally dispatches and combines across expert ranks. + ``dispatches_experts`` gates the expert-parallel all-to-all. Only a mixture + layer dispatches tokens to expert ranks; the three dense-FFN layers compute + their whole FFN locally and emit the all-reduce alone. Charging them an + all-to-all would put 5.5 MB of wire traffic per layer on a block that has no + experts to send anything to. + ``serial_launches`` is withheld when the SKU carries no interconnect bandwidth, so an unpriced collective still predicts zero time and :attr:`Graph.has_unpriced_collectives` keeps reporting it. A latency floor @@ -779,7 +790,7 @@ def add_link(name_op: str, byts: float) -> None: ) ) - if sh.ep > 1 and op == "tp_all_reduce_mlp": + if sh.ep > 1 and dispatches_experts: off_rank = (sh.ep - 1) / sh.ep add_link( "moe_all_to_all", diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index 4d75b58..b33e403 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -20,10 +20,10 @@ from __future__ import annotations -import pytest - from dataclasses import replace +import pytest + from gitm.planner.glm_graph import ( GlmMoeDsaModelSpec, core_read_entries, @@ -292,6 +292,25 @@ def test_two_collectives_per_layer_not_one(): assert len(_ops(g, "tp_all_reduce_mlp")) == n_blocks +def test_dense_layers_dispatch_no_experts(): + """Only a mixture layer sends tokens to expert ranks. + + The three dense-FFN layers compute their whole FFN locally. Charging them an + expert-parallel all-to-all puts wire traffic on a block with no experts to + send anything to — and on this model that node is half of prefill, so a + spurious three layers of it is not a rounding error. + """ + spec = _spec() + g = predict_glm_graph( + spec, batch=BatchConfig(batch=1, kv_cache_len=4096), + sharding=ShardingConfig(tp=8, ep=8), + ) + assert len(_ops(g, "moe_all_to_all")) == ( + spec.n_sparse_mlp_layers + spec.num_nextn_predict_layers + ) + assert not [n for n in g.nodes if n.op == "moe_all_to_all" and n.layer < 3] + + def test_unpriced_collectives_stay_visible_under_the_launch_floor(): """A launch floor must not quietly price a collective the SKU cannot price. From 776663b05f80fdbefd6ed32e4b59ed54449691b2 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 13:28:46 -0700 Subject: [PATCH 05/35] GLM-5.2: restore the indexer's 32 head scores; no draft head on prefill Two corrections found by reading Appendix A's per-node dump against the graph, both in the node the note's headline claim is about. - attn_index_score dropped index_n_heads in the prefill rewrite, scoring each candidate once instead of once per index head. The 32 heads each dot against the single shared 128-d key wk produces per token (MQA-style), which is why the key bytes carry no head factor and the score FLOPs do. The decode picture is unchanged -- the node was memory-bound at every context and stays so, AI 2.0 -> 64.0, and the scan is still 54.4% of a 1M step. Prefill moves: 5.77 TF against 22 MB of keys, emphatically compute-bound, and the whole-pass AI goes 341 -> 358. - The draft head ran on a pure-prefill step, adding 18 zero-work nodes whose only cost was their launches -- a launch facet made of kernels that never ran. A draft proposes continuations; a prefill chunk has nothing yet to continue. Also: the doc's prefill reproduce command now names the shape its table is labelled with (--batch 0 --kv-len 0), and every affected figure in the note is re-derived from the fixed graph. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 81 +++++++++++++++++++------------------ gitm/planner/glm_graph.py | 14 ++++++- tests/test_glm_graph.py | 39 ++++++++++++++++++ 3 files changed, 93 insertions(+), 41 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index e8f27ad..9712dee 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -13,7 +13,7 @@ Reproduce any figure here: ```bash gitm plan glm-5.2-fp8 --gpu H200 --batch 32 --kv-len 8192 --tp 8 --ep 8 gitm plan glm-5.2-fp8 --gpu H200 --batch 32 --kv-len 8192 --tp 8 --ep 8 --spec-tokens 5 -gitm plan glm-5.2-fp8 --gpu H200 --prefill-tokens 8192 --tp 8 --ep 8 +gitm plan glm-5.2-fp8 --gpu H200 --prefill-tokens 8192 --batch 0 --kv-len 0 --tp 8 --ep 8 gitm plan glm-5.2-fp8 --gpu H200 --sweep 1,4,16,32,64,128,256 --kv-len 8192 --tp 8 --ep 8 gitm plan glm-5.2 --gpu H200 --batch 32 --kv-len 8192 --tp 8 --ep 8 # what bf16 costs ``` @@ -243,7 +243,7 @@ tokens issues `8P` token→expert assignments. Once `8P ≫ 256` — P above a f hundred — every expert receives at least one token, so **every layer reads its entire expert bank**, 1.26 GB per rank per layer at EP8, **constant in P**. That is 95.7 GB per pass, 29 % of all prefill traffic. But it is not the top line: **the EP -all-to-all is**, at 105.7 GB and 49 % of predicted prefill time. Three quarters of +all-to-all is**, at 105.7 GB and 48 % of predicted prefill time. Three quarters of prefill cost is the MoE path, and under expert parallelism most of that is *wire*, not DRAM. @@ -279,7 +279,7 @@ vocabulary weights *every step* rather than once per request. │ full-indexer layer (21 of 78)? │ │ yes → wq_b/wk/gate BF16, then │ ← the ONLY term that grows with S. │ score the WHOLE history │ 0.9 % of the step at 8K, - │ B×C×128 bytes, top-2048 │ 13.0 % at 128K, 54.4 % at 1M + │ B×C×128 bytes, 32 heads, │ 13.0 % at 128K, 54.4 % at 1M │ no → reuse. NO KERNEL. (57 layers) │ └────┬─────────────────────────────────────┘ │ @@ -406,9 +406,11 @@ against P = 1 decode row per sequence: - Every projection crosses into **compute-bound**: `q_a`/`o_proj` run AI 1,403, `q_b` 963, `kv_a` 488 — all far past the fp8 ridge of 412. -- **The indexer scan flips from memory to compute.** At decode it streams keys for - one query per sequence (AI 2.0); at prefill it scores 8,192 queries against the - same keys (AI 473). +- **The indexer scan flips from memory to compute, and by two orders of + magnitude.** At decode 32 index heads score one query per sequence against a + streamed key set (AI 64, memory-bound at every context). At prefill the same keys + are read once and scored by 8,192 queries — **5.77 TF against 22 MB**, and + emphatically compute-bound. - **The attention core's bytes and FLOPs stop moving together.** FLOPs are capped at 2,048 keys per query either way. Bytes are not: at decode one sequence reads its own 2,048-entry window; at prefill 8,192 queries each select a *different* @@ -426,11 +428,11 @@ against P = 1 decode row per sequence: | Prefill, P=8192, C=0, TP8/EP8, per rank | value | | --- | --- | -| predicted floor | **240.4 ms** for the chunk (34.1 k tok/s) | -| bytes moved | **331.0 GB** | -| FLOPs | **112.8 TF** | -| whole-pass AI | **341** — below the fp8 ridge of 412, so **memory-bound overall** | -| facets | compute 666 n / 83.5 ms / 34.7 % · memory 539 n / 156.7 ms / 65.2 % · launch 89 n / 0.2 ms | +| predicted floor | **243.1 ms** for the chunk (33.7 k tok/s) | +| bytes moved | **330.6 GB** | +| FLOPs | **118.3 TF** | +| whole-pass AI | **358** — below the fp8 ridge of 412, so **memory-bound overall** | +| facets | compute 666 n / 86.3 ms / 35.5 % · memory 535 n / 156.6 ms / 64.4 % · launch 75 n / 0.1 ms | > ⚠ **The prefill rows overturn the dense-model intuition, and the reversal is > worth keeping visible.** Dense intuition says prefill is compute-bound. It is @@ -446,12 +448,12 @@ against P = 1 decode row per sequence: | chunking | bytes | floor | | ------------------- | ----------- | ---------- | -| 1 × 8,192 | **331 GB** | 240 ms | -| 2 × 4,096 | 427 GB | 256 ms | -| 8 × 1,024 | 1,000 GB | 372 ms | -| 64 × 128 | **6,249 GB**| **1,505 ms** | +| 1 × 8,192 | **331 GB** | 243 ms | +| 2 × 4,096 | 426 GB | 259 ms | +| 8 × 1,024 | 996 GB | 374 ms | +| 64 × 128 | **6,221 GB**| **1,499 ms** | -**18.9× the bytes for identical FLOPs.** The expert bank is read per *chunk*, not per +**18.8× the bytes for identical FLOPs.** The expert bank is read per *chunk*, not per token — so a small `--max-num-batched-tokens`, chosen to protect decode latency, is paid for here at a rate nothing in a per-token cost model shows. @@ -562,7 +564,7 @@ ridge 412 F/B (fp8) · 206 (bf16) · 14 (fp32) · launch floor 2.0 µs (graph-re moe_router GEMM **fp32**, replicated 6.7 MB 15.0 76 0.152 launch ▏ 1.0% moe_topk sigmoid+bias+top-8 0.03 MB 0.8 76 0.152 launch ▏ 1.0% ← the only data-dependent shape moe_shared grouped GEMM fp8 5.5 MB 54.5 76 0.152 launch ▏ 1.0% - attn_index_score index scan + top-k 33.6 MB 2.0 21 0.147 memory ▏ 0.9% ← the only term that grows with S + attn_index_score index scan + top-k 33.6 MB 64.0 21 0.147 memory ▏ 0.9% ← the only term that grows with S lm_head GEMM bf16, tall-skinny 238.0 MB 32.0 2 0.100 memory ▏ 0.6% ────────────────────── ─────────────────────── ──────── ──── ─── ───── ────── ──────────────────────── 1,294 nodes 15.887 ms/step · 2,014 tok/s @ B=32 @@ -615,16 +617,16 @@ at TP8/EP8, FP8. | Phase | Region | Bound | Why (point at a number) | Precision / peak | Flip variable | |---|---|---|---|---|---| -| **Pre** | **EP dispatch/combine all-to-all** | **comm — BANDWIDTH** | 1.39 GB/layer × 76 = **105.7 GB**, 117.4 ms = **48.9 % of the step** | BF16 payload | ⚑ **fp8 dispatch halves it**; EP degree; TP-only removes the node and doubles the bank | -| **Pre** | **MoE expert grouped GEMMs** | compute (AI 485) | 1.26 GB/layer **constant in P**, 95.7 GB/pass = 29 % of traffic | **FP8 block-scaled** | **chunk size** (18.9× across 1→64 chunks); imbalance | -| **Pre** | **router GEMM** | **compute** | 26 GF/layer at **fp32's 67 TF/s** → 28.8 ms = **12.0 %** | ⚠ **FP32** — see Q3 | ⚑ whether the engine runs the GEMM in fp32 or only accumulates there | +| **Pre** | **EP dispatch/combine all-to-all** | **comm — BANDWIDTH** | 1.39 GB/layer × 76 = **105.7 GB**, 117.4 ms = **48.3 % of the step** | BF16 payload | ⚑ **fp8 dispatch halves it**; EP degree; TP-only removes the node and doubles the bank | +| **Pre** | **MoE expert grouped GEMMs** | compute (AI 485) | 1.26 GB/layer **constant in P**, 95.7 GB/pass = 29 % of traffic | **FP8 block-scaled** | **chunk size** (18.8× across 1→64 chunks); imbalance | +| **Pre** | **router GEMM** | **compute** | 26 GF/layer at **fp32's 67 TF/s** → 28.8 ms = **11.9 %** | ⚠ **FP32** — see Q3 | ⚑ whether the engine runs the GEMM in fp32 or only accumulates there | | **Pre** | `all_reduce` ×158 | **comm — BANDWIDTH** | 174 MB each, 27.5 GB/pass = 30.5 ms | BF16 payload | P; below P≈256 flips to latency | | **Pre** | projections (`q_a`,`o_proj`,`q_b`,`kv_a`,`kv_b`) | **compute** | AI 1,403 / 963 / 488 vs the fp8 ridge 412 | FP8 e4m3 | prompt length — below P≈512 memory-bound | -| **Pre** | indexer scan ×21 | **compute** | AI 473; `O(P·C + P²/2)` — **the quadratic lives here, not in the core** | **BF16** (unquantised) | P **and** C; at C=131k it is 3.4 ms more | +| **Pre** | indexer scan ×21 | **compute** | **5.77 TF** against 22 MB of keys — `O(P·C + P²/2)` × 32 heads. **The quadratic lives here, not in the core** | **BF16** proj, fp8 keys | P **and** C | | **Pre** | attention core ×79 | compute, **linear in C** | FLOPs capped at 2,048 keys/query; **bytes are the whole cache once per request** | FP8 KV | ⚑ **`index_topk`**; C. Not P² | | **Pre** | permute / combine | memory | 8.5 GB/layer-pass on the `8P`-row expanded tensor, near-zero FLOPs | BF16 activations | top-k; **expert imbalance** | | **Pre** | *everything else* — embed gather, norms, `lm_head`, `moe_topk` | memory or launch | each under 1.5 %; `lm_head` reads 239.5 MB for **one row per request** | BF16; FP32 top-k | none of them flips | -| **Pre** | **whole prefill pass** | **memory** | **AI 341 vs ridge 412**; 65 % memory / 35 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | +| **Pre** | **whole prefill pass** | **memory** | **AI 358 vs ridge 412**; 64 % memory / 36 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | | **MTP** | verify expert GEMMs | **memory** | 256 distinct experts at 192 rows vs 163 at 32 — **+57 %/layer, ~85 % of MTP's cost** | FP8 block-scaled | `B(1+D)` vs E=256; **D**; imbalance | | **MTP** | draft expert GEMMs ×D | **memory** | the MTP block carries a **full 256-expert bank**; 5 stages × 163 experts, **linear in D, no saturation** | FP8 block-scaled | **D**; batch | | **MTP** | draft `lm_head` ×5 | memory | 239.5 MB × 5 = **1.20 GB = 19 % of the draft's bytes** | **BF16** | sharded sampling; draft vocab | @@ -642,12 +644,12 @@ compute-bound, and its bandwidth moves the decode floor directly. | Kernel | Prefill | Decode | Why the same kernel flips | | --------------------- | ------------------------------------ | --------------------------------------------- | ---------------------------------------------------------------- | | expert grouped GEMM | **compute** (AI 485) | **memory** (AI 3.1) | M = P vs M = B. **156× apart in AI**, same weights | -| indexer scan | **compute** (AI 473) | **memory** (AI 2.0) | the query count collapses; the key set does not | +| indexer scan | **compute** (5.77 TF / 22 MB) | **memory** (AI 64) | the query count collapses; the key set does not | | attention core | compute, **whole cache** in bytes | **launch/memory**, one 2,048-window | 8,192 queries' selections union to everything; one query's do not | | `all_reduce` ×158 | **comm-bandwidth** (30.5 ms wire) | **comm-latency** (0.16 ms floor, 0.8 µs wire) | payload 174 MB vs 688 kB | -| EP all-to-all | **the top line** (48.9 %) | 3.0 % | payload scales with rows; the ring latency does not | +| EP all-to-all | **the top line** (48.3 %) | 3.0 % | payload scales with rows; the ring latency does not | | `lm_head` | memory, 1 row per **request** | memory, **every row every step** | the epilogue is free in prefill and is not in decode | -| `moe_router` (fp32) | **compute** (12.0 % of prefill) | launch (1.0 %) | 26 GF/layer against 67 TF/s only matters when P is large | +| `moe_router` (fp32) | **compute** (11.9 % of prefill) | launch (1.0 %) | 26 GF/layer against 67 TF/s only matters when P is large | ### 4.4 Flip-variable index @@ -655,10 +657,10 @@ compute-bound, and its bandwidth moves the decode floor directly. | ------------- | ------------- | ----------------------- | | **Decode batch B** | every decode GEMM, expert hit-rate, all collectives, the whole-step label (launch below B≈16, memory above), the sign of the MTP decision | expert bytes **sub-linear** in B: 8 experts at B=1, 163 at B=32, 252 at B=128. 303 → 7,623 tok/s across 1→256 | | **Sequence length S** | the indexer scan, and **only** the indexer scan | 0.3 % of the step at 8K → 13.0 % at 128K → **54.4 % at 1M**. The attention core does not move at all | -| **Prompt length P** | every prefill projection (memory→compute above P≈512), the router, the indexer's quadratic, all-reduce (latency→bandwidth above P≈256) | whole-pass AI 341 at P=8k | -| ⚑ **Chunked prefill / chunk size** | prefill expert GEMMs, permute, and through them the whole prefill label | 8,192 tokens in 1 chunk = **331 GB**; in 64 chunks of 128 = **6,249 GB** for identical FLOPs | +| **Prompt length P** | every prefill projection (memory→compute above P≈512), the router, the indexer's quadratic, all-reduce (latency→bandwidth above P≈256) | whole-pass AI 358 at P=8k | +| ⚑ **Chunked prefill / chunk size** | prefill expert GEMMs, permute, and through them the whole prefill label | 8,192 tokens in 1 chunk = **331 GB**; in 64 chunks of 128 = **6,221 GB** for identical FLOPs | | ⚑ **CUDA-graph capture** | the whole decode step, the whole MTP step, every launch row | at B=1 it is 63 % of the floor; at 5 µs eager it is 81 %. Decides whether MTP is a 3× win or a net loss | -| **EP vs TP** | every collective row, every MoE memory row | under EP8 the a2a is **49 % of prefill** but the per-rank bank is **8× smaller**. Since the bank is ~86 % of decode DRAM, this is a **trade, not a cost** | +| **EP vs TP** | every collective row, every MoE memory row | under EP8 the a2a is **48 % of prefill** but the per-rank bank is **8× smaller**. Since the bank is ~86 % of decode DRAM, this is a **trade, not a cost** | | ⚑ **Precision (fp8 vs bf16)** | every weight term, and whether the model fits at all | 1.84× on the decode floor (15.887 ms vs 29.227 ms) and **10.7 → 5.4 H200s** for weights | | ⚑ **KV dtype** | the attention core, and the 1M-context footprint | 55.2 GB vs 99.9 GB per rank at 1M — the difference between fitting and not | | ⚑ **Absorbed vs unabsorbed MLA** | `attn_kv_b` + `attn_out_proj` | drops one node and doubles the other's input width: ±2× on 2.4 % of decode, more at prefill | @@ -675,14 +677,14 @@ graph**, to be confirmed against a capture. | Rank | Region | Prediction | Why | Evidence to inspect | What would prove it wrong | |---|---|---|---|---|---| -| **1** | **EP all-to-all at prefill** | **≥49 % of prefill time is wire, and roughly half of it is recoverable** | 105.7 GB/pass in BF16. An fp8 dispatch halves the payload outright; overlapping dispatch with the shared-expert GEMM hides more | NCCL kernel duration vs payload at prefill (§6.4 row 5); whether dispatch is bf16 | duration ≪ payload/900 GB/s → the engine already fuses or compresses it | +| **1** | **EP all-to-all at prefill** | **≥48 % of prefill time is wire, and roughly half of it is recoverable** | 105.7 GB/pass in BF16. An fp8 dispatch halves the payload outright; overlapping dispatch with the shared-expert GEMM hides more | NCCL kernel duration vs payload at prefill (§6.4 row 5); whether dispatch is bf16 | duration ≪ payload/900 GB/s → the engine already fuses or compresses it | | **2** | **Grouped-GEMM group sizing (the fork)** | either **76 D2H per token** (no graph capture possible) **or** fixed-capacity padding (**all 256 experts read every step**) | The only data-dependent shape in the graph is `moe_topk`. A stack does one or the other — see §5.2 | `cuda_api_sum` D2H count per decode step | 0 D2H **and** MoE bytes that track `distinct_experts(B)` → a device-side path, nothing to recover | | **3** | ⚑ **CUDA-graph capture at low batch** | **63 % of the B=1 floor is launch overhead** (81 % at eager 5 µs) | 1,033 nodes × 2 µs = 2.07 ms against a 1.24 ms memory term | `cuda_api_sum` launch count with `--cuda-graph-trace=node`; expect **1** `cudaGraphLaunch` | already captured → this rank is worth nothing, and rank 2's fork is already resolved to "padded" | | **4** | **The indexer scan at long context** | at 1M it is **54 % of the step**, and IndexShare's 3.7× is already banked | 90.2 GB/step of index keys at S=1M. Levers: fp8 index keys (2×), temporal reuse across steps, a smaller `index_topk_freq` group | `dram__bytes_read.sum` on the scan vs 90.2 GB; whether keys are fp8 | keys already fp8 and no temporal reuse available → architectural | | **5** | **`moe_routed` — the memory-bound heart** | 76.8 % of decode; **EPLB placement and the grouped-GEMM backend are the levers** | Weight traffic scales with *distinct* experts woken, not with FLOPs. Good placement cuts per-rank distinct traffic; DeepGEMM cuts the constant | measured per-rank expert traffic vs `distinct_experts`; **the real EP imbalance** | measured traffic already at the union prediction with balance ≈1.0 → the bank is the bank | -| **6** | ⚑ **fp32 router at prefill** | **12.0 % of prefill** rests on the reading that the router *GEMM* runs in fp32 | 26 GF/layer at 67 TF/s. If only the softmax/top-k accumulates in fp32 and the GEMM is bf16, this row shrinks ~15× | the engine's router implementation; `cuda_gpu_kern_sum` dtype of the gate GEMM | the GEMM is genuinely fp32 → architectural, and the row stays | +| **6** | ⚑ **fp32 router at prefill** | **11.9 % of prefill** rests on the reading that the router *GEMM* runs in fp32 | 26 GF/layer at 67 TF/s. If only the softmax/top-k accumulates in fp32 and the GEMM is bf16, this row shrinks ~15× | the engine's router implementation; `cuda_gpu_kern_sum` dtype of the gate GEMM | the GEMM is genuinely fp32 → architectural, and the row stays | | **7** | **Decode collective placement** | 158 all-reduces + 76 all-to-alls per step on the compute stream, **latency-bound**, with idle SMs around them | 688 kB payloads: the gap *is* the cost. Nothing prevents other layers' work overlapping | NCCL ranges and stream ids on the timeline (S2/S3) | already on a separate stream with overlap → nothing to recover | -| **8** | **Chunk size** | a decode-latency-protecting `--max-num-batched-tokens` can cost **18.9× the prefill bytes** | the expert bank is read per chunk | total prefill MoE DRAM ÷ 95.7 GB — the quotient **is** the chunk count | quotient ≈ 1 → prefill is already unchunked | +| **8** | **Chunk size** | a decode-latency-protecting `--max-num-batched-tokens` can cost **18.8× the prefill bytes** | the expert bank is read per chunk | total prefill MoE DRAM ÷ 95.7 GB — the quotient **is** the chunk count | quotient ≈ 1 → prefill is already unchunked | | **9** | **`attn_q_a` / `attn_kv_a` replication** | 2.8 % of decode is paid **in full on every rank** and TP does not reduce it | they produce the shared latent, which has nothing to split | per-rank duration of `q_a` vs `q_b` under TP8 | already sharded via DP-attention → the graph is wrong here, not the engine | | **10** | **IndexShare selection handoff (S9)** | if the top-k round-trips the host, **21 syncs/step** to save 57 kernels | the selection must reach three downstream layers | D2H count attributable to the indexer region | 0 → device-side, and IndexShare is pure win | @@ -823,7 +825,7 @@ Key: **R:** recoverable → the rank it feeds · **A:** architectural, do not ch | **8** | **Tensor-pipe active %** | C1, C3 | **near-idle at every decode batch**; prefill well below peak with DRAM and NVLink busy | **R:** pipe busy at low batch → something does far more FLOPs than the graph predicts. **A:** near-idle at decode — that is what decode *is* | | **9** | **CPU thread sampling, between steps** | C1 | no gap between step *N* and *N*+1 | **R:** CPU-shaped inter-step gap with the scheduler hot → scheduler-bound (S5). **A:** the single sampling D2H | | **10** | **`spec_decode` bucket counts + acceptance** | C4 | **D = 5** draft stages, 5 extra `lm_head`-shaped GEMMs, verify KV **flat** as D rises | **R:** KV scales with 1+D → rows treated as sequences; use a multi-query kernel. **R:** one `lm_head` for five stages → the draft samples on a sharded vocab already. **A:** the five serial gaps | -| **11** | **Router GEMM dtype** | C3 | fp32 if the config is honoured | **R:** bf16 GEMM with fp32 accumulate → **rank 6 evaporates and §4.2's 12.0 % row shrinks 15×.** **A:** genuinely fp32 → it is the model | +| **11** | **Router GEMM dtype** | C3 | fp32 if the config is honoured | **R:** bf16 GEMM with fp32 accumulate → **rank 6 evaporates and §4.2's 11.9 % row shrinks 15×.** **A:** genuinely fp32 → it is the model | **Rows 0–3 are the thirty-minute version.** @@ -897,10 +899,11 @@ gitm/planner/model_catalogue.py nested tuple coercion for op_dtype_overrides gitm/planner/models/ glm-5.2.yaml the bf16 model fact + the fp32 router glm-5.2-fp8.yaml NEW — the vendor's recommended deployment -tests/test_glm_graph.py 20 tests: the fp8 footprint, the precision map, the +tests/test_glm_graph.py 23 tests: the fp8 footprint, the precision map, the prefill/decode byte inversion, the D-deep chain, two - collectives, and that a launch floor does not hide an - unpriced collective + collectives, the 32-head index scan, that a pure-prefill + step runs no draft head, and that a launch floor + does not hide an unpriced collective docs/glm-5.2/artifacts/*.json DELETED — 29k lines of node dumps that go stale on every graph change; the commands at the top of this note regenerate any of them @@ -916,11 +919,11 @@ docs/glm-5.2/artifacts/*.json DELETED — 29k lines of node dumps that go sta |---|---|---|---| | **Q1** | Does the engine run **absorbed** MLA at decode? | drops `attn_kv_b` and **doubles** `attn_out_proj`'s input width (16384→32768). ±2× on the #4 and #8 lines | serving image / C6 | | **Q2** | Is the decode step **CUDA-graph captured**? | at B=1 the difference between 1.24 ms and 3.30 ms per step, and it decides whether MTP is a 3× win or a net loss | engine config + `--cuda-graph-trace=node` | -| **Q3** | Is the **router GEMM** fp32, or only its accumulation? | **12.0 % of prefill.** At bf16 the row shrinks ~15× | the engine's MoE gate implementation | +| **Q3** | Is the **router GEMM** fp32, or only its accumulation? | **11.9 % of prefill.** At bf16 the row shrinks ~15× | the engine's MoE gate implementation | | **Q4** | Is the grouped GEMM **device-sized or host-sized**? | decides **rank 2 *or* rank 3** — the fork in §5.2 | trace D2H count, or the kernel source | | **Q5** | Is the **EP dispatch** bf16 or fp8? | **half of 105.7 GB** at prefill — the largest single recoverable number here | serving image / C6 | | **Q6** | Is the **IndexShare selection** passed device-side? | 21 syncs/step if not (S9) | trace D2H attribution | -| **Q7** | Chunked prefill on, at what chunk size? | **up to 18.9× on prefill bytes** | engine launch args (C6) | +| **Q7** | Chunked prefill on, at what chunk size? | **up to 18.8× on prefill bytes** | engine launch args (C6) | | **Q8** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is 0.142; the range 0.5→0.9 is 4,121→6,476 tok/s | engine metrics (C4) — **not predictable from a config** | | **Q9** | Are the **index keys** cached in fp8 or bf16? | 2× on 90.2 GB/step at 1M context | C6 / trace | | **Q10** | Is `eh_proj` **TP-sharded** in the MTP block? | 151 MB → 19 MB per rank per draft stage | serving image | @@ -1040,7 +1043,7 @@ launches is about this row. | id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | |---|---|---|---|---|---|---|---| | .4a | `attn_index_proj` | GEMM ×3 — `wq_b` (2048→4096), `wk` (6144→128), `weights_proj` (6144→32), **replicated** | 599.8 MF | 19.937 MB | **BF16** | 4.15 | memory | -| .4b | `attn_index_score` | index scan over the whole history + top-2048 | 67.1 MF | 33.563 MB | FP8 KV | 6.99 | memory | +| .4b | `attn_index_score` | index scan over the whole history + top-2048 | 2,147.5 MF | 33.563 MB | FP8 KV | 6.99 | memory | `.4a` is BF16 because the indexer is named in `modules_to_not_convert` — at FP8 it would price at 10.0 MB, and this is the layer type whose cost grows with context. @@ -1054,7 +1057,7 @@ small: | 131,072 | 537.0 MB | 0.112 ms | 2.349 ms | 13.0 % | | 1,048,576 | **4,296.0 MB** | **0.895 ms** | **18.795 ms** | **54.4 %** | -**Σ per layer at S=8192: 6.2 GF, 923.7 MB, 0.214 ms** — 5 % more than `Ls,sh`, and +**Σ per layer at S=8192: 8.3 GF, 923.7 MB, 0.214 ms** — 5 % more than `Ls,sh`, and that 5 % is the whole price of IndexShare's 21-of-78 schedule at short context. ### A.3 — Archetype `Ld,f`, 3 layers (full indexer + DENSE FFN) diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 2f228ea..5883a16 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -597,7 +597,12 @@ def w_bytes(op: str, default: str) -> float: add( "attn_index_score", - 2.0 * scan_pairs * spec.index_head_dim, + # Every one of the 32 index heads scores each candidate against the + # single shared 128-d key ``wk`` produces per token — MQA-style, which + # is why the key term below is not multiplied by the head count and + # this one is. Dropping ``index_n_heads`` here understates the scan 32x + # and would leave it looking free at every context. + 2.0 * scan_pairs * spec.index_n_heads * spec.index_head_dim, # Index keys live in the cache: read once per sequence (or per # prefilling request), not per position, and replicated across ranks # alongside the KV latent. @@ -887,7 +892,12 @@ def add_lm_head(rows: float, layer: int | None) -> None: # a full ``mlp.experts.*`` bank in the checkpoint, so every draft stage draws on # a 256-expert mixture — the draft's cost is dominated by expert weight traffic # it pays ``D`` times over, not by its arithmetic. - if spec.num_nextn_predict_layers > 0: + # Gated on there being decode positions at all: on a pure-prefill step the + # draft head does not run — it proposes continuations, and there is nothing yet + # to continue. Emitting it anyway put 18 zero-work nodes in the graph whose only + # cost was their launches, which is a launch facet made of kernels that never + # ran. + if spec.num_nextn_predict_layers > 0 and batch.positions_per_step > 0: draft_batch = replace(batch, prefill_tokens=0, speculative_tokens=0) stages = max(1, batch.speculative_tokens) for stage in range(stages): diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index b33e403..90aece5 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -365,6 +365,45 @@ def total_bytes(g, op): assert all(n.prediction.bound == "launch" for n in short_scan) +def test_indexer_scans_with_every_index_head(): + """32 query heads against one shared key per token — the head count is a factor. + + ``wk`` produces a single 128-d key per token (MQA-style), which is why the key + *bytes* carry no head factor and the score *FLOPs* do. Dropping it understates + the scan 32x and leaves the one node that grows with context looking free. + """ + spec = _spec() + batch = BatchConfig(batch=32, kv_cache_len=8192) + g = predict_glm_graph(spec, batch=batch) + scan = [n for n in g.nodes if n.op == "attn_index_score"][0] + pairs = 32 * 8192 + assert scan.prediction.flops == pytest.approx( + 2.0 * pairs * spec.index_n_heads * spec.index_head_dim + ) + # The cached keys are read once per sequence and are NOT per-head. + assert scan.prediction.bytes == pytest.approx(pairs * spec.index_head_dim * 2) + + +def test_a_pure_prefill_step_runs_no_draft_head(): + """The draft proposes continuations; a prefill chunk has nothing to continue. + + Emitting it anyway puts nodes in the graph that never ran, and since a draft + stage is almost all launch cost at one row, it shows up as a launch facet made + of absent kernels. + """ + spec = _spec() + g = predict_glm_graph( + spec, + batch=BatchConfig(batch=0, kv_cache_len=0, prefill_tokens=8192, + prefill_requests=1), + ) + assert not [n for n in g.nodes if n.layer is not None and n.layer >= spec.n_layers] + assert len(_ops(g, "mtp_eh_proj")) == 0 + # The backbone still runs, and the epilogue still projects one row per prompt. + assert len(_ops(g, "attn_q_a")) == spec.n_layers + assert len(_ops(g, "lm_head")) == 1 + + def test_detect_family_routes_glm_before_sparse_moe(): """Both families carry index_topk + n_routed_experts; model_type must win.""" assert detect_family(GLM_CONFIG) == "glm_moe_dsa" From 01a0857b37e013f207ffb6657e098b4c6011dbdf Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 14:01:21 -0700 Subject: [PATCH 06/35] GLM-5.2: lower a layer to its real kernel sequence, not just its GEMMs The graph emitted 16 nodes per layer where the design note's spec lowers a layer to 23. The missing seven were all pointwise, and the note's entire low-batch argument is a claim about how many kernels a layer *is* -- so folding them into the GEMMs they precede reported a decode step as more memory-bound than it is. Now emitted, in issue order: input_layernorm, act_quant_attn, attn_residual, post_attention_layernorm, moe_sigmoid_bias (split from moe_topk, because they are the same size and the same cost and only one of them is a CUDA-graph hazard), act_quant_moe, moe_silu, mlp_silu, mlp_residual. Plus the prologue/epilogue the step actually has: embed_tokens, final_norm, and the logits_all_gather that a vocabulary-sharded lm_head forces onto the critical path before anything can be sampled. And mtp_norms for the draft block's enorm/hnorm. act_quant is emitted only where the consuming GEMM is genuinely fp8 -- GLM-5.2-FP8 declares activation_scheme "dynamic", and on the bf16 checkpoint the kernel does not exist. That is a difference a single model-wide dtype cannot express, and it falls out of op_dtype_overrides. What it changes: a MoE layer is 24 kernels and two of them cost anything. At B=32 the step goes 15.887 -> 17.173 ms with the launch facet at 17% (was 11%); at B=1 launches are 73% of the floor, not 63%, and 87% at the eager 5 us. Node count 1,294 -> 1,927. Every figure in the note is re-derived, and the layer's node sequence is now pinned by a test so the code cannot drift from Appendix A. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 368 ++++++++++++++++++++---------------- gitm/planner/glm_graph.py | 139 +++++++++++++- tests/test_glm_graph.py | 77 ++++++++ 3 files changed, 416 insertions(+), 168 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 9712dee..10c57dc 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -242,8 +242,8 @@ flowchart TD tokens issues `8P` token→expert assignments. Once `8P ≫ 256` — P above a few hundred — every expert receives at least one token, so **every layer reads its entire expert bank**, 1.26 GB per rank per layer at EP8, **constant in P**. That is -95.7 GB per pass, 29 % of all prefill traffic. But it is not the top line: **the EP -all-to-all is**, at 105.7 GB and 48 % of predicted prefill time. Three quarters of +95.7 GB per pass, 23 % of all prefill traffic. But it is not the top line: **the EP +all-to-all is**, at 105.7 GB and 45 % of predicted prefill time. Three quarters of prefill cost is the MoE path, and under expert parallelism most of that is *wire*, not DRAM. @@ -296,7 +296,7 @@ vocabulary weights *every step* rather than once per request. │ in the graph. Blocks CUDA-graph │ capture. §5 rank 2. EP dispatch a2a ─▶ permute ─▶ grouped GEMM fp8 ×2 ─▶ SiLU ─▶ grouped GEMM fp8 - │ 163/256 experts woken at B=32 · 771 MB/layer · 86 % of DRAM + │ 163/256 experts woken at B=32 · 771 MB/layer · 85 % of DRAM scatter-add ×2.5 ─▶ EP combine a2a ─▶ all_reduce #2 ▼ … ×78 layers, then: @@ -335,10 +335,10 @@ never read. flowchart LR subgraph V["VANILLA DECODE"] direction TB - v1["1 row per seq"] --> v2["78 layers + 1 MTP
1,294 nodes"] --> v3["lm_head"] --> v4["sample"] + v1["1 row per seq"] --> v2["78 layers + 1 MTP
1,927 nodes"] --> v3["lm_head"] --> v4["sample"] v4 --> v5(("1 token")) end - subgraph D["DRAFT — 5 SERIAL stages, 90 nodes"] + subgraph D["DRAFT — 5 SERIAL stages, 135 nodes"] direction TB d0["h at last accepted pos
+ its token"] --> eh["eh_proj [12288→6144] BF16"] eh --> d1["MTP block
MLA+DSA (shared index) + FULL MoE"] @@ -347,7 +347,7 @@ flowchart LR end subgraph W["VERIFY = decode at 1+D rows"] direction TB - w1["6 rows per seq"] --> w2["THE SAME 78 layers
THE SAME 1,276 nodes"] + w1["6 rows per seq"] --> w2["THE SAME 78 layers
THE SAME 1,900 nodes"] w2 --> w3["lm_head, 6 rows"] --> w4["compare vs draft"] w4 --> w5(("1..6 accepted")) end @@ -357,8 +357,8 @@ flowchart LR ``` **Verify is not a new graph.** It is the decode graph with the row dimension -multiplied by `1+D`. The only genuinely new subgraph is the draft chain — 90 nodes -against the backbone's 1,276. +multiplied by `1+D`. The only genuinely new subgraph is the draft chain — 135 nodes +against the backbone's 1,900. **The dependency point:** the loop is strictly serial and now has `D+1` sampling points instead of one. Nothing in stage `k` can start before stage `k-1`'s token id @@ -387,16 +387,29 @@ argued rather than looked up. ### 3.1 Prologue and epilogue -| id | operator | kernel class | shape | FLOPs | bytes | stream | conf | -| --- | ----------------- | ------------------------- | ------------------------------ | -------- | ------------- | ------- | ------ | -| D0 | `embed_tokens` | gather (index_select) | `[32] → [32,6144]` | 0 F | 393 kB | compute | high | -| E0 | final norm | fused RMSNorm | `[32,6144]` **every sequence** | 590 kF | 786 kB | compute | high | -| E1 | `lm_head` | GEMM (BF16), tall-skinny | `[32,6144] × [6144,19360]` | 7.61 GF | **239.5 MB**/rank | memory | high | -| E2 | sample + D2H | argmax/top-p + host copy | `[32,154880] → [32]` | — | small | compute | high | - -`lm_head` is TP-sharded on the vocabulary, so per rank it is 239.5 MB of the 1.903 GB -whole-model matrix. It is 0.6 % of a decode step and **0.02 %** of a prefill one — -the epilogue is nearly free in prefill and is not in decode. +The step does not begin at layer 0 or end at layer 77. Four nodes bracket it, and +two of them are on the critical path between the last layer and the sample. + +| id | operator | kernel class | shape | FLOPs | bytes | stream | bound | +| --- | ------------------- | ------------------------- | ------------------------------ | -------- | ------------- | ------ | ------ | +| D0 | `embed_tokens` | gather (index_select) | `[32] → [32,6144]` | 0 F | 0.393 MB | compute | launch | +| E0 | `final_norm` | fused RMSNorm | `[32,6144]`, **logits rows only** | 0.6 MF | 0.786 MB | compute | launch | +| E1 | `lm_head` | GEMM (BF16), tall-skinny | `[32,6144] × [6144,19360]` | 7.61 GF | **239.5 MB** | compute | memory | +| E2 | `logits_all_gather` | collective (all-gather) | `[32,19360] → [32,154880]` fp32 | 0 F | **17.3 MB** | comm | memory | + +Three things worth naming: + +- **D0 reads what it selects, not the table.** An `index_select` over a 1.9 GB + embedding costs 393 kB at 32 rows. The resident 1.9 GB matters for the fit math + (§1), not for the step. +- **E0 runs over `logits_rows`, not over every row in the step.** At decode those + are the same number; at prefill they are 1 per prompt against a 8,192-token + chunk, and charging the chunk would make the epilogue the largest thing in a + prefill step. +- **E2 exists because E1 is vocabulary-sharded.** 17.3 MB of fp32 logits gathered + across 8 ranks before anything can be sampled — small in bytes, unavoidable in + position, and the second-most-expensive single node in a decode step after the + expert bank (19.3 µs). ### 3.2 What prefill changes @@ -428,11 +441,11 @@ against P = 1 decode row per sequence: | Prefill, P=8192, C=0, TP8/EP8, per rank | value | | --- | --- | -| predicted floor | **243.1 ms** for the chunk (33.7 k tok/s) | -| bytes moved | **330.6 GB** | -| FLOPs | **118.3 TF** | -| whole-pass AI | **358** — below the fp8 ridge of 412, so **memory-bound overall** | -| facets | compute 666 n / 86.3 ms / 35.5 % · memory 535 n / 156.6 ms / 64.4 % · launch 75 n / 0.1 ms | +| predicted floor | **262.4 ms** for the chunk (31.2 k tok/s) | +| bytes moved | **422.6 GB** | +| FLOPs | **118.4 TF** | +| whole-pass AI | **280** — below the fp8 ridge of 412, so **memory-bound overall** | +| facets | compute 666 n / 86.3 ms / 32.9 % · memory 1,082 n / 175.7 ms / 67.0 % · launch 152 n / 0.3 ms | > ⚠ **The prefill rows overturn the dense-model intuition, and the reversal is > worth keeping visible.** Dense intuition says prefill is compute-bound. It is @@ -448,12 +461,12 @@ against P = 1 decode row per sequence: | chunking | bytes | floor | | ------------------- | ----------- | ---------- | -| 1 × 8,192 | **331 GB** | 243 ms | -| 2 × 4,096 | 426 GB | 259 ms | -| 8 × 1,024 | 996 GB | 374 ms | -| 64 × 128 | **6,221 GB**| **1,499 ms** | +| 1 × 8,192 | **423 GB** | 262 ms | +| 2 × 4,096 | 518 GB | 278 ms | +| 8 × 1,024 | 1,088 GB | 394 ms | +| 64 × 128 | **6,313 GB**| **1,579 ms** | -**18.8× the bytes for identical FLOPs.** The expert bank is read per *chunk*, not per +**14.9× the bytes for identical FLOPs.** The expert bank is read per *chunk*, not per token — so a small `--max-num-batched-tokens`, chosen to protect decode latency, is paid for here at a rate nothing in a per-token cost model shows. @@ -463,42 +476,42 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): | pass | nodes | bytes | floor | | ---------------------- | --------- | ------------ | ------------- | -| vanilla decode (D=0) | 1,294 | 68.22 GB | 15.887 ms | -| — of which the draft | 18 | 1.26 GB | 0.285 ms | -| draft chain, D=5 | 90 | **6.31 GB** | 1.423 ms | -| verify, 192 rows | 1,276 | 104.2 GB | 25.756 ms | -| **MTP step total** | **1,366** | **110.5 GB** | **27.179 ms** | +| vanilla decode (D=0) | 1,927 | 68.61 GB | 17.173 ms | +| — of which the draft | 27 | 1.27 GB | 0.305 ms | +| draft chain, D=5 | 135 | **6.34 GB** | 1.523 ms | +| verify, 192 rows | 1,900 | 106.4 GB | 27.118 ms | +| **MTP step total** | **2,035** | **112.8 GB** | **28.641 ms** | -**Cost ratio 1.71× for up to 6 tokens.** Where the extra 42.3 GB goes: +**Cost ratio 1.67× for up to 6 tokens.** Where the extra 44.2 GB goes: -- **The verify pass, +36.0 GB.** Almost all of it is one line: expert weights go +- **The verify pass, +37.8 GB.** Almost all of it is one line: expert weights go from 163 distinct experts at 32 rows to 256 at 192 rows — **the union saturates**, so 6× the rows costs 1.57× the expert bytes. KV read does **not** move at all (0.43 GB either way, because it is read per *sequence*, not per row), and verify `lm_head` does not move either (239.5 MB regardless of rows). -- **The draft chain, +5.05 GB.** And this is where GLM differs from a dense-draft - model: **the draft is 5.2 % of the MTP step, not 1–2 %**, because each of its 5 +- **The draft chain, +5.07 GB.** And this is where GLM differs from a dense-draft + model: **the draft is 5.3 % of the MTP step, not 1–2 %**, because each of its 5 stages draws on a full 256-expert bank. Its cost is linear in D with no saturation to help it — 32 rows wakes ~163 experts every stage, five times over. -**So ~85 % of the price of speculation is the MoE expert bank** — charged because +**So ~86 % of the price of speculation is the MoE expert bank** — charged because more rows and more stages touch more experts, not because more work is done per token. -**Break-even.** The step costs 1.71×; it produces up to 6 tokens instead of 1, so -acceptance α must exceed `(1.71 − 1)/5 = 0.142` to pay. Predicted throughput: +**Break-even.** The step costs 1.67×; it produces up to 6 tokens instead of 1, so +acceptance α must exceed `(1.67 − 1)/5 = 0.134` to pay. Predicted throughput: | α | 0.0 | 0.5 | 0.7 | 0.9 | | ---- | ----- | ----- | ----- | ----- | -| tok/s| 1,177 | 4,121 | 5,298 | **6,476** | +| tok/s| 1,117 | 3,911 | 5,028 | **6,145** | -against 2,014 tok/s with MTP off. The vendor claims the GLM-5.2 MTP layer raises +against 1,863 tok/s with MTP off. The vendor claims the GLM-5.2 MTP layer raises accepted length by up to 20 % over its predecessor, which puts the operating point well past break-even — but **α is a serving observable and this graph does not predict it.** It prices the cost and leaves the payoff to a measurement (§6.2, C4). ⚑ **All of this assumes the step is memory-bound.** At B ≤ 8 it is not (§4.1), and -in the launch regime the draft's 90 extra launches are pure cost against a step +in the launch regime the draft's 135 extra launches are pure cost against a step that was never moving bytes. The sign of the MTP decision flips with batch. ### 3.4 Predicted synchronization points @@ -515,7 +528,7 @@ that was never moving bytes. The sign of the MTP decision flips with batch. | S8 | KV rollback | discard rejected rows | **low** on mechanism | pointer rewind (free) or real memmove (not free) — the trace tells you which | | S9 | indexer selection handoff | the `full` layer's top-k must be visible to its 3 `shared` layers | **low** | if it round-trips the host, IndexShare costs a sync it should not — **21 per step** | -**S5 is the decode-specific one worth chasing.** Decode runs ~1,294 kernels in +**S5 is the decode-specific one worth chasing.** Decode runs ~1,927 kernels in ~16 ms and then hands control back to a Python scheduler. If the scheduler takes longer than the step, the GPU idles and no kernel-level work matters. @@ -550,32 +563,38 @@ ridge 412 F/B (fp8) · 206 (bf16) · 14 (fp32) · launch floor 2.0 µs (graph-re node kernel class bytes AI xN Σ ms bound share ────────────────────── ─────────────────────── ──────── ──── ─── ───── ────── ──────────────────────── - moe_routed grouped GEMM fp8 ×3 771.1 MB 3.1 76 12.208 memory ██████████████████████ 76.8% - attn_score_value paged decode attention 42.0 MB 12.8 79 0.690 memory █ 4.3% - moe_all_to_all collective (EP a2a) 5.5 MB — 76 0.465 comm █ 2.9% - attn_q_a GEMM fp8, REPLICATED 13.1 MB 61.4 79 0.216 memory ▏ 1.4% - attn_out_proj GEMM fp8, tall-skinny 13.1 MB 61.4 79 0.216 memory ▏ 1.4% - attn_q_b GEMM fp8, tall-skinny 4.5 MB 60.2 79 0.158 launch ▏ 1.0% - attn_kv_a GEMM fp8 + cache append 4.0 MB 56.8 79 0.158 launch ▏ 1.0% - attn_kv_b GEMM fp8 (unabsorbed) 2.1 MB 56.0 79 0.158 launch ▏ 1.0% - attn_qnorm_rope_insert fused norm+RoPE+insert 0.4 MB 0.9 79 0.158 launch ▏ 1.0% - tp_all_reduce_attn collective NCCL ring 0.7 MB — 79 0.158 launch ▏ 1.0% - tp_all_reduce_mlp collective NCCL ring 0.7 MB — 79 0.158 launch ▏ 1.0% - moe_router GEMM **fp32**, replicated 6.7 MB 15.0 76 0.152 launch ▏ 1.0% - moe_topk sigmoid+bias+top-8 0.03 MB 0.8 76 0.152 launch ▏ 1.0% ← the only data-dependent shape - moe_shared grouped GEMM fp8 5.5 MB 54.5 76 0.152 launch ▏ 1.0% + moe_routed grouped GEMM fp8 ×3 771.1 MB 3.1 76 12.208 memory ████████████████████ 71.1% + attn_score_value paged decode attention 42.0 MB 12.8 79 0.690 memory █ 4.0% + moe_all_to_all collective (EP a2a) 5.5 MB — 76 0.465 comm ▊ 2.7% + attn_q_a GEMM fp8, REPLICATED 13.1 MB 61.4 79 0.216 memory ▌ 1.3% + attn_out_proj GEMM fp8, tall-skinny 13.1 MB 61.4 79 0.216 memory ▌ 1.3% + ── 17 nodes at the 2 µs launch floor, 0.158 ms each (0.9%) ──────────────────────────────── + input_layernorm · act_quant_attn · attn_q_b · attn_kv_a · attn_kv_b · + attn_qnorm_rope_insert · tp_all_reduce_attn · attn_residual · + post_attention_layernorm · moe_router (fp32) · moe_sigmoid_bias · + moe_topk · act_quant_moe · moe_shared · moe_permute · moe_silu · + moe_combine · tp_all_reduce_mlp · mlp_residual + ─────────────────────────────────────────────────────────────────────────────────────────── attn_index_score index scan + top-k 33.6 MB 64.0 21 0.147 memory ▏ 0.9% ← the only term that grows with S - lm_head GEMM bf16, tall-skinny 238.0 MB 32.0 2 0.100 memory ▏ 0.6% + lm_head GEMM bf16, tall-skinny 239.5 MB 31.8 2 0.100 memory ▏ 0.6% + mtp_eh_proj GEMM bf16, REPLICATED 152.2 MB 31.8 1 0.032 memory ▏ 0.2% + logits_all_gather collective (all-gather) 17.3 MB — 1 0.019 comm ▏ 0.1% ────────────────────── ─────────────────────── ──────── ──── ─── ───── ────── ──────────────────────── - 1,294 nodes 15.887 ms/step · 2,014 tok/s @ B=32 - moe_routed alone 12.208 ms = 76.8% of the step + 1,927 nodes 17.173 ms/step · 1,863 tok/s @ B=32 + moe_routed alone 12.208 ms = 71.1% of the step facets - memory 440 nodes 14.179 ms 89.2% - launch 854 nodes 1.708 ms 10.8% ← 66% of all nodes, an ninth of the time - compute 0 nodes 0.000 ms 0.0% ← the entire roofline claim, one row + memory 441 nodes 14.199 ms 82.7% ← five node types + launch 1,486 nodes 2.974 ms 17.3% ← 77% of all nodes, a sixth of the time + compute 0 nodes 0.000 ms 0.0% ← the entire roofline claim, one row ``` +**A MoE layer is 24 kernels and two of them cost anything.** `moe_routed` is 73 % +of the layer's time; `attn_score_value`, `attn_q_a`, `attn_out_proj` and the EP +all-to-all are most of the rest; the other 19 are at the launch floor. That +distribution is the model, not an artefact — and it is why every ranked hypothesis +in §5 is about either the expert bank or the launch count. + Two rows are kept **despite** being small: `moe_topk` because it is the only data-dependent shape in the graph and therefore the thing that blocks CUDA-graph capture, and `attn_index_score` because "the only term that grows with S" is the @@ -583,9 +602,9 @@ architecture's whole payoff — and because it does not stay small: | context S | `attn_index_score` | share of step | step floor | | --------- | ------------------ | ------------- | ---------- | -| 8,192 | 0.147 ms | 0.9 % | 15.887 ms | -| 131,072 | 2.349 ms | 13.0 % | 18.090 ms | -| **1,048,576** | **18.795 ms** | **54.4 %** | 34.536 ms | +| 8,192 | 0.147 ms | 0.9 % | 17.173 ms | +| 131,072 | 2.349 ms | 12.1 % | 19.375 ms | +| **1,048,576** | **18.795 ms** | **52.5 %** | 35.821 ms | **At the model's advertised context the indexer scan is the largest node in the step**, and it is the node IndexShare already cut by 3.7×. Everything anyone says @@ -596,19 +615,19 @@ the step. | B | floor | tok/s | launch nodes | launch time | compute nodes | | --- | ---------- | ----- | ------------ | ----------- | ------------- | -| 1 | 3.303 ms | 303 | 1,033 | 2.066 ms = **63 %** | 0 | -| 4 | 4.990 ms | 802 | 1,033 | 2.066 ms = 41 % | 0 | -| 16 | 10.637 ms | 1,504 | 854 | 1.708 ms = 16 % | 0 | -| 32 | 15.887 ms | 2,014 | 854 | 1.708 ms = 11 % | 0 | -| 64 | 21.713 ms | 2,948 | 778 | 1.556 ms = 7 % | 76 | -| 128 | 26.948 ms | 4,750 | 620 | 1.240 ms = 5 % | 76 | -| 256 | 33.583 ms | 7,623 | 544 | 1.088 ms = 4 % | 79 | +| 1 | 4.571 ms | 219 | 1,666 | 3.334 ms = **73 %** | 0 | +| 4 | 6.258 ms | 639 | 1,665 | 3.332 ms = 53 % | 0 | +| 16 | 11.913 ms | 1,343 | 1,486 | 2.974 ms = 25 % | 0 | +| 32 | 17.173 ms | 1,863 | 1,486 | 2.974 ms = 17 % | 0 | +| 64 | 23.017 ms | 2,781 | 1,410 | 2.822 ms = 12 % | 76 | +| 128 | 28.291 ms | 4,524 | 1,252 | 2.506 ms = 9 % | 76 | +| 256 | 35.004 ms | 7,314 | 1,176 | 2.354 ms = 7 % | 79 | **Below B≈16 the step is a launch-bound step wearing a memory-bound model's -clothes.** At B=1, 63 % of the predicted floor is 1,033 kernel launches at 2 µs — -and that figure already assumes CUDA-graph replay. At the eager 5 µs it is 5.17 ms -of launches against a 1.24 ms memory term, and the whole low-batch analysis changes -sign (A4, §5 rank 3). +clothes.** At B=1, **73 %** of the predicted floor is 1,666 kernel launches at 2 µs +— and that already assumes CUDA-graph replay. At the eager 5 µs it is 8.33 ms of +launches against a 1.24 ms memory term: **87 %**, and the whole low-batch analysis +changes sign (A4, §5 rank 3). ### 4.2 Prefill and MTP — regions and what flips them @@ -617,23 +636,23 @@ at TP8/EP8, FP8. | Phase | Region | Bound | Why (point at a number) | Precision / peak | Flip variable | |---|---|---|---|---|---| -| **Pre** | **EP dispatch/combine all-to-all** | **comm — BANDWIDTH** | 1.39 GB/layer × 76 = **105.7 GB**, 117.4 ms = **48.3 % of the step** | BF16 payload | ⚑ **fp8 dispatch halves it**; EP degree; TP-only removes the node and doubles the bank | -| **Pre** | **MoE expert grouped GEMMs** | compute (AI 485) | 1.26 GB/layer **constant in P**, 95.7 GB/pass = 29 % of traffic | **FP8 block-scaled** | **chunk size** (18.8× across 1→64 chunks); imbalance | -| **Pre** | **router GEMM** | **compute** | 26 GF/layer at **fp32's 67 TF/s** → 28.8 ms = **11.9 %** | ⚠ **FP32** — see Q3 | ⚑ whether the engine runs the GEMM in fp32 or only accumulates there | +| **Pre** | **EP dispatch/combine all-to-all** | **comm — BANDWIDTH** | 1.39 GB/layer × 76 = **105.7 GB**, 117.4 ms = **44.8 % of the step** | BF16 payload | ⚑ **fp8 dispatch halves it**; EP degree; TP-only removes the node and doubles the bank | +| **Pre** | **MoE expert grouped GEMMs** | compute (AI 485) | 1.26 GB/layer **constant in P**, 95.7 GB/pass = 23 % of traffic | **FP8 block-scaled** | **chunk size** (14.9× across 1→64 chunks); imbalance | +| **Pre** | **router GEMM** | **compute** | 26 GF/layer at **fp32's 67 TF/s** → 28.8 ms = **11.0 %** | ⚠ **FP32** — see Q3 | ⚑ whether the engine runs the GEMM in fp32 or only accumulates there | | **Pre** | `all_reduce` ×158 | **comm — BANDWIDTH** | 174 MB each, 27.5 GB/pass = 30.5 ms | BF16 payload | P; below P≈256 flips to latency | | **Pre** | projections (`q_a`,`o_proj`,`q_b`,`kv_a`,`kv_b`) | **compute** | AI 1,403 / 963 / 488 vs the fp8 ridge 412 | FP8 e4m3 | prompt length — below P≈512 memory-bound | | **Pre** | indexer scan ×21 | **compute** | **5.77 TF** against 22 MB of keys — `O(P·C + P²/2)` × 32 heads. **The quadratic lives here, not in the core** | **BF16** proj, fp8 keys | P **and** C | | **Pre** | attention core ×79 | compute, **linear in C** | FLOPs capped at 2,048 keys/query; **bytes are the whole cache once per request** | FP8 KV | ⚑ **`index_topk`**; C. Not P² | | **Pre** | permute / combine | memory | 8.5 GB/layer-pass on the `8P`-row expanded tensor, near-zero FLOPs | BF16 activations | top-k; **expert imbalance** | | **Pre** | *everything else* — embed gather, norms, `lm_head`, `moe_topk` | memory or launch | each under 1.5 %; `lm_head` reads 239.5 MB for **one row per request** | BF16; FP32 top-k | none of them flips | -| **Pre** | **whole prefill pass** | **memory** | **AI 358 vs ridge 412**; 64 % memory / 36 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | +| **Pre** | **whole prefill pass** | **memory** | **AI 280 vs ridge 412**; 67 % memory / 33 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | | **MTP** | verify expert GEMMs | **memory** | 256 distinct experts at 192 rows vs 163 at 32 — **+57 %/layer, ~85 % of MTP's cost** | FP8 block-scaled | `B(1+D)` vs E=256; **D**; imbalance | | **MTP** | draft expert GEMMs ×D | **memory** | the MTP block carries a **full 256-expert bank**; 5 stages × 163 experts, **linear in D, no saturation** | FP8 block-scaled | **D**; batch | | **MTP** | draft `lm_head` ×5 | memory | 239.5 MB × 5 = **1.20 GB = 19 % of the draft's bytes** | **BF16** | sharded sampling; draft vocab | | **MTP** | draft `eh_proj` ×5 | memory | `[12288,6144]` BF16, **replicated per rank** | **BF16** — in `modules_to_not_convert` | whether it is TP-sharded | | **MTP** | verify attention | **memory, unchanged** | 0.43 GB — read **per sequence, not per row**; 1+D rows share one block table | FP8 KV | seq length; explicitly *not* D | | **MTP** | accept/reject + KV rollback | launch + **host sync** | tiny tensors, but a data-dependent host-visible seq length (S7) | n/a | pointer rewind vs memmove | -| **MTP** | **whole MTP step** | ⚑ **memory above B≈16, launch below — and the two regimes disagree about whether MTP helps** | **1.71×** cost for ≤6 tokens at B=32; break-even α = **0.142** | mixed | **graph capture**; batch; α; D | +| **MTP** | **whole MTP step** | ⚑ **memory above B≈16, launch below — and the two regimes disagree about whether MTP helps** | **1.67×** cost for ≤6 tokens at B=32; break-even α = **0.134** | mixed | **graph capture**; batch; α; D | **Hardware sensitivity:** nothing flips between H200 SXM and H20 on the compute rows — but H20's much lower FP8 peak moves every prefill projection further into @@ -647,9 +666,9 @@ compute-bound, and its bandwidth moves the decode floor directly. | indexer scan | **compute** (5.77 TF / 22 MB) | **memory** (AI 64) | the query count collapses; the key set does not | | attention core | compute, **whole cache** in bytes | **launch/memory**, one 2,048-window | 8,192 queries' selections union to everything; one query's do not | | `all_reduce` ×158 | **comm-bandwidth** (30.5 ms wire) | **comm-latency** (0.16 ms floor, 0.8 µs wire) | payload 174 MB vs 688 kB | -| EP all-to-all | **the top line** (48.3 %) | 3.0 % | payload scales with rows; the ring latency does not | +| EP all-to-all | **the top line** (44.8 %) | 3.0 % | payload scales with rows; the ring latency does not | | `lm_head` | memory, 1 row per **request** | memory, **every row every step** | the epilogue is free in prefill and is not in decode | -| `moe_router` (fp32) | **compute** (11.9 % of prefill) | launch (1.0 %) | 26 GF/layer against 67 TF/s only matters when P is large | +| `moe_router` (fp32) | **compute** (11.0 % of prefill) | launch (1.0 %) | 26 GF/layer against 67 TF/s only matters when P is large | ### 4.4 Flip-variable index @@ -657,15 +676,15 @@ compute-bound, and its bandwidth moves the decode floor directly. | ------------- | ------------- | ----------------------- | | **Decode batch B** | every decode GEMM, expert hit-rate, all collectives, the whole-step label (launch below B≈16, memory above), the sign of the MTP decision | expert bytes **sub-linear** in B: 8 experts at B=1, 163 at B=32, 252 at B=128. 303 → 7,623 tok/s across 1→256 | | **Sequence length S** | the indexer scan, and **only** the indexer scan | 0.3 % of the step at 8K → 13.0 % at 128K → **54.4 % at 1M**. The attention core does not move at all | -| **Prompt length P** | every prefill projection (memory→compute above P≈512), the router, the indexer's quadratic, all-reduce (latency→bandwidth above P≈256) | whole-pass AI 358 at P=8k | -| ⚑ **Chunked prefill / chunk size** | prefill expert GEMMs, permute, and through them the whole prefill label | 8,192 tokens in 1 chunk = **331 GB**; in 64 chunks of 128 = **6,221 GB** for identical FLOPs | +| **Prompt length P** | every prefill projection (memory→compute above P≈512), the router, the indexer's quadratic, all-reduce (latency→bandwidth above P≈256) | whole-pass AI 280 at P=8k | +| ⚑ **Chunked prefill / chunk size** | prefill expert GEMMs, permute, and through them the whole prefill label | 8,192 tokens in 1 chunk = **423 GB**; in 64 chunks of 128 = **6,313 GB** for identical FLOPs | | ⚑ **CUDA-graph capture** | the whole decode step, the whole MTP step, every launch row | at B=1 it is 63 % of the floor; at 5 µs eager it is 81 %. Decides whether MTP is a 3× win or a net loss | -| **EP vs TP** | every collective row, every MoE memory row | under EP8 the a2a is **48 % of prefill** but the per-rank bank is **8× smaller**. Since the bank is ~86 % of decode DRAM, this is a **trade, not a cost** | -| ⚑ **Precision (fp8 vs bf16)** | every weight term, and whether the model fits at all | 1.84× on the decode floor (15.887 ms vs 29.227 ms) and **10.7 → 5.4 H200s** for weights | +| **EP vs TP** | every collective row, every MoE memory row | under EP8 the a2a is **45 % of prefill** but the per-rank bank is **8× smaller**. Since the bank is ~85 % of decode DRAM, this is a **trade, not a cost** | +| ⚑ **Precision (fp8 vs bf16)** | every weight term, and whether the model fits at all | 1.76× on the decode floor (17.173 ms vs 30.196 ms) and **10.7 → 5.4 H200s** for weights | | ⚑ **KV dtype** | the attention core, and the 1M-context footprint | 55.2 GB vs 99.9 GB per rank at 1M — the difference between fitting and not | | ⚑ **Absorbed vs unabsorbed MLA** | `attn_kv_b` + `attn_out_proj` | drops one node and doubles the other's input width: ±2× on 2.4 % of decode, more at prefill | | **`index_topk`** | the attention core's FLOPs in both phases; its bytes in neither | 2,048 → 4,096 doubles core FLOPs and changes no byte term | -| **Draft depth D, acceptance α** | verify expert GEMMs, whole MTP step | 1.71× cost at D=5; break-even α = 0.142; 1,177 → 6,476 tok/s across α | +| **Draft depth D, acceptance α** | verify expert GEMMs, whole MTP step | 1.67× cost at D=5; break-even α = 0.134; 1,117 → 6,145 tok/s across α | | **Expert imbalance** | prefill + decode expert GEMMs, permute, the a2a, grouped-GEMM tail | skew *reduces* bytes while *increasing* tail latency and stalling every other EP rank | --- @@ -677,14 +696,14 @@ graph**, to be confirmed against a capture. | Rank | Region | Prediction | Why | Evidence to inspect | What would prove it wrong | |---|---|---|---|---|---| -| **1** | **EP all-to-all at prefill** | **≥48 % of prefill time is wire, and roughly half of it is recoverable** | 105.7 GB/pass in BF16. An fp8 dispatch halves the payload outright; overlapping dispatch with the shared-expert GEMM hides more | NCCL kernel duration vs payload at prefill (§6.4 row 5); whether dispatch is bf16 | duration ≪ payload/900 GB/s → the engine already fuses or compresses it | +| **1** | **EP all-to-all at prefill** | **≥45 % of prefill time is wire, and roughly half of it is recoverable** | 105.7 GB/pass in BF16. An fp8 dispatch halves the payload outright; overlapping dispatch with the shared-expert GEMM hides more | NCCL kernel duration vs payload at prefill (§6.4 row 5); whether dispatch is bf16 | duration ≪ payload/900 GB/s → the engine already fuses or compresses it | | **2** | **Grouped-GEMM group sizing (the fork)** | either **76 D2H per token** (no graph capture possible) **or** fixed-capacity padding (**all 256 experts read every step**) | The only data-dependent shape in the graph is `moe_topk`. A stack does one or the other — see §5.2 | `cuda_api_sum` D2H count per decode step | 0 D2H **and** MoE bytes that track `distinct_experts(B)` → a device-side path, nothing to recover | | **3** | ⚑ **CUDA-graph capture at low batch** | **63 % of the B=1 floor is launch overhead** (81 % at eager 5 µs) | 1,033 nodes × 2 µs = 2.07 ms against a 1.24 ms memory term | `cuda_api_sum` launch count with `--cuda-graph-trace=node`; expect **1** `cudaGraphLaunch` | already captured → this rank is worth nothing, and rank 2's fork is already resolved to "padded" | | **4** | **The indexer scan at long context** | at 1M it is **54 % of the step**, and IndexShare's 3.7× is already banked | 90.2 GB/step of index keys at S=1M. Levers: fp8 index keys (2×), temporal reuse across steps, a smaller `index_topk_freq` group | `dram__bytes_read.sum` on the scan vs 90.2 GB; whether keys are fp8 | keys already fp8 and no temporal reuse available → architectural | -| **5** | **`moe_routed` — the memory-bound heart** | 76.8 % of decode; **EPLB placement and the grouped-GEMM backend are the levers** | Weight traffic scales with *distinct* experts woken, not with FLOPs. Good placement cuts per-rank distinct traffic; DeepGEMM cuts the constant | measured per-rank expert traffic vs `distinct_experts`; **the real EP imbalance** | measured traffic already at the union prediction with balance ≈1.0 → the bank is the bank | -| **6** | ⚑ **fp32 router at prefill** | **11.9 % of prefill** rests on the reading that the router *GEMM* runs in fp32 | 26 GF/layer at 67 TF/s. If only the softmax/top-k accumulates in fp32 and the GEMM is bf16, this row shrinks ~15× | the engine's router implementation; `cuda_gpu_kern_sum` dtype of the gate GEMM | the GEMM is genuinely fp32 → architectural, and the row stays | -| **7** | **Decode collective placement** | 158 all-reduces + 76 all-to-alls per step on the compute stream, **latency-bound**, with idle SMs around them | 688 kB payloads: the gap *is* the cost. Nothing prevents other layers' work overlapping | NCCL ranges and stream ids on the timeline (S2/S3) | already on a separate stream with overlap → nothing to recover | -| **8** | **Chunk size** | a decode-latency-protecting `--max-num-batched-tokens` can cost **18.8× the prefill bytes** | the expert bank is read per chunk | total prefill MoE DRAM ÷ 95.7 GB — the quotient **is** the chunk count | quotient ≈ 1 → prefill is already unchunked | +| **5** | **`moe_routed` — the memory-bound heart** | 71.1 % of decode; **EPLB placement and the grouped-GEMM backend are the levers** | Weight traffic scales with *distinct* experts woken, not with FLOPs. Good placement cuts per-rank distinct traffic; DeepGEMM cuts the constant | measured per-rank expert traffic vs `distinct_experts`; **the real EP imbalance** | measured traffic already at the union prediction with balance ≈1.0 → the bank is the bank | +| **6** | ⚑ **fp32 router at prefill** | **11.0 % of prefill** rests on the reading that the router *GEMM* runs in fp32 | 26 GF/layer at 67 TF/s. If only the softmax/top-k accumulates in fp32 and the GEMM is bf16, this row shrinks ~15× | the engine's router implementation; `cuda_gpu_kern_sum` dtype of the gate GEMM | the GEMM is genuinely fp32 → architectural, and the row stays | +| **7** | **Decode collective placement** | 158 all-reduces + 76 all-to-alls + 1 logits all-gather per step on the compute stream, **latency-bound**, with idle SMs around them | 688 kB payloads: the gap *is* the cost. Nothing prevents other layers' work overlapping | NCCL ranges and stream ids on the timeline (S2/S3) | already on a separate stream with overlap → nothing to recover | +| **8** | **Chunk size** | a decode-latency-protecting `--max-num-batched-tokens` can cost **14.9× the prefill bytes** | the expert bank is read per chunk | total prefill MoE DRAM ÷ 95.7 GB — the quotient **is** the chunk count | quotient ≈ 1 → prefill is already unchunked | | **9** | **`attn_q_a` / `attn_kv_a` replication** | 2.8 % of decode is paid **in full on every rank** and TP does not reduce it | they produce the shared latent, which has nothing to split | per-rank duration of `q_a` vs `q_b` under TP8 | already sharded via DP-attention → the graph is wrong here, not the engine | | **10** | **IndexShare selection handoff (S9)** | if the top-k round-trips the host, **21 syncs/step** to save 57 kernels | the selection must reach three downstream layers | D2H count attributable to the indexer region | 0 → device-side, and IndexShare is pure win | @@ -759,12 +778,12 @@ Two worked examples, because the rule is easy to agree with and hard to apply: - **The draft chain shows five gaps with no kernel spanning them.** Q1: *yes* — stage `k` consumes stage `k-1`'s token id. **Architectural.** -- **The 234 decode collectives sit on the compute stream with idle gaps around +- **The 235 decode collectives sit on the compute stream with idle gaps around them.** Q1: *no* — the output feeds the next layer, but nothing prevents *other* layers' work overlapping. Q2: *yes* — stream assignment. **Recoverable** (rank 7). **The trap runs in both directions.** The accept/reject readback will look alarming -and is architectural. The 1,294 kernel launches are entirely *expected* from §3 and +and is architectural. The 1,927 kernel launches are entirely *expected* from §3 and are the largest recoverable item at low batch. **Neither surprise nor familiarity is evidence.** @@ -816,16 +835,16 @@ Key: **R:** recoverable → the rank it feeds · **A:** architectural, do not ch |---|---|---|---|---| | **0** | **Launch args, as text** — not a measurement | C6 | chunk size, graph capture, KV dtype, D, TP/EP, absorbed MLA | Resolves or reframes **ranks 1, 2, 3, 6, 8 before a timeline is opened** | | **1** | `cuda_api_sum` — **D2H count per decode step** | C1 | **0** in the MoE region | **R:** 76/token → host-resolved group sizes, no graph capture → **rank 2**. **R:** 0 but MoE bytes flat in B → padded capacity → **rank 3** instead (§5.2). **A:** none | -| **2** | `cuda_api_sum` — **launches per step**, needs `--cuda-graph-trace=node` | C1 | **1** `cudaGraphLaunch` | **R:** ~1,294 individual launches + CPU gaps below B≈16 → **rank 3**. **F1:** count wildly off 1,294 → the lowering in §3 is wrong by an order of magnitude | -| **3** | `dram__bytes_read.sum` — **MoE region** | C1, C3 | 1.26 GB/layer/rank = **95.7 GB/pass**; **≈86 %** of decode | **R:** prefill total ÷ 95.7 GB > 1 → chunked re-read, and the quotient **is** the chunk count → **rank 8**. **R:** flat in B → padded capacity → **rank 3**. **F2:** much lower → L2 residency, and the central claim of both phases is wrong | +| **2** | `cuda_api_sum` — **launches per step**, needs `--cuda-graph-trace=node` | C1 | **1** `cudaGraphLaunch` | **R:** ~1,927 individual launches + CPU gaps below B≈16 → **rank 3**. **F1:** count wildly off 1,927 → the lowering in §3 is wrong by an order of magnitude | +| **3** | `dram__bytes_read.sum` — **MoE region** | C1, C3 | 1.26 GB/layer/rank = **95.7 GB/pass**; **≈85 %** of decode | **R:** prefill total ÷ 95.7 GB > 1 → chunked re-read, and the quotient **is** the chunk count → **rank 8**. **R:** flat in B → padded capacity → **rank 3**. **F2:** much lower → L2 residency, and the central claim of both phases is wrong | | **4** | `dram__bytes_read.sum` — **indexer region**, swept in S | C2 | 33.6 MB/layer at 8K → **90.2 GB/step at 1M**, on **21 layers only** | **R:** keys read on 78 layers → IndexShare is not being honoured, a hidden 3.7× → **rank 4**. **R:** 2× the prediction → keys are bf16 where fp8 would do. **A:** growth on 21 layers is the architecture | -| **5** | **NCCL kernel duration vs payload** | C1, C3, C5 | prefill **∝ payload** (~900 GB/s, 117 ms a2a); decode **flat, ~2 µs × 234** | **R:** prefill a2a at bf16 payload → fp8 dispatch → **rank 1**. **R:** decode collectives on the compute stream with idle SMs → **rank 7**. **A:** the ring latency floor | +| **5** | **NCCL kernel duration vs payload** | C1, C3, C5 | prefill **∝ payload** (~900 GB/s, 117 ms a2a); decode **flat, ~2 µs × 235** | **R:** prefill a2a at bf16 payload → fp8 dispatch → **rank 1**. **R:** decode collectives on the compute stream with idle SMs → **rank 7**. **A:** the ring latency floor | | **6** | **Kernel-name coverage** — every kernel maps to a §3 node | C1 | **complete** | **A/F:** an unmapped kernel block is not headroom, it is a node this graph does not have — and unlike a multimodal model (§2.3) there is no external encoder to explain it away, so it is a **model-validity failure** | | **7** | `cuda_gpu_kern_sum` — **attention core duration vs S** | C2 | **flat** from 8K to 1M | **R:** grows with S → `index_topk` is not being applied and the core is reading the whole cache. **A:** flat — that is DSA working | | **8** | **Tensor-pipe active %** | C1, C3 | **near-idle at every decode batch**; prefill well below peak with DRAM and NVLink busy | **R:** pipe busy at low batch → something does far more FLOPs than the graph predicts. **A:** near-idle at decode — that is what decode *is* | | **9** | **CPU thread sampling, between steps** | C1 | no gap between step *N* and *N*+1 | **R:** CPU-shaped inter-step gap with the scheduler hot → scheduler-bound (S5). **A:** the single sampling D2H | | **10** | **`spec_decode` bucket counts + acceptance** | C4 | **D = 5** draft stages, 5 extra `lm_head`-shaped GEMMs, verify KV **flat** as D rises | **R:** KV scales with 1+D → rows treated as sequences; use a multi-query kernel. **R:** one `lm_head` for five stages → the draft samples on a sharded vocab already. **A:** the five serial gaps | -| **11** | **Router GEMM dtype** | C3 | fp32 if the config is honoured | **R:** bf16 GEMM with fp32 accumulate → **rank 6 evaporates and §4.2's 11.9 % row shrinks 15×.** **A:** genuinely fp32 → it is the model | +| **11** | **Router GEMM dtype** | C3 | fp32 if the config is honoured | **R:** bf16 GEMM with fp32 accumulate → **rank 6 evaporates and §4.2's 11.0 % row shrinks 15×.** **A:** genuinely fp32 → it is the model | **Rows 0–3 are the thirty-minute version.** @@ -899,7 +918,7 @@ gitm/planner/model_catalogue.py nested tuple coercion for op_dtype_overrides gitm/planner/models/ glm-5.2.yaml the bf16 model fact + the fp32 router glm-5.2-fp8.yaml NEW — the vendor's recommended deployment -tests/test_glm_graph.py 23 tests: the fp8 footprint, the precision map, the +tests/test_glm_graph.py 26 tests: the fp8 footprint, the precision map, the prefill/decode byte inversion, the D-deep chain, two collectives, the 32-head index scan, that a pure-prefill step runs no draft head, and that a launch floor @@ -919,12 +938,12 @@ docs/glm-5.2/artifacts/*.json DELETED — 29k lines of node dumps that go sta |---|---|---|---| | **Q1** | Does the engine run **absorbed** MLA at decode? | drops `attn_kv_b` and **doubles** `attn_out_proj`'s input width (16384→32768). ±2× on the #4 and #8 lines | serving image / C6 | | **Q2** | Is the decode step **CUDA-graph captured**? | at B=1 the difference between 1.24 ms and 3.30 ms per step, and it decides whether MTP is a 3× win or a net loss | engine config + `--cuda-graph-trace=node` | -| **Q3** | Is the **router GEMM** fp32, or only its accumulation? | **11.9 % of prefill.** At bf16 the row shrinks ~15× | the engine's MoE gate implementation | +| **Q3** | Is the **router GEMM** fp32, or only its accumulation? | **11.0 % of prefill.** At bf16 the row shrinks ~15× | the engine's MoE gate implementation | | **Q4** | Is the grouped GEMM **device-sized or host-sized**? | decides **rank 2 *or* rank 3** — the fork in §5.2 | trace D2H count, or the kernel source | | **Q5** | Is the **EP dispatch** bf16 or fp8? | **half of 105.7 GB** at prefill — the largest single recoverable number here | serving image / C6 | | **Q6** | Is the **IndexShare selection** passed device-side? | 21 syncs/step if not (S9) | trace D2H attribution | -| **Q7** | Chunked prefill on, at what chunk size? | **up to 18.8× on prefill bytes** | engine launch args (C6) | -| **Q8** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is 0.142; the range 0.5→0.9 is 4,121→6,476 tok/s | engine metrics (C4) — **not predictable from a config** | +| **Q7** | Chunked prefill on, at what chunk size? | **up to 14.9× on prefill bytes** | engine launch args (C6) | +| **Q8** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is 0.134; the range 0.5→0.9 is 3,911→6,145 tok/s | engine metrics (C4) — **not predictable from a config** | | **Q9** | Are the **index keys** cached in fp8 or bf16? | 2× on 90.2 GB/step at 1M context | C6 / trace | | **Q10** | Is `eh_proj` **TP-sharded** in the MTP block? | 151 MB → 19 MB per rank per draft stage | serving image | | **Q11** | Does the prefill attention kernel read the selected KV **once per request** or once per query tile? | up to 64× on the prefill core's bytes — the graph takes the optimistic floor | `dram__bytes_read.sum` on the prefill core | @@ -1005,75 +1024,87 @@ the collectives (**medium**, a TP/EP convention) and the S1 histogram readback ### A.1 — Archetype `Ls,sh`, 57 layers (shared indexer + MoE) -Read straight off the graph, not retyped: every row is a `PredictedNode` at this -shape. **Norms are not nodes here** — `input_layernorm` and -`post_attention_layernorm` are folded into the projection that consumes them and -into `attn_qnorm_rope_insert`, which is one fused kernel in every serving path this -targets. That is a modelling choice, and it is the reason a trace will show ~2 -more small kernels per layer than this table has rows. +Read straight off the graph, in issue order — every row is a `PredictedNode` at +this shape, and `tests/test_glm_graph.py::test_layer_lowers_to_the_documented_node_sequence` +pins this exact sequence so the code cannot drift from the table. | id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | |---|---|---|---|---|---|---|---| -| .1 | `attn_q_a` | GEMM, **replicated** | 805.3 MF | 13.110 MB | FP8 | 2.73 | memory | -| .2 | `attn_q_b` | GEMM, head-sharded | 268.4 MF | 4.457 MB | FP8 | 2.00 | launch | -| .3 | `attn_kv_a` | GEMM + cache append | 226.5 MF | 3.990 MB | FP8 | 2.00 | launch | -| .4 | `attn_kv_b` | GEMM, head-sharded (**unabsorbed**) | 117.4 MF | 2.098 MB | FP8 | 2.00 | launch | -| .5 | `attn_score_value` | paged decode attn over ≤2048 entries | 536.9 MF | 41.951 MB | FP8 KV | 8.74 | memory | -| .6 | `attn_qnorm_rope_insert` | fused norm + partial RoPE + insert | 0.4 MF | 0.410 MB | BF16 | 2.00 | launch | -| .7 | `attn_out_proj` | GEMM, tall-skinny (16384→6144) | 805.3 MF | 13.110 MB | FP8 | 2.73 | memory | -| .8 | `tp_all_reduce_attn` | NCCL ring | — | 0.688 MB | BF16 | 2.00 | launch | -| .9 | `moe_router` | GEMM, **replicated** | 100.7 MF | 6.701 MB | **FP32** | 2.00 | launch | -| .10 | `moe_topk` | sigmoid + bias + top-8 + renorm | 0.02 MF | 0.033 MB | BF16 | 2.00 | launch ← **the only data-dependent shape** | -| .11 | `moe_shared` | grouped GEMM ×3, always on | 302.0 MF | 5.539 MB | FP8 | 2.00 | launch | -| .12 | `moe_permute` | gather into expert-major order | 0 F | 0.442 MB | BF16 | 2.00 | launch | -| .13 | `moe_routed` | grouped GEMM ×3, **163 distinct of 256** | 2,415.9 MF | **771.062 MB** | FP8 | **160.64** | **memory** | -| .14 | `moe_combine` | scatter-add × `routed_scaling 2.5` | 3.1 MF | 0.442 MB | BF16 | 2.00 | launch | -| .15 | `moe_all_to_all` | EP dispatch + combine | — | 5.505 MB | BF16 | 6.12 | comm | -| .16 | `tp_all_reduce_mlp` | NCCL ring | — | 0.688 MB | BF16 | 2.00 | launch | - -**Σ per layer: 5.6 GF, 870.2 MB, 0.203 ms.** `.13` alone is **89 % of the layer's -bytes and 79 % of its time**, and it is the same 771 MB whether the layer is one of -57 or one of 75 — which is why every ranked hypothesis in §5 that is not about -launches is about this row. +| .1 | `input_layernorm` | fused RMSNorm | 0.6 MF | 0.786 MB | BF16 | 2.00 | launch | +| .2 | `act_quant_attn` | dynamic FP8 quant + row reduction | 0.4 MF | 0.590 MB | BF16→FP8 | 2.00 | launch | +| .3 | `attn_q_a` | GEMM, **replicated** (6144→2048) | 805.3 MF | 13.110 MB | FP8 | 2.73 | memory | +| .4 | `attn_q_b` | GEMM, head-sharded (2048→2048) | 268.4 MF | 4.457 MB | FP8 | 2.00 | launch | +| .5 | `attn_kv_a` | GEMM + cache append (6144→576) | 226.5 MF | 3.990 MB | FP8 | 2.00 | launch | +| .6 | `attn_kv_b` | GEMM, head-sharded (**unabsorbed**) | 117.4 MF | 2.098 MB | FP8 | 2.00 | launch | +| .7 | `attn_score_value` | paged decode attn over ≤2048 entries | 536.9 MF | 41.951 MB | FP8 KV | 8.74 | memory | +| .8 | `attn_qnorm_rope_insert` | fused q/kv norm + partial RoPE + insert | 0.4 MF | 0.410 MB | BF16 | 2.00 | launch | +| .9 | `attn_out_proj` | GEMM, tall-skinny (16384→6144) | 805.3 MF | 13.110 MB | FP8 | 2.73 | memory | +| .10 | `tp_all_reduce_attn` | NCCL ring | — | 0.688 MB | BF16 | 2.00 | launch | +| .11 | `attn_residual` | pointwise add | 0.2 MF | 0.786 MB | BF16 | 2.00 | launch | +| .12 | `post_attention_layernorm` | fused RMSNorm | 0.6 MF | 0.786 MB | BF16 | 2.00 | launch | +| .13 | `moe_router` | GEMM, **replicated** (6144→256) | 100.7 MF | 6.701 MB | **FP32** | 2.00 | launch | +| .14 | `moe_sigmoid_bias` | pointwise + `e_score_correction_bias` | 0.02 MF | 0.033 MB | BF16 | 2.00 | launch | +| .15 | `moe_topk` | **top-8 of 256 + renorm** | 0 F | 0.017 MB | BF16 | 2.00 | launch ← **the only data-dependent shape** | +| | ⚠ **Blocks CUDA-graph capture. ↯ S1: host readback of the expert histogram? 76 D2H per token if real.** | | | | | | | +| .16 | `act_quant_moe` | dynamic FP8 quant | 0.4 MF | 0.590 MB | BF16→FP8 | 2.00 | launch | +| .17 | `moe_shared` | grouped GEMM ×3, always on | 302.0 MF | 5.539 MB | FP8 | 2.00 | launch | +| .18 | `moe_permute` | gather into expert-major order | 0 F | 0.442 MB | BF16 | 2.00 | launch | +| .19 | `moe_routed` | grouped GEMM ×3, **163 distinct of 256** | 2,415.9 MF | **771.062 MB** | FP8 | **160.64** | **memory** | +| .20 | `moe_silu` | SwiGLU over the expanded tensor | 0.3 MF | 0.262 MB | BF16 | 2.00 | launch | +| .21 | `moe_combine` | scatter-add × `routed_scaling 2.5` | 3.1 MF | 0.442 MB | BF16 | 2.00 | launch | +| .22 | `moe_all_to_all` | EP dispatch + combine | — | 5.505 MB | BF16 | 6.12 | comm | +| .23 | `tp_all_reduce_mlp` | NCCL ring | — | 0.688 MB | BF16 | 2.00 | launch | +| .24 | `mlp_residual` | pointwise add | 0.2 MF | 0.786 MB | BF16 | 2.00 | launch | + +**Σ per layer: 5.6 GF, 874.8 MB, 0.219 ms.** + +**Twenty-four kernels, and two of them cost anything.** `.19` alone is **88 % of +the layer's bytes and 73 % of its time**; `.7`, `.3`, `.9` and `.22` are most of +the remainder; the other **19 nodes sit at the 2 µs launch floor** — 0.038 ms per +layer of pure launch, which is 17 % of it. Those 19 rows are why §4.1's launch +facet exists, and folding any of them into the GEMM it precedes would report this +layer as more memory-bound than it is. ### A.2 — Archetype `Ls,f`, 18 layers (full indexer + MoE) -**Delta from A.1: two nodes inserted after `.4`.** Everything else is identical. +**Delta from A.1: two nodes inserted after `.6`.** Everything else is identical. | id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | |---|---|---|---|---|---|---|---| -| .4a | `attn_index_proj` | GEMM ×3 — `wq_b` (2048→4096), `wk` (6144→128), `weights_proj` (6144→32), **replicated** | 599.8 MF | 19.937 MB | **BF16** | 4.15 | memory | -| .4b | `attn_index_score` | index scan over the whole history + top-2048 | 2,147.5 MF | 33.563 MB | FP8 KV | 6.99 | memory | +| .6a | `attn_index_proj` | GEMM ×3 — `wq_b` (2048→4096), `wk` (6144→128), `weights_proj` (6144→32), **replicated** | 599.8 MF | 19.937 MB | **BF16** | 4.15 | memory | +| .6b | `attn_index_score` | 32 heads score the whole history + top-2048 | 2,147.5 MF | 33.563 MB | FP8 KV | 6.99 | memory | -`.4a` is BF16 because the indexer is named in `modules_to_not_convert` — at FP8 it +`.6a` is BF16 because the indexer is named in `modules_to_not_convert` — at FP8 it would price at 10.0 MB, and this is the layer type whose cost grows with context. -**`.4b` is the only node in the model that grows with S**, and it does not stay +**`.6b` is the only node in the model that grows with S**, and it does not stay small: -| context S | `.4b` bytes/layer | `.4b` time/layer | Σ over 21 layers | share of step | +| context S | `.6b` bytes/layer | `.6b` time/layer | Σ over 21 layers | share of step | | --------- | ----------------- | ---------------- | ---------------- | ------------- | | 8,192 | 33.6 MB | 6.99 µs | 0.147 ms | 0.9 % | -| 131,072 | 537.0 MB | 0.112 ms | 2.349 ms | 13.0 % | -| 1,048,576 | **4,296.0 MB** | **0.895 ms** | **18.795 ms** | **54.4 %** | +| 131,072 | 537.0 MB | 0.112 ms | 2.349 ms | 12.1 % | +| 1,048,576 | **4,296.0 MB** | **0.895 ms** | **18.795 ms** | **52.5 %** | -**Σ per layer at S=8192: 8.3 GF, 923.7 MB, 0.214 ms** — 5 % more than `Ls,sh`, and +**Σ per layer at S=8192: 8.3 GF, 928.3 MB, 0.230 ms** — 5 % more than `Ls,sh`, and that 5 % is the whole price of IndexShare's 21-of-78 schedule at short context. ### A.3 — Archetype `Ld,f`, 3 layers (full indexer + DENSE FFN) -**Delta from A.2: `.9`–`.15` replaced by two nodes.** No router, no top-k, no -expert bank — and therefore **no data-dependent shape and no expert-parallel -traffic** on these three layers. They are the only blocks in the model a CUDA graph -could capture unconditionally. +**Delta from A.2: `.13`–`.22` replaced by four nodes.** No router, no sigmoid, no +top-k, no expert bank, no all-to-all — and therefore **no data-dependent shape and +no expert-parallel traffic**. These three are the only blocks in the model a CUDA +graph could capture unconditionally. | id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | |---|---|---|---|---|---|---|---| -| .9′ | `mlp_gate_up` | GEMM (6144→3072/rank) | 1,208.0 MF | 19.469 MB | FP8 | 4.06 | memory | -| .10′ | `mlp_down` | GEMM (1536/rank→6144) | 604.0 MF | 9.931 MB | FP8 | 2.07 | memory | +| .13′ | `act_quant_mlp` | dynamic FP8 quant | 0.4 MF | 0.590 MB | BF16→FP8 | 2.00 | launch | +| .14′ | `mlp_gate_up` | GEMM (6144→3072/rank) | 1,208.0 MF | 19.469 MB | FP8 | 4.06 | memory | +| .15′ | `mlp_silu` | SwiGLU | 0.2 MF | 0.197 MB | BF16 | 2.00 | launch | +| .16′ | `mlp_down` | GEMM (1536/rank→6144) | 604.0 MF | 9.931 MB | FP8 | 2.07 | memory | -**Σ per layer: 5.2 GF, 168.9 MB, 0.050 ms** — **a quarter the time of a MoE layer -at a fifth the bytes.** Three of 78 layers are 0.9 % of the step. +**Σ per layer: 7.3 GF, 167.9 MB, 0.057 ms** — **a quarter the time of a MoE layer +at a fifth the bytes.** Three of 78 layers are 1.0 % of the step. ### A.4 — MTP draft chain, per stage (×5 at D=5) @@ -1082,24 +1113,39 @@ for the sequence, not for the verify rows. | id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | |---|---|---|---|---|---|---|---| -| M.1 | `mtp_eh_proj` | GEMM `[12288→6144]`, **replicated** | 4,831.8 MF | **152.175 MB** | **BF16** | 31.70 | memory | -| M.2–M.17 | the whole `Ls,sh` block (A.1) | as A.1 | 5.6 GF | 870.2 MB | mixed | 203 | memory | -| M.18 | `lm_head` | GEMM, vocab-sharded `[6144→19360]` | 7,612.7 MF | **239.528 MB** | **BF16** | 49.90 | memory | -| M.19 | `argmax` + D2H | sampling + host round-trip | — | small | BF16 | — | **sync (S6)** | +| M.1 | `mtp_norms` | `enorm` + `hnorm`, fused RMSNorm ×2 | 1.2 MF | 1.573 MB | **BF16** | 4.00 | launch | +| M.2 | `mtp_eh_proj` | GEMM `[12288→6144]`, **replicated** | 4,831.8 MF | **152.175 MB** | **BF16** | 31.70 | memory | +| M.3–M.26 | the whole `Ls,sh` block (A.1) | as A.1 | 5.6 GF | 874.8 MB | mixed | 219 | memory | +| M.27 | `lm_head` | GEMM, vocab-sharded `[6144→19360]` | 7,612.7 MF | **239.528 MB** | **BF16** | 49.90 | memory | +| M.28 | `argmax` + D2H | sampling + host round-trip | — | small | BF16 | — | **sync (S6)** | -**Σ per stage: 18.0 GF, 1,261.9 MB, 0.285 ms — ×5 = 6.31 GB, 1.423 ms.** +**Σ per stage: 18.0 GF, 1,268.1 MB, 0.305 ms — ×5 = 6.34 GB, 1.523 ms.** Three things in that table are the whole §3.3 argument: -- **M.2–M.17 is a full MoE block.** The MTP module carries its own 256-expert +- **M.3–M.26 is a full MoE block.** The MTP module carries its own 256-expert `mlp.experts.*` bank in the checkpoint, so 69 % of a draft stage's bytes are expert weights it re-reads every stage. There is no saturation to help: 32 rows wakes ~163 experts, five times over. -- **M.1 and M.18 are BF16**, both named in `modules_to_not_convert`, and together - they are **31 %** of the stage. `eh_proj` is replicated per rank (Q10); the - vocabulary projection is shared with the backbone and gets no cheaper for being - a draft. +- **M.1, M.2 and M.27 are BF16**, all named in `modules_to_not_convert`, and + together they are **31 %** of the stage. `eh_proj` is replicated per rank (Q10); + the vocabulary projection is shared with the backbone and gets no cheaper for + being a draft. - **What is absent:** no `attn_index_proj`, no `attn_index_score`. The MTP block has no indexer tensors in the weight map, which is - `index_share_for_mtp_iteration: true` made visible — and it means the draft - inherits the top-2048 selection the backbone already paid for. + `index_share_for_mtp_iteration: true` made visible — the draft inherits the + top-2048 selection the backbone already paid for. + +### A.5 — Node budget for the whole step + +| region | ×N | nodes each | Σ nodes | Σ ms | share | +| ------ | -- | ---------- | ------- | ---- | ----- | +| prologue + epilogue | 1 | 4 | 4 | 0.073 | 0.4 % | +| `Ld,f` dense layers | 3 | 20 | 60 | 0.172 | 1.0 % | +| `Ls,f` full-indexer MoE | 18 | 26 | 468 | 4.142 | 24.1 % | +| `Ls,sh` shared-indexer MoE | 57 | 24 | 1,368 | 12.481 | 72.7 % | +| `Lmtp` draft (D=1) | 1 | 27 | 27 | 0.305 | 1.8 % | +| **total** | | | **1,927** | **17.173** | | + +At D=5 the draft region becomes 5 × 27 = 135 nodes and 1.523 ms, and the backbone +runs at 192 rows instead of 32 — **2,035 nodes, 28.641 ms.** diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 5883a16..67e107d 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -95,6 +95,19 @@ weight_bytes, ) + +def _canon(dtype: str) -> str: + """Canonical dtype name, so ``"e4m3"`` and ``"fp8"`` answer the same question.""" + d = dtype.lower() + if d in ("fp8", "e4m3", "e5m2"): + return "fp8" + if d in ("bf16", "fp16", "float16", "half"): + return "fp16" + if d in ("fp32", "float32", "tf32"): + return "fp32" + return d + + FULL_INDEXER = "full" SHARED_INDEXER = "shared" DENSE_MLP = "dense" @@ -466,6 +479,23 @@ def _linear(rows: float, k: int, n: int, act_b: float, w_b: float) -> tuple[floa return 2.0 * rows * k * n, act_b * rows * k + w_b * k * n + act_b * rows * n +def _pointwise(rows: float, elems: float, act_b: float, *, ops: float = 1.0) -> tuple[float, float]: + """(flops, bytes) for an elementwise kernel over ``rows x elems``. + + Read once, written once — the traffic that makes a norm or a residual add + cost anything at all. ``ops`` is arithmetic per element (a residual add is 1, + an RMSNorm is ~3 counting the reduction and the rescale). + + These nodes are individually a rounding error and collectively the low-batch + story: at batch 1 there is no useful arithmetic and no useful bandwidth in any + of them, so each one costs exactly one kernel launch and the step is the sum + of its launches. A graph that folds them into the GEMM they precede cannot + show that, and will report a decode step as memory-bound when it is not. + """ + n = rows * elems + return ops * n, 2.0 * n * act_b + + def _emit_layer( g: Graph, spec: GlmMoeDsaModelSpec, @@ -535,7 +565,30 @@ def w_bytes(op: str, default: str) -> float: """Bytes per stored weight for ``op``, after any precision override.""" return weight_bytes(spec.dtype_for(op, default)) + def add_pointwise(op: str, elems: float, *, ops: float = 1.0) -> None: + f_p, b_p = _pointwise(rows, elems, aw, ops=ops) + add(op, f_p, b_p, spec.act_dtype) + + def add_act_quant(op: str, elems: float, gemm_op: str) -> None: + """Dynamic FP8 activation scaling ahead of a quantised GEMM. + + GLM-5.2-FP8 declares ``activation_scheme: "dynamic"``, so the activation + is quantised at run time — a pointwise pass plus a per-row reduction for + the scale, as its own kernel, once per group of fp8 GEMMs that share an + input. Emitted only where the consuming GEMM is actually fp8: on the bf16 + checkpoint there is nothing to quantise and the kernel does not exist, + which is the sort of difference a single model-wide dtype cannot express. + """ + if _canon(spec.dtype_for(gemm_op, wd)) != "fp8": + return + # Read the bf16 activation, write the fp8 one plus its scales. + add(op, 2.0 * rows * elems, + rows * elems * (aw + 1.0) + rows * 4.0, spec.act_dtype) + # ── MLA attention: low-rank query, compressed KV latent ────────────────── + add_pointwise("input_layernorm", h, ops=3.0) + add_act_quant("act_quant_attn", h, "attn_q_a") + # q_a and kv_a are replicated across TP ranks: they produce the shared latent, # which has nothing to split when there is one KV latent. Every rank pays them # in full, so TP's speedup on attention is strictly less than ``tp``. @@ -653,17 +706,22 @@ def w_bytes(op: str, default: str) -> float: add("attn_out_proj", f, b, wd) _emit_collective(g, spec, hw, layer, "tp_all_reduce_attn", rows, sh, prefix) + add_pointwise("attn_residual", h) + add_pointwise("post_attention_layernorm", h, ops=3.0) # ── FFN: dense on the leading layers, mixture on the rest ──────────────── if not spec.is_sparse_mlp(layer): # Dense FFN (first_k_dense_replace). gate+up then down over the wide # intermediate. Canonical dense-graph names so residuals stay comparable. inter = spec.intermediate_size + add_act_quant("act_quant_mlp", h, "mlp_gate_up") f_gu, b_gu = _linear(rows, h, 2 * inter // tp, aw, w_bytes("mlp_gate_up", wd)) add("mlp_gate_up", f_gu, b_gu, wd) + add_pointwise("mlp_silu", inter / tp, ops=4.0) f_d, b_d = _linear(rows, inter // tp, h, aw, w_bytes("mlp_down", wd)) add("mlp_down", f_d, b_d, wd) _emit_collective(g, spec, hw, layer, "tp_all_reduce_mlp", rows, sh, prefix) + add_pointwise("mlp_residual", h) return # Router is replicated: every rank scores every expert to know what to keep. @@ -672,17 +730,24 @@ def w_bytes(op: str, default: str) -> float: f, b = _linear(rows, h, spec.n_routed_experts, aw, w_bytes("moe_router", wd)) add("moe_router", f, b, wd) - # Sigmoid scoring, the noaux_tc bias correction, top-8 and the renorm. Almost - # no bytes and almost no arithmetic — but it is the **only data-dependent - # shape in the step**, so it is the node that decides whether the step can be - # CUDA-graph captured at all. Kept as its own node for that reason alone. + # Sigmoid scoring and the noaux_tc bias correction — a plain pointwise pass + # over the [rows, 256] score matrix. + add_pointwise("moe_sigmoid_bias", spec.n_routed_experts, ops=3.0) + + # The top-8 selection and renorm. Almost no bytes and almost no arithmetic — + # but it is the **only data-dependent shape in the step**, and therefore the + # node that decides whether the step can be CUDA-graph captured at all. Split + # from the sigmoid above for exactly that reason: they are the same size and + # the same cost, and only one of them is a hazard. add( "moe_topk", - 3.0 * rows * spec.n_routed_experts, - 2.0 * rows * spec.n_routed_experts * aw, + 0.0, + rows * (spec.n_routed_experts + 2.0 * spec.top_k) * aw, spec.act_dtype, ) + add_act_quant("act_quant_moe", h, "moe_routed") + inter = spec.moe_intermediate_size per_expert_weights = 3.0 * h * inter per_position_flops = 6.0 * h * inter # 2 * (gate + up + down) * h * inter @@ -725,6 +790,10 @@ def w_bytes(op: str, default: str) -> float: ed, ) + # SwiGLU between the gate/up and down grouped GEMMs. Its own kernel in every + # grouped-GEMM backend this targets, over the expanded ``rows x top_k`` tensor. + add_pointwise("moe_silu", spec.top_k * inter / es, ops=4.0) + # Weighted scatter-add back to ``rows x hidden``, including the # routed_scaling_factor multiply. add( @@ -738,6 +807,7 @@ def w_bytes(op: str, default: str) -> float: g, spec, hw, layer, "tp_all_reduce_mlp", rows, sh, prefix, dispatches_experts=True, ) + add_pointwise("mlp_residual", h) def _emit_collective( @@ -857,11 +927,35 @@ def predict_glm_graph( raise ValueError("n_layers must be positive — an empty model predicts nothing") g = Graph(model=spec, hw=hw, batch=batch, sharding=sh) # type: ignore[arg-type] + aw = weight_bytes(spec.act_dtype) + rows = float(batch.positions_per_step + batch.prefill_tokens) + + # Prologue: one gather from the untied input embedding. No FLOPs, and the + # bytes are the rows it touches, not the 1.9 GB table — an index_select reads + # what it selects. + g.nodes.append( + PredictedNode( + "embed_tokens", None, + roofline("embed_tokens", 0.0, rows * spec.hidden * aw, hw, + spec.act_dtype, serial_launches=1), + ) + ) for layer in range(spec.n_layers): _emit_layer(g, spec, hw, layer, batch=batch, sh=sh) - aw = weight_bytes(spec.act_dtype) + # Epilogue: the final norm runs over every row that needs logits, not every + # row in the step — the same count lm_head uses, and at prefill that is one + # row per prompt rather than the whole chunk. + logit_rows = float(batch.logits_rows) + f_n, b_n = _pointwise(logit_rows, spec.hidden, aw, ops=3.0) + g.nodes.append( + PredictedNode( + "final_norm", None, + roofline("final_norm", f_n, b_n, hw, spec.act_dtype, serial_launches=1), + ) + ) + lm_w = weight_bytes(spec.dtype_for("lm_head", spec.weight_dtype)) lm_dtype = spec.dtype_for("lm_head", spec.weight_dtype) @@ -876,6 +970,26 @@ def add_lm_head(rows: float, layer: int | None) -> None: add_lm_head(batch.logits_rows, None) + # The vocabulary projection is sharded across TP ranks, so the ranks must + # gather each other's slices before sampling. FP32 logits, full vocabulary — + # 19.8 MB at 32 rows, which is small in bytes and is one more unavoidable + # collective on the critical path between the last layer and the sample. + if sh.tp > 1: + link = replace(hw, peak_mem_bw_bytes_per_s=hw.interconnect_bw_bytes_per_s) + priced = hw.interconnect_bw_bytes_per_s > 0 + g.nodes.append( + PredictedNode( + "logits_all_gather", None, + roofline( + "logits_all_gather", 0.0, + batch.logits_rows * spec.vocab * 4.0 * (sh.tp - 1) / sh.tp, + link, "fp32", estimated=True, + serial_launches=1 if priced else 0, + ), + expected_stream_id=1, + ) + ) + # ── the draft chain ────────────────────────────────────────────────────── # The MTP module is invoked once per drafted token, serially. Each stage sees # one row per sequence (it proposes for the sequence, not for the verify rows) @@ -906,6 +1020,17 @@ def add_lm_head(rows: float, layer: int | None) -> None: batch=draft_batch, sh=sh, force_full_indexer=not spec.index_share_for_mtp_iteration, ) + # ``enorm`` and ``hnorm``: the MTP block normalises the embedding and + # the carried hidden state separately before fusing them. Two kernels, + # bf16 on the FP8 checkpoint, and they sit inside the serial chain. + f_n2, b_n2 = _pointwise(draft_batch.batch, 2.0 * spec.hidden, aw, ops=3.0) + g.nodes.append( + PredictedNode( + "mtp_norms", spec.n_layers + stage, + roofline("mtp_norms", f_n2, b_n2, hw, spec.act_dtype, + serial_launches=2), + ) + ) # ``eh_proj``: the [2*hidden, hidden] fusion of the previous hidden # state with the embedding of the token just drafted. bf16 on the FP8 # checkpoint (named in ``modules_to_not_convert``), and replicated per diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index 90aece5..d9f96be 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -404,6 +404,83 @@ def test_a_pure_prefill_step_runs_no_draft_head(): assert len(_ops(g, "lm_head")) == 1 +#: The node a GLM-5.2 MoE layer lowers to, in issue order. Pinned because the +#: design note's whole low-batch argument is a claim about *how many kernels* a +#: layer is, not just how many bytes it moves — and a graph that quietly folds the +#: pointwise work into the GEMM it precedes reports a decode step as memory-bound +#: when it is launch-bound. +MOE_LAYER_NODES = ( + "input_layernorm", "act_quant_attn", + "attn_q_a", "attn_q_b", "attn_kv_a", "attn_kv_b", + "attn_score_value", "attn_qnorm_rope_insert", "attn_out_proj", + "tp_all_reduce_attn", "attn_residual", "post_attention_layernorm", + "moe_router", "moe_sigmoid_bias", "moe_topk", "act_quant_moe", + "moe_shared", "moe_permute", "moe_routed", "moe_silu", "moe_combine", + "moe_all_to_all", "tp_all_reduce_mlp", "mlp_residual", +) + + +def test_layer_lowers_to_the_documented_node_sequence(): + # The FP8 entry, because two of the 24 nodes are the dynamic activation + # quantisation the bf16 checkpoint does not run. + spec = load_spec("glm-5.2-fp8") + g = predict_glm_graph( + spec, batch=BatchConfig(batch=32, kv_cache_len=8192), + sharding=ShardingConfig(tp=8, ep=8), + ) + shared = tuple(n.op for n in g.nodes if n.layer == 5) # Ls,sh + assert shared == MOE_LAYER_NODES + + # A full-indexer layer is the same sequence with two nodes inserted. + full = tuple(n.op for n in g.nodes if n.layer == 6) # Ls,f + assert len(full) == len(shared) + 2 + assert "attn_index_proj" in full and "attn_index_score" in full + + # A dense layer swaps the whole mixture for three nodes, and so is the only + # block in the model with no data-dependent shape and no expert traffic. + dense = tuple(n.op for n in g.nodes if n.layer == 0) # Ld,f + assert {"act_quant_mlp", "mlp_gate_up", "mlp_silu", "mlp_down"} <= set(dense) + assert "moe_topk" not in dense and "moe_all_to_all" not in dense + + +def test_prologue_and_epilogue_are_nodes(): + """The step does not begin at layer 0 or end at the last one. + + A gather, a final norm, the vocabulary projection and — under TP — the logits + all-gather that has to complete before anything can be sampled. + """ + spec = _spec() + g = predict_glm_graph( + spec, batch=BatchConfig(batch=32, kv_cache_len=8192), + sharding=ShardingConfig(tp=8), + ) + ends = [n.op for n in g.nodes if n.layer is None] + assert ends == ["embed_tokens", "final_norm", "lm_head", "logits_all_gather"] + # Without TP there is nothing to gather. + solo = predict_glm_graph(spec, batch=BatchConfig(batch=32, kv_cache_len=8192)) + assert "logits_all_gather" not in [n.op for n in solo.nodes] + + +def test_act_quant_exists_only_where_a_gemm_is_actually_fp8(): + """Dynamic activation scaling is a kernel the bf16 checkpoint does not run. + + ``activation_scheme: "dynamic"`` means the activation is quantised at run + time, once per group of fp8 GEMMs sharing an input. On the unquantised + checkpoint there is nothing to quantise — the sort of difference a single + model-wide dtype cannot express. + """ + bf16 = _spec() + fp8 = replace( + bf16, weight_dtype="fp8", expert_dtype="fp8", + op_dtype_overrides=(("lm_head", "bf16"), ("moe_router", "fp32")), + ) + batch = BatchConfig(batch=32, kv_cache_len=8192) + assert not _ops(predict_glm_graph(bf16, batch=batch), "act_quant_attn") + assert len(_ops(predict_glm_graph(fp8, batch=batch), "act_quant_attn")) == ( + fp8.n_layers + fp8.num_nextn_predict_layers + ) + + def test_detect_family_routes_glm_before_sparse_moe(): """Both families carry index_topk + n_routed_experts; model_type must win.""" assert detect_family(GLM_CONFIG) == "glm_moe_dsa" From d39a66ae78d01a10b8481222f3efc912d9487f4b Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 14:13:58 -0700 Subject: [PATCH 07/35] GLM-5.2: op names a capture can pair against; scan math dtype MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the pointwise lowering. That commit added twelve op names without checking them against deviation.classify_op, which is the fallback identity for a capture with no NVTX ranges (docs/kernel_identity.md). A name it cannot classify leaves the predicted node permanently unmatched AND the real kernel filed as unmodeled -- two errors in opposite directions, in the per-op diff this family exists to support. The note asserts that diff in three places, so the names are load-bearing. Three of the new names could not resolve and three collided with decisions the other families depend on. Resolved by following the canonical names rather than redefining them: - One rms_norm op for all three norm sites. They are one kernel name; only an NVTX range can say which site a launch belongs to. The residual add is inside it, because vLLM runs RMSNorm.forward(x, residual) as one fused_add_rms_norm kernel -- a separate residual node predicted a launch that never happens. - SwiGLU folded back into mlp_gate_up and moe_routed: silu_and_mul was already that op's needle. - The fused gating kernel emitted as a second moe_router instance, not a private moe_topk: moe_align/topk_softmax -> moe_router is tested and the dense-MoE and hybrid families rely on it. It is still its own node -- it is still the only data-dependent shape in the step. _OP_RULES then gains only what is genuinely new and unclaimed: rms_norm, act_quant, embed_tokens, moe_permute/moe_combine, attn_index_proj (was shadowed by the scan's own "indexer" needle), attn_kv_b (absent while only absorbed MLA was modelled; GLM models it unabsorbed, so the kernel exists), mtp_eh_proj. A test asserts every op the graph emits resolves. Separately: attn_index_score carried one dtype for two questions. Its bytes follow how the keys are stored (fp8); its FLOPs follow what the indexer computes in, and the indexer is one of the modules the quantiser skipped. Invisible at decode -- memory-bound at every context -- and 1.2% -> 2.2% of prefill, where it is compute-bound. Full suite back to the pre-existing baseline of 4 failures. Every figure in the note re-derived; §7 gains the two gaps this pass actually was. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 324 +++++++++++++++++------------- gitm/optimizer/deviation.py | 51 ++++- gitm/planner/glm_graph.py | 103 ++++++---- tests/test_deviation_alignment.py | 7 +- tests/test_glm_graph.py | 59 ++++-- 5 files changed, 355 insertions(+), 189 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 10c57dc..f9f7340 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -243,7 +243,7 @@ tokens issues `8P` token→expert assignments. Once `8P ≫ 256` — P above a f hundred — every expert receives at least one token, so **every layer reads its entire expert bank**, 1.26 GB per rank per layer at EP8, **constant in P**. That is 95.7 GB per pass, 23 % of all prefill traffic. But it is not the top line: **the EP -all-to-all is**, at 105.7 GB and 45 % of predicted prefill time. Three quarters of +all-to-all is**, at 105.7 GB and 44 % of predicted prefill time. Three quarters of prefill cost is the MoE path, and under expert parallelism most of that is *wire*, not DRAM. @@ -335,10 +335,10 @@ never read. flowchart LR subgraph V["VANILLA DECODE"] direction TB - v1["1 row per seq"] --> v2["78 layers + 1 MTP
1,927 nodes"] --> v3["lm_head"] --> v4["sample"] + v1["1 row per seq"] --> v2["78 layers + 1 MTP
1,614 nodes"] --> v3["lm_head"] --> v4["sample"] v4 --> v5(("1 token")) end - subgraph D["DRAFT — 5 SERIAL stages, 135 nodes"] + subgraph D["DRAFT — 5 SERIAL stages, 115 nodes"] direction TB d0["h at last accepted pos
+ its token"] --> eh["eh_proj [12288→6144] BF16"] eh --> d1["MTP block
MLA+DSA (shared index) + FULL MoE"] @@ -347,7 +347,7 @@ flowchart LR end subgraph W["VERIFY = decode at 1+D rows"] direction TB - w1["6 rows per seq"] --> w2["THE SAME 78 layers
THE SAME 1,900 nodes"] + w1["6 rows per seq"] --> w2["THE SAME 78 layers
THE SAME 1,591 nodes"] w2 --> w3["lm_head, 6 rows"] --> w4["compare vs draft"] w4 --> w5(("1..6 accepted")) end @@ -357,8 +357,8 @@ flowchart LR ``` **Verify is not a new graph.** It is the decode graph with the row dimension -multiplied by `1+D`. The only genuinely new subgraph is the draft chain — 135 nodes -against the backbone's 1,900. +multiplied by `1+D`. The only genuinely new subgraph is the draft chain — 115 nodes +against the backbone's 1,591. **The dependency point:** the loop is strictly serial and now has `D+1` sampling points instead of one. Nothing in stage `k` can start before stage `k-1`'s token id @@ -441,11 +441,11 @@ against P = 1 decode row per sequence: | Prefill, P=8192, C=0, TP8/EP8, per rank | value | | --- | --- | -| predicted floor | **262.4 ms** for the chunk (31.2 k tok/s) | -| bytes moved | **422.6 GB** | +| predicted floor | **264.0 ms** for the chunk (31.0 k tok/s) | +| bytes moved | **422.0 GB** | | FLOPs | **118.4 TF** | -| whole-pass AI | **280** — below the fp8 ridge of 412, so **memory-bound overall** | -| facets | compute 666 n / 86.3 ms / 32.9 % · memory 1,082 n / 175.7 ms / 67.0 % · launch 152 n / 0.3 ms | +| whole-pass AI | **281** — below the fp8 ridge of 412, so **memory-bound overall** | +| facets | compute 666 n / 89.3 ms / 33.8 % · memory 848 n / 174.6 ms / 66.1 % · launch 77 n / 0.2 ms | > ⚠ **The prefill rows overturn the dense-model intuition, and the reversal is > worth keeping visible.** Dense intuition says prefill is compute-bound. It is @@ -461,10 +461,10 @@ against P = 1 decode row per sequence: | chunking | bytes | floor | | ------------------- | ----------- | ---------- | -| 1 × 8,192 | **423 GB** | 262 ms | -| 2 × 4,096 | 518 GB | 278 ms | -| 8 × 1,024 | 1,088 GB | 394 ms | -| 64 × 128 | **6,313 GB**| **1,579 ms** | +| 1 × 8,192 | **422 GB** | 264 ms | +| 2 × 4,096 | 517 GB | 281 ms | +| 8 × 1,024 | 1,088 GB | 396 ms | +| 64 × 128 | **6,313 GB**| **1,543 ms** | **14.9× the bytes for identical FLOPs.** The expert bank is read per *chunk*, not per token — so a small `--max-num-batched-tokens`, chosen to protect decode latency, @@ -476,13 +476,13 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): | pass | nodes | bytes | floor | | ---------------------- | --------- | ------------ | ------------- | -| vanilla decode (D=0) | 1,927 | 68.61 GB | 17.173 ms | -| — of which the draft | 27 | 1.27 GB | 0.305 ms | -| draft chain, D=5 | 135 | **6.34 GB** | 1.523 ms | -| verify, 192 rows | 1,900 | 106.4 GB | 27.118 ms | -| **MTP step total** | **2,035** | **112.8 GB** | **28.641 ms** | +| vanilla decode (D=0) | 1,614 | 68.60 GB | 16.551 ms | +| — of which the draft | 23 | 1.27 GB | 0.297 ms | +| draft chain, D=5 | 115 | **6.34 GB** | 1.483 ms | +| verify, 192 rows | 1,591 | 106.4 GB | 26.652 ms | +| **MTP step total** | **1,706** | **112.8 GB** | **28.135 ms** | -**Cost ratio 1.67× for up to 6 tokens.** Where the extra 44.2 GB goes: +**Cost ratio 1.70× for up to 6 tokens.** Where the extra 44.2 GB goes: - **The verify pass, +37.8 GB.** Almost all of it is one line: expert weights go from 163 distinct experts at 32 rows to 256 at 192 rows — **the union saturates**, @@ -498,20 +498,20 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): more rows and more stages touch more experts, not because more work is done per token. -**Break-even.** The step costs 1.67×; it produces up to 6 tokens instead of 1, so -acceptance α must exceed `(1.67 − 1)/5 = 0.134` to pay. Predicted throughput: +**Break-even.** The step costs 1.70×; it produces up to 6 tokens instead of 1, so +acceptance α must exceed `(1.70 − 1)/5 = 0.140` to pay. Predicted throughput: | α | 0.0 | 0.5 | 0.7 | 0.9 | | ---- | ----- | ----- | ----- | ----- | -| tok/s| 1,117 | 3,911 | 5,028 | **6,145** | +| tok/s| 1,137 | 3,981 | 5,118 | **6,256** | -against 1,863 tok/s with MTP off. The vendor claims the GLM-5.2 MTP layer raises +against 1,933 tok/s with MTP off. The vendor claims the GLM-5.2 MTP layer raises accepted length by up to 20 % over its predecessor, which puts the operating point well past break-even — but **α is a serving observable and this graph does not predict it.** It prices the cost and leaves the payoff to a measurement (§6.2, C4). ⚑ **All of this assumes the step is memory-bound.** At B ≤ 8 it is not (§4.1), and -in the launch regime the draft's 135 extra launches are pure cost against a step +in the launch regime the draft's 115 extra launches are pure cost against a step that was never moving bytes. The sign of the MTP decision flips with batch. ### 3.4 Predicted synchronization points @@ -528,7 +528,7 @@ that was never moving bytes. The sign of the MTP decision flips with batch. | S8 | KV rollback | discard rejected rows | **low** on mechanism | pointer rewind (free) or real memmove (not free) — the trace tells you which | | S9 | indexer selection handoff | the `full` layer's top-k must be visible to its 3 `shared` layers | **low** | if it round-trips the host, IndexShare costs a sync it should not — **21 per step** | -**S5 is the decode-specific one worth chasing.** Decode runs ~1,927 kernels in +**S5 is the decode-specific one worth chasing.** Decode runs ~1,614 kernels in ~16 ms and then hands control back to a Python scheduler. If the scheduler takes longer than the step, the GPU idles and no kernel-level work matters. @@ -563,48 +563,58 @@ ridge 412 F/B (fp8) · 206 (bf16) · 14 (fp32) · launch floor 2.0 µs (graph-re node kernel class bytes AI xN Σ ms bound share ────────────────────── ─────────────────────── ──────── ──── ─── ───── ────── ──────────────────────── - moe_routed grouped GEMM fp8 ×3 771.1 MB 3.1 76 12.208 memory ████████████████████ 71.1% - attn_score_value paged decode attention 42.0 MB 12.8 79 0.690 memory █ 4.0% - moe_all_to_all collective (EP a2a) 5.5 MB — 76 0.465 comm ▊ 2.7% + moe_routed grouped GEMM fp8 ×3 771.3 MB 3.1 76 12.213 memory ████████████████████ 73.8% + attn_score_value paged decode attention 42.0 MB 12.8 79 0.690 memory █ 4.2% + moe_all_to_all collective (EP a2a) 5.5 MB — 76 0.465 comm ▊ 2.8% + rms_norm fused_add_rms_norm 1.6 MB 0.8 160 0.322 launch ▌ 1.9% + act_quant dynamic fp8 quant 0.6 MB 0.7 158 0.316 launch ▌ 1.9% + moe_router GEMM fp32 + fused gating 3.4 MB 15.0 152 0.304 launch ▌ 1.8% attn_q_a GEMM fp8, REPLICATED 13.1 MB 61.4 79 0.216 memory ▌ 1.3% attn_out_proj GEMM fp8, tall-skinny 13.1 MB 61.4 79 0.216 memory ▌ 1.3% - ── 17 nodes at the 2 µs launch floor, 0.158 ms each (0.9%) ──────────────────────────────── - input_layernorm · act_quant_attn · attn_q_b · attn_kv_a · attn_kv_b · - attn_qnorm_rope_insert · tp_all_reduce_attn · attn_residual · - post_attention_layernorm · moe_router (fp32) · moe_sigmoid_bias · - moe_topk · act_quant_moe · moe_shared · moe_permute · moe_silu · - moe_combine · tp_all_reduce_mlp · mlp_residual + ── 8 more nodes at the 2 µs launch floor, 0.158 ms each (1.0%) ──────────────────────────── + attn_q_b · attn_kv_a · attn_kv_b · attn_qnorm_rope_insert · + tp_all_reduce_attn · tp_all_reduce_mlp · moe_shared · moe_permute · moe_combine ─────────────────────────────────────────────────────────────────────────────────────────── attn_index_score index scan + top-k 33.6 MB 64.0 21 0.147 memory ▏ 0.9% ← the only term that grows with S lm_head GEMM bf16, tall-skinny 239.5 MB 31.8 2 0.100 memory ▏ 0.6% mtp_eh_proj GEMM bf16, REPLICATED 152.2 MB 31.8 1 0.032 memory ▏ 0.2% logits_all_gather collective (all-gather) 17.3 MB — 1 0.019 comm ▏ 0.1% ────────────────────── ─────────────────────── ──────── ──── ─── ───── ────── ──────────────────────── - 1,927 nodes 17.173 ms/step · 1,863 tok/s @ B=32 - moe_routed alone 12.208 ms = 71.1% of the step + 1,614 nodes 16.551 ms/step · 1,933 tok/s @ B=32 + moe_routed alone 12.213 ms = 73.8% of the step facets - memory 441 nodes 14.199 ms 82.7% ← five node types - launch 1,486 nodes 2.974 ms 17.3% ← 77% of all nodes, a sixth of the time + memory 441 nodes 14.203 ms 85.8% ← five node types + launch 1,173 nodes 2.348 ms 14.2% ← 73% of all nodes, a seventh of the time compute 0 nodes 0.000 ms 0.0% ← the entire roofline claim, one row ``` -**A MoE layer is 24 kernels and two of them cost anything.** `moe_routed` is 73 % +**A MoE layer is 20 kernels and two of them cost anything.** `moe_routed` is 76 % of the layer's time; `attn_score_value`, `attn_q_a`, `attn_out_proj` and the EP -all-to-all are most of the rest; the other 19 are at the launch floor. That +all-to-all are most of the rest; the other 15 are at the launch floor. That distribution is the model, not an artefact — and it is why every ranked hypothesis in §5 is about either the expert bank or the launch count. -Two rows are kept **despite** being small: `moe_topk` because it is the only -data-dependent shape in the graph and therefore the thing that blocks CUDA-graph -capture, and `attn_index_score` because "the only term that grows with S" is the -architecture's whole payoff — and because it does not stay small: +**Three op names appear more than once per layer, deliberately.** `rms_norm` twice +(entry and post-attention, plus once in the epilogue), `act_quant` twice (ahead of +the attention GEMMs and ahead of the experts), `moe_router` twice (the `h→256` +GEMM, then the fused gating kernel that scores and selects). They are the *same +kernel name* in a trace, so they are the same op here — which is the only way the +per-op residual diff in §6 can pair them at all. Which of the two a launch belongs +to is recoverable from an NVTX range and never from a name; see +`docs/kernel_identity.md`. + +Two rows are kept **despite** being small. The second `moe_router` instance is the +fused gating kernel — **the only data-dependent shape in the graph**, and therefore +the thing that blocks CUDA-graph capture. And `attn_index_score`, because "the only +term that grows with S" is the architecture's whole payoff, and it does not stay +small: | context S | `attn_index_score` | share of step | step floor | | --------- | ------------------ | ------------- | ---------- | -| 8,192 | 0.147 ms | 0.9 % | 17.173 ms | -| 131,072 | 2.349 ms | 12.1 % | 19.375 ms | -| **1,048,576** | **18.795 ms** | **52.5 %** | 35.821 ms | +| 8,192 | 0.147 ms | 0.9 % | 16.551 ms | +| 131,072 | 2.349 ms | 12.5 % | 18.753 ms | +| **1,048,576** | **18.795 ms** | **53.4 %** | 35.199 ms | **At the model's advertised context the indexer scan is the largest node in the step**, and it is the node IndexShare already cut by 3.7×. Everything anyone says @@ -615,18 +625,18 @@ the step. | B | floor | tok/s | launch nodes | launch time | compute nodes | | --- | ---------- | ----- | ------------ | ----------- | ------------- | -| 1 | 4.571 ms | 219 | 1,666 | 3.334 ms = **73 %** | 0 | -| 4 | 6.258 ms | 639 | 1,665 | 3.332 ms = 53 % | 0 | -| 16 | 11.913 ms | 1,343 | 1,486 | 2.974 ms = 25 % | 0 | -| 32 | 17.173 ms | 1,863 | 1,486 | 2.974 ms = 17 % | 0 | -| 64 | 23.017 ms | 2,781 | 1,410 | 2.822 ms = 12 % | 76 | -| 128 | 28.291 ms | 4,524 | 1,252 | 2.506 ms = 9 % | 76 | -| 256 | 35.004 ms | 7,314 | 1,176 | 2.354 ms = 7 % | 79 | +| 1 | 3.946 ms | 253 | 1,353 | 2.708 ms = **69 %** | 0 | +| 4 | 5.633 ms | 710 | 1,352 | 2.706 ms = 48 % | 0 | +| 16 | 11.289 ms | 1,417 | 1,173 | 2.348 ms = 21 % | 0 | +| 32 | 16.551 ms | 1,933 | 1,173 | 2.348 ms = 14 % | 0 | +| 64 | 22.400 ms | 2,857 | 1,097 | 2.196 ms = 10 % | 76 | +| 128 | 27.682 ms | 4,624 | 939 | 1.880 ms = 7 % | 76 | +| 256 | 34.510 ms | 7,418 | 705 | 1.412 ms = 4 % | 79 | **Below B≈16 the step is a launch-bound step wearing a memory-bound model's -clothes.** At B=1, **73 %** of the predicted floor is 1,666 kernel launches at 2 µs -— and that already assumes CUDA-graph replay. At the eager 5 µs it is 8.33 ms of -launches against a 1.24 ms memory term: **87 %**, and the whole low-batch analysis +clothes.** At B=1, **69 %** of the predicted floor is 1,353 kernel launches at 2 µs +— and that already assumes CUDA-graph replay. At the eager 5 µs it is 6.77 ms of +launches against a 1.24 ms memory term: **85 %**, and the whole low-batch analysis changes sign (A4, §5 rank 3). ### 4.2 Prefill and MTP — regions and what flips them @@ -636,23 +646,23 @@ at TP8/EP8, FP8. | Phase | Region | Bound | Why (point at a number) | Precision / peak | Flip variable | |---|---|---|---|---|---| -| **Pre** | **EP dispatch/combine all-to-all** | **comm — BANDWIDTH** | 1.39 GB/layer × 76 = **105.7 GB**, 117.4 ms = **44.8 % of the step** | BF16 payload | ⚑ **fp8 dispatch halves it**; EP degree; TP-only removes the node and doubles the bank | +| **Pre** | **EP dispatch/combine all-to-all** | **comm — BANDWIDTH** | 1.39 GB/layer × 76 = **105.7 GB**, 117.4 ms = **44.5 % of the step** | BF16 payload | ⚑ **fp8 dispatch halves it**; EP degree; TP-only removes the node and doubles the bank | | **Pre** | **MoE expert grouped GEMMs** | compute (AI 485) | 1.26 GB/layer **constant in P**, 95.7 GB/pass = 23 % of traffic | **FP8 block-scaled** | **chunk size** (14.9× across 1→64 chunks); imbalance | -| **Pre** | **router GEMM** | **compute** | 26 GF/layer at **fp32's 67 TF/s** → 28.8 ms = **11.0 %** | ⚠ **FP32** — see Q3 | ⚑ whether the engine runs the GEMM in fp32 or only accumulates there | +| **Pre** | **router GEMM** | **compute** | 26 GF/layer at **fp32's 67 TF/s** → 29.0 ms = **11.0 %** | ⚠ **FP32** — see Q3 | ⚑ whether the engine runs the GEMM in fp32 or only accumulates there | | **Pre** | `all_reduce` ×158 | **comm — BANDWIDTH** | 174 MB each, 27.5 GB/pass = 30.5 ms | BF16 payload | P; below P≈256 flips to latency | | **Pre** | projections (`q_a`,`o_proj`,`q_b`,`kv_a`,`kv_b`) | **compute** | AI 1,403 / 963 / 488 vs the fp8 ridge 412 | FP8 e4m3 | prompt length — below P≈512 memory-bound | -| **Pre** | indexer scan ×21 | **compute** | **5.77 TF** against 22 MB of keys — `O(P·C + P²/2)` × 32 heads. **The quadratic lives here, not in the core** | **BF16** proj, fp8 keys | P **and** C | +| **Pre** | indexer scan ×21 | **compute** | 5.77 TF against 22 MB of keys — `O(P·C + P²/2)` × 32 heads, and **2.2 % of the step**. **The quadratic lives here, not in the core** | **BF16** arithmetic, fp8 keys — two dtypes, one node | P **and** C | | **Pre** | attention core ×79 | compute, **linear in C** | FLOPs capped at 2,048 keys/query; **bytes are the whole cache once per request** | FP8 KV | ⚑ **`index_topk`**; C. Not P² | | **Pre** | permute / combine | memory | 8.5 GB/layer-pass on the `8P`-row expanded tensor, near-zero FLOPs | BF16 activations | top-k; **expert imbalance** | | **Pre** | *everything else* — embed gather, norms, `lm_head`, `moe_topk` | memory or launch | each under 1.5 %; `lm_head` reads 239.5 MB for **one row per request** | BF16; FP32 top-k | none of them flips | -| **Pre** | **whole prefill pass** | **memory** | **AI 280 vs ridge 412**; 67 % memory / 33 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | +| **Pre** | **whole prefill pass** | **memory** | **AI 281 vs ridge 412**; 66 % memory / 34 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | | **MTP** | verify expert GEMMs | **memory** | 256 distinct experts at 192 rows vs 163 at 32 — **+57 %/layer, ~85 % of MTP's cost** | FP8 block-scaled | `B(1+D)` vs E=256; **D**; imbalance | | **MTP** | draft expert GEMMs ×D | **memory** | the MTP block carries a **full 256-expert bank**; 5 stages × 163 experts, **linear in D, no saturation** | FP8 block-scaled | **D**; batch | | **MTP** | draft `lm_head` ×5 | memory | 239.5 MB × 5 = **1.20 GB = 19 % of the draft's bytes** | **BF16** | sharded sampling; draft vocab | | **MTP** | draft `eh_proj` ×5 | memory | `[12288,6144]` BF16, **replicated per rank** | **BF16** — in `modules_to_not_convert` | whether it is TP-sharded | | **MTP** | verify attention | **memory, unchanged** | 0.43 GB — read **per sequence, not per row**; 1+D rows share one block table | FP8 KV | seq length; explicitly *not* D | | **MTP** | accept/reject + KV rollback | launch + **host sync** | tiny tensors, but a data-dependent host-visible seq length (S7) | n/a | pointer rewind vs memmove | -| **MTP** | **whole MTP step** | ⚑ **memory above B≈16, launch below — and the two regimes disagree about whether MTP helps** | **1.67×** cost for ≤6 tokens at B=32; break-even α = **0.134** | mixed | **graph capture**; batch; α; D | +| **MTP** | **whole MTP step** | ⚑ **memory above B≈16, launch below — and the two regimes disagree about whether MTP helps** | **1.70×** cost for ≤6 tokens at B=32; break-even α = **0.140** | mixed | **graph capture**; batch; α; D | **Hardware sensitivity:** nothing flips between H200 SXM and H20 on the compute rows — but H20's much lower FP8 peak moves every prefill projection further into @@ -666,7 +676,7 @@ compute-bound, and its bandwidth moves the decode floor directly. | indexer scan | **compute** (5.77 TF / 22 MB) | **memory** (AI 64) | the query count collapses; the key set does not | | attention core | compute, **whole cache** in bytes | **launch/memory**, one 2,048-window | 8,192 queries' selections union to everything; one query's do not | | `all_reduce` ×158 | **comm-bandwidth** (30.5 ms wire) | **comm-latency** (0.16 ms floor, 0.8 µs wire) | payload 174 MB vs 688 kB | -| EP all-to-all | **the top line** (44.8 %) | 3.0 % | payload scales with rows; the ring latency does not | +| EP all-to-all | **the top line** (44.5 %) | 3.0 % | payload scales with rows; the ring latency does not | | `lm_head` | memory, 1 row per **request** | memory, **every row every step** | the epilogue is free in prefill and is not in decode | | `moe_router` (fp32) | **compute** (11.0 % of prefill) | launch (1.0 %) | 26 GF/layer against 67 TF/s only matters when P is large | @@ -676,15 +686,15 @@ compute-bound, and its bandwidth moves the decode floor directly. | ------------- | ------------- | ----------------------- | | **Decode batch B** | every decode GEMM, expert hit-rate, all collectives, the whole-step label (launch below B≈16, memory above), the sign of the MTP decision | expert bytes **sub-linear** in B: 8 experts at B=1, 163 at B=32, 252 at B=128. 303 → 7,623 tok/s across 1→256 | | **Sequence length S** | the indexer scan, and **only** the indexer scan | 0.3 % of the step at 8K → 13.0 % at 128K → **54.4 % at 1M**. The attention core does not move at all | -| **Prompt length P** | every prefill projection (memory→compute above P≈512), the router, the indexer's quadratic, all-reduce (latency→bandwidth above P≈256) | whole-pass AI 280 at P=8k | -| ⚑ **Chunked prefill / chunk size** | prefill expert GEMMs, permute, and through them the whole prefill label | 8,192 tokens in 1 chunk = **423 GB**; in 64 chunks of 128 = **6,313 GB** for identical FLOPs | +| **Prompt length P** | every prefill projection (memory→compute above P≈512), the router, the indexer's quadratic, all-reduce (latency→bandwidth above P≈256) | whole-pass AI 281 at P=8k | +| ⚑ **Chunked prefill / chunk size** | prefill expert GEMMs, permute, and through them the whole prefill label | 8,192 tokens in 1 chunk = **422 GB**; in 64 chunks of 128 = **6,313 GB** for identical FLOPs | | ⚑ **CUDA-graph capture** | the whole decode step, the whole MTP step, every launch row | at B=1 it is 63 % of the floor; at 5 µs eager it is 81 %. Decides whether MTP is a 3× win or a net loss | -| **EP vs TP** | every collective row, every MoE memory row | under EP8 the a2a is **45 % of prefill** but the per-rank bank is **8× smaller**. Since the bank is ~85 % of decode DRAM, this is a **trade, not a cost** | -| ⚑ **Precision (fp8 vs bf16)** | every weight term, and whether the model fits at all | 1.76× on the decode floor (17.173 ms vs 30.196 ms) and **10.7 → 5.4 H200s** for weights | +| **EP vs TP** | every collective row, every MoE memory row | under EP8 the a2a is **44 % of prefill** but the per-rank bank is **8× smaller**. Since the bank is ~85 % of decode DRAM, this is a **trade, not a cost** | +| ⚑ **Precision (fp8 vs bf16)** | every weight term, and whether the model fits at all | 1.79× on the decode floor (16.551 ms vs 29.574 ms) and **10.7 → 5.4 H200s** for weights | | ⚑ **KV dtype** | the attention core, and the 1M-context footprint | 55.2 GB vs 99.9 GB per rank at 1M — the difference between fitting and not | | ⚑ **Absorbed vs unabsorbed MLA** | `attn_kv_b` + `attn_out_proj` | drops one node and doubles the other's input width: ±2× on 2.4 % of decode, more at prefill | | **`index_topk`** | the attention core's FLOPs in both phases; its bytes in neither | 2,048 → 4,096 doubles core FLOPs and changes no byte term | -| **Draft depth D, acceptance α** | verify expert GEMMs, whole MTP step | 1.67× cost at D=5; break-even α = 0.134; 1,117 → 6,145 tok/s across α | +| **Draft depth D, acceptance α** | verify expert GEMMs, whole MTP step | 1.70× cost at D=5; break-even α = 0.140; 1,137 → 6,256 tok/s across α | | **Expert imbalance** | prefill + decode expert GEMMs, permute, the a2a, grouped-GEMM tail | skew *reduces* bytes while *increasing* tail latency and stalling every other EP rank | --- @@ -696,11 +706,11 @@ graph**, to be confirmed against a capture. | Rank | Region | Prediction | Why | Evidence to inspect | What would prove it wrong | |---|---|---|---|---|---| -| **1** | **EP all-to-all at prefill** | **≥45 % of prefill time is wire, and roughly half of it is recoverable** | 105.7 GB/pass in BF16. An fp8 dispatch halves the payload outright; overlapping dispatch with the shared-expert GEMM hides more | NCCL kernel duration vs payload at prefill (§6.4 row 5); whether dispatch is bf16 | duration ≪ payload/900 GB/s → the engine already fuses or compresses it | +| **1** | **EP all-to-all at prefill** | **≥44 % of prefill time is wire, and roughly half of it is recoverable** | 105.7 GB/pass in BF16. An fp8 dispatch halves the payload outright; overlapping dispatch with the shared-expert GEMM hides more | NCCL kernel duration vs payload at prefill (§6.4 row 5); whether dispatch is bf16 | duration ≪ payload/900 GB/s → the engine already fuses or compresses it | | **2** | **Grouped-GEMM group sizing (the fork)** | either **76 D2H per token** (no graph capture possible) **or** fixed-capacity padding (**all 256 experts read every step**) | The only data-dependent shape in the graph is `moe_topk`. A stack does one or the other — see §5.2 | `cuda_api_sum` D2H count per decode step | 0 D2H **and** MoE bytes that track `distinct_experts(B)` → a device-side path, nothing to recover | | **3** | ⚑ **CUDA-graph capture at low batch** | **63 % of the B=1 floor is launch overhead** (81 % at eager 5 µs) | 1,033 nodes × 2 µs = 2.07 ms against a 1.24 ms memory term | `cuda_api_sum` launch count with `--cuda-graph-trace=node`; expect **1** `cudaGraphLaunch` | already captured → this rank is worth nothing, and rank 2's fork is already resolved to "padded" | | **4** | **The indexer scan at long context** | at 1M it is **54 % of the step**, and IndexShare's 3.7× is already banked | 90.2 GB/step of index keys at S=1M. Levers: fp8 index keys (2×), temporal reuse across steps, a smaller `index_topk_freq` group | `dram__bytes_read.sum` on the scan vs 90.2 GB; whether keys are fp8 | keys already fp8 and no temporal reuse available → architectural | -| **5** | **`moe_routed` — the memory-bound heart** | 71.1 % of decode; **EPLB placement and the grouped-GEMM backend are the levers** | Weight traffic scales with *distinct* experts woken, not with FLOPs. Good placement cuts per-rank distinct traffic; DeepGEMM cuts the constant | measured per-rank expert traffic vs `distinct_experts`; **the real EP imbalance** | measured traffic already at the union prediction with balance ≈1.0 → the bank is the bank | +| **5** | **`moe_routed` — the memory-bound heart** | 73.8 % of decode; **EPLB placement and the grouped-GEMM backend are the levers** | Weight traffic scales with *distinct* experts woken, not with FLOPs. Good placement cuts per-rank distinct traffic; DeepGEMM cuts the constant | measured per-rank expert traffic vs `distinct_experts`; **the real EP imbalance** | measured traffic already at the union prediction with balance ≈1.0 → the bank is the bank | | **6** | ⚑ **fp32 router at prefill** | **11.0 % of prefill** rests on the reading that the router *GEMM* runs in fp32 | 26 GF/layer at 67 TF/s. If only the softmax/top-k accumulates in fp32 and the GEMM is bf16, this row shrinks ~15× | the engine's router implementation; `cuda_gpu_kern_sum` dtype of the gate GEMM | the GEMM is genuinely fp32 → architectural, and the row stays | | **7** | **Decode collective placement** | 158 all-reduces + 76 all-to-alls + 1 logits all-gather per step on the compute stream, **latency-bound**, with idle SMs around them | 688 kB payloads: the gap *is* the cost. Nothing prevents other layers' work overlapping | NCCL ranges and stream ids on the timeline (S2/S3) | already on a separate stream with overlap → nothing to recover | | **8** | **Chunk size** | a decode-latency-protecting `--max-num-batched-tokens` can cost **14.9× the prefill bytes** | the expert bank is read per chunk | total prefill MoE DRAM ÷ 95.7 GB — the quotient **is** the chunk count | quotient ≈ 1 → prefill is already unchunked | @@ -783,7 +793,7 @@ Two worked examples, because the rule is easy to agree with and hard to apply: layers' work overlapping. Q2: *yes* — stream assignment. **Recoverable** (rank 7). **The trap runs in both directions.** The accept/reject readback will look alarming -and is architectural. The 1,927 kernel launches are entirely *expected* from §3 and +and is architectural. The 1,614 kernel launches are entirely *expected* from §3 and are the largest recoverable item at low batch. **Neither surprise nor familiarity is evidence.** @@ -835,7 +845,7 @@ Key: **R:** recoverable → the rank it feeds · **A:** architectural, do not ch |---|---|---|---|---| | **0** | **Launch args, as text** — not a measurement | C6 | chunk size, graph capture, KV dtype, D, TP/EP, absorbed MLA | Resolves or reframes **ranks 1, 2, 3, 6, 8 before a timeline is opened** | | **1** | `cuda_api_sum` — **D2H count per decode step** | C1 | **0** in the MoE region | **R:** 76/token → host-resolved group sizes, no graph capture → **rank 2**. **R:** 0 but MoE bytes flat in B → padded capacity → **rank 3** instead (§5.2). **A:** none | -| **2** | `cuda_api_sum` — **launches per step**, needs `--cuda-graph-trace=node` | C1 | **1** `cudaGraphLaunch` | **R:** ~1,927 individual launches + CPU gaps below B≈16 → **rank 3**. **F1:** count wildly off 1,927 → the lowering in §3 is wrong by an order of magnitude | +| **2** | `cuda_api_sum` — **launches per step**, needs `--cuda-graph-trace=node` | C1 | **1** `cudaGraphLaunch` | **R:** ~1,614 individual launches + CPU gaps below B≈16 → **rank 3**. **F1:** count wildly off 1,614 → the lowering in §3 is wrong by an order of magnitude | | **3** | `dram__bytes_read.sum` — **MoE region** | C1, C3 | 1.26 GB/layer/rank = **95.7 GB/pass**; **≈85 %** of decode | **R:** prefill total ÷ 95.7 GB > 1 → chunked re-read, and the quotient **is** the chunk count → **rank 8**. **R:** flat in B → padded capacity → **rank 3**. **F2:** much lower → L2 residency, and the central claim of both phases is wrong | | **4** | `dram__bytes_read.sum` — **indexer region**, swept in S | C2 | 33.6 MB/layer at 8K → **90.2 GB/step at 1M**, on **21 layers only** | **R:** keys read on 78 layers → IndexShare is not being honoured, a hidden 3.7× → **rank 4**. **R:** 2× the prediction → keys are bf16 where fp8 would do. **A:** growth on 21 layers is the architecture | | **5** | **NCCL kernel duration vs payload** | C1, C3, C5 | prefill **∝ payload** (~900 GB/s, 117 ms a2a); decode **flat, ~2 µs × 235** | **R:** prefill a2a at bf16 payload → fp8 dispatch → **rank 1**. **R:** decode collectives on the compute stream with idle SMs → **rank 7**. **A:** the ring latency floor | @@ -886,7 +896,9 @@ gaps below are narrower because of it. | **G5** | **`gitm plan` dropping the launch bound and mispricing the ridge** | `_render_table` recomputed `bound` as compute-vs-memory, **discarding `"launch"` entirely**, and divided the ridge by `peak_flops_bf16_per_s` regardless of op dtype. So **854 launch-bound nodes printed as memory-bound**, against ridge 206 where fp8 answers to 412 | use the node's own `bound`; print one ridge per dtype present in the graph; add a launch-bound count and a `*` marker where an op's instances disagree | **yes** | | **G6** | **Two collectives per layer, not one** | `_emit_layer` folded the post-attention and post-FFN all-reduces into one node with double the payload. Bytes right, **count wrong** — and at 688 kB a decode collective is bounded by its ring latency, so the count *is* the cost | `_emit_collective` called at both sub-block boundaries, emitting `tp_all_reduce_attn` and `tp_all_reduce_mlp` separately, with the EP all-to-all on the MoE half only | **yes** | | **G7** | **An MTP chain D stages deep, each with its own vocabulary projection** | The graph emitted **one** draft block and **one** `lm_head` for what the vendor recipe runs **five** deep. `lm_head` is 19 % of the draft's bytes, so a D-deep chain was understated by ~5× on its largest term | a stage loop in `predict_glm_graph` driven by `BatchConfig.speculative_tokens`, with `mtp_eh_proj` and an `lm_head` per stage; `--spec-tokens` on the CLI | **yes** | -| **G8** | **Which adjacent nodes may overlap and which may not** | `Graph.total_pred_s` is a sum, not a DAG. GLM puts both kinds of serialisation in one step — the draft chain is genuinely serial, the 158 collectives are not — and **both appear as the same positive residual today**. §6.1 is unanswerable without telling them apart | *Not shipped.* It is a cross-family IR change and it is already sequenced on the roadmap. What this note adds is the requirement, and `expected_stream_id=1` on collectives so the stream-concurrency invariant has something to read | **no — deliberately** | +| **G8** | **A graph that is only its GEMMs, priced against a bound it cannot express** | The family emitted 16 nodes per layer where a layer lowers to ~20 kernels, and the seven missing ones were all pointwise: the norms, the dynamic fp8 activation scaling, the fused gating, the prologue gather and the epilogue's all-gather. Every one is a rounding error in bytes and **a full kernel launch in time** — so at B=1 the graph reported a step as memory-bound that is 69 % launches. A roofline with a launch bound and a graph with no launches in it cannot both be right | Emit them. `_pointwise`, `add_rms_norm` (with the residual fused in, as vLLM runs it), `add_act_quant` gated on the consuming GEMM actually being fp8, plus `embed_tokens` / `rms_norm` / `logits_all_gather` around the stack. Node names constrained by G9 | **yes** | +| **G9** | **Op names a capture can actually pair against** | G8's new nodes needed names, and `deviation.classify_op` is a *name guess* (`docs/kernel_identity.md`): a name it cannot classify leaves the predicted node permanently unmatched **and** the real kernel filed as unmodeled — two errors in opposite directions, in the diff the family exists to support. Three norm sites are one kernel name; `silu_and_mul` was already claimed by `mlp_gate_up`; `moe_align`/`topk_softmax` were already claimed by `moe_router`, a decision the dense-MoE and hybrid families depend on | Follow the canonical names rather than redefine them: one `rms_norm` op for all three sites, SwiGLU folded back into the GEMM that owns its needle, gating emitted as a second `moe_router` instance. Then `_OP_RULES` gains only what is genuinely new and unclaimed — `rms_norm`, `act_quant`, `embed_tokens`, `moe_permute`/`moe_combine`, `attn_index_proj`, `attn_kv_b`, `mtp_eh_proj`. A test asserts every op the graph emits resolves | **yes** | +| **G10** | **Which adjacent nodes may overlap and which may not** | `Graph.total_pred_s` is a sum, not a DAG. GLM puts both kinds of serialisation in one step — the draft chain is genuinely serial, the 158 collectives are not — and **both appear as the same positive residual today**. §6.1 is unanswerable without telling them apart | *Not shipped.* It is a cross-family IR change and it is already sequenced on the roadmap. What this note adds is the requirement, and `expected_stream_id=1` on collectives so the stream-concurrency invariant has something to read | **no — deliberately** | ### 7.2 The one that needed more than a table row @@ -906,11 +918,23 @@ will get suggested — hence four helpers with four docstrings rather than one a ``` gitm/planner/glm_graph.py op_dtype_overrides + dtype_for; four DSA phase helpers; rows = positions + prefill_tokens throughout; - serial_launches on every node; moe_topk / moe_permute / - moe_combine as their own nodes; _emit_collective ×2 per - layer; the D-stage draft chain with per-stage lm_head; - indexer wk + weights_proj in both the graph and the - footprint; the quantisation-map reader + serial_launches on every node; the full pointwise + lowering -- rms_norm (residual fused in, as vLLM runs + it), act_quant gated on the GEMM being fp8, the fused + gating kernel, moe_permute / moe_combine, and the + embed_tokens / rms_norm / lm_head / logits_all_gather + that bracket the stack; _emit_collective x2 per layer, + EP dispatch only where a layer has experts; the D-stage + draft chain with per-stage lm_head and eh_proj; the + indexer's wk + weights_proj, its 32 head scores, and + its storage dtype held apart from its math dtype; + the quantisation-map reader +gitm/optimizer/deviation.py _OP_RULES entries for the ops the pointwise lowering + added, plus attn_index_proj (was shadowed by the scan), + attn_kv_b (present once MLA is modelled unabsorbed) and + mtp_eh_proj. Existing mappings left alone: the + gating->moe_router and silu_and_mul->mlp_gate_up + decisions are shared with the other families gitm/planner/context.py _FP32_PEAKS + fp32_peak_for_sku, wired to hardware_spec_for gitm/planner/registry.py node-owned bound labels, per-dtype ridges, launch count, --spec-tokens @@ -918,11 +942,13 @@ gitm/planner/model_catalogue.py nested tuple coercion for op_dtype_overrides gitm/planner/models/ glm-5.2.yaml the bf16 model fact + the fp32 router glm-5.2-fp8.yaml NEW — the vendor's recommended deployment -tests/test_glm_graph.py 26 tests: the fp8 footprint, the precision map, the +tests/test_glm_graph.py 29 tests: the fp8 footprint, the precision map, the prefill/decode byte inversion, the D-deep chain, two collectives, the 32-head index scan, that a pure-prefill - step runs no draft head, and that a launch floor - does not hide an unpriced collective + step runs no draft head, that a launch floor does not + hide an unpriced collective, the exact node sequence a + layer lowers to, and that every op the graph emits + resolves from some kernel name docs/glm-5.2/artifacts/*.json DELETED — 29k lines of node dumps that go stale on every graph change; the commands at the top of this note regenerate any of them @@ -943,7 +969,7 @@ docs/glm-5.2/artifacts/*.json DELETED — 29k lines of node dumps that go sta | **Q5** | Is the **EP dispatch** bf16 or fp8? | **half of 105.7 GB** at prefill — the largest single recoverable number here | serving image / C6 | | **Q6** | Is the **IndexShare selection** passed device-side? | 21 syncs/step if not (S9) | trace D2H attribution | | **Q7** | Chunked prefill on, at what chunk size? | **up to 14.9× on prefill bytes** | engine launch args (C6) | -| **Q8** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is 0.134; the range 0.5→0.9 is 3,911→6,145 tok/s | engine metrics (C4) — **not predictable from a config** | +| **Q8** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is 0.140; the range 0.5→0.9 is 3,981→6,256 tok/s | engine metrics (C4) — **not predictable from a config** | | **Q9** | Are the **index keys** cached in fp8 or bf16? | 2× on 90.2 GB/step at 1M context | C6 / trace | | **Q10** | Is `eh_proj` **TP-sharded** in the MTP block? | 151 MB → 19 MB per rank per draft stage | serving image | | **Q11** | Does the prefill attention kernel read the selected KV **once per request** or once per query tile? | up to 64× on the prefill core's bytes — the graph takes the optimistic floor | `dram__bytes_read.sum` on the prefill core | @@ -1028,10 +1054,16 @@ Read straight off the graph, in issue order — every row is a `PredictedNode` a this shape, and `tests/test_glm_graph.py::test_layer_lowers_to_the_documented_node_sequence` pins this exact sequence so the code cannot drift from the table. +**Three op names repeat, deliberately** (`rms_norm` ×2, `act_quant` ×2, +`moe_router` ×2). They are the same kernel name in a trace, so they are the same +op here; a private name per site would emit a node no capture could pair against +while the real kernel filed as unmodeled. Which instance a launch belongs to is an +NVTX question, not a name question — `docs/kernel_identity.md`. + | id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | |---|---|---|---|---|---|---|---| -| .1 | `input_layernorm` | fused RMSNorm | 0.6 MF | 0.786 MB | BF16 | 2.00 | launch | -| .2 | `act_quant_attn` | dynamic FP8 quant + row reduction | 0.4 MF | 0.590 MB | BF16→FP8 | 2.00 | launch | +| .1 | `rms_norm` | `fused_add_rms_norm` — input norm **with the residual carried in** | 1.2 MF | 1.573 MB | BF16 | 2.00 | launch | +| .2 | `act_quant` | dynamic FP8 quant + per-row scale | 0.4 MF | 0.590 MB | BF16→FP8 | 2.00 | launch | | .3 | `attn_q_a` | GEMM, **replicated** (6144→2048) | 805.3 MF | 13.110 MB | FP8 | 2.73 | memory | | .4 | `attn_q_b` | GEMM, head-sharded (2048→2048) | 268.4 MF | 4.457 MB | FP8 | 2.00 | launch | | .5 | `attn_kv_a` | GEMM + cache append (6144→576) | 226.5 MF | 3.990 MB | FP8 | 2.00 | launch | @@ -1040,30 +1072,34 @@ pins this exact sequence so the code cannot drift from the table. | .8 | `attn_qnorm_rope_insert` | fused q/kv norm + partial RoPE + insert | 0.4 MF | 0.410 MB | BF16 | 2.00 | launch | | .9 | `attn_out_proj` | GEMM, tall-skinny (16384→6144) | 805.3 MF | 13.110 MB | FP8 | 2.73 | memory | | .10 | `tp_all_reduce_attn` | NCCL ring | — | 0.688 MB | BF16 | 2.00 | launch | -| .11 | `attn_residual` | pointwise add | 0.2 MF | 0.786 MB | BF16 | 2.00 | launch | -| .12 | `post_attention_layernorm` | fused RMSNorm | 0.6 MF | 0.786 MB | BF16 | 2.00 | launch | -| .13 | `moe_router` | GEMM, **replicated** (6144→256) | 100.7 MF | 6.701 MB | **FP32** | 2.00 | launch | -| .14 | `moe_sigmoid_bias` | pointwise + `e_score_correction_bias` | 0.02 MF | 0.033 MB | BF16 | 2.00 | launch | -| .15 | `moe_topk` | **top-8 of 256 + renorm** | 0 F | 0.017 MB | BF16 | 2.00 | launch ← **the only data-dependent shape** | +| .11 | `rms_norm` | `fused_add_rms_norm` — post-attention | 1.2 MF | 1.573 MB | BF16 | 2.00 | launch | +| .12 | `moe_router` | GEMM, **replicated** (6144→256) | 100.7 MF | 6.701 MB | **FP32** | 2.00 | launch | +| .13 | `moe_router` | fused gating: sigmoid + `e_score_correction_bias` + **top-8 of 256** + renorm | 0 F | 0.034 MB | **FP32** | 2.00 | launch ← **the only data-dependent shape** | | | ⚠ **Blocks CUDA-graph capture. ↯ S1: host readback of the expert histogram? 76 D2H per token if real.** | | | | | | | -| .16 | `act_quant_moe` | dynamic FP8 quant | 0.4 MF | 0.590 MB | BF16→FP8 | 2.00 | launch | -| .17 | `moe_shared` | grouped GEMM ×3, always on | 302.0 MF | 5.539 MB | FP8 | 2.00 | launch | -| .18 | `moe_permute` | gather into expert-major order | 0 F | 0.442 MB | BF16 | 2.00 | launch | -| .19 | `moe_routed` | grouped GEMM ×3, **163 distinct of 256** | 2,415.9 MF | **771.062 MB** | FP8 | **160.64** | **memory** | -| .20 | `moe_silu` | SwiGLU over the expanded tensor | 0.3 MF | 0.262 MB | BF16 | 2.00 | launch | -| .21 | `moe_combine` | scatter-add × `routed_scaling 2.5` | 3.1 MF | 0.442 MB | BF16 | 2.00 | launch | -| .22 | `moe_all_to_all` | EP dispatch + combine | — | 5.505 MB | BF16 | 6.12 | comm | -| .23 | `tp_all_reduce_mlp` | NCCL ring | — | 0.688 MB | BF16 | 2.00 | launch | -| .24 | `mlp_residual` | pointwise add | 0.2 MF | 0.786 MB | BF16 | 2.00 | launch | - -**Σ per layer: 5.6 GF, 874.8 MB, 0.219 ms.** - -**Twenty-four kernels, and two of them cost anything.** `.19` alone is **88 % of -the layer's bytes and 73 % of its time**; `.7`, `.3`, `.9` and `.22` are most of -the remainder; the other **19 nodes sit at the 2 µs launch floor** — 0.038 ms per -layer of pure launch, which is 17 % of it. Those 19 rows are why §4.1's launch -facet exists, and folding any of them into the GEMM it precedes would report this -layer as more memory-bound than it is. +| .14 | `act_quant` | dynamic FP8 quant, expert input | 0.4 MF | 0.590 MB | BF16→FP8 | 2.00 | launch | +| .15 | `moe_shared` | grouped GEMM ×3, always on | 302.0 MF | 5.539 MB | FP8 | 2.00 | launch | +| .16 | `moe_permute` | gather into expert-major order | 0 F | 0.442 MB | BF16 | 2.00 | launch | +| .17 | `moe_routed` | grouped GEMM ×3 + SwiGLU, **163 distinct of 256** | 2,416.2 MF | **771.324 MB** | FP8 | **160.69** | **memory** | +| .18 | `moe_combine` | scatter-add × `routed_scaling 2.5` | 3.1 MF | 0.442 MB | BF16 | 2.00 | launch | +| .19 | `moe_all_to_all` | EP dispatch + combine | — | 5.505 MB | BF16 | 6.12 | comm | +| .20 | `tp_all_reduce_mlp` | NCCL ring | — | 0.688 MB | BF16 | 2.00 | launch | + +**Σ per layer: 5.6 GF, 874.9 MB, 0.211 ms.** + +**Twenty kernels, and two of them cost anything.** `.17` alone is **88 % of the +layer's bytes and 76 % of its time**; `.7`, `.3`, `.9` and `.19` are most of the +remainder; the other **15 nodes sit at the 2 µs launch floor** — 0.030 ms per layer +of pure launch, 14 % of it. Those 15 rows are why §4.1's launch facet exists, and +folding any of them into the GEMM it precedes would report this layer as more +memory-bound than it is. + +Two folds are deliberate and named, because they are the places a reader will +expect a row and not find one. **SwiGLU is inside `.17`** and inside `mlp_gate_up` +on the dense layers: `silu_and_mul` is already one of `mlp_gate_up`'s needles in +`deviation._OP_RULES`, so a separate node would be a prediction with no observed +kernel to pair against. **The residual adds are inside `.1` and `.11`**: vLLM runs +`RMSNorm.forward(x, residual)` as one `fused_add_rms_norm` kernel, so a separate +residual node would predict a launch that never happens. ### A.2 — Archetype `Ls,f`, 18 layers (full indexer + MoE) @@ -1072,39 +1108,46 @@ layer as more memory-bound than it is. | id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | |---|---|---|---|---|---|---|---| | .6a | `attn_index_proj` | GEMM ×3 — `wq_b` (2048→4096), `wk` (6144→128), `weights_proj` (6144→32), **replicated** | 599.8 MF | 19.937 MB | **BF16** | 4.15 | memory | -| .6b | `attn_index_score` | 32 heads score the whole history + top-2048 | 2,147.5 MF | 33.563 MB | FP8 KV | 6.99 | memory | +| .6b | `attn_index_score` | 32 heads score the whole history + top-2048 | 2,147.5 MF | 33.563 MB | **BF16 math, FP8 keys** | 6.99 | memory | `.6a` is BF16 because the indexer is named in `modules_to_not_convert` — at FP8 it would price at 10.0 MB, and this is the layer type whose cost grows with context. -**`.6b` is the only node in the model that grows with S**, and it does not stay -small: +**`.6b` carries two dtypes and they answer different questions.** Its *bytes* are +governed by how the keys are stored (fp8, under the vendor recipe's fp8 cache); +its *FLOPs* by what the indexer computes in, which is the projection's bf16. At +decode the distinction is invisible — the node is memory-bound at every context — +and at prefill it is the difference between 1.2 % and 2.2 % of the step. + +**`.6b` is also the only node in the model that grows with S**, and it does not +stay small: | context S | `.6b` bytes/layer | `.6b` time/layer | Σ over 21 layers | share of step | | --------- | ----------------- | ---------------- | ---------------- | ------------- | | 8,192 | 33.6 MB | 6.99 µs | 0.147 ms | 0.9 % | -| 131,072 | 537.0 MB | 0.112 ms | 2.349 ms | 12.1 % | -| 1,048,576 | **4,296.0 MB** | **0.895 ms** | **18.795 ms** | **52.5 %** | +| 131,072 | 537.0 MB | 0.112 ms | 2.349 ms | 12.5 % | +| 1,048,576 | **4,296.0 MB** | **0.895 ms** | **18.795 ms** | **53.4 %** | -**Σ per layer at S=8192: 8.3 GF, 928.3 MB, 0.230 ms** — 5 % more than `Ls,sh`, and +**Σ per layer at S=8192: 8.3 GF, 928.4 MB, 0.222 ms** — 5 % more than `Ls,sh`, and that 5 % is the whole price of IndexShare's 21-of-78 schedule at short context. ### A.3 — Archetype `Ld,f`, 3 layers (full indexer + DENSE FFN) -**Delta from A.2: `.13`–`.22` replaced by four nodes.** No router, no sigmoid, no -top-k, no expert bank, no all-to-all — and therefore **no data-dependent shape and -no expert-parallel traffic**. These three are the only blocks in the model a CUDA +**Delta from A.2: `.12`–`.19` replaced by three nodes.** No router, no gating, no +expert bank, no all-to-all — and therefore **no data-dependent shape and no +expert-parallel traffic**. These three are the only blocks in the model a CUDA graph could capture unconditionally. | id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | |---|---|---|---|---|---|---|---| -| .13′ | `act_quant_mlp` | dynamic FP8 quant | 0.4 MF | 0.590 MB | BF16→FP8 | 2.00 | launch | -| .14′ | `mlp_gate_up` | GEMM (6144→3072/rank) | 1,208.0 MF | 19.469 MB | FP8 | 4.06 | memory | -| .15′ | `mlp_silu` | SwiGLU | 0.2 MF | 0.197 MB | BF16 | 2.00 | launch | -| .16′ | `mlp_down` | GEMM (1536/rank→6144) | 604.0 MF | 9.931 MB | FP8 | 2.07 | memory | +| .12′ | `act_quant` | dynamic FP8 quant | 0.4 MF | 0.590 MB | BF16→FP8 | 2.00 | launch | +| .13′ | `mlp_gate_up` | GEMM (6144→3072/rank) **+ SwiGLU** | 1,208.2 MF | 19.665 MB | FP8 | 4.10 | memory | +| .14′ | `mlp_down` | GEMM (1536/rank→6144) | 604.0 MF | 9.931 MB | FP8 | 2.07 | memory | -**Σ per layer: 7.3 GF, 167.9 MB, 0.057 ms** — **a quarter the time of a MoE layer -at a fifth the bytes.** Three of 78 layers are 1.0 % of the step. +**Σ per layer: 7.3 GF, 167.6 MB, 0.052 ms** — **a quarter the time of a MoE layer +at a fifth the bytes.** Three of 78 layers are 0.9 % of the step. (Layer 0's entry +norm is the one `rms_norm` in the model with no residual to carry — nothing +precedes it.) ### A.4 — MTP draft chain, per stage (×5 at D=5) @@ -1113,21 +1156,21 @@ for the sequence, not for the verify rows. | id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | |---|---|---|---|---|---|---|---| -| M.1 | `mtp_norms` | `enorm` + `hnorm`, fused RMSNorm ×2 | 1.2 MF | 1.573 MB | **BF16** | 4.00 | launch | +| M.1 | `rms_norm` | `enorm` + `hnorm`, two launches | 1.2 MF | 1.573 MB | **BF16** | 4.00 | launch | | M.2 | `mtp_eh_proj` | GEMM `[12288→6144]`, **replicated** | 4,831.8 MF | **152.175 MB** | **BF16** | 31.70 | memory | -| M.3–M.26 | the whole `Ls,sh` block (A.1) | as A.1 | 5.6 GF | 874.8 MB | mixed | 219 | memory | -| M.27 | `lm_head` | GEMM, vocab-sharded `[6144→19360]` | 7,612.7 MF | **239.528 MB** | **BF16** | 49.90 | memory | -| M.28 | `argmax` + D2H | sampling + host round-trip | — | small | BF16 | — | **sync (S6)** | +| M.3–M.22 | the whole `Ls,sh` block (A.1) | as A.1 | 5.6 GF | 874.9 MB | mixed | 211 | memory | +| M.23 | `lm_head` | GEMM, vocab-sharded `[6144→19360]` | 7,612.7 MF | **239.528 MB** | **BF16** | 49.90 | memory | +| M.24 | `argmax` + D2H | sampling + host round-trip | — | small | BF16 | — | **sync (S6)** | -**Σ per stage: 18.0 GF, 1,268.1 MB, 0.305 ms — ×5 = 6.34 GB, 1.523 ms.** +**Σ per stage: 18.0 GF, 1,268.2 MB, 0.297 ms — ×5 = 6.34 GB, 1.483 ms.** Three things in that table are the whole §3.3 argument: -- **M.3–M.26 is a full MoE block.** The MTP module carries its own 256-expert +- **M.3–M.22 is a full MoE block.** The MTP module carries its own 256-expert `mlp.experts.*` bank in the checkpoint, so 69 % of a draft stage's bytes are expert weights it re-reads every stage. There is no saturation to help: 32 rows wakes ~163 experts, five times over. -- **M.1, M.2 and M.27 are BF16**, all named in `modules_to_not_convert`, and +- **M.1, M.2 and M.23 are BF16**, all named in `modules_to_not_convert`, and together they are **31 %** of the stage. `eh_proj` is replicated per rank (Q10); the vocabulary projection is shared with the backbone and gets no cheaper for being a draft. @@ -1141,11 +1184,16 @@ Three things in that table are the whole §3.3 argument: | region | ×N | nodes each | Σ nodes | Σ ms | share | | ------ | -- | ---------- | ------- | ---- | ----- | | prologue + epilogue | 1 | 4 | 4 | 0.073 | 0.4 % | -| `Ld,f` dense layers | 3 | 20 | 60 | 0.172 | 1.0 % | -| `Ls,f` full-indexer MoE | 18 | 26 | 468 | 4.142 | 24.1 % | -| `Ls,sh` shared-indexer MoE | 57 | 24 | 1,368 | 12.481 | 72.7 % | -| `Lmtp` draft (D=1) | 1 | 27 | 27 | 0.305 | 1.8 % | -| **total** | | | **1,927** | **17.173** | | - -At D=5 the draft region becomes 5 × 27 = 135 nodes and 1.523 ms, and the backbone -runs at 192 rows instead of 32 — **2,035 nodes, 28.641 ms.** +| `Ld,f` dense layers | 3 | 17 | 51 | 0.155 | 0.9 % | +| `Ls,f` full-indexer MoE | 18 | 22 | 396 | 3.999 | 24.2 % | +| `Ls,sh` shared-indexer MoE | 57 | 20 | 1,140 | 12.028 | 72.7 % | +| `Lmtp` draft (D=1) | 1 | 23 | 23 | 0.297 | 1.8 % | +| **total** | | | **1,614** | **16.551** | | + +At D=5 the draft region becomes 5 × 23 = 115 nodes and 1.483 ms, and the backbone +runs at 192 rows instead of 32 — **1,706 nodes, 28.135 ms.** + +The prologue/epilogue rows are 4 nodes and 0.4 % of the step, and two of them are +on the critical path between the last layer and the sample: `lm_head` re-reads +239.5 MB of vocabulary weights **every step**, and `logits_all_gather` cannot start +until it finishes. diff --git a/gitm/optimizer/deviation.py b/gitm/optimizer/deviation.py index 3834b90..527b996 100644 --- a/gitm/optimizer/deviation.py +++ b/gitm/optimizer/deviation.py @@ -88,6 +88,12 @@ # completely different cost curve; and the indexer must never fall through to # a bare "index" rule, which is how it gets misfiled as elementwise in the # coarse taxonomy. + # The indexer's own projections, before the score entry below — whose + # "indexer" needle would otherwise claim ``indexers_proj`` and file a GEMM as + # a scan. They are separate kernels with opposite bounds: on GLM-5.2 the + # projection is bf16 and flat in context, the scan grows with it. + "attn_index_proj": ("indexer_proj", "indexers_proj", "index_proj", "wq_b", + "weights_proj"), "attn_index_score": ("indexer", "lightning_index", "index_topk", "topk_indices"), "moe_shared": ("shared_expert", "moe_shared"), # `topkGating` is vLLM's fused routing kernel. Without it the generic "moe" @@ -96,6 +102,17 @@ # traffic dominates the step. "moe_router": ("moe_align", "topk_softmax", "topkgating", "gating", "router", "routing", "sinkhorn", "expert_bias"), + # Dispatch/gather into expert-major order and the weighted scatter back. + # Before the generic "moe" needle below, which would claim both as expert + # GEMMs — they move real bytes and do no arithmetic, so folding them into the + # dominant weight-traffic row hides a term that chunk size and expert + # imbalance both move. + # Combine first: "unpermute" contains "permute", so the reverse direction has + # to be tested before the forward one or every combine kernel files as a + # dispatch. + "moe_combine": ("moe_sum", "finalize_moe", "unpermute", "scatter_add", + "index_add", "moe_combine"), + "moe_permute": ("permute", "expert_sort", "shuffle_rows", "gather_rows"), "moe_routed": ("moe", "expert", "grouped_gemm", "group_gemm", "groupedgemm"), "dspark": ("dspark",), @@ -120,17 +137,45 @@ # checkpoints use (`_triton_mrope_forward`). "attn_qnorm_rope_insert": ("qnorm", "q_norm", "qk_norm", "mrope", "rope", "rotary"), + # ── pointwise work that is its own kernel ──────────────────────────────── + # These used to classify as ``None`` — "a norm/activation/copy" — and land as + # unmodeled. That was right while no graph emitted them. The GLM-5.2 graph + # does, because on a sparse model at low batch the pointwise kernels are the + # majority of the launches and a step bounded by its launches cannot be + # explained by a graph that only has GEMMs in it. + # + # ``rms_norm`` covers every norm site in a block. They are one kernel name in + # the trace, so they are one op here; which site a given launch belongs to is + # recoverable only from an NVTX range (``docs/kernel_identity.md``), never + # from the name. + "rms_norm": ("rms_norm", "rmsnorm", "layernorm", "layer_norm", "fused_add_rms"), + # Dynamic FP8 activation scaling ahead of a quantised GEMM. "dequant" is + # excluded on purpose: it is the epilogue of the GEMM, not this kernel. + "act_quant": ("scaled_fp8_quant", "per_token_quant", "act_quant", + "quant_fp8", "dynamic_scaled"), + # Before ``lm_head``, whose "embed" needle would otherwise claim the input + # gather and attribute it to the vocabulary projection. + "embed_tokens": ("embedding", "embed_tokens", "index_select"), + # ── projections ────────────────────────────────────────────────────────── "attn_q_a": ("q_a_proj", "q_lora", "q_down"), "attn_q_b": ("q_b_proj", "q_up"), - # `kv_b_proj` is absent on purpose: in the absorbed decode form it is folded - # into the query and output projections, so there is no node to map it to and - # a guess would attribute real work to the wrong op. "attn_kv_a": ("kv_a_proj", "kv_lora", "kv_down", "compress_kv"), + # `kv_b_proj` was absent here while the only MLA families modelled the + # *absorbed* decode form, where W^UK folds into the query and W^UV into the + # output projection and no such kernel is launched. The GLM-5.2 graph models + # it unabsorbed, so the kernel exists and has a node to land on. The entry is + # safe either way: an absorbed deployment launches nothing these needles + # match, so it stays absent rather than mis-attributing. + "attn_kv_b": ("kv_b_proj", "kv_up", "w_uk", "w_uv"), "qkv_proj": ("qkv",), "attn_out_proj": ("o_proj", "out_proj", "attn_out"), "mlp_gate_up": ("gate_up", "gate_proj", "up_proj", "swiglu", "silu_and_mul"), "mlp_down": ("down_proj", "mlp_down"), + # The MTP block's [2h, h] fusion of the carried hidden state with the + # embedding of the token just drafted. Before ``lm_head``, which is the other + # bf16 GEMM in a draft stage. + "mtp_eh_proj": ("eh_proj", "mtp_proj"), "lm_head": ("lm_head", "logits", "vocab_proj", "embed"), } diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 67e107d..c3196fa 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -569,6 +569,26 @@ def add_pointwise(op: str, elems: float, *, ops: float = 1.0) -> None: f_p, b_p = _pointwise(rows, elems, aw, ops=ops) add(op, f_p, b_p, spec.act_dtype) + def add_rms_norm(*, with_residual: bool) -> None: + """One ``rms_norm`` node per norm site — and the residual add is inside it. + + vLLM runs ``RMSNorm.forward(x, residual)`` as a single + ``fused_add_rms_norm`` kernel, so a separate residual node would predict a + launch that never happens. ``with_residual`` adds the extra read the fused + form does, and costs nothing else. + + All three norm sites in a layer share this op name deliberately. They are + the *same kernel* in the trace — ``classify_op`` matches on the kernel + name, and three distinct op names for one name would leave two of them + permanently unmatched while the third absorbed all three sites' time. The + position that distinguishes them is carried by ``layer`` and by issue + order, which is where it belongs; see ``docs/kernel_identity.md`` on why a + name is a guess and an NVTX range is an identity. + """ + elems = h * (2.0 if with_residual else 1.0) + f_p, b_p = _pointwise(rows, elems, aw, ops=3.0) + add("rms_norm", f_p, b_p, spec.act_dtype) + def add_act_quant(op: str, elems: float, gemm_op: str) -> None: """Dynamic FP8 activation scaling ahead of a quantised GEMM. @@ -586,8 +606,8 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: rows * elems * (aw + 1.0) + rows * 4.0, spec.act_dtype) # ── MLA attention: low-rank query, compressed KV latent ────────────────── - add_pointwise("input_layernorm", h, ops=3.0) - add_act_quant("act_quant_attn", h, "attn_q_a") + add_rms_norm(with_residual=layer > 0) + add_act_quant("act_quant", h, "attn_q_a") # q_a and kv_a are replicated across TP ranks: they produce the shared latent, # which has nothing to split when there is one KV latent. Every rank pays them @@ -648,6 +668,15 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: wd, ) + # The scan's two dtypes are different questions and this node answers + # both. Its **bytes** are governed by how the index keys are *stored* + # (``kv_dtype`` — fp8 under the vendor recipe's fp8 cache); its **FLOPs** + # are governed by what the indexer *computes* in, and the indexer is one + # of the modules the quantiser skipped, so that is the projection's dtype. + # One field cannot do both: pricing 2.1 GF of bf16 arithmetic against the + # fp8 peak halves it, which is invisible at decode (the node is + # memory-bound at every context) and doubles the prefill row, where it is + # compute-bound. add( "attn_index_score", # Every one of the 32 index heads scores each candidate against the @@ -661,7 +690,7 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: # alongside the KV latent. index_scan_entries(batch) * spec.index_head_dim * weight_bytes(spec.kv_dtype), - spec.kv_dtype, + spec.dtype_for("attn_index_proj", wd), ) # ── attention core over the selected positions ────────────────────────── @@ -706,22 +735,23 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: add("attn_out_proj", f, b, wd) _emit_collective(g, spec, hw, layer, "tp_all_reduce_attn", rows, sh, prefix) - add_pointwise("attn_residual", h) - add_pointwise("post_attention_layernorm", h, ops=3.0) + add_rms_norm(with_residual=True) # ── FFN: dense on the leading layers, mixture on the rest ──────────────── if not spec.is_sparse_mlp(layer): # Dense FFN (first_k_dense_replace). gate+up then down over the wide # intermediate. Canonical dense-graph names so residuals stay comparable. inter = spec.intermediate_size - add_act_quant("act_quant_mlp", h, "mlp_gate_up") + add_act_quant("act_quant", h, "mlp_gate_up") + # SwiGLU stays inside ``mlp_gate_up``: ``silu_and_mul`` is already one of + # that op's needles in ``deviation._OP_RULES``, so a separate node would + # be a prediction the pairing has no way to receive. f_gu, b_gu = _linear(rows, h, 2 * inter // tp, aw, w_bytes("mlp_gate_up", wd)) - add("mlp_gate_up", f_gu, b_gu, wd) - add_pointwise("mlp_silu", inter / tp, ops=4.0) + f_act, b_act = _pointwise(rows, inter / tp, aw, ops=4.0) + add("mlp_gate_up", f_gu + f_act, b_gu + b_act, wd) f_d, b_d = _linear(rows, inter // tp, h, aw, w_bytes("mlp_down", wd)) add("mlp_down", f_d, b_d, wd) _emit_collective(g, spec, hw, layer, "tp_all_reduce_mlp", rows, sh, prefix) - add_pointwise("mlp_residual", h) return # Router is replicated: every rank scores every expert to know what to keep. @@ -730,23 +760,28 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: f, b = _linear(rows, h, spec.n_routed_experts, aw, w_bytes("moe_router", wd)) add("moe_router", f, b, wd) - # Sigmoid scoring and the noaux_tc bias correction — a plain pointwise pass - # over the [rows, 256] score matrix. - add_pointwise("moe_sigmoid_bias", spec.n_routed_experts, ops=3.0) - - # The top-8 selection and renorm. Almost no bytes and almost no arithmetic — - # but it is the **only data-dependent shape in the step**, and therefore the - # node that decides whether the step can be CUDA-graph captured at all. Split - # from the sigmoid above for exactly that reason: they are the same size and - # the same cost, and only one of them is a hazard. + # Sigmoid scoring, the noaux_tc bias correction, the top-8 selection and the + # renorm — vLLM's fused gating kernel (``topk_softmax`` / + # ``moe_align_block_size``), a second launch after the router GEMM above. + # + # It carries the same op name on purpose. Both kernels classify to + # ``moe_router`` in ``deviation._OP_RULES``, which is a decision the + # dense-MoE and hybrid families already depend on; giving this one a private + # name would emit a node no capture can pair against and strand the gating + # kernel on the GEMM's row. Two instances of one op, distinguished by issue + # order and — where NVTX ranges exist — by identity. + # + # Kept as its own node regardless of the name it shares: it is the **only + # data-dependent shape in the step**, and therefore the node that decides + # whether the step can be CUDA-graph captured at all. add( - "moe_topk", - 0.0, - rows * (spec.n_routed_experts + 2.0 * spec.top_k) * aw, + "moe_router", + 3.0 * rows * spec.n_routed_experts, + rows * (2.0 * spec.n_routed_experts + 2.0 * spec.top_k) * aw, spec.act_dtype, ) - add_act_quant("act_quant_moe", h, "moe_routed") + add_act_quant("act_quant", h, "moe_routed") inter = spec.moe_intermediate_size per_expert_weights = 3.0 * h * inter @@ -782,18 +817,19 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: int(rows), spec.n_routed_experts, spec.num_experts_per_tok ) skew = sh.ep_imbalance if sh.ep > 1 else 1.0 + # SwiGLU is inside this node, not beside it: ``silu_and_mul`` is one of + # ``mlp_gate_up``'s needles in ``deviation._OP_RULES``, so a separate SwiGLU + # node would be a prediction with no observed kernel to pair against. + swiglu_f, swiglu_b = _pointwise(rows, spec.top_k * inter / es, aw, ops=4.0) add( "moe_routed", - per_position_flops * rows * spec.num_experts_per_tok * skew / es, + per_position_flops * rows * spec.num_experts_per_tok * skew / es + swiglu_f, per_expert_weights * distinct * ew * skew / es - + aw * (rows * inter * 2 * spec.num_experts_per_tok / es), + + aw * (rows * inter * 2 * spec.num_experts_per_tok / es) + + swiglu_b, ed, ) - # SwiGLU between the gate/up and down grouped GEMMs. Its own kernel in every - # grouped-GEMM backend this targets, over the expanded ``rows x top_k`` tensor. - add_pointwise("moe_silu", spec.top_k * inter / es, ops=4.0) - # Weighted scatter-add back to ``rows x hidden``, including the # routed_scaling_factor multiply. add( @@ -807,7 +843,6 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: g, spec, hw, layer, "tp_all_reduce_mlp", rows, sh, prefix, dispatches_experts=True, ) - add_pointwise("mlp_residual", h) def _emit_collective( @@ -948,11 +983,11 @@ def predict_glm_graph( # row in the step — the same count lm_head uses, and at prefill that is one # row per prompt rather than the whole chunk. logit_rows = float(batch.logits_rows) - f_n, b_n = _pointwise(logit_rows, spec.hidden, aw, ops=3.0) + f_n, b_n = _pointwise(logit_rows, 2.0 * spec.hidden, aw, ops=3.0) g.nodes.append( PredictedNode( - "final_norm", None, - roofline("final_norm", f_n, b_n, hw, spec.act_dtype, serial_launches=1), + "rms_norm", None, + roofline("rms_norm", f_n, b_n, hw, spec.act_dtype, serial_launches=1), ) ) @@ -1026,8 +1061,8 @@ def add_lm_head(rows: float, layer: int | None) -> None: f_n2, b_n2 = _pointwise(draft_batch.batch, 2.0 * spec.hidden, aw, ops=3.0) g.nodes.append( PredictedNode( - "mtp_norms", spec.n_layers + stage, - roofline("mtp_norms", f_n2, b_n2, hw, spec.act_dtype, + "rms_norm", spec.n_layers + stage, + roofline("rms_norm", f_n2, b_n2, hw, spec.act_dtype, serial_launches=2), ) ) diff --git a/tests/test_deviation_alignment.py b/tests/test_deviation_alignment.py index b884cee..6243710 100644 --- a/tests/test_deviation_alignment.py +++ b/tests/test_deviation_alignment.py @@ -38,7 +38,12 @@ def test_classify_op(): assert classify_op("triton_qkv_proj_gemm") == "qkv_proj" assert classify_op("cutlass_down_proj_kernel") == "mlp_down" assert classify_op("lm_head_logits") == "lm_head" - assert classify_op("triton_rms_norm") is None # not a modeled op + # Modelled since the GLM-5.2 graph began emitting the pointwise work: on a + # sparse model at low batch the norms are most of the launches, and a step + # bounded by its launches cannot be explained by a graph of GEMMs alone. + # All three norm sites in a block share this op — they are one kernel name, + # and only an NVTX range can say which site a launch belongs to. + assert classify_op("triton_rms_norm") == "rms_norm" def test_classify_op_matches_real_vllm_kernel_names(): diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index d9f96be..1521631 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -129,7 +129,12 @@ def test_dense_prefix_runs_ffn_not_mixture(): # Exactly the 3 dense layers carry an mlp_gate_up/down; the router runs on the # 75 sparse layers plus the sparse MTP head. assert len(_ops(g, "mlp_gate_up")) == 3 - assert len(_ops(g, "moe_router")) == spec.n_sparse_mlp_layers + spec.num_nextn_predict_layers + # Two per sparse block: the h->256 GEMM, then the fused gating kernel that + # scores and selects. Same op name because the trace cannot tell them apart + # (see MOE_LAYER_NODES), so the count is doubled, not the node list. + assert len(_ops(g, "moe_router")) == 2 * ( + spec.n_sparse_mlp_layers + spec.num_nextn_predict_layers + ) def test_precision_is_bf16_no_fp4_leak(): @@ -410,13 +415,13 @@ def test_a_pure_prefill_step_runs_no_draft_head(): #: pointwise work into the GEMM it precedes reports a decode step as memory-bound #: when it is launch-bound. MOE_LAYER_NODES = ( - "input_layernorm", "act_quant_attn", + "rms_norm", "act_quant", "attn_q_a", "attn_q_b", "attn_kv_a", "attn_kv_b", "attn_score_value", "attn_qnorm_rope_insert", "attn_out_proj", - "tp_all_reduce_attn", "attn_residual", "post_attention_layernorm", - "moe_router", "moe_sigmoid_bias", "moe_topk", "act_quant_moe", - "moe_shared", "moe_permute", "moe_routed", "moe_silu", "moe_combine", - "moe_all_to_all", "tp_all_reduce_mlp", "mlp_residual", + "tp_all_reduce_attn", "rms_norm", + "moe_router", "moe_router", "act_quant", + "moe_shared", "moe_permute", "moe_routed", "moe_combine", + "moe_all_to_all", "tp_all_reduce_mlp", ) @@ -439,8 +444,35 @@ def test_layer_lowers_to_the_documented_node_sequence(): # A dense layer swaps the whole mixture for three nodes, and so is the only # block in the model with no data-dependent shape and no expert traffic. dense = tuple(n.op for n in g.nodes if n.layer == 0) # Ld,f - assert {"act_quant_mlp", "mlp_gate_up", "mlp_silu", "mlp_down"} <= set(dense) - assert "moe_topk" not in dense and "moe_all_to_all" not in dense + assert {"act_quant", "mlp_gate_up", "mlp_down"} <= set(dense) + assert "moe_router" not in dense and "moe_all_to_all" not in dense + + +def test_every_emitted_op_name_resolves_from_a_kernel_name(): + """A node the pairing cannot receive is a prediction that never gets checked. + + ``classify_op`` is the fallback identity for a capture with no NVTX ranges + (``docs/kernel_identity.md``), and it matches on the kernel name. An op this + graph emits that no kernel name can classify to would sit in the predicted + graph permanently unmatched while the real kernel landed as unmodeled — two + errors in opposite directions, and the per-op residual diff this whole family + exists to support would be quietly decorative. + + Collectives are the documented exception: NCCL kernel names carry no hint of + *which* of a layer's two all-reduces they are, so they are matched by the + coarse taxonomy rather than by op. + """ + from gitm.optimizer.deviation import _OP_RULES + + g = predict_glm_graph( + load_spec("glm-5.2-fp8"), + batch=BatchConfig(batch=32, kv_cache_len=8192, speculative_tokens=2), + sharding=ShardingConfig(tp=8, ep=8), + ) + collectives = {"tp_all_reduce_attn", "tp_all_reduce_mlp", "moe_all_to_all", + "logits_all_gather"} + emitted = {n.op for n in g.nodes} - collectives + assert emitted <= set(_OP_RULES), sorted(emitted - set(_OP_RULES)) def test_prologue_and_epilogue_are_nodes(): @@ -455,7 +487,7 @@ def test_prologue_and_epilogue_are_nodes(): sharding=ShardingConfig(tp=8), ) ends = [n.op for n in g.nodes if n.layer is None] - assert ends == ["embed_tokens", "final_norm", "lm_head", "logits_all_gather"] + assert ends == ["embed_tokens", "rms_norm", "lm_head", "logits_all_gather"] # Without TP there is nothing to gather. solo = predict_glm_graph(spec, batch=BatchConfig(batch=32, kv_cache_len=8192)) assert "logits_all_gather" not in [n.op for n in solo.nodes] @@ -475,10 +507,11 @@ def test_act_quant_exists_only_where_a_gemm_is_actually_fp8(): op_dtype_overrides=(("lm_head", "bf16"), ("moe_router", "fp32")), ) batch = BatchConfig(batch=32, kv_cache_len=8192) - assert not _ops(predict_glm_graph(bf16, batch=batch), "act_quant_attn") - assert len(_ops(predict_glm_graph(fp8, batch=batch), "act_quant_attn")) == ( - fp8.n_layers + fp8.num_nextn_predict_layers - ) + assert not _ops(predict_glm_graph(bf16, batch=batch), "act_quant") + # Two per block: one ahead of the attention GEMMs, one ahead of the FFN — + # the two groups of fp8 GEMMs, with bf16 work in between. + blocks = fp8.n_layers + fp8.num_nextn_predict_layers + assert len(_ops(predict_glm_graph(fp8, batch=batch), "act_quant")) == 2 * blocks def test_detect_family_routes_glm_before_sparse_moe(): From 62a47580d62082cc9546f9384126a8ecba4a0a5e Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 14:32:36 -0700 Subject: [PATCH 08/35] GLM-5.2: fix a truncated schedule, dedupe the catalogue, prune prose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rigorous sweep for stale artifacts from the earlier attempts, plus a validation pass against the published config and the transformers docs. The real find: glm-5.2.yaml's indexer_types had 78 layers' worth of evidence in 77 entries. Layer 77 fell through to the modulo fallback and landed on the right answer, so nothing failed and the count still came out 21 -- a plausible total resting on evidence that is not there, which is the exact failure explicit schedules exist to prevent. Restored from the published array; the catalogue loader now validates schedule length the way spec_from_hf_config already did, and a test pins it. Related: the FP8 entry carried its own 78-entry copy of both schedules -- 158 lines of duplicated evidence, in the file where the drift happened. It now `extends: glm-5.2` (new, in model_catalogue), so the two entries differ only where the checkpoints do: four dtypes. provenance is deliberately not inherited; each checkpoint is validated against its own published size. Validated against HuggingFace, and two corrections fall out: - transformers documents indexer_types "shared" as reusing *the previous full layer's* top-k -- what the graph does, now quoted rather than asserted. - Z.ai publishes 744B params; the checkpoint is 753.3B by its own bytes. The gap is the MTP block, and this graph predicts 744.2B with the draft removed (0.03%). That puts the MTP block at 9.9B against 0.23B for a dense draft head -- confirming from a number published for another reason what the weight map already said: the draft is a full MoE block. - IndexShare's published figure is 2.9x whole-model per-token FLOPs at 1M; the 3.7x used here is the indexer's own ratio. Both now stated. Pruning: module docstring 80 -> 48 lines (it restated the note), the long comment blocks trimmed to the hazard they guard, glm_graph 1261 -> 1169. Doc: §2.3, §7.3 and §9 cut where they repeated the header or the hardware section; four stale op names fixed. Comment density unchanged at 17%, matching the sibling families. No behaviour change: the decode floor is 16.551 ms before and after. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 183 +++++++-------- gitm/planner/glm_graph.py | 332 ++++++++++----------------- gitm/planner/model_catalogue.py | 39 +++- gitm/planner/models/glm-5.2-fp8.yaml | 289 ++++------------------- gitm/planner/models/glm-5.2.yaml | 1 + tests/test_glm_graph.py | 54 ++++- 6 files changed, 328 insertions(+), 570 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index f9f7340..173cae7 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -70,7 +70,7 @@ the planner change that made it representable. | MTP | 1 module, `index_share_for_mtp_iteration: true`, **carries a full MoE** | | Vocab | 154,880, untied `lm_head` | | Max context | 1,048,576 | -| Total / active params | ~754 B / ~39 B | +| Total / active params | **744 B** published + a **9.9 B** MTP block / ~39 B active | ``` L: 0 1 2 3 6 10 14 ... 74 77 @@ -90,13 +90,21 @@ IndexShare period is 4. No single modulo rule reproduces either (§7, G3). `config.json`: - `indexer_types[i] == "full"` → the layer **computes** its own top-2048 selection -- `indexer_types[i] == "shared"` → the layer **reuses** the group's selection +- `indexer_types[i] == "shared"` → the layer **reuses the previous `full` layer's + top-k** — the semantics `transformers` documents for the field, verbatim - `mlp_layer_types[i]` → `"dense"` | `"sparse"`, and it agrees with `first_k_dense_replace: 3` - `moe_router_dtype: "float32"` → the router is fp32 **on every variant**, because this is a field of the base config and not of any quantisation config -**This is proven from the weight map, not inferred.** Indexer tensors +IndexShare is a published mechanism ([paper 2603.12201](https://huggingface.co/papers/2603.12201)): +Z.ai reports it "reuses the same indexer across every four sparse attention +layers, reducing per-token FLOPs by **2.9× at a 1M context length**". That 2.9× is +a whole-model figure; the **3.7×** used throughout this note is the *indexer's +own* ratio (78 layers ÷ 21), which is the one that matters when ranking the +indexer against the MoE term. Two denominators, both correct. + +**The schedule is proven from the weight map, not inferred.** Indexer tensors (`*.indexer.wq_b`, `.wk`, `.weights_proj`, `.k_norm`) exist on exactly the 21 `full` layers, on **none** of the 57 `shared` layers, and on **none** of the MTP module. A shared layer that recomputed the index would need those weights; it does @@ -150,6 +158,15 @@ stronger check than either alone: an error in the shape arithmetic would have to be precision-proportional to survive both. A 2:4-sparsity-compressed checkpoint would be roughly half the fp8 size. It is not — which is the evidence behind A2. +**And a third check falls out of the gap between them.** Z.ai publishes **744 B** +parameters; the bf16 checkpoint is 753.3 B by its own byte count. The difference +is the MTP block, which the published figure excludes — and this graph predicts +**744.2 B with the draft head removed**, a 0.03 % match. That is not just another +size check: it says the MTP block is **9.9 B**, and a *dense* draft head at +`intermediate_size` would be **0.23 B**. The draft carrying a full 256-expert +mixture (§2.4) is therefore confirmed twice over — once from the weight map, once +from arithmetic on a number the vendor published for another reason. + ### KV cache — the number that drives decode ``` @@ -305,28 +322,23 @@ vocabulary weights *every step* rather than once per request. ### 2.3 Encoders — there are none, and the absence is worth stating -GLM-5.2 is a **text-only** decoder. There is no vision tower, no audio encoder, no -patch embedding, no multimodal scatter into `inputs_embeds`. The design template -this note follows devotes two sections to encoder cost; here they collapse to a -single fact, and it changes three things downstream: +GLM-5.2 is a **text-only** decoder: no vision tower, no audio encoder, no +multimodal scatter into `inputs_embeds`. Three consequences, because the absence +is load-bearing: -1. **There is no encoder→backbone seam**, so the one unavoidable serial dependency - that dominates a multimodal prefill does not exist. Prefill starts at - `embed_tokens`. +1. **Prefill starts at `embed_tokens`.** The encoder→backbone seam that dominates + a multimodal prefill — a strict serial prefix nothing can overlap — does not + exist here. 2. **Prompt length is the only input-side variable.** No image or video token - count feeds P, so the flip-variable index (§4.4) is one row shorter than a - multimodal model's and the remaining rows are correspondingly better - constrained. -3. **No second model is hiding off-checkpoint.** A multimodal note has to carry an - open question for an external audio codec it cannot see. Here the checkpoint is - the whole model, so every FLOP in the trace should map to a node in §3 — which - makes an unexplained kernel block a much stronger signal than it would be - elsewhere (§6.4 row 6). - -The GLM family does ship vision variants (GLM-4.5V and successors). **They are a -different checkpoint with a different `model_type` and this graph does not model -them** — `is_glm_moe_dsa_config` would decline them rather than price a tower it -never read. + count feeds P, so §4.4's flip-variable index is one row shorter and the + remaining rows are correspondingly better constrained. +3. **No second model is hiding off-checkpoint.** Every FLOP in a trace should map + to a node in §3, which makes an unexplained kernel block a much stronger + signal than it would be elsewhere (§6.4 row 6). + +The GLM family does ship vision variants. **They are a different checkpoint with a +different `model_type` and this graph does not model them** — +`is_glm_moe_dsa_config` declines them rather than pricing a tower it never read. ### 2.4 MTP-on decode — draft and verify @@ -393,7 +405,7 @@ two of them are on the critical path between the last layer and the sample. | id | operator | kernel class | shape | FLOPs | bytes | stream | bound | | --- | ------------------- | ------------------------- | ------------------------------ | -------- | ------------- | ------ | ------ | | D0 | `embed_tokens` | gather (index_select) | `[32] → [32,6144]` | 0 F | 0.393 MB | compute | launch | -| E0 | `final_norm` | fused RMSNorm | `[32,6144]`, **logits rows only** | 0.6 MF | 0.786 MB | compute | launch | +| E0 | `rms_norm` | fused RMSNorm | `[32,6144]`, **logits rows only** | 0.6 MF | 0.786 MB | compute | launch | | E1 | `lm_head` | GEMM (BF16), tall-skinny | `[32,6144] × [6144,19360]` | 7.61 GF | **239.5 MB** | compute | memory | | E2 | `logits_all_gather` | collective (all-gather) | `[32,19360] → [32,154880]` fp32 | 0 F | **17.3 MB** | comm | memory | @@ -518,7 +530,7 @@ that was never moving bytes. The sign of the MTP decision flips with batch. | # | Where | Kind | conf | Trace signature if real | | ------ | --------------------------- | ----------------------------------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------ | -| **S1** | after top-k (`moe_topk`) | host readback of the expert histogram to size the grouped GEMM | **low** | **76 D2H per decoded token** — fatal for graph capture | +| **S1** | after the gating kernel | host readback of the expert histogram to size the grouped GEMM | **low** | **76 D2H per decoded token** — fatal for graph capture | | S2 | around each all-reduce | stream-to-stream event wait | medium | 158 event pairs/step; at 688 kB **the gap *is* the cost** | | S3 | around each EP all-to-all | dispatch/combine barrier | medium | 76 more, and **none on the 3 dense layers**; an imbalanced rank stalls every other one | | S4 | sampling / detokenisation | D2H of sampled ids every step | **high** | one D2H + host round-trip per step; unavoidable, but its *placement* decides overlap | @@ -654,7 +666,7 @@ at TP8/EP8, FP8. | **Pre** | indexer scan ×21 | **compute** | 5.77 TF against 22 MB of keys — `O(P·C + P²/2)` × 32 heads, and **2.2 % of the step**. **The quadratic lives here, not in the core** | **BF16** arithmetic, fp8 keys — two dtypes, one node | P **and** C | | **Pre** | attention core ×79 | compute, **linear in C** | FLOPs capped at 2,048 keys/query; **bytes are the whole cache once per request** | FP8 KV | ⚑ **`index_topk`**; C. Not P² | | **Pre** | permute / combine | memory | 8.5 GB/layer-pass on the `8P`-row expanded tensor, near-zero FLOPs | BF16 activations | top-k; **expert imbalance** | -| **Pre** | *everything else* — embed gather, norms, `lm_head`, `moe_topk` | memory or launch | each under 1.5 %; `lm_head` reads 239.5 MB for **one row per request** | BF16; FP32 top-k | none of them flips | +| **Pre** | *everything else* — embed gather, norms, `lm_head`, gating | memory or launch | each under 1.5 %; `lm_head` reads 239.5 MB for **one row per request** | BF16; FP32 top-k | none of them flips | | **Pre** | **whole prefill pass** | **memory** | **AI 281 vs ridge 412**; 66 % memory / 34 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | | **MTP** | verify expert GEMMs | **memory** | 256 distinct experts at 192 rows vs 163 at 32 — **+57 %/layer, ~85 % of MTP's cost** | FP8 block-scaled | `B(1+D)` vs E=256; **D**; imbalance | | **MTP** | draft expert GEMMs ×D | **memory** | the MTP block carries a **full 256-expert bank**; 5 stages × 163 experts, **linear in D, no saturation** | FP8 block-scaled | **D**; batch | @@ -707,7 +719,7 @@ graph**, to be confirmed against a capture. | Rank | Region | Prediction | Why | Evidence to inspect | What would prove it wrong | |---|---|---|---|---|---| | **1** | **EP all-to-all at prefill** | **≥44 % of prefill time is wire, and roughly half of it is recoverable** | 105.7 GB/pass in BF16. An fp8 dispatch halves the payload outright; overlapping dispatch with the shared-expert GEMM hides more | NCCL kernel duration vs payload at prefill (§6.4 row 5); whether dispatch is bf16 | duration ≪ payload/900 GB/s → the engine already fuses or compresses it | -| **2** | **Grouped-GEMM group sizing (the fork)** | either **76 D2H per token** (no graph capture possible) **or** fixed-capacity padding (**all 256 experts read every step**) | The only data-dependent shape in the graph is `moe_topk`. A stack does one or the other — see §5.2 | `cuda_api_sum` D2H count per decode step | 0 D2H **and** MoE bytes that track `distinct_experts(B)` → a device-side path, nothing to recover | +| **2** | **Grouped-GEMM group sizing (the fork)** | either **76 D2H per token** (no graph capture possible) **or** fixed-capacity padding (**all 256 experts read every step**) | The only data-dependent shape in the graph is the fused gating kernel. A stack does one or the other — see §5.2 | `cuda_api_sum` D2H count per decode step | 0 D2H **and** MoE bytes that track `distinct_experts(B)` → a device-side path, nothing to recover | | **3** | ⚑ **CUDA-graph capture at low batch** | **63 % of the B=1 floor is launch overhead** (81 % at eager 5 µs) | 1,033 nodes × 2 µs = 2.07 ms against a 1.24 ms memory term | `cuda_api_sum` launch count with `--cuda-graph-trace=node`; expect **1** `cudaGraphLaunch` | already captured → this rank is worth nothing, and rank 2's fork is already resolved to "padded" | | **4** | **The indexer scan at long context** | at 1M it is **54 % of the step**, and IndexShare's 3.7× is already banked | 90.2 GB/step of index keys at S=1M. Levers: fp8 index keys (2×), temporal reuse across steps, a smaller `index_topk_freq` group | `dram__bytes_read.sum` on the scan vs 90.2 GB; whether keys are fp8 | keys already fp8 and no temporal reuse available → architectural | | **5** | **`moe_routed` — the memory-bound heart** | 73.8 % of decode; **EPLB placement and the grouped-GEMM backend are the levers** | Weight traffic scales with *distinct* experts woken, not with FLOPs. Good placement cuts per-rank distinct traffic; DeepGEMM cuts the constant | measured per-rank expert traffic vs `distinct_experts`; **the real EP imbalance** | measured traffic already at the union prediction with balance ≈1.0 → the bank is the bank | @@ -913,46 +925,28 @@ which is exactly what makes them survivable. **A prefill path that is wrong in a self-cancelling way is worse than no prefill path**, and it is the version that will get suggested — hence four helpers with four docstrings rather than one alias. -### 7.3 What this branch actually changed +### 7.3 What this branch changed, in kind -``` -gitm/planner/glm_graph.py op_dtype_overrides + dtype_for; four DSA phase helpers; - rows = positions + prefill_tokens throughout; - serial_launches on every node; the full pointwise - lowering -- rms_norm (residual fused in, as vLLM runs - it), act_quant gated on the GEMM being fp8, the fused - gating kernel, moe_permute / moe_combine, and the - embed_tokens / rms_norm / lm_head / logits_all_gather - that bracket the stack; _emit_collective x2 per layer, - EP dispatch only where a layer has experts; the D-stage - draft chain with per-stage lm_head and eh_proj; the - indexer's wk + weights_proj, its 32 head scores, and - its storage dtype held apart from its math dtype; - the quantisation-map reader -gitm/optimizer/deviation.py _OP_RULES entries for the ops the pointwise lowering - added, plus attn_index_proj (was shadowed by the scan), - attn_kv_b (present once MLA is modelled unabsorbed) and - mtp_eh_proj. Existing mappings left alone: the - gating->moe_router and silu_and_mul->mlp_gate_up - decisions are shared with the other families -gitm/planner/context.py _FP32_PEAKS + fp32_peak_for_sku, wired to hardware_spec_for -gitm/planner/registry.py node-owned bound labels, per-dtype ridges, launch count, - --spec-tokens -gitm/planner/model_catalogue.py nested tuple coercion for op_dtype_overrides -gitm/planner/models/ - glm-5.2.yaml the bf16 model fact + the fp32 router - glm-5.2-fp8.yaml NEW — the vendor's recommended deployment -tests/test_glm_graph.py 29 tests: the fp8 footprint, the precision map, the - prefill/decode byte inversion, the D-deep chain, two - collectives, the 32-head index scan, that a pure-prefill - step runs no draft head, that a launch floor does not - hide an unpriced collective, the exact node sequence a - layer lowers to, and that every op the graph emits - resolves from some kernel name -docs/glm-5.2/artifacts/*.json DELETED — 29k lines of node dumps that go stale on - every graph change; the commands at the top of this - note regenerate any of them -``` +Five things, in the order they re-rank the tables. `git log` has the file list. + +1. **Precision became per-op** (`op_dtype_overrides`), read from + `modules_to_not_convert` and `moe_router_dtype`. Everything downstream is + priced against it, and the FP8 catalogue entry became the one to plan against. +2. **Prefill exists**, with DSA's own asymptotics rather than a dense family's — + four helpers, because FLOPs and bytes stop moving together (G2, §7.2). +3. **A layer lowers to its kernels, not just its GEMMs** (G8): norms, activation + quantisation, the fused gating, the prologue and epilogue. Without them the + launch bound the roofline already supported had nothing to bind. +4. **Node names follow the pairing contract** (G9) rather than redefining it, so + the per-op residual diff in §6 can actually pair what the graph predicts. +5. **The MTP chain is D stages deep**, each with its own vocabulary projection + (G7), driven by `--spec-tokens`. + +Two supporting fixes outside the family: an fp32 peak for modern SKUs (G4), and +`gitm plan` keeping the launch bound and pricing the ridge per dtype (G5). + +**Deleted:** three committed JSON node dumps, 29k lines that went stale on every +graph change. The commands at the top of this note regenerate any of them. --- @@ -993,45 +987,24 @@ docs/glm-5.2/artifacts/*.json DELETED — 29k lines of node dumps that go sta ## 9. How to run it -**A. Predict-only (no GPU, free, works now):** - -```bash -pip install -e . -gitm plan glm-5.2-fp8 --gpu H200 --batch 32 --kv-len 8192 --tp 8 --ep 8 -gitm plan glm-5.2-fp8 --gpu B200 --batch 32 --kv-len 131072 --tp 8 --ep 8 --json -gitm plan --list -``` - -**B. Serve + capture (to get the numbers §6 wants validated).** The footprint math -decides the hardware: **fp8 → one 8×H200 node** (755.9 GB of weights against -1,128 GB, leaving ~370 GB for KV and activations); **bf16 → two nodes** (1,508 GB, -10.7 H200s for weights alone). Full 1M context wants B200/B300 for the extra HBM. - -1. On RunPod take an **8×H200 SXM** pod with a **network volume ≥ 1 TB** for the - 141-shard fp8 checkpoint. -2. Serve with the vendor's own recipe — reproduced here verbatim because §4's - constants assume it: - ```bash - vllm serve zai-org/GLM-5.2-FP8 \ - --kv-cache-dtype fp8 \ - --tensor-parallel-size 8 \ - --speculative-config.method mtp \ - --speculative-config.num_speculative_tokens 5 \ - --tool-call-parser glm47 --reasoning-parser glm45 --enable-auto-tool-choice - ``` - Add `--enable-expert-parallel` for the EP8 shape §4 prices; **without it the - graph's `moe_all_to_all` rows should not appear at all**, and rank 1 does not - exist. That difference is capture C5. -3. Attach the GitM collector and capture a bounded decode window: - ```bash - gitm capture serve # or: gitm capture attach - ``` -4. Import and diff observed-vs-predicted per op. **A residual here is a lead, not a - defect.** - -> ⚠ Take path A first. It is free, and it answers the two questions that gate -> everything else — does the fp8 shape fit (yes, 755.9 GB on 1,128 GB), and is the -> step launch-bound at your batch (yes, below B≈16). +**Predict-only — free, no GPU, and it answers the two questions that gate +everything else.** Does the fp8 shape fit (yes: 755.9 GB of weights on 1,128 GB), +and is the step launch-bound at your batch (yes, below B≈16). The commands are at +the top of this note. + +**Serve and capture.** The footprint decides the hardware: **fp8 → one 8×H200 +node**, leaving ~370 GB for KV and activations; **bf16 → two nodes** (10.7 H200s +for weights alone). Full 1M context wants B200/B300 for the extra HBM — 55 GB of +KV per rank on top of a 96 GB weight share. + +1. An **8×H200 SXM** pod with a **network volume ≥ 1 TB** for the 141-shard fp8 + checkpoint. +2. Serve with the vendor recipe quoted in the hardware section — §4's constants + assume it. Add `--enable-expert-parallel` for the EP8 shape §4 prices; + **without it the `moe_all_to_all` rows should not appear at all**, and rank 1 + does not exist. That difference is capture C5. +3. `gitm capture serve` (or `gitm capture attach`) for a bounded decode window. +4. Diff observed-vs-predicted per op. **A residual is a lead, not a defect.** --- diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index c3196fa..4c23a9e 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -1,83 +1,58 @@ -"""Predicted execution graph for a GLM-5.2-class (``glm_moe_dsa``) decode step. +"""Predicted execution graph for a GLM-5.2-class (``glm_moe_dsa``) engine step. -A fork of :mod:`gitm.planner.moe_graph`, specialised for ZhipuAI's +A fork of :mod:`gitm.planner.moe_graph`, specialised for Z.ai's ``GlmMoeDsaForCausalLM``. Both families are sparse-MoE with a lightning indexer, but the attention differs in kind, not degree, so a shared spec would carry a -field for each that is dead on the other — the exact "a default silently -activates" hazard the roofline module warns about. The two graphs therefore live -apart, and share only the canonical op names (so residuals stay comparable) and -the :func:`~gitm.planner.roofline.distinct_experts` union term (one owner, no -drift). - -What GLM-5.2 is, read from ``config.json`` and the checkpoint's own tensor index -(``model.safetensors.index.json``), never from a trace: - -**Attention is MLA + DeepSeek Sparse Attention, with no per-layer compression.** -Unlike DeepSeek-V4's CSA/HCA schedule (``compress_ratios``), every GLM layer runs -the *same* attention: a compressed KV latent (``kv_lora_rank=512``, shared across -all 64 query heads) plus a lightning indexer that scores the whole history and -keeps ``index_topk=2048`` positions for the core. There is no ``m``/``m'`` split, -no HCA, no sliding window. So the V4 fields that encode that schedule are simply -absent here. - -**IndexShare — the mechanism this fork exists to price.** The checkpoint declares -``indexer_types`` per layer: ``full`` or ``shared``. A ``full`` layer computes its -own indexer (projection + score) and selects the top-k; the next three ``shared`` -layers *reuse that selection* and run no indexer at all. This is not an inference: -the ``shared`` layers physically **carry no indexer tensors** in the weight map. -Only 21 of 78 layers run the indexer; 57 skip it. ``index_topk_freq=4`` is the -period of that grouping (one full + three shared), and ``index_skip_topk_offset`` -its offset. Pricing every layer's indexer at full rate — as a naive reading of -``index_topk`` would — overstates the indexer's share of the step roughly 4x and -mis-ranks it against the MoE weight traffic that actually dominates decode. - -**The MLP schedule is dense-then-sparse.** ``first_k_dense_replace=3``: the first -three layers run a conventional dense FFN (``intermediate_size=12288``) with no -router and no experts; the remaining 75 are MoE (256 routed experts, top-8, one -shared, ``moe_intermediate_size=2048``). Modelling the dense layers as MoE would -invent a router GEMM and expert traffic that the weight map shows are not there. - -**Three precisions in one block, and which op runs in which is read from the -checkpoint.** ``zai-org/GLM-5.2`` is bf16 (1.507 TB on disk, no -``quantization_config``); ``zai-org/GLM-5.2-FP8`` is the vendor's *recommended* -deployment (753.33 GB, e4m3, 128x128 block-scaled) and its -``quantization_config.modules_to_not_convert`` names what stays wide. On the FP8 -checkpoint the backbone GEMMs — ``q_a``/``q_b``/``kv_a``/``kv_b``, **``o_proj``**, -the dense FFN, the shared expert and all 256 routed experts — are fp8, while -``lm_head``, ``embed_tokens``, the MTP ``eh_proj``, every norm and — the outlier -worth naming — the **lightning indexer's projections** are bf16. The router is -fp32 on both checkpoints (``moe_router_dtype: "float32"``, a field of the *base* -config, so it is a model fact and not a quantisation choice). - -That is three widths inside one attention block, which one ``weight_dtype`` per -spec cannot say. :attr:`GlmMoeDsaModelSpec.op_dtype_overrides` says it per op, and -it is load-bearing in both directions: pricing ``lm_head`` at 1 byte/weight -understates 154,880 x 6,144 of real traffic, and pricing the indexer at fp8 -understates the one node whose cost grows with context. Note the inversion against -the fp8-backbone models this planner has seen before — here ``o_proj`` is *inside* -the quantised set and the *indexer* is outside it. +field for each that is dead on the other. The two graphs live apart and share +only the canonical op names (so residuals stay comparable) and +:func:`~gitm.planner.roofline.distinct_experts` (one owner, no drift). + +Shape and provenance are in ``docs/glm-5.2/DESIGN-NOTE.md`` and in the catalogue +entries' ``provenance`` blocks; this docstring carries only what constrains an +edit to *this file*. + +Four properties do the work, and each has a plausible wrong reading: + +**MLA + DSA on every layer, no compression schedule.** One KV latent +(``kv_lora_rank``) shared across all query heads. Deriving cache traffic from +``num_key_value_heads * head_dim`` is the classic MLA error and overstates it 50x +on GLM-5.2 — the config's ``num_key_value_heads: 64`` is a red herring. + +**IndexShare.** ``indexer_types`` is ``full`` | ``shared`` per layer; a ``shared`` +layer reuses the previous ``full`` layer's top-k and physically carries no indexer +tensors. Emitting an indexer on all 78 layers overstates it ~3.7x. + +**Dense-then-sparse MLP.** ``first_k_dense_replace`` leading layers have no router +and no experts. Modelling them as MoE invents traffic the weight map denies. + +**Three precisions in one block**, read from +``quantization_config.modules_to_not_convert`` and ``moe_router_dtype`` rather +than assumed. On GLM-5.2-FP8 the backbone GEMMs *including* ``o_proj`` are fp8 +while the *indexer*, ``lm_head`` and the MTP ``eh_proj`` are not — the inversion +of the usual fp8-backbone layout. :attr:`GlmMoeDsaModelSpec.op_dtype_overrides` +carries it; one ``weight_dtype`` cannot. Known limits, stated rather than hidden: -* **Prefill is modelled, and it is not decode with a bigger M.** DSA makes the two - phases disagree about what ``index_topk`` buys: at decode the core reads at most - ``index_topk`` cached entries, so attention is flat in context; at prefill every - query in the chunk selects a *different* top-k, so their union is the whole - history and the core streams the entire cache once per request. Top-k bounds - prefill FLOPs, not prefill bytes. See :func:`core_qk_pairs` / - :func:`core_read_entries`. -* **Uniform routing.** :func:`~gitm.planner.roofline.distinct_experts` assumes a - balanced router; real skew touches fewer distinct experts, moving *less* traffic - than predicted — the conservative direction. -* **Expert-parallel skew is calibrated, not predicted.** - :attr:`ShardingConfig.ep_imbalance` stays at 1.0 until a trace measures it. -* **Collectives are bandwidth-plus-launch**: a ring latency floor of one launch - each, which is what a 262 kB decode all-reduce is actually bounded by. Still - flagged ``estimated``, and still reported unpriced when the SKU carries no - interconnect bandwidth. -* **The MTP economics are a prediction, not a measurement.** Acceptance rate is a - serving observable; the graph prices the *cost* of D drafts and a 1+D verify and - leaves the payoff to :attr:`BatchConfig.tokens_per_step`. +* **Prefill is not decode with a bigger M.** ``index_topk`` bounds the core's + FLOPs in both phases and its *bytes* in neither: at prefill every query selects + a different top-k and their union is the whole cache. See :func:`core_qk_pairs` + and :func:`core_read_entries` — reusing ``BatchConfig.attention_qk_pairs`` here + is wrong in both directions at once, and the errors partly cancel. +* **Uniform routing.** ``distinct_experts`` assumes a balanced router; real skew + touches fewer experts, moving *less* traffic — the conservative direction. +* **``ep_imbalance`` is calibrated, not predicted.** It stays 1.0 until a trace + measures it. +* **Collectives are bandwidth-plus-one-launch**, flagged ``estimated``, and still + reported unpriced when the SKU carries no interconnect bandwidth. +* **MTP cost is predicted; acceptance is not.** The graph prices D drafts and a + 1+D verify and leaves the payoff to :attr:`BatchConfig.tokens_per_step`. +* **Node names are constrained by the pairing contract.** Every op emitted here + must be classifiable by :func:`gitm.optimizer.deviation.classify_op`, or the + predicted node goes permanently unmatched while the real kernel files as + unmodeled. That is why three norm sites share one ``rms_norm`` op and why + SwiGLU lives inside the GEMM that owns its needle. See + ``docs/kernel_identity.md``. """ from __future__ import annotations @@ -174,21 +149,17 @@ class GlmMoeDsaModelSpec: moe_intermediate_size: int = 256 #: Dense-FFN width for the leading ``first_k_dense_replace`` layers. intermediate_size: int = 768 - #: Leading layers that run a dense FFN instead of the mixture (GLM - #: ``first_k_dense_replace``). Those layers carry no router and no experts. + #: Leading layers with a dense FFN: no router, no experts. first_k_dense_replace: int = 1 - #: Per-layer ``dense`` | ``sparse`` when the checkpoint declares it; overrides - #: :attr:`first_k_dense_replace` if both are given. + #: Per-layer ``dense`` | ``sparse``; overrides :attr:`first_k_dense_replace`. mlp_layer_types: tuple[str, ...] = () - #: Scales routed-expert outputs (GLM ``routed_scaling_factor``). Numerics only — - #: no effect on the FLOP/byte roofline, carried for completeness. + #: Numerics only — no effect on the FLOP/byte roofline. routed_scaling_factor: float = 1.0 # ── multi-token prediction ─────────────────────────────────────────────── num_nextn_predict_layers: int = 0 - #: The MTP layer reuses the main model's index rather than recomputing it - #: (GLM ``index_share_for_mtp_iteration``). The tensor exists but the iteration - #: shares — carried as a headroom lever, not banked into the floor. + #: The MTP block reuses the main model's index instead of recomputing it, + #: and the weight map agrees: it carries no indexer tensors. index_share_for_mtp_iteration: bool = True # ── precision ──────────────────────────────────────────────────────────── @@ -196,17 +167,9 @@ class GlmMoeDsaModelSpec: expert_dtype: str = "bf16" kv_dtype: str = "bf16" act_dtype: str = "bf16" - #: Per-op precision, for the ops that do not run in :attr:`weight_dtype`. - #: ``(op_name, dtype)`` pairs — a tuple rather than a mapping so the spec - #: stays hashable, matching how the per-layer schedules are carried. - #: - #: One dtype per model is a fiction on this checkpoint. GLM-5.2-FP8 quantises - #: the backbone GEMMs and the experts to e4m3 but leaves ``lm_head``, the MTP - #: ``eh_proj`` and the **lightning indexer** in bf16 - #: (``quantization_config.modules_to_not_convert``), and the router is fp32 on - #: every variant (``moe_router_dtype``). Without this the indexer — the one - #: attention node whose cost grows with context — is priced at half its real - #: weight traffic and against a peak it never sees. + #: ``(op_name, dtype)`` for the ops that do not run in :attr:`weight_dtype`. + #: A tuple, not a mapping, so the spec stays hashable. Read from the + #: checkpoint, never assumed — see the module docstring. op_dtype_overrides: tuple[tuple[str, str], ...] = () # ── derived shapes / schedule ──────────────────────────────────────────── @@ -509,25 +472,15 @@ def _emit_layer( ) -> None: """Append one transformer layer's predicted nodes to ``g``. - ``batch`` is the phase this layer runs in, already adjusted by the caller: a - decode step, a chunked-prefill step, or a draft stage (prefill stripped). The - node *set* is identical across all three — what changes is the class of four - of them, which is why phase is a parameter here rather than a second emitter. - - Three row counts, and conflating any two is a real error: - - ``rows`` - Sequence positions this layer computes — decode positions (one per - sequence per speculative slot) plus whatever prefill chunk rides along. - Every projection and every FFN scales with this. - ``batch.batch`` - Distinct KV caches those rows read from. Cache traffic is charged per - *sequence*, not per row, which is why verifying 1+D drafted positions in - one step reads the cache once — and is what makes MTP pay at all on a - memory-bound step. - ``batch.prefill_requests`` - Prompts the prefill tokens belong to — the denominator for anything read - once per request rather than once per token. + ``batch`` is the phase, already adjusted by the caller: decode, chunked + prefill, or a draft stage (prefill stripped). The node *set* is identical + across all three; only the class of four of them changes. + + Three row counts, and conflating any two is a real error: ``rows`` (positions + computed — every projection scales with it), ``batch.batch`` (distinct KV + caches read — charged per *sequence*, which is what makes a 1+D verify pay), + and ``batch.prefill_requests`` (the denominator for anything read once per + request). """ h = spec.hidden aw = weight_bytes(spec.act_dtype) @@ -626,13 +579,10 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: f, b = _linear(rows, h, spec.kv_entry_dim, aw, w_bytes("attn_kv_a", wd)) add("attn_kv_a", f, b + rows * kv_entry_bytes(spec), wd) - # KV up-projection: reconstruct per-head K_nope and V from the cached latent - # (W^UK, W^UV). Modelled *unabsorbed* — it runs as its own GEMM and the output - # projection stays narrow (n_heads * v_head_dim). A serving engine that absorbs - # MLA folds W^UK into the query and W^UV into attn_out_proj instead, dropping - # this node and doubling attn_out_proj's input width; that is a serving-path - # variant, flagged in the catalogue provenance. Kept as a node because the - # weight physically exists in the checkpoint and model_weight_bytes counts it. + # Reconstruct per-head K_nope and V from the cached latent (W^UK, W^UV), + # modelled *unabsorbed*: its own GEMM, and attn_out_proj stays narrow. An + # engine that absorbs MLA drops this node and doubles attn_out_proj's input + # width instead — a serving variant, flagged in the catalogue provenance. f, b = _linear( rows, spec.kv_lora_rank, spec.n_heads * (spec.qk_nope_head_dim + spec.v_head_dim) // tp, aw, @@ -668,15 +618,10 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: wd, ) - # The scan's two dtypes are different questions and this node answers - # both. Its **bytes** are governed by how the index keys are *stored* - # (``kv_dtype`` — fp8 under the vendor recipe's fp8 cache); its **FLOPs** - # are governed by what the indexer *computes* in, and the indexer is one - # of the modules the quantiser skipped, so that is the projection's dtype. - # One field cannot do both: pricing 2.1 GF of bf16 arithmetic against the - # fp8 peak halves it, which is invisible at decode (the node is - # memory-bound at every context) and doubles the prefill row, where it is - # compute-bound. + # Two dtypes, two questions: the *bytes* follow how the keys are stored + # (``kv_dtype``), the *FLOPs* follow what the indexer computes in — and the + # indexer is one of the modules the quantiser skipped. Invisible at decode + # (memory-bound at every context); doubles the prefill row. add( "attn_index_score", # Every one of the 32 index heads scores each candidate against the @@ -724,11 +669,9 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: spec.act_dtype, ) - # Output projection: plain dense from the per-head value space back to hidden. - # No o_lora/o_groups here (unlike V4), so no ``estimated`` grouping guess. On - # the FP8 checkpoint this one *is* quantised — absent from - # ``modules_to_not_convert`` — the opposite of the fp8-backbone checkpoints - # that keep o_proj wide. + # Per-head value space back to hidden. No o_lora/o_groups here (unlike V4). + # On the FP8 checkpoint this one *is* quantised — the opposite of the + # fp8-backbone checkpoints that keep o_proj wide. f, b = _linear( rows, spec.n_heads * spec.v_head_dim // tp, h, aw, w_bytes("attn_out_proj", wd) ) @@ -760,20 +703,13 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: f, b = _linear(rows, h, spec.n_routed_experts, aw, w_bytes("moe_router", wd)) add("moe_router", f, b, wd) - # Sigmoid scoring, the noaux_tc bias correction, the top-8 selection and the - # renorm — vLLM's fused gating kernel (``topk_softmax`` / - # ``moe_align_block_size``), a second launch after the router GEMM above. + # vLLM's fused gating kernel (sigmoid + noaux_tc bias + top-8 + renorm): a + # second launch after the router GEMM, sharing its op name because both + # classify to ``moe_router`` — a mapping the other families depend on, and a + # private name here would emit a node no capture can pair against. # - # It carries the same op name on purpose. Both kernels classify to - # ``moe_router`` in ``deviation._OP_RULES``, which is a decision the - # dense-MoE and hybrid families already depend on; giving this one a private - # name would emit a node no capture can pair against and strand the gating - # kernel on the GEMM's row. Two instances of one op, distinguished by issue - # order and — where NVTX ranges exist — by identity. - # - # Kept as its own node regardless of the name it shares: it is the **only - # data-dependent shape in the step**, and therefore the node that decides - # whether the step can be CUDA-graph captured at all. + # Its own node regardless: the **only data-dependent shape in the step**, and + # so the thing that decides whether the step is CUDA-graph capturable. add( "moe_router", 3.0 * rows * spec.n_routed_experts, @@ -798,12 +734,10 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: ed, ) - # Gather rows into expert-major order and scatter the results back. Zero - # arithmetic in the gather, and at decode a rounding error — but the expanded - # tensor is ``rows x top_k`` wide, so at an 8,192-token prefill chunk this pair - # moves hundreds of megabytes per layer for essentially no FLOPs. A graph that - # folds it into the expert GEMM cannot show that, and it is a real and - # separately addressable share of prefill traffic. + # Gather into expert-major order. Zero arithmetic, a rounding error at + # decode — but the expanded tensor is ``rows x top_k`` wide, so at an + # 8,192-token chunk this and its scatter move hundreds of MB per layer for no + # FLOPs. Folded into the expert GEMM, that share of prefill is invisible. expanded = rows * spec.top_k add("moe_permute", 0.0, aw * (rows * h + expanded * h) / es, spec.act_dtype) @@ -866,17 +800,14 @@ def _emit_collective( bytes, so the count is the cost. Under expert parallelism the MoE half additionally dispatches and combines across expert ranks. - ``dispatches_experts`` gates the expert-parallel all-to-all. Only a mixture - layer dispatches tokens to expert ranks; the three dense-FFN layers compute - their whole FFN locally and emit the all-reduce alone. Charging them an - all-to-all would put 5.5 MB of wire traffic per layer on a block that has no - experts to send anything to. + ``dispatches_experts`` gates the expert-parallel all-to-all: only a mixture + layer sends tokens to expert ranks, and charging the dense layers for one puts + wire traffic on a block with no experts to send to. ``serial_launches`` is withheld when the SKU carries no interconnect bandwidth, so an unpriced collective still predicts zero time and :attr:`Graph.has_unpriced_collectives` keeps reporting it. A latency floor - applied there would quietly convert "we cannot price this" into "it costs two - microseconds". + there would convert "cannot price this" into "costs two microseconds". """ tp = max(1, sh.tp) if tp <= 1 and sh.ep <= 1: @@ -920,25 +851,18 @@ def predict_glm_graph( One step, three passes, and they are not three graphs: - **The backbone** runs over every position in the step — decode positions plus - any prefill chunk riding along under chunked prefill. With speculative - decoding on, "decode positions" is ``batch x (1 + D)``: **verify is not a new - graph, it is this one with the row dimension multiplied by 1+D**. - - **The draft chain** is the one genuinely new subgraph. GLM-5.2 has a single - MTP module (``num_nextn_predict_layers: 1``) invoked ``D`` times serially, - EAGLE-style — stage *k* cannot start until stage *k-1*'s token id exists — and - each stage runs its own vocabulary projection. Emitting one draft block and - one ``lm_head`` for a ``D``-deep chain understates it by ``D``, and the - ``lm_head`` term is not small: it reads the whole untied vocabulary matrix - per stage regardless of how few rows ride on it. - - **The epilogue** projects only the rows that need logits — - :attr:`BatchConfig.logits_rows`, which is one row per prefilling *request* - plus every decode position. Charging every prefill token would overstate a - 154,880-wide projection by the chunk size. - - With ``sharding`` left at its default the graph is whole-model; given a real + **The backbone** runs over every position — decode positions plus any prefill + chunk riding along. Under speculative decoding that is ``batch x (1 + D)``: + **verify is not a new graph, it is this one at 1+D rows**. + + **The draft chain** is the one genuinely new subgraph: a single MTP module + invoked ``D`` times serially, EAGLE-style, each stage running its own + vocabulary projection over the whole untied matrix. + + **The epilogue** projects only :attr:`BatchConfig.logits_rows` — one row per + prefilling *request* plus every decode position, not the whole chunk. + + With ``sharding`` at its default the graph is whole-model; given a real sharding it predicts what *one rank* does. """ spec = model or GlmMoeDsaModelSpec() @@ -1026,26 +950,18 @@ def add_lm_head(rows: float, layer: int | None) -> None: ) # ── the draft chain ────────────────────────────────────────────────────── - # The MTP module is invoked once per drafted token, serially. Each stage sees - # one row per sequence (it proposes for the sequence, not for the verify rows) - # and carries no prefill: a draft head proposes continuations, it does not - # ingest a prompt. + # One MTP module invoked once per drafted token, serially, at one row per + # sequence and no prefill — a draft proposes continuations, so on a + # pure-prefill step it does not run at all (nodes that never launch would show + # up as a launch facet made of absent kernels). # - # ``index_share_for_mtp_iteration`` says the iteration reuses the main model's - # selection rather than recomputing it, and the weight map agrees — the MTP - # block carries **no** ``self_attn.indexer.*`` tensors, exactly like the 57 - # ``shared`` layers. So the draft is emitted as a shared block; banking the - # projection and the scan into the floor would price a scan the runtime skips. + # Emitted as a *shared* block: ``index_share_for_mtp_iteration`` says the + # iteration reuses the main selection, and the weight map agrees — no + # ``self_attn.indexer.*`` tensors, exactly like the 57 ``shared`` layers. # - # What the draft is *not* is a smaller copy of the model. The MTP block carries - # a full ``mlp.experts.*`` bank in the checkpoint, so every draft stage draws on - # a 256-expert mixture — the draft's cost is dominated by expert weight traffic - # it pays ``D`` times over, not by its arithmetic. - # Gated on there being decode positions at all: on a pure-prefill step the - # draft head does not run — it proposes continuations, and there is nothing yet - # to continue. Emitting it anyway put 18 zero-work nodes in the graph whose only - # cost was their launches, which is a launch facet made of kernels that never - # ran. + # It is not a smaller copy of the model. The block carries a full + # ``mlp.experts.*`` bank, so its cost is expert weight traffic paid ``D`` times + # over, not arithmetic. if spec.num_nextn_predict_layers > 0 and batch.positions_per_step > 0: draft_batch = replace(batch, prefill_tokens=0, speculative_tokens=0) stages = max(1, batch.speculative_tokens) @@ -1125,22 +1041,14 @@ def _op_dtype_overrides( ) -> tuple[tuple[str, str], ...]: """Per-op precision, read from the checkpoint rather than assumed. - Two independent sources, and they answer different questions: - - ``quantization_config.modules_to_not_convert`` - *What the quantiser skipped.* Those tensors stay at the model dtype while - the backbone drops to fp8 — on GLM-5.2-FP8 that is ``lm_head``, - ``embed_tokens``, the MTP ``eh_proj`` and, the one worth naming, the - **lightning indexer**. Pricing the indexer at fp8 would halve the weight - traffic of the one attention node whose cost grows with context. - - ``moe_router_dtype`` - *What the model computes in regardless.* fp32 on every GLM-5.2 variant, - including the unquantised one, so it is emitted whether or not a - quantisation config exists. + Two sources answering different questions. + ``quantization_config.modules_to_not_convert`` is *what the quantiser + skipped* — on GLM-5.2-FP8 that is ``lm_head``, ``embed_tokens``, the MTP + ``eh_proj`` and, the one worth naming, the **lightning indexer**. + ``moe_router_dtype`` is *what the model computes in regardless*: fp32 on every + variant, so it is emitted with or without a quantisation config. - Returns an empty tuple when the checkpoint says nothing — an unquantised model - needs no overrides beyond the router, and inventing entries would put a + Empty when the checkpoint says nothing — inventing entries would put a precision in the graph that the checkpoint never declared. """ found: dict[str, str] = {} diff --git a/gitm/planner/model_catalogue.py b/gitm/planner/model_catalogue.py index 3b272b3..674804a 100644 --- a/gitm/planner/model_catalogue.py +++ b/gitm/planner/model_catalogue.py @@ -69,13 +69,34 @@ def _expand_layer_types(value: Any, n_layers: int) -> tuple[str, ...]: return expanded -def load_entry(name_or_path: str | Path) -> dict[str, Any]: - """The raw catalogue entry, validated for structure but not yet a spec.""" +def load_entry(name_or_path: str | Path, _seen: frozenset[str] = frozenset()) -> dict[str, Any]: + """The raw catalogue entry, validated for structure but not yet a spec. + + ``extends: `` merges this entry's ``spec`` over that of another. It + exists for the case where two entries describe *the same architecture at a + different precision* — a bf16 release and its FP8 sibling, which share a + 78-entry indexer schedule and a 78-entry MLP schedule verbatim. Copying those + into both files is 158 lines of duplicated evidence that can drift apart + silently, and it hides the thing worth seeing: the two entries differ only in + their dtypes. ``provenance`` is deliberately *not* merged — each checkpoint + was validated against its own published size and has its own open questions. + """ path = _resolve(name_or_path) data = yaml.safe_load(path.read_text()) or {} if not isinstance(data, dict): raise ValueError(f"{path}: expected a mapping at the top level") + base_name = data.pop("extends", None) + if base_name is not None: + key = str(base_name) + if key in _seen: + raise ValueError(f"{path}: 'extends' cycle through {key!r}") + base = load_entry(key, _seen | {key}) + merged = dict(base.get("spec") or {}) + merged.update(data.get("spec") or {}) + data = {**{k: v for k, v in base.items() if k != "provenance"}, **data} + data["spec"] = merged + family = data.get("family") if family not in _FAMILIES: raise ValueError( @@ -118,6 +139,20 @@ def load_spec(name_or_path: str | Path): for key in ("compress_ratios", "dspark_layer_ids", "indexer_types", "mlp_layer_types"): if key in raw and isinstance(raw[key], list): raw[key] = tuple(raw[key]) + # Per-layer schedules must cover the model exactly. ``spec_from_hf_config`` + # already refuses a short one; the catalogue path did not, and a schedule one + # entry short does not fail — the missing layers fall through to the modulo + # fallback and can land on the right answer by luck, which is a plausible + # total resting on evidence that is not there. That is the exact failure the + # explicit schedules exist to prevent, so it is an error here too. + n_layers = int(raw.get("n_layers", 0) or 0) + for key in ("indexer_types", "mlp_layer_types"): + sched = raw.get(key) + if sched and n_layers and len(sched) != n_layers: + raise ValueError( + f"{name_or_path}: {key} has {len(sched)} entries for {n_layers} " + "layers — the schedule must cover the model exactly" + ) # Nested one level: ``op_dtype_overrides`` is a list of ``[op, dtype]`` pairs # in YAML and must reach the frozen spec as a tuple of tuples. Coercing only # the outer list would leave inner lists inside a frozen dataclass — hashable diff --git a/gitm/planner/models/glm-5.2-fp8.yaml b/gitm/planner/models/glm-5.2-fp8.yaml index 225c6ca..5e4d64a 100644 --- a/gitm/planner/models/glm-5.2-fp8.yaml +++ b/gitm/planner/models/glm-5.2-fp8.yaml @@ -1,238 +1,37 @@ # zai-org/GLM-5.2-FP8 — the vendor's recommended deployment shape. # Source: https://huggingface.co/zai-org/GLM-5.2-FP8/blob/main/config.json -# https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json -# Read from config.json and model.safetensors.index.json only — no traces. # -# Identical architecture to `glm-5.2`; the difference is precision, and precision -# is the difference between a model that fits one 8xH200 node and one that does -# not. 753.33 GB on disk against the bf16 release's 1.507 TB. +# Identical architecture to `glm-5.2`, which this extends; the difference is +# precision, and precision is the difference between a model that fits one +# 8xH200 node and one that does not. 753.33 GB on disk against the bf16 +# release's 1.507 TB. Everything not listed below is inherited, so the fields +# that ARE listed are exactly what the FP8 checkpoint changes. name: zai-org/GLM-5.2-FP8 family: glm_moe_dsa +extends: glm-5.2 description: > - GLM-5.2 quantised to FP8 e4m3 with 128x128 weight blocks — 78 layers, hidden - 6144, 256 routed experts top-8, MLA + DeepSeek Sparse Attention with - IndexShare (21 of 78 layers compute the index). The backbone GEMMs, the dense - FFN, the shared expert and all routed experts are fp8; lm_head, embed_tokens, - the MTP eh_proj and the lightning indexer stay bf16; the router is fp32. This - is the entry to plan against — the vendor recommends it and it is what fits - 8xH200. + GLM-5.2 quantised to FP8 e4m3 with 128x128 weight blocks. The backbone GEMMs, + the dense FFN, the shared expert and all routed experts are fp8; lm_head, + embed_tokens, the MTP eh_proj and the lightning indexer stay bf16; the router + is fp32. This is the entry to plan against — the vendor recommends it and it + is what fits 8xH200. spec: - hidden: 6144 - n_layers: 78 - vocab: 154880 - - # MLA attention. head_dim (192) is the nope width; q_head_dim widens to - # 192+64=256, and v_head_dim (256) differs from it — score and value reads use - # different per-head widths. num_key_value_heads is 64 in the config but is a - # red herring: the cache holds one kv_lora_rank latent per token, shared across - # all 64 query heads, so KV traffic derives from kv_lora_rank, never 64*head_dim. - n_heads: 64 - q_lora_rank: 2048 - kv_lora_rank: 512 - qk_nope_head_dim: 192 - qk_rope_head_dim: 64 - v_head_dim: 256 - - # DeepSeek Sparse Attention indexer. index_topk_freq 4 is the IndexShare period: - # one 'full' layer computes the top-2048 selection, the next three reuse it. - index_n_heads: 32 - index_head_dim: 128 - index_topk: 2048 - index_topk_freq: 4 - # Read verbatim from the checkpoint (period-4 past the 3-layer 'full' prefix). - # The 'shared' layers physically carry no indexer weights — proven, not fitted. - indexer_types: - - full - - full - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - full - - shared - - shared - - shared - - # Mixture of experts. moe_intermediate_size 2048 per expert; the first 3 layers - # are dense FFN (intermediate_size 12288) with no router or experts. - n_routed_experts: 256 - n_shared_experts: 1 - num_experts_per_tok: 8 - moe_intermediate_size: 2048 - intermediate_size: 12288 - first_k_dense_replace: 3 - routed_scaling_factor: 2.5 - mlp_layer_types: - - dense - - dense - - dense - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - sparse - - # Multi-token prediction. One MTP module, invoked once per drafted token; the - # vendor recipe runs it 5 deep (--speculative-config.num_speculative_tokens 5). - # index_share_for_mtp_iteration: the draft reuses the main model's selection, - # and the weight map agrees — the MTP block carries no indexer tensors. - num_nextn_predict_layers: 1 - index_share_for_mtp_iteration: true - # FP8 e4m3, 128x128 block-scaled, dynamic activation scaling. weight_dtype: fp8 expert_dtype: fp8 - act_dtype: bf16 + # A serving choice, not a model fact — but it is the one the vendor recipe # makes (`--kv-cache-dtype fp8`), and it halves the latent half of every cached # entry. The decoupled RoPE key stays bf16 either way; kv_entry_bytes prices # the two halves separately for exactly this reason. kv_dtype: fp8 - # The ops the quantiser skipped, from quantization_config.modules_to_not_convert, - # plus the router's own dtype from the base config. Without these the indexer — - # the one attention node whose cost grows with context — is priced at half its - # real weight traffic, and lm_head at half of 154,880 x 6,144. + # From quantization_config.modules_to_not_convert, plus the router's own dtype + # from the base config. Without these the indexer — the one attention node + # whose cost grows with context — is priced at half its real weight traffic, + # and lm_head at half of 154,880 x 6,144. op_dtype_overrides: - [attn_index_proj, bf16] - [lm_head, bf16] @@ -244,63 +43,53 @@ provenance: - claim: predicted weight bytes match the published FP8 checkpoint detail: > model.safetensors totals 753,329,940,480 B (753.33 GB) across 141 shards. - model_weight_bytes predicts 755.9 GB — +0.34%, and the same spec at bf16 + model_weight_bytes predicts 755.9 GB — +0.34%, and the same shape at bf16 predicts 1.5079 TB against the bf16 release's 1,506,659,919,872 B (+0.08%). Two checkpoints at two precisions agreeing to under half a percent is a stronger check on the shape arithmetic than either one alone. - claim: the unquantised op list is read, not assumed detail: > - quantization_config.modules_to_not_convert names lm_head, embed_tokens, - eh_proj, enorm/hnorm, every layernorm, mlp.gate (+ e_score_correction_bias) - and the indexer's projections and k_norm. o_proj is absent from that list — - it IS quantised here, the opposite of the fp8-backbone checkpoints that - keep the output projection wide. - - claim: the router is fp32 on every variant - detail: > - moe_router_dtype "float32" is a field of the base GLM-5.2 config, not of - the quantisation config, so it is a model fact and applies to the bf16 - release too. - - claim: IndexShare is read from the weight map + modules_to_not_convert names lm_head, embed_tokens, eh_proj, enorm/hnorm, + every layernorm, mlp.gate (+ e_score_correction_bias) and the indexer's + projections and k_norm. o_proj is absent from that list — it IS quantised + here, the opposite of the fp8-backbone checkpoints that keep the output + projection wide. + - claim: activations are quantised at run time, so the quant kernel is real detail: > - Indexer tensors (*.indexer.wq_b / .wk / .weights_proj / .k_norm) exist on - exactly the 21 layers whose indexer_types is 'full' and on none of the 57 - 'shared' layers — nor on the MTP block, which is what - index_share_for_mtp_iteration=true predicts. + activation_scheme "dynamic" — the graph emits an act_quant node ahead of + each group of fp8 GEMMs. On the bf16 entry there is nothing to quantise + and the node does not exist. estimated: - field: kv_dtype value: fp8 detail: > - A deployment choice, taken from the vendor's own vLLM recipe + A deployment choice, taken from the vendor's vLLM recipe (`--kv-cache-dtype fp8`), not from the checkpoint. Set it back to bf16 to price the conservative cache; on this architecture that moves the latent half of a 576-element entry, not the RoPE half. - field: MLA absorption (attn_kv_b vs attn_out_proj width) value: unabsorbed — kv_b runs as its own GEMM, o_proj narrow detail: > - Modelled unabsorbed: attn_kv_b reconstructs per-head K/V from the latent - and attn_out_proj is n_heads*v_head_dim -> hidden (16384->6144). An engine - that absorbs MLA folds W^UK into the query and W^UV into the output, - dropping attn_kv_b and doubling attn_out_proj's input width to + An engine that absorbs MLA folds W^UK into the query and W^UV into the + output, dropping attn_kv_b and doubling attn_out_proj's input width to n_heads*kv_lora_rank (32768->6144). Same resident weights either way; the two readings move attn_out_proj by 2x, so it re-ranks the attention side. - field: expert-parallel imbalance value: 1.0 (perfect balance) detail: > - Trace-calibrated by design. Declared rather than fitted while the - no-traces constraint holds; real skew moves less weight traffic than - predicted (fewer distinct experts) but lengthens the grouped-GEMM tail. + Trace-calibrated by design, declared while the no-traces constraint holds. + Real skew moves less weight traffic than predicted (fewer distinct + experts) but lengthens the grouped-GEMM tail. unmodelled: - - Absorbed-MLA decode. See the estimated entry above; it is the one open - question that re-ranks the attention side by 2x in either direction. + - Absorbed-MLA decode. See the estimated entry above; the one open question + that re-ranks the attention side by 2x in either direction. - Acceptance rate under MTP. The graph prices D drafts and a 1+D verify; what - fraction is kept is a serving observable, carried in BatchConfig rather than - predicted here. + fraction is kept is a serving observable, not a config-derivable one. - index_topk_freq as a *temporal* reuse across decode steps. Modelled as the - spatial layer-group period, which the weight map proves; any additional - step-to-step reuse would only reduce indexer cost further. + spatial layer-group period, which the weight map proves; extra step-to-step + reuse would only reduce indexer cost. - Expert-capacity padding. If the grouped-GEMM backend pads to a fixed - capacity to keep static shapes, every step reads all 256 experts and the - distinct_experts union term is an underestimate at low batch — the fork to - settle with one D2H count. + capacity, every step reads all 256 experts and the distinct_experts union + term is an underestimate at low batch — the fork one D2H count settles. diff --git a/gitm/planner/models/glm-5.2.yaml b/gitm/planner/models/glm-5.2.yaml index dcc5632..bf97f65 100644 --- a/gitm/planner/models/glm-5.2.yaml +++ b/gitm/planner/models/glm-5.2.yaml @@ -115,6 +115,7 @@ spec: - full - shared - shared + - shared # Mixture of experts. moe_intermediate_size 2048 per expert; the first 3 layers # are dense FFN (intermediate_size 12288) with no router or experts. diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index 1521631..b54b7d6 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -20,9 +20,12 @@ from __future__ import annotations +import os +import tempfile from dataclasses import replace import pytest +import yaml from gitm.planner.glm_graph import ( GlmMoeDsaModelSpec, @@ -33,7 +36,7 @@ predict_glm_graph, spec_from_hf_config, ) -from gitm.planner.model_catalogue import available, load_spec, predict +from gitm.planner.model_catalogue import available, load_entry, load_spec, predict from gitm.planner.registry import detect_family from gitm.planner.roofline import BatchConfig, ShardingConfig @@ -534,6 +537,55 @@ def test_catalogue_entry_loads_and_predicts(entry): assert g.total_pred_s > 0 +def test_catalogue_schedules_cover_the_model_exactly(): + """A schedule one entry short does not fail — it falls through and may be right. + + The missing layers take the modulo fallback, which on GLM-5.2 happens to + produce the correct count. A plausible total resting on evidence that is not + there is the exact failure explicit schedules exist to prevent, so the + catalogue path refuses it rather than accepting the luck. + """ + published = ["full"] * 3 + (["shared"] * 3 + ["full"]) * 18 + ["shared"] * 3 + for entry in ("glm-5.2", "glm-5.2-fp8"): + spec = load_spec(entry) + assert len(spec.indexer_types) == spec.n_layers == 78 + assert list(spec.indexer_types) == published + + from gitm.planner.model_catalogue import CATALOGUE_DIR + from gitm.planner.model_catalogue import load_spec as _load + + short = yaml.safe_load((CATALOGUE_DIR / "glm-5.2.yaml").read_text()) + short["spec"]["indexer_types"] = short["spec"]["indexer_types"][:-1] + with tempfile.NamedTemporaryFile("w", suffix=".yaml", delete=False) as fh: + yaml.safe_dump(short, fh) + path = fh.name + try: + with pytest.raises(ValueError, match="must cover the model exactly"): + _load(path) + finally: + os.unlink(path) + + +def test_fp8_entry_inherits_the_schedule_it_shares(): + """Two precisions of one architecture, and only the dtypes are written twice. + + The FP8 entry ``extends`` the bf16 one. Copying a 78-entry schedule into both + files is duplicated evidence that can drift apart silently — and it did: + the schedule was one entry short for a while, in the file that had it twice. + ``provenance`` is deliberately not inherited; each checkpoint is validated + against its own published size. + """ + bf16, fp8 = load_spec("glm-5.2"), load_spec("glm-5.2-fp8") + assert fp8.indexer_types == bf16.indexer_types + assert fp8.n_layers == bf16.n_layers and fp8.hidden == bf16.hidden + assert (fp8.weight_dtype, fp8.kv_dtype) == ("fp8", "fp8") + assert (bf16.weight_dtype, bf16.kv_dtype) == ("bf16", "bf16") + + entry = load_entry("glm-5.2-fp8") + fields = {e["field"] for e in entry["provenance"]["estimated"]} + assert "kv_dtype" in fields # its own, not the base's + + def test_fp8_entry_is_the_deployable_one(): """Precision is what decides whether the model fits a node, so it is checked. From 6dcc7e8465aad75f3bdabbe415cb8489018e958a Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 14:36:17 -0700 Subject: [PATCH 09/35] GLM-5.2: restore the vendor launch args; correct two overstated claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-ups from reviewing the prune and the validation pass. The §9 prune took the `vllm serve` block with it, and the replacement pointed at "the recipe quoted in the hardware section" where it no longer was. That block is evidence, not prose: it is what makes the hardware assumption auditable as the vendor's own rather than mine, and §6.2's C6 and §6.4 row 0 are built on it. Restored into the hardware section, with the note that --enable-expert-parallel is NOT in the vendor recipe even though §4 prices the EP8 shape -- which is capture C5 and re-ranks the largest line in prefill. Two claims were stronger than the code supports: - `expected_stream_id=1` on collectives does not give "the stream-concurrency invariant something to read". docs/invariants.md §3 defines that invariant, but optimizer/monitor.py tests overlap using the *observed* kernel's stream_id; nothing reads the predicted field. It is a hook, not a wiring, and both the code comment and §7.1 now say so. - Reported as G11, found not fixed: kernels/library.yaml scopes every lever with `applies_to_kernels` from a canonical op list that has no `moe_routed`/`moe_shared`, so the two levers meant for routed-expert traffic scope to `[mlp_gate_up, mlp_down]` -- true of a dense FFN, false of either MoE family. §5 rank 5 targets 74% of a decode step through tooling that cannot match it. Pre-existing and shared with DeepSeek-V4, so it should land where both families can be checked. Also: §1 now carries the truncated-schedule incident as the worked demonstration of why the schedules are read verbatim -- the wrong evidence produced the right total and a byte-identical floor, which is the whole argument. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 40 +++++++++++++++++++++++++++++++++++-- gitm/planner/glm_graph.py | 6 ++++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 173cae7..db17aee 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -35,6 +35,19 @@ some bound labels do. Regions whose label would flip are marked ⚑ throughout. | Memory | 141 GB × 8 = 1,128 GB | one node | | Kernel launch | ~2 µs graph-replay / ~5 µs eager | ⚑ the crossover hinge — see §5 rank 3 | +The recipe, verbatim, because every constant above and every bound label in §4 +assumes it — and because §6.2's C6 ("the engine's launch arguments, as text") is +worth more than most of the traces: + +```bash +vllm serve zai-org/GLM-5.2-FP8 --kv-cache-dtype fp8 --tensor-parallel-size 8 --speculative-config.method mtp --speculative-config.num_speculative_tokens 5 --tool-call-parser glm47 --reasoning-parser glm45 --enable-auto-tool-choice +``` + +`--enable-expert-parallel` is **not** in it. §4 prices the EP8 shape anyway, +because it is the shape that makes the `moe_all_to_all` rows exist at all — +TP8-only removes them and doubles the per-rank expert bank instead. Which of the +two is running is capture C5, and it re-ranks the largest line in prefill. + ``` FP8 ridge = 1,979e12 / 4.8e12 = 412 FLOP/byte BF16 ridge = 989.5e12 / 4.8e12 = 206 FLOP/byte @@ -104,6 +117,13 @@ a whole-model figure; the **3.7×** used throughout this note is the *indexer's own* ratio (78 layers ÷ 21), which is the one that matters when ranking the indexer against the MoE term. Two denominators, both correct. +**A worked demonstration of why "read verbatim" is not pedantry:** the catalogue +entry carried this schedule one entry short for a while. Layer 77 fell through to +the modulo fallback, landed on `shared`, and the total came out 21 full-indexer +layers — the right answer from evidence that was not there. Nothing failed, and +the predicted floor was byte-identical before and after the fix. The loader now +refuses a schedule that does not cover the model exactly. + **The schedule is proven from the weight map, not inferred.** Indexer tensors (`*.indexer.wq_b`, `.wk`, `.weights_proj`, `.k_norm`) exist on exactly the 21 `full` layers, on **none** of the 57 `shared` layers, and on **none** of the MTP @@ -910,7 +930,22 @@ gaps below are narrower because of it. | **G7** | **An MTP chain D stages deep, each with its own vocabulary projection** | The graph emitted **one** draft block and **one** `lm_head` for what the vendor recipe runs **five** deep. `lm_head` is 19 % of the draft's bytes, so a D-deep chain was understated by ~5× on its largest term | a stage loop in `predict_glm_graph` driven by `BatchConfig.speculative_tokens`, with `mtp_eh_proj` and an `lm_head` per stage; `--spec-tokens` on the CLI | **yes** | | **G8** | **A graph that is only its GEMMs, priced against a bound it cannot express** | The family emitted 16 nodes per layer where a layer lowers to ~20 kernels, and the seven missing ones were all pointwise: the norms, the dynamic fp8 activation scaling, the fused gating, the prologue gather and the epilogue's all-gather. Every one is a rounding error in bytes and **a full kernel launch in time** — so at B=1 the graph reported a step as memory-bound that is 69 % launches. A roofline with a launch bound and a graph with no launches in it cannot both be right | Emit them. `_pointwise`, `add_rms_norm` (with the residual fused in, as vLLM runs it), `add_act_quant` gated on the consuming GEMM actually being fp8, plus `embed_tokens` / `rms_norm` / `logits_all_gather` around the stack. Node names constrained by G9 | **yes** | | **G9** | **Op names a capture can actually pair against** | G8's new nodes needed names, and `deviation.classify_op` is a *name guess* (`docs/kernel_identity.md`): a name it cannot classify leaves the predicted node permanently unmatched **and** the real kernel filed as unmodeled — two errors in opposite directions, in the diff the family exists to support. Three norm sites are one kernel name; `silu_and_mul` was already claimed by `mlp_gate_up`; `moe_align`/`topk_softmax` were already claimed by `moe_router`, a decision the dense-MoE and hybrid families depend on | Follow the canonical names rather than redefine them: one `rms_norm` op for all three sites, SwiGLU folded back into the GEMM that owns its needle, gating emitted as a second `moe_router` instance. Then `_OP_RULES` gains only what is genuinely new and unclaimed — `rms_norm`, `act_quant`, `embed_tokens`, `moe_permute`/`moe_combine`, `attn_index_proj`, `attn_kv_b`, `mtp_eh_proj`. A test asserts every op the graph emits resolves | **yes** | -| **G10** | **Which adjacent nodes may overlap and which may not** | `Graph.total_pred_s` is a sum, not a DAG. GLM puts both kinds of serialisation in one step — the draft chain is genuinely serial, the 158 collectives are not — and **both appear as the same positive residual today**. §6.1 is unanswerable without telling them apart | *Not shipped.* It is a cross-family IR change and it is already sequenced on the roadmap. What this note adds is the requirement, and `expected_stream_id=1` on collectives so the stream-concurrency invariant has something to read | **no — deliberately** | +| **G10** | **Which adjacent nodes may overlap and which may not** | `Graph.total_pred_s` is a sum, not a DAG. GLM puts both kinds of serialisation in one step — the draft chain is genuinely serial, the 158 collectives are not — and **both appear as the same positive residual today**. §6.1 is unanswerable without telling them apart | *Not shipped.* It is a cross-family IR change and it is already sequenced on the roadmap. What this note adds is the requirement. `expected_stream_id=1` is set on collectives, but note that **nothing reads it today** — `optimizer/monitor.py` tests overlap using the *observed* kernel's stream, so the predicted field is carried by the IR and consumed by no one. It is a hook for the invariant in `docs/invariants.md` §3, not a wiring of it | **no — deliberately** | + +**G11 — found, not fixed: the intervention library has no vocabulary for the +expert term.** `gitm/kernels/library.yaml` scopes each lever with +`applies_to_kernels`, drawn from a canonical op list (`_decode_step_ops`) that is +`qkv_proj · attn_score_value · attn_out_proj · mlp_gate_up · mlp_down · lm_head`. +**`moe_routed` and `moe_shared` are not in it**, and the two entries that mean to +target expert traffic scope themselves to `[mlp_gate_up, mlp_down]` with the +comment "the routed-expert GEMMs" — which is true of a dense FFN and not of this +graph. So §5 rank 5, the lever list on the node that is **74 % of a decode step**, +cannot be matched by the tooling that is supposed to act on it. + +This is **pre-existing and not GLM-specific** — `moe_graph.py` emits the same op +names, so DeepSeek-V4 has it identically — which is why it is reported here rather +than fixed on this branch: the fix is a change to a shared vocabulary, and it +should land where both families' coverage can be checked at once. ### 7.2 The one that needed more than a table row @@ -1016,7 +1051,8 @@ A.2–A.4 are stated as **deltas** from A.1, because everything not listed is byte-for-byte identical. **Two columns are omitted rather than repeated.** Every node runs on the compute -stream except the collectives (`expected_stream_id=1`). Confidence is **high** +stream except the collectives, which the graph marks `expected_stream_id=1` — +a declaration, not yet a check (§7.1 G10). Confidence is **high** throughout — these rows are read from `config.json` and the tensor index — except the collectives (**medium**, a TP/EP convention) and the S1 histogram readback (**low**, a hypothesis about a stack nobody has opened). diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 4c23a9e..43dc760 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -825,8 +825,10 @@ def add_link(name_op: str, byts: float) -> None: name, 0.0, byts, link, spec.act_dtype, estimated=True, serial_launches=1 if priced else 0, ), - # Collectives are the one region this graph expects off the - # compute stream; the stream-concurrency invariant reads this. + # The one region this graph expects off the compute stream. + # A declaration, not a check: nothing reads this field today + # (monitor.py tests overlap on the *observed* stream), so it is a + # hook for the invariant in docs/invariants.md §3, not a wiring. expected_stream_id=1, ) ) From 0e3db394c0356e582973b1aa7e76020ee386c634 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 15:07:11 -0700 Subject: [PATCH 10/35] GLM-5.2: prune the note and delete dead helpers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last "prune" was token-level and the doc grew back. This one deletes rather than rewrites. Doc 1208 -> 1047. Structural cuts, not word-smithing: - The node list appeared three times -- §4.1's ASCII table, Appendix A.1's markdown table, and §2.1's flowchart subgraph. It now appears once, in A.1, with shapes and dtypes; §4.1 keeps the same nodes sorted by cost, which is the argument, and §2.1's diagram points at A.1 instead of redrawing it. - §4.3 ("same kernel, opposite label") was §2.2's four-node table and §3.2's bullets a third time. Its two unique rows moved into §2.2 and the section is gone. - §4.4 was a reverse index of §4.2's flip column; it is now the eight variables that move more than one row, with magnitudes. - §5.1's gate-check table, §6.1's four-box flowchart, §6.3's instrument table and §7.0's "already gets right" table were all bookkeeping in table form. Prose, one to four lines each. - §1's ASCII layer diagram, and three paragraphs restating the table immediately above them, deleted outright. Composition is now 292 table + 110 diagram + 399 prose + headings/blanks: the tables and diagrams are the content, and the prose no longer restates them. Code 1171 -> 1137, and two of those lines were dead. effective_kv_tokens and index_candidates were the decode-only helpers superseded by core_read_entries / index_scan_pairs when prefill landed; each was referenced only by its own definition. The earlier artifact sweep missed them because it grepped GLM names, not dead functions. Also dropped a local _canon that duplicated roofline._canon_dtype. No behaviour change: floor still 16.551 ms, suite still at the pre-existing 4 failures. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 741 ++++++++++++++---------------------- gitm/planner/glm_graph.py | 39 +- 2 files changed, 292 insertions(+), 488 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index db17aee..90a7901 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -85,51 +85,27 @@ the planner change that made it representable. | Max context | 1,048,576 | | Total / active params | **744 B** published + a **9.9 B** MTP block / ~39 B active | -``` -L: 0 1 2 3 6 10 14 ... 74 77 - F F F ···F ···F ···F ···F ···F ··· - │ │ │ F = full indexer (recomputes top-2048) - └──┴──┴─ the only DENSE MLP layers · = shared indexer (reuses, no weights) - everything from 3 up: MoE 256e / top-8 / 1 shared -``` - -**A clean 3-full prefix then a strict period-4 cycle** — -`indexer_types` is read verbatim, and the two schedules do *not* line up: the -dense prefix is 3 layers, the `full` prefix is also 3 layers, and then the -IndexShare period is 4. No single modulo rule reproduces either (§7, G3). - ### Semantics read from the checkpoint, not guessed -`config.json`: - -- `indexer_types[i] == "full"` → the layer **computes** its own top-2048 selection -- `indexer_types[i] == "shared"` → the layer **reuses the previous `full` layer's - top-k** — the semantics `transformers` documents for the field, verbatim -- `mlp_layer_types[i]` → `"dense"` | `"sparse"`, and it agrees with - `first_k_dense_replace: 3` -- `moe_router_dtype: "float32"` → the router is fp32 **on every variant**, because - this is a field of the base config and not of any quantisation config - -IndexShare is a published mechanism ([paper 2603.12201](https://huggingface.co/papers/2603.12201)): -Z.ai reports it "reuses the same indexer across every four sparse attention -layers, reducing per-token FLOPs by **2.9× at a 1M context length**". That 2.9× is -a whole-model figure; the **3.7×** used throughout this note is the *indexer's -own* ratio (78 layers ÷ 21), which is the one that matters when ranking the -indexer against the MoE term. Two denominators, both correct. - -**A worked demonstration of why "read verbatim" is not pedantry:** the catalogue -entry carried this schedule one entry short for a while. Layer 77 fell through to -the modulo fallback, landed on `shared`, and the total came out 21 full-indexer -layers — the right answer from evidence that was not there. Nothing failed, and -the predicted floor was byte-identical before and after the fix. The loader now -refuses a schedule that does not cover the model exactly. +From `config.json`: `indexer_types[i]` is `"full"` (the layer computes its own +top-2048) or `"shared"` (it **reuses the previous `full` layer's top-k** — the +semantics `transformers` documents, verbatim); `mlp_layer_types[i]` is +`"dense"` | `"sparse"`, agreeing with `first_k_dense_replace: 3`; and +`moe_router_dtype: "float32"` makes the router fp32 **on every variant**, being a +field of the base config rather than of any quantisation config. **The schedule is proven from the weight map, not inferred.** Indexer tensors (`*.indexer.wq_b`, `.wk`, `.weights_proj`, `.k_norm`) exist on exactly the 21 `full` layers, on **none** of the 57 `shared` layers, and on **none** of the MTP -module. A shared layer that recomputed the index would need those weights; it does -not have them. Pricing all 78 at full rate — the naive reading of `index_topk` — -overstates the indexer ~3.7× and mis-ranks it against the MoE term. +module. Pricing all 78 at full rate — the naive reading of `index_topk` — +overstates the indexer ~3.7×. (Z.ai publishes **2.9×** for +[IndexShare](https://huggingface.co/papers/2603.12201); that is whole-model +per-token FLOPs at 1M context, where 3.7× is the indexer's own ratio, 78 ÷ 21.) + +Why "read verbatim" is not pedantry: this catalogue entry carried the schedule one +entry short for a while. Layer 77 took the modulo fallback, landed on `shared`, +and the count still came out 21 — the right answer from evidence that was not +there, with a byte-identical floor. The loader now refuses a short schedule. Attention shapes, per token per layer: @@ -143,9 +119,9 @@ Attention shapes, per token per layer: | After `o_proj` | 6,144 | back to `d_model` | | Index key (`full` only) | 128 | cached alongside the latent | -`num_key_value_heads: 64` is in the config and is a **red herring**. A GQA reading -of the cache gives `64 × 448 = 28,672` elements per token per layer against the -real 576 — a **50× overstatement of the single quantity decode is bound by**. +`num_key_value_heads: 64` is a **red herring**: a GQA reading gives 28,672 +elements per token per layer against the real 576 — **50× on the single quantity +decode is bound by**. ### The 79-row table, collapsed to four archetypes @@ -159,33 +135,28 @@ is byte-for-byte identical between them. | `Ls,sh` | **57**| MLA+DSA | shared | MoE 256×2048 | 576 | 2× all-reduce (+2× a2a under EP) | | `Lmtp` | **1** | MLA+DSA | shared | MoE 256×2048 | 576 | 2× all-reduce, ×D stages | -Note what is *absent*, because the absences are the design: **no sliding window, -no compression schedule, no attention-type alternation, no vision encoder, no -audio encoder.** GLM-5.2 is a text-only decoder in which every layer runs the same -attention. All of its structural variation is in two schedules — dense/sparse MLP -and full/shared indexer — and one of those has no cost consequence at all past -layer 2. - -### Verification — both checkpoints agree - -- **bf16:** predicted **1,508.1 GB** against `total_size` **1,506,659,919,872 B** - (282 shards) — **+0.08 %**. -- **fp8:** predicted **755.9 GB** against **753,329,940,480 B** (141 shards) — - **+0.34 %**. - -Two published checkpoints at two precisions agreeing to under half a percent is a -stronger check than either alone: an error in the shape arithmetic would have to -be precision-proportional to survive both. A 2:4-sparsity-compressed checkpoint -would be roughly half the fp8 size. It is not — which is the evidence behind A2. - -**And a third check falls out of the gap between them.** Z.ai publishes **744 B** -parameters; the bf16 checkpoint is 753.3 B by its own byte count. The difference -is the MTP block, which the published figure excludes — and this graph predicts -**744.2 B with the draft head removed**, a 0.03 % match. That is not just another -size check: it says the MTP block is **9.9 B**, and a *dense* draft head at -`intermediate_size` would be **0.23 B**. The draft carrying a full 256-expert -mixture (§2.4) is therefore confirmed twice over — once from the weight map, once -from arithmetic on a number the vendor published for another reason. +The two schedules do not line up — a 3-layer dense prefix, a 3-layer `full` +prefix, then an IndexShare period of 4 — so no single modulo rule reproduces +either, which is why both are read verbatim (§7, G3). And note the absences: no +sliding window, no compression schedule, no attention-type alternation, no +encoders. Every layer runs the same attention. + +### Verification — three independent checks + +| check | predicted | published | error | +| --- | --- | --- | --- | +| bf16 checkpoint | 1,508.1 GB | 1,506,659,919,872 B (282 shards) | **+0.08 %** | +| fp8 checkpoint | 755.9 GB | 753,329,940,480 B (141 shards) | **+0.34 %** | +| params, MTP block removed | **744.2 B** | **744 B** (Z.ai) | **+0.03 %** | + +The third is the interesting one. The checkpoint is 753.3 B by its own bytes but +Z.ai publishes 744 B — the gap is the MTP block, which the published figure +excludes. So the block is **9.9 B**, where a *dense* draft head would be **0.23 B**: +the draft carrying a full 256-expert mixture (§2.4) is confirmed twice, once from +the weight map and once from arithmetic on a number published for another reason. + +Two precisions agreeing to under half a percent also rules out sparsity — a +2:4-compressed checkpoint would be roughly half the fp8 size (A2). ### KV cache — the number that drives decode @@ -205,11 +176,10 @@ whole model, bytes per token of context (fp8 latent, bf16 rope key + index key): | 131,072 | 6.90 GB | 12.48 GB | | 1,048,576 | **55.2 GB** | **99.9 GB** | -**And it is replicated, not sharded.** One shared latent cannot be split across -tensor-parallel ranks, so every rank holds and reads the whole cache: **TP buys no -KV bandwidth on this architecture.** At the advertised 1M context that is 55 GB per -rank on top of a 96 GB per-rank weight share — which is the real reason the vendor -recipe reaches for B200s and `--max-num-seqs 32` when it wants full context. +**Replicated, not sharded**: one shared latent cannot be split, so every rank +reads the whole cache — **TP buys no KV bandwidth here**. At 1M that is 55 GB per +rank on top of a 96 GB weight share, which is why the vendor recipe reaches for +B200s and `--max-num-seqs 32` for full context. ### FP8 — what is and is not quantized @@ -239,39 +209,25 @@ is 54 % of the step (§4.1). This is why the planner grew ```mermaid %%{init: {'theme':'neutral'}}%% flowchart TD - T["input_ids"] --> EMB["embed_tokens gather — BF16"] + T["input_ids"] --> EMB["embed_tokens gather"] EMB --> L0["layers 0-2 — MLA+DSA + DENSE FFN"] L0 --> LB["layers 3-77 — MLA+DSA + MoE 256/top-8"] - LB --> FN["final RMSNorm
LAST TOKEN OF EACH PROMPT ONLY"] - FN --> LM["lm_head GEMM BF16
1.9 GB of weights for 1 row per request"] + LB --> FN["final RMSNorm — LAST TOKEN OF EACH PROMPT ONLY"] + FN --> LM["lm_head BF16 — 239.5 MB for one row per request"] - subgraph LB + subgraph LB["one MoE layer (node list: Appendix A.1)"] direction TB - N1["input_layernorm"] --> QA["q_a fp8 → q_a_layernorm"] - QA --> QB["q_b fp8 — M=P, COMPUTE-BOUND"] - N1 --> KA["kv_a fp8 → latent[576] → CACHE WRITE"] - KA --> KB["kv_b fp8 — reconstruct K_nope,V"] - QA --> IX{"full indexer layer?"} - IX -- "21 layers" --> IP["indexer wq_b/wk/weights_proj — BF16"] - IP --> IS["index_score over the WHOLE history
O(P·C + P²/2) — the quadratic lives HERE"] - IS --> TK2["top-2048 per query"] - IX -- "57 layers" --> RE["reuse the group's selection
NO KERNEL AT ALL"] - TK2 --> ATT - RE --> ATT - KB --> ATT{"attention core over ≤2048 selected keys
FLOPs capped · BYTES ARE NOT"} - ATT --> OP["o_proj fp8"] - OP --> AR1{{"all_reduce #1 — 174 MB, BANDWIDTH-bound"}} - AR1 --> N2["post_attention_layernorm"] - N2 --> G["router GEMM — FP32, 26 GF/layer"] - G --> SIG["sigmoid + e_score_correction_bias"] - SIG --> TK["top-8 of 256 + renorm
DATA-DEPENDENT SHAPE"] - TK --> A2A{{"EP dispatch all-to-all
695 MB/layer — the largest single term"}} - A2A --> PERM["permute/gather — 8P rows"] - PERM --> EG["grouped GEMM gate+up fp8
ALL 256 experts hit"] - EG --> SW["SiLU × up"] --> ED["grouped GEMM down fp8"] - ED --> COMB["scatter-add × routed_scaling 2.5"] - COMB --> A2B{{"EP combine all-to-all"}} - A2B --> AR2{{"all_reduce #2"}} + QKV["q_a → q_b · kv_a → latent[576] → CACHE WRITE · kv_b
fp8, M=P — COMPUTE-BOUND"] --> IX + IX{"full indexer layer?"} + IX -- "21 layers" --> IS["index_score over the WHOLE history
O(P·C + P²/2) × 32 heads — the quadratic lives HERE"] + IX -- "57 layers" --> RE["reuse the group's selection — NO KERNEL"] + IS --> ATT + RE --> ATT{"attention core over ≤2048 selected keys
FLOPs capped · BYTES ARE NOT"} + ATT --> AR1{{"o_proj → all_reduce — 174 MB, BANDWIDTH-bound"}} + AR1 --> G["router GEMM FP32 → fused gating → top-8 of 256
DATA-DEPENDENT SHAPE"] + G --> A2A{{"EP dispatch all-to-all — 1.39 GB/layer, the largest single term"}} + A2A --> EG["grouped GEMM fp8 ×3 + SwiGLU — ALL 256 experts hit"] + EG --> A2B{{"EP combine → all_reduce #2"}} end ``` @@ -300,6 +256,8 @@ Identical node set to prefill. Four nodes change **kind**: | indexer scan | `O(P·C + P²/2)`, **compute-bound** | `O(B·C)`, **memory-bound** — streams the whole key set | the query count collapses from P to B; the key set does not | | **attention bytes** | whole cache, once **per request** | **top-2048 window, per sequence** | prefill queries' selections union to everything; one query's does not | | every GEMM | M = P ≈ 8192, compute-bound | M = B, **weight-streaming** | AI falls ~1,400 → ~60; same kernel name, different regime | +| collectives | **bandwidth** — 174 MB, 30.5 ms wire | **latency** — 688 kB, a ring floor | payload 250× apart; the EP a2a goes from the top line to 2.8 % | +| `lm_head` | one row per **request** | **every row, every step** | the epilogue is free in prefill and is not in decode | Plus one epilogue change: **in prefill only the last position of each prompt runs `lm_head`. At decode every row is a last position**, so it reads 1.9 GB of bf16 @@ -343,22 +301,17 @@ vocabulary weights *every step* rather than once per request. ### 2.3 Encoders — there are none, and the absence is worth stating GLM-5.2 is a **text-only** decoder: no vision tower, no audio encoder, no -multimodal scatter into `inputs_embeds`. Three consequences, because the absence -is load-bearing: - -1. **Prefill starts at `embed_tokens`.** The encoder→backbone seam that dominates - a multimodal prefill — a strict serial prefix nothing can overlap — does not - exist here. -2. **Prompt length is the only input-side variable.** No image or video token - count feeds P, so §4.4's flip-variable index is one row shorter and the - remaining rows are correspondingly better constrained. -3. **No second model is hiding off-checkpoint.** Every FLOP in a trace should map - to a node in §3, which makes an unexplained kernel block a much stronger - signal than it would be elsewhere (§6.4 row 6). - -The GLM family does ship vision variants. **They are a different checkpoint with a -different `model_type` and this graph does not model them** — -`is_glm_moe_dsa_config` declines them rather than pricing a tower it never read. +multimodal scatter into `inputs_embeds`. The absence is load-bearing three times +over. **Prefill starts at `embed_tokens`** — the encoder→backbone seam that +dominates a multimodal prefill does not exist. **Prompt length is the only +input-side variable**, so §4.4 is one row shorter and the rest better constrained. +And **no second model is hiding off-checkpoint**, so every FLOP in a trace should +map to a node in §3 — which makes an unexplained kernel block a far stronger +signal than it would be elsewhere (§6.4 row 6). + +The GLM family does ship vision variants. They are a different checkpoint with a +different `model_type`, and `is_glm_moe_dsa_config` declines them rather than +pricing a tower it never read. ### 2.4 MTP-on decode — draft and verify @@ -410,66 +363,48 @@ expert bank once per stage. §3.3 puts a number on it. ## 3. Predicted execution graph Detailed enough to put a trace next to. The 79 blocks collapse to **four -archetypes** exactly (§1) plus a prologue, an epilogue and, under MTP, a 90-node -draft chain. All figures at **B=32, S=8192, TP8/EP8, FP8 weights and KV, per -rank** unless stated. - -**The per-node tables live in Appendix A.** What stays here is the part that is -argued rather than looked up. +archetypes** (§1) plus a prologue, an epilogue and, under MTP, a 115-node draft +chain. All figures at **B=32, S=8192, TP8/EP8, FP8 weights and KV, per rank** +unless stated. **Per-node tables are in Appendix A**; what stays here is argued +rather than looked up. ### 3.1 Prologue and epilogue -The step does not begin at layer 0 or end at layer 77. Four nodes bracket it, and -two of them are on the critical path between the last layer and the sample. - -| id | operator | kernel class | shape | FLOPs | bytes | stream | bound | -| --- | ------------------- | ------------------------- | ------------------------------ | -------- | ------------- | ------ | ------ | -| D0 | `embed_tokens` | gather (index_select) | `[32] → [32,6144]` | 0 F | 0.393 MB | compute | launch | -| E0 | `rms_norm` | fused RMSNorm | `[32,6144]`, **logits rows only** | 0.6 MF | 0.786 MB | compute | launch | -| E1 | `lm_head` | GEMM (BF16), tall-skinny | `[32,6144] × [6144,19360]` | 7.61 GF | **239.5 MB** | compute | memory | -| E2 | `logits_all_gather` | collective (all-gather) | `[32,19360] → [32,154880]` fp32 | 0 F | **17.3 MB** | comm | memory | - -Three things worth naming: - -- **D0 reads what it selects, not the table.** An `index_select` over a 1.9 GB - embedding costs 393 kB at 32 rows. The resident 1.9 GB matters for the fit math - (§1), not for the step. -- **E0 runs over `logits_rows`, not over every row in the step.** At decode those - are the same number; at prefill they are 1 per prompt against a 8,192-token - chunk, and charging the chunk would make the epilogue the largest thing in a - prefill step. -- **E2 exists because E1 is vocabulary-sharded.** 17.3 MB of fp32 logits gathered - across 8 ranks before anything can be sampled — small in bytes, unavoidable in - position, and the second-most-expensive single node in a decode step after the - expert bank (19.3 µs). +The step does not begin at layer 0 or end at layer 77. + +| id | operator | kernel class | shape | FLOPs | bytes | bound | +| --- | ------------------- | ------------------------- | ------------------------------ | -------- | ------------- | ------ | +| D0 | `embed_tokens` | gather (index_select) | `[32] → [32,6144]` | 0 F | 0.393 MB | launch | +| E0 | `rms_norm` | fused RMSNorm | `[32,6144]`, **logits rows only** | 0.6 MF | 0.786 MB | launch | +| E1 | `lm_head` | GEMM (BF16), tall-skinny | `[32,6144] × [6144,19360]` | 7.61 GF | **239.5 MB** | memory | +| E2 | `logits_all_gather` | collective (all-gather) | `[32,19360] → [32,154880]` fp32 | 0 F | **17.3 MB** | memory | + +**D0 reads what it selects, not the table** — 393 kB at 32 rows; the resident +1.9 GB matters for the fit math, not the step. **E0 runs over `logits_rows`**, so +at prefill it is one row per prompt and not the chunk. **E2 exists because E1 is +vocabulary-sharded**: 17.3 MB gathered across 8 ranks before anything can be +sampled — small in bytes, unavoidable in position, and at 19.3 µs the second most +expensive single node in a decode step after the expert bank. ### 3.2 What prefill changes -**Same node ids, same order — `M` becomes `P` instead of `B`, and four nodes change -in kind rather than degree** (§2.2's table). Concretely, at P = 8,192 in one chunk -against P = 1 decode row per sequence: - -- Every projection crosses into **compute-bound**: `q_a`/`o_proj` run AI 1,403, - `q_b` 963, `kv_a` 488 — all far past the fp8 ridge of 412. -- **The indexer scan flips from memory to compute, and by two orders of - magnitude.** At decode 32 index heads score one query per sequence against a - streamed key set (AI 64, memory-bound at every context). At prefill the same keys - are read once and scored by 8,192 queries — **5.77 TF against 22 MB**, and - emphatically compute-bound. -- **The attention core's bytes and FLOPs stop moving together.** FLOPs are capped - at 2,048 keys per query either way. Bytes are not: at decode one sequence reads - its own 2,048-entry window; at prefill 8,192 queries each select a *different* - 2,048 and their union is the whole history, so the kernel streams the entire - cache once per request. **`index_topk` bounds prefill FLOPs, not prefill bytes** — - and a prefill path copied mechanically from a dense family would charge - `P × index_topk` here and understate long-context prefill traffic by C/2048. -- **The collectives change character completely.** `all_reduce` carries 688 kB at - decode (latency-bound) against **174 MB each, 27.5 GB/pass** at prefill. The EP - all-to-all goes from 5.5 MB/layer to **1.39 GB/layer** — 105.7 GB per pass, and - the single largest line in prefill at 50 % of predicted time. -- **The epilogue runs the opposite way:** at prefill `E0/E1` process the last token - of each prompt only, so `lm_head` reads 239.5 MB of weights to produce one row per - request; at decode every row is a last position. +Same node ids, same order; `M` becomes `P`, and the four nodes in §2.2's table +change kind. At P = 8,192 in one chunk: + +- **Every projection crosses into compute-bound** — `q_a`/`o_proj` at AI 1,403, + `q_b` 963, `kv_a` 488, all past the fp8 ridge of 412. +- **The indexer scan flips from memory to compute by two orders of magnitude**: + the same keys, read once and scored by 8,192 queries instead of 32 — 5.77 TF + against 22 MB. +- **The attention core's bytes and FLOPs stop moving together.** FLOPs stay capped + at 2,048 keys per query; bytes do not, because 8,192 queries each select a + *different* 2,048 and their union is the whole cache. `index_topk` bounds + prefill FLOPs, not prefill bytes — a path copied from a dense family charges + `P × index_topk` and understates long-context prefill traffic by C/2048. +- **The collectives change character.** `all_reduce` goes 688 kB → 174 MB each; + the EP all-to-all 5.5 MB/layer → **1.39 GB/layer**, 105.7 GB per pass. +- **The epilogue inverts:** `lm_head` reads 239.5 MB to produce one row per + *request*, where at decode every row is a last position. | Prefill, P=8192, C=0, TP8/EP8, per rank | value | | --- | --- | @@ -479,28 +414,23 @@ against P = 1 decode row per sequence: | whole-pass AI | **281** — below the fp8 ridge of 412, so **memory-bound overall** | | facets | compute 666 n / 89.3 ms / 33.8 % · memory 848 n / 174.6 ms / 66.1 % · launch 77 n / 0.2 ms | -> ⚠ **The prefill rows overturn the dense-model intuition, and the reversal is -> worth keeping visible.** Dense intuition says prefill is compute-bound. It is -> **not**, for two structural reasons: 256 experts × top-8 means the whole bank is -> read per layer regardless of P, and expert parallelism turns the MoE dispatch -> into a wire-bound all-to-all that no amount of arithmetic hides. -> `confidence: high for the arithmetic, medium for the conclusion` — the soft link -> is "essentially all 256 experts are hit", which assumes routing is not -> pathologically concentrated. `e_score_correction_bias` exists precisely to spread -> load, so concentration is unlikely, but it is an assumption (A5). - -**Chunk size is a multiplier on the whole MoE term.** The same 8,192 tokens: - -| chunking | bytes | floor | -| ------------------- | ----------- | ---------- | -| 1 × 8,192 | **422 GB** | 264 ms | -| 2 × 4,096 | 517 GB | 281 ms | -| 8 × 1,024 | 1,088 GB | 396 ms | -| 64 × 128 | **6,313 GB**| **1,543 ms** | - -**14.9× the bytes for identical FLOPs.** The expert bank is read per *chunk*, not per -token — so a small `--max-num-batched-tokens`, chosen to protect decode latency, -is paid for here at a rate nothing in a per-token cost model shows. +> ⚠ **Dense intuition says prefill is compute-bound. Here it is not**, for two +> structural reasons: 256 experts × top-8 reads the whole bank per layer +> regardless of P, and expert parallelism turns the dispatch into a wire-bound +> all-to-all. `confidence: high for the arithmetic, medium for the conclusion` — +> the soft link is that essentially all 256 experts are hit, which assumes routing +> is not pathologically concentrated (A5). + +**Chunk size multiplies the whole MoE term.** The same 8,192 tokens: + +| chunking | 1 × 8,192 | 2 × 4,096 | 8 × 1,024 | 64 × 128 | +| --- | --- | --- | --- | --- | +| bytes | **422 GB** | 517 GB | 1,088 GB | **6,313 GB** | +| floor | 264 ms | 281 ms | 396 ms | **1,543 ms** | + +**14.9× the bytes for identical FLOPs** — the bank is read per *chunk*. A small +`--max-num-batched-tokens`, chosen to protect decode latency, is paid for here at +a rate no per-token cost model shows. ### 3.3 MTP — the whole-step economics @@ -516,31 +446,25 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): **Cost ratio 1.70× for up to 6 tokens.** Where the extra 44.2 GB goes: -- **The verify pass, +37.8 GB.** Almost all of it is one line: expert weights go - from 163 distinct experts at 32 rows to 256 at 192 rows — **the union saturates**, - so 6× the rows costs 1.57× the expert bytes. KV read does **not** move at all - (0.43 GB either way, because it is read per *sequence*, not per row), and verify - `lm_head` does not move either (239.5 MB regardless of rows). -- **The draft chain, +5.07 GB.** And this is where GLM differs from a dense-draft - model: **the draft is 5.3 % of the MTP step, not 1–2 %**, because each of its 5 - stages draws on a full 256-expert bank. Its cost is linear in D with no - saturation to help it — 32 rows wakes ~163 experts every stage, five times over. +- **Verify, +37.8 GB**, almost all one line: the expert union saturates, so 6× the + rows costs 1.57× the expert bytes (163 → 256 distinct). KV read does **not** + move — 0.43 GB either way, read per *sequence* — and neither does `lm_head`. +- **The draft chain, +5.07 GB.** Each of 5 stages draws on a full 256-expert bank, + linear in D with no saturation to help, so **the draft is 5.3 % of the MTP step + where a dense-draft model's would be 1–2 %**. -**So ~86 % of the price of speculation is the MoE expert bank** — charged because -more rows and more stages touch more experts, not because more work is done per -token. +**~86 % of the price of speculation is the MoE expert bank** — charged because +more rows and stages touch more experts, not because more work is done per token. -**Break-even.** The step costs 1.70×; it produces up to 6 tokens instead of 1, so -acceptance α must exceed `(1.70 − 1)/5 = 0.140` to pay. Predicted throughput: +**Break-even** needs α > `(1.70 − 1)/5 = 0.140`, against 1,933 tok/s with MTP off: | α | 0.0 | 0.5 | 0.7 | 0.9 | | ---- | ----- | ----- | ----- | ----- | | tok/s| 1,137 | 3,981 | 5,118 | **6,256** | -against 1,933 tok/s with MTP off. The vendor claims the GLM-5.2 MTP layer raises -accepted length by up to 20 % over its predecessor, which puts the operating point -well past break-even — but **α is a serving observable and this graph does not -predict it.** It prices the cost and leaves the payoff to a measurement (§6.2, C4). +Z.ai claims the GLM-5.2 MTP layer raises accepted length up to 20 % over its +predecessor, which puts the operating point well past break-even — but **α is a +serving observable this graph does not predict** (§6.2, C4). ⚑ **All of this assumes the step is memory-bound.** At B ≤ 8 it is not (§4.1), and in the launch regime the draft's 115 extra launches are pure cost against a step @@ -560,17 +484,12 @@ that was never moving bytes. The sign of the MTP decision flips with batch. | S8 | KV rollback | discard rejected rows | **low** on mechanism | pointer rewind (free) or real memmove (not free) — the trace tells you which | | S9 | indexer selection handoff | the `full` layer's top-k must be visible to its 3 `shared` layers | **low** | if it round-trips the host, IndexShare costs a sync it should not — **21 per step** | -**S5 is the decode-specific one worth chasing.** Decode runs ~1,614 kernels in -~16 ms and then hands control back to a Python scheduler. If the scheduler takes -longer than the step, the GPU idles and no kernel-level work matters. - -**S7 is the one that breaks CUDA graphs.** MTP adds a per-step, host-visible, -data-dependent sequence length. A stack that captures the decode step needs two -captured shapes plus a padded accept path, or no capture at all. - -**S9 is GLM-specific and cheap to check.** IndexShare's whole value is that 57 -layers run no indexer. If the selection is passed device-side (a tensor handed -down the stack) it is free; if it is materialised through the host it costs 21 +**S5** is the decode-specific one worth chasing: ~1,614 kernels in ~16 ms, then +control returns to a Python scheduler — if the scheduler is slower than the step, +no kernel-level work matters. **S7** is the one that breaks CUDA graphs: MTP adds a +per-step, host-visible, data-dependent sequence length, so a capturing stack needs +two shapes plus a padded accept path, or no capture. **S9** is GLM-specific and +cheap: if the `full` layer's selection round-trips the host, IndexShare costs 21 syncs a step to save 57 kernels. --- @@ -578,69 +497,45 @@ syncs a step to save 57 kernels. ## 4. Execution-bound / roofline hypotheses Five labels: **compute · memory-bandwidth · communication · launch/sync/latency · -mixed or shape-dependent.** Every row names the precision its dominant kernels run -in, the peak it is bounded against, and the variable that flips it. Labels are -**against peak**; a realistic achievable fraction is never used to move a row -across a bound boundary. +mixed**. Every row names its precision, the peak it is bounded against, and the +variable that flips it. Labels are **against peak** — a realistic achievable +fraction is never used to move a row across a boundary. -**Two notations, on purpose.** §4.1 is a **node** table — decode is where cost -concentrates, so the useful question is *which nodes own the step*. §4.2 is a -**region** table — for prefill and MTP the useful question is *what would flip this -label*, which needs a flip-variable column and not a cost ranking. +Two notations on purpose: §4.1 is a **node** table (decode concentrates cost, so +the question is which nodes own the step), §4.2 a **region** table (for prefill and +MTP the question is what flips the label, which needs a flip column, not a +ranking). ### 4.1 Decode as a node table — B=32, S=8192, TP8/EP8, FP8, per rank -``` -ridge 412 F/B (fp8) · 206 (bf16) · 14 (fp32) · launch floor 2.0 µs (graph-replay) - - node kernel class bytes AI xN Σ ms bound share - ────────────────────── ─────────────────────── ──────── ──── ─── ───── ────── ──────────────────────── - moe_routed grouped GEMM fp8 ×3 771.3 MB 3.1 76 12.213 memory ████████████████████ 73.8% - attn_score_value paged decode attention 42.0 MB 12.8 79 0.690 memory █ 4.2% - moe_all_to_all collective (EP a2a) 5.5 MB — 76 0.465 comm ▊ 2.8% - rms_norm fused_add_rms_norm 1.6 MB 0.8 160 0.322 launch ▌ 1.9% - act_quant dynamic fp8 quant 0.6 MB 0.7 158 0.316 launch ▌ 1.9% - moe_router GEMM fp32 + fused gating 3.4 MB 15.0 152 0.304 launch ▌ 1.8% - attn_q_a GEMM fp8, REPLICATED 13.1 MB 61.4 79 0.216 memory ▌ 1.3% - attn_out_proj GEMM fp8, tall-skinny 13.1 MB 61.4 79 0.216 memory ▌ 1.3% - ── 8 more nodes at the 2 µs launch floor, 0.158 ms each (1.0%) ──────────────────────────── - attn_q_b · attn_kv_a · attn_kv_b · attn_qnorm_rope_insert · - tp_all_reduce_attn · tp_all_reduce_mlp · moe_shared · moe_permute · moe_combine - ─────────────────────────────────────────────────────────────────────────────────────────── - attn_index_score index scan + top-k 33.6 MB 64.0 21 0.147 memory ▏ 0.9% ← the only term that grows with S - lm_head GEMM bf16, tall-skinny 239.5 MB 31.8 2 0.100 memory ▏ 0.6% - mtp_eh_proj GEMM bf16, REPLICATED 152.2 MB 31.8 1 0.032 memory ▏ 0.2% - logits_all_gather collective (all-gather) 17.3 MB — 1 0.019 comm ▏ 0.1% - ────────────────────── ─────────────────────── ──────── ──── ─── ───── ────── ──────────────────────── - 1,614 nodes 16.551 ms/step · 1,933 tok/s @ B=32 - moe_routed alone 12.213 ms = 73.8% of the step - - facets - memory 441 nodes 14.203 ms 85.8% ← five node types - launch 1,173 nodes 2.348 ms 14.2% ← 73% of all nodes, a seventh of the time - compute 0 nodes 0.000 ms 0.0% ← the entire roofline claim, one row -``` - -**A MoE layer is 20 kernels and two of them cost anything.** `moe_routed` is 76 % -of the layer's time; `attn_score_value`, `attn_q_a`, `attn_out_proj` and the EP -all-to-all are most of the rest; the other 15 are at the launch floor. That -distribution is the model, not an artefact — and it is why every ranked hypothesis -in §5 is about either the expert bank or the launch count. - -**Three op names appear more than once per layer, deliberately.** `rms_norm` twice -(entry and post-attention, plus once in the epilogue), `act_quant` twice (ahead of -the attention GEMMs and ahead of the experts), `moe_router` twice (the `h→256` -GEMM, then the fused gating kernel that scores and selects). They are the *same -kernel name* in a trace, so they are the same op here — which is the only way the -per-op residual diff in §6 can pair them at all. Which of the two a launch belongs -to is recoverable from an NVTX range and never from a name; see -`docs/kernel_identity.md`. - -Two rows are kept **despite** being small. The second `moe_router` instance is the -fused gating kernel — **the only data-dependent shape in the graph**, and therefore -the thing that blocks CUDA-graph capture. And `attn_index_score`, because "the only -term that grows with S" is the architecture's whole payoff, and it does not stay -small: +Ridge 412 F/B (fp8) · 206 (bf16) · 14 (fp32); launch floor 2.0 µs (graph-replay). +Shapes and dtypes per node are in Appendix A.1; this is the same nodes sorted by +what they cost. + +| node | bytes | AI | ×N | Σ ms | bound | share | +| --- | ---: | ---: | ---: | ---: | --- | ---: | +| `moe_routed` | 771.3 MB | 3.1 | 76 | **12.213** | memory | **73.8 %** | +| `attn_score_value` | 42.0 MB | 12.8 | 79 | 0.690 | memory | 4.2 % | +| `moe_all_to_all` | 5.5 MB | — | 76 | 0.465 | comm | 2.8 % | +| `rms_norm` | 1.6 MB | 0.8 | 160 | 0.322 | launch | 1.9 % | +| `act_quant` | 0.6 MB | 0.7 | 158 | 0.316 | launch | 1.9 % | +| `moe_router` (GEMM + gating) | 3.4 MB | 15.0 | 152 | 0.304 | launch | 1.8 % | +| `attn_q_a` · `attn_out_proj` | 13.1 MB each | 61.4 | 79 each | 0.216 each | memory | 1.3 % each | +| 9 more nodes at the launch floor | <5 MB | — | 79 each | 0.158 each | launch | 1.0 % each | +| `attn_index_score` | 33.6 MB | 64.0 | 21 | 0.147 | memory | 0.9 % ← the only term that grows with S | +| `lm_head` · `mtp_eh_proj` · `logits_all_gather` | 239.5 / 152.2 / 17.3 MB | — | 2 / 1 / 1 | 0.100 / 0.032 / 0.019 | memory | 0.9 % total | +| **1,614 nodes** | **68.60 GB** | | | **16.551** | | **1,933 tok/s** | + +| facet | nodes | Σ ms | share | | +| --- | ---: | ---: | ---: | --- | +| memory | 441 | 14.203 | 85.8 % | five node types | +| launch | 1,173 | 2.348 | 14.2 % | 73 % of all nodes, a seventh of the time | +| compute | 0 | 0.000 | 0.0 % | the entire roofline claim, one row | + +**A MoE layer is 20 kernels and two of them cost anything** (Appendix A.1). Two +small rows are kept anyway: the second `moe_router` instance is the fused gating +kernel — **the only data-dependent shape in the graph** — and `attn_index_score` +is the only term that grows with S, which does not stay small: | context S | `attn_index_score` | share of step | step floor | | --------- | ------------------ | ------------- | ---------- | @@ -648,10 +543,9 @@ small: | 131,072 | 2.349 ms | 12.5 % | 18.753 ms | | **1,048,576** | **18.795 ms** | **53.4 %** | 35.199 ms | -**At the model's advertised context the indexer scan is the largest node in the -step**, and it is the node IndexShare already cut by 3.7×. Everything anyone says -about GLM-5.2 being "flat in context" is true of the attention *core* and false of -the step. +**At 1M the indexer scan is the largest node in the step** — and it is the node +IndexShare already cut 3.7×. "Flat in context" is true of the attention *core* and +false of the step. **The batch story, and where the labels flip:** @@ -665,11 +559,11 @@ the step. | 128 | 27.682 ms | 4,624 | 939 | 1.880 ms = 7 % | 76 | | 256 | 34.510 ms | 7,418 | 705 | 1.412 ms = 4 % | 79 | -**Below B≈16 the step is a launch-bound step wearing a memory-bound model's -clothes.** At B=1, **69 %** of the predicted floor is 1,353 kernel launches at 2 µs -— and that already assumes CUDA-graph replay. At the eager 5 µs it is 6.77 ms of -launches against a 1.24 ms memory term: **85 %**, and the whole low-batch analysis -changes sign (A4, §5 rank 3). +**Below B≈16 the step is launch-bound in memory-bound clothes.** At B=1 that +69 % already assumes CUDA-graph replay; at the eager 5 µs it is **85 %**, and the +whole low-batch analysis changes sign (A4, §5 rank 3). + + ### 4.2 Prefill and MTP — regions and what flips them @@ -700,34 +594,25 @@ at TP8/EP8, FP8. rows — but H20's much lower FP8 peak moves every prefill projection further into compute-bound, and its bandwidth moves the decode floor directly. -### 4.3 Same kernel, opposite label — the contradictions worth naming - -| Kernel | Prefill | Decode | Why the same kernel flips | -| --------------------- | ------------------------------------ | --------------------------------------------- | ---------------------------------------------------------------- | -| expert grouped GEMM | **compute** (AI 485) | **memory** (AI 3.1) | M = P vs M = B. **156× apart in AI**, same weights | -| indexer scan | **compute** (5.77 TF / 22 MB) | **memory** (AI 64) | the query count collapses; the key set does not | -| attention core | compute, **whole cache** in bytes | **launch/memory**, one 2,048-window | 8,192 queries' selections union to everything; one query's do not | -| `all_reduce` ×158 | **comm-bandwidth** (30.5 ms wire) | **comm-latency** (0.16 ms floor, 0.8 µs wire) | payload 174 MB vs 688 kB | -| EP all-to-all | **the top line** (44.5 %) | 3.0 % | payload scales with rows; the ring latency does not | -| `lm_head` | memory, 1 row per **request** | memory, **every row every step** | the epilogue is free in prefill and is not in decode | -| `moe_router` (fp32) | **compute** (11.0 % of prefill) | launch (1.0 %) | 26 GF/layer against 67 TF/s only matters when P is large | - ### 4.4 Flip-variable index -| Flip variable | Rows it flips | Direction and magnitude | -| ------------- | ------------- | ----------------------- | -| **Decode batch B** | every decode GEMM, expert hit-rate, all collectives, the whole-step label (launch below B≈16, memory above), the sign of the MTP decision | expert bytes **sub-linear** in B: 8 experts at B=1, 163 at B=32, 252 at B=128. 303 → 7,623 tok/s across 1→256 | -| **Sequence length S** | the indexer scan, and **only** the indexer scan | 0.3 % of the step at 8K → 13.0 % at 128K → **54.4 % at 1M**. The attention core does not move at all | -| **Prompt length P** | every prefill projection (memory→compute above P≈512), the router, the indexer's quadratic, all-reduce (latency→bandwidth above P≈256) | whole-pass AI 281 at P=8k | -| ⚑ **Chunked prefill / chunk size** | prefill expert GEMMs, permute, and through them the whole prefill label | 8,192 tokens in 1 chunk = **422 GB**; in 64 chunks of 128 = **6,313 GB** for identical FLOPs | -| ⚑ **CUDA-graph capture** | the whole decode step, the whole MTP step, every launch row | at B=1 it is 63 % of the floor; at 5 µs eager it is 81 %. Decides whether MTP is a 3× win or a net loss | -| **EP vs TP** | every collective row, every MoE memory row | under EP8 the a2a is **44 % of prefill** but the per-rank bank is **8× smaller**. Since the bank is ~85 % of decode DRAM, this is a **trade, not a cost** | -| ⚑ **Precision (fp8 vs bf16)** | every weight term, and whether the model fits at all | 1.79× on the decode floor (16.551 ms vs 29.574 ms) and **10.7 → 5.4 H200s** for weights | -| ⚑ **KV dtype** | the attention core, and the 1M-context footprint | 55.2 GB vs 99.9 GB per rank at 1M — the difference between fitting and not | -| ⚑ **Absorbed vs unabsorbed MLA** | `attn_kv_b` + `attn_out_proj` | drops one node and doubles the other's input width: ±2× on 2.4 % of decode, more at prefill | -| **`index_topk`** | the attention core's FLOPs in both phases; its bytes in neither | 2,048 → 4,096 doubles core FLOPs and changes no byte term | -| **Draft depth D, acceptance α** | verify expert GEMMs, whole MTP step | 1.70× cost at D=5; break-even α = 0.140; 1,137 → 6,256 tok/s across α | -| **Expert imbalance** | prefill + decode expert GEMMs, permute, the a2a, grouped-GEMM tail | skew *reduces* bytes while *increasing* tail latency and stalling every other EP rank | +§4.2's flip column, reverse-indexed to the eight variables that move more than one +row, with the magnitude each is worth: + +| Flip variable | Direction and magnitude | +| --- | --- | +| **Decode batch B** | expert bytes are **sub-linear**: 8 distinct experts at B=1, 163 at 32, 252 at 128. 253 → 7,418 tok/s across 1→256, and the whole-step label goes launch → memory at B≈16 | +| **Sequence length S** | moves the indexer scan and **nothing else**: 0.9 % of the step at 8K → 12.5 % at 128K → **53.4 % at 1M** | +| **Prompt length P** | every prefill projection (memory→compute above P≈512), the router, the indexer's quadratic, all-reduce (latency→bandwidth above P≈256) | +| ⚑ **Chunk size** | **14.9×** on prefill bytes across 1→64 chunks, for identical FLOPs | +| ⚑ **CUDA-graph capture** | 69 % of the B=1 floor, 85 % at eager 5 µs. Decides whether MTP is a 3× win or a net loss | +| ⚑ **EP vs TP** | the a2a is 45 % of prefill under EP8, but the per-rank bank is **8× smaller** — a trade, not a cost, since the bank is 85 % of decode DRAM | +| ⚑ **Precision, and KV dtype** | 1.79× on the decode floor and **10.7 → 5.4 H200s** for weights; 55 GB vs 100 GB of KV per rank at 1M | +| **D and acceptance α** | 1.70× cost at D=5, break-even α = 0.140, 1,137 → 6,256 tok/s across α | + +Two more move single rows and are named where they appear: **absorbed-vs-unabsorbed +MLA** (±2× on `attn_kv_b` + `attn_out_proj`, Q1) and **expert imbalance** (skew +*reduces* bytes while lengthening the grouped-GEMM tail and stalling EP ranks). --- @@ -749,47 +634,30 @@ graph**, to be confirmed against a capture. | **9** | **`attn_q_a` / `attn_kv_a` replication** | 2.8 % of decode is paid **in full on every rank** and TP does not reduce it | they produce the shared latent, which has nothing to split | per-rank duration of `q_a` vs `q_b` under TP8 | already sharded via DP-attention → the graph is wrong here, not the engine | | **10** | **IndexShare selection handoff (S9)** | if the top-k round-trips the host, **21 syncs/step** to save 57 kernels | the selection must reach three downstream layers | D2H count attributable to the indexer region | 0 → device-side, and IndexShare is pure win | -### 5.1 Gate check — every row maps to a GitM-observable category - -| GitM category | Rows | -| -------------------------------- | --------- | -| launch gaps | 3 | -| needless syncs | 2, 10 | -| serialised work | 1, 7 | -| stream underuse / overlap misses | 1, 7 | -| collective placement | 1, 7 | -| dispatch/combine cost | 1, 5 | -| routing imbalance | 5 | -| phase transitions | 8 | -| precision selection | 4, 6 | +### 5.1 Gate check -No row sits outside the list. +Every row above maps to a category GitM can observe and act on — launch gaps (3), +needless syncs (2, 10), serialised work and stream/overlap misses (1, 7), +collective placement (1, 7), dispatch/combine cost (1, 5), routing imbalance (5), +phase transitions (8), precision selection (4, 6). No row sits outside the list. ### 5.2 Ranks 2 and 3 are the same fork, not two independent bets -Either group sizes are resolved on the **host** (rank 2) *or* the kernel pads to -**fixed capacity** to keep static shapes (rank 3). A stack does one or the other: - -- host-resolved sizes → exact shapes, no padding, **but a sync per layer and no graph** -- fixed capacity → no sync, graph-capturable, **but all 256 experts read every step** - +Either group sizes are resolved on the **host** (rank 2) — exact shapes, no +padding, but a sync per layer and no graph — *or* the kernel pads to **fixed +capacity** (rank 3): no sync, capturable, but all 256 experts read every step. **You cannot pay both, and you cannot escape both without a device-side grouped -GEMM.** Counting D2H per step is the single cheapest measurement in this document. +GEMM.** Counting D2H per step is the cheapest measurement in this document. ### 5.3 Deliberately excluded — architectural, not recoverable -These will look alarming on a timeline and are **not** actionable: - -- **The five MTP draft gaps.** Stage `k` consumes stage `k-1`'s sampled id. A - producer→consumer edge; no scheduling closes it. -- **The accept/reject host readback (S7).** Genuinely data-dependent and genuinely - host-visible. -- **The sampling D2H (S4).** One host round-trip per step is the floor for any - autoregressive decoder. -- **The KV cache being replicated across TP ranks.** One shared MLA latent cannot - be split. That is the architecture doing what it was designed to do. -- **The indexer scan growing with context.** IndexShare already cut it 3.7×; the - remainder is the cost of selecting from an uncompressed history. +These look alarming on a timeline and are not actionable: **the five MTP draft +gaps** (stage `k` consumes stage `k-1`'s id — a producer→consumer edge no +scheduling closes), **the accept/reject readback** (S7), **the sampling D2H** (S4 — +one host round-trip per step is the floor for any autoregressive decoder), **the +KV cache replicated across TP ranks** (one shared MLA latent cannot be split), and +**the indexer scan growing with context** (IndexShare already cut it 3.7×; the +remainder is the cost of selecting from an uncompressed history). --- @@ -799,22 +667,13 @@ Assume the Nsight Systems / CUPTI trace arrives tomorrow. ### 6.1 The classification rule — *unexpected ≠ recoverable* -```mermaid -%%{init: {'theme':'neutral'}}%% -flowchart TD - Q1{"Is there a producer→consumer edge
across this gap?"} - Q1 -- yes --> A1["ARCHITECTURAL
the §2/§3 graphs exist precisely
to answer this without guessing"] - Q1 -- no --> Q2{"Does the gap scale with something
the deployment controls?
batch · chunk size · graph capture
KV dtype · D · EP degree · stream"} - Q2 -- yes --> R1["RECOVERABLE
name the knob AND the expected delta"] - Q2 -- no --> Q3{"Would the gap survive a perfect
implementation of the same model?"} - Q3 -- yes --> A2["ARCHITECTURAL
it is the model, not the stack"] - Q3 -- no --> R2["RECOVERABLE"] - - classDef arch fill:#5a2a2a,stroke:#c88,color:#fff - classDef rec fill:#24543a,stroke:#7c9,color:#fff - class A1,A2 arch - class R1,R2 rec -``` +Three questions, in order; the first that answers decides it. +**(1) Is there a producer→consumer edge across the gap?** Yes → **architectural**; +the §2/§3 graphs exist precisely to answer this without guessing. **(2) Does the +gap scale with something the deployment controls** — batch, chunk size, graph +capture, KV dtype, D, EP degree, stream assignment? Yes → **recoverable**, and name +the knob *and* the expected delta. **(3) Would the gap survive a perfect +implementation of the same model?** Yes → **architectural**; no → **recoverable**. Two worked examples, because the rule is easy to agree with and hard to apply: @@ -852,20 +711,21 @@ unobservable. ### 6.3 Instrument map — three tools, three questions -| Question shape | Instrument | What it gives | -| --- | --- | --- | -| *Where are the gaps, syncs and serialisations?* | **Nsight Systems** (`nsys`) | timeline, CUDA API calls, launch counts, `cudaMemcpyAsync` D2H, NCCL ranges, stream assignment, CPU scheduler time | -| *How many bytes did that kernel actually move?* | **Nsight Compute** (`ncu`) | per-kernel DRAM read/write, L2 traffic, achieved bandwidth, tensor-pipe activity | -| *What happened across the whole run, cheaply?* | **CUPTI activity records** | kernel/memcpy/NCCL counts and durations without `ncu`'s serialising replay — and GitM's own `spec_decode` bucket already separates the MTP scaffolding kernels from ordinary sampling | +**`nsys`** answers *where are the gaps, syncs and serialisations* — timeline, API +calls, launch counts, D2H, NCCL ranges, stream assignment, CPU scheduler time. +**`ncu`** answers *how many bytes did that kernel move* — per-kernel DRAM, L2, +achieved bandwidth, tensor-pipe activity. **CUPTI activity records** answer *what +happened across the run, cheaply*, without `ncu`'s serialising replay — and GitM's +`spec_decode` bucket already separates the MTP scaffolding from ordinary sampling. -Counters: `dram__bytes_read.sum`, `dram__bytes_write.sum`, `lts__t_bytes.sum` (L2 — +Counters: `dram__bytes_read.sum`, `dram__bytes_write.sum`, `lts__t_bytes.sum` (L2, the escape hatch for the expert-bank claim), `gpu__time_duration.sum`, tensor-pipe -active %. **Exact names vary by architecture and `ncu` version.** +active %. Exact names vary by architecture and `ncu` version. -**Two cautions.** `ncu` serialises kernels and destroys exactly the overlap -information ranks 1 and 7 depend on — **profile bytes with `ncu`, overlap with -`nsys`.** And DRAM counters measure traffic that missed L2; a low reading is -ambiguous between "did not read it" and "read it from cache". +**Two cautions.** `ncu` serialises kernels and destroys the overlap information +ranks 1 and 7 depend on — profile bytes with `ncu`, overlap with `nsys`. And DRAM +counters miss L2: a low reading is ambiguous between "did not read it" and "read +it from cache". ### 6.4 Trace triage — what to measure, in order @@ -899,20 +759,17 @@ right; §7.1 is what GLM-5.2 broke; §7.2 is the code that now exists. ### 7.0 What the planner already gets right -Listed first because several findings here turn out to be things GitM already -models, and proposing them as gaps would waste the pilot's time. - -| Already in the IR | Where | The step that independently derived it | -| --- | --- | --- | -| `positions` vs `sequences` — multi-row verify reads the cache **once per sequence** | `_emit_layer` docstring | §3.3's central MTP result | -| Coupon-collector distinct-expert traffic: FLOPs ∝ `positions·top_k`, bytes ∝ *distinct* experts | `roofline.distinct_experts` | the decode sub-linearity finding, and MTP's break-even | -| EP vs TP as "a collective trade, not a memory trade", with `ep_imbalance` **calibrated from a trace, not predicted** | `roofline.ShardingConfig` | §4.4 | -| A three-way bound — compute / memory / **launch** — via `serial_launches` | `roofline.roofline` | §4.1's launch facet | -| FP8 block-scale overhead: `weight_bytes("fp8") = 1.000244` | `roofline` | the constants section | -| Self-reported model debt: `has_fallback_peaks`, `has_unpriced_collectives` | `graph.Graph` | — | -| Per-checkpoint `provenance: verified / estimated / unmodelled` | `models/*.yaml` | mirrors this note's source/confidence columns | -| Explicit per-layer schedules preferred over a modulo rule | `glm_graph.indexer_kind` | §1 — and GLM has *two* schedules that do not align | -| Prefill as `rows = positions + prefill_tokens` with `logits_rows` for the epilogue | `hybrid_graph` | §3.2 | +Listed first because several findings here turned out to be things GitM already +models, and proposing them as gaps would waste the pilot's time: `positions` vs +`sequences` (a multi-row verify reads the cache **once per sequence** — §3.3's +central MTP result, and the planner had it first); the coupon-collector +distinct-expert term in `roofline.distinct_experts`; EP-vs-TP as a collective +trade with `ep_imbalance` **calibrated from a trace, not predicted**; the +three-way compute/memory/**launch** bound via `serial_launches`; fp8 block-scale +overhead at 1.000244 bytes/weight; `has_fallback_peaks` / `has_unpriced_collectives` +as self-reported debt; per-checkpoint `provenance`; explicit per-layer schedules +over modulo rules; and prefill as `rows = positions + prefill_tokens` with +`logits_rows` for the epilogue. **This is a planner built by someone who has been wrong about these before.** The gaps below are narrower because of it. @@ -950,15 +807,12 @@ should land where both families' coverage can be checked at once. ### 7.2 The one that needed more than a table row **G2 must not ship as a copy of another family's prefill path.** Aliasing -`BatchConfig.attention_qk_pairs` into the DSA core is a two-line change that -produces a complete, plausible graph — and it would be wrong in *both* directions -at once: the core's FLOPs over-charged by the ratio of context to `index_topk`, -and its bytes under-charged by the same ratio, because the two errors come from the -same false premise (that a query's selection is the whole cache, or that the -chunk's selection is one query's). The two mistakes partly cancel in the total, -which is exactly what makes them survivable. **A prefill path that is wrong in a -self-cancelling way is worse than no prefill path**, and it is the version that -will get suggested — hence four helpers with four docstrings rather than one alias. +`BatchConfig.attention_qk_pairs` into the DSA core is a two-line change producing a +complete, plausible graph — and wrong in *both* directions at once: the core's +FLOPs over-charged by context ÷ `index_topk`, its bytes under-charged by the same +ratio, both from the same false premise. **The two mistakes partly cancel in the +total, which is what makes them survivable** — and a prefill path that is wrong in +a self-cancelling way is worse than none. Hence four helpers rather than one alias. ### 7.3 What this branch changed, in kind @@ -1046,16 +900,14 @@ KV per rank on top of a 96 GB weight share. ## Appendix A — Predicted node tables Trace-day reference for §3. **B=32, S=8192, TP8/EP8, FP8 weights and KV, per -rank.** The 79 blocks are exactly `Ld,f` ×3 + `Ls,f` ×18 + `Ls,sh` ×57 + `Lmtp` ×1; -A.2–A.4 are stated as **deltas** from A.1, because everything not listed is -byte-for-byte identical. +rank.** The 79 blocks are `Ld,f` ×3 + `Ls,f` ×18 + `Ls,sh` ×57 + `Lmtp` ×1; A.2–A.4 +are **deltas** from A.1, since everything unlisted is byte-for-byte identical. -**Two columns are omitted rather than repeated.** Every node runs on the compute -stream except the collectives, which the graph marks `expected_stream_id=1` — -a declaration, not yet a check (§7.1 G10). Confidence is **high** -throughout — these rows are read from `config.json` and the tensor index — except -the collectives (**medium**, a TP/EP convention) and the S1 histogram readback -(**low**, a hypothesis about a stack nobody has opened). +Two columns are omitted rather than repeated. Every node runs on the compute +stream except the collectives (`expected_stream_id=1` — a declaration, not yet a +check, §7.1 G10). Confidence is **high** throughout, these rows being read from +`config.json` and the tensor index, except the collectives (**medium**, a TP/EP +convention) and the S1 histogram readback (**low**, a hypothesis). ### A.1 — Archetype `Ls,sh`, 57 layers (shared indexer + MoE) @@ -1063,11 +915,10 @@ Read straight off the graph, in issue order — every row is a `PredictedNode` a this shape, and `tests/test_glm_graph.py::test_layer_lowers_to_the_documented_node_sequence` pins this exact sequence so the code cannot drift from the table. -**Three op names repeat, deliberately** (`rms_norm` ×2, `act_quant` ×2, -`moe_router` ×2). They are the same kernel name in a trace, so they are the same -op here; a private name per site would emit a node no capture could pair against -while the real kernel filed as unmodeled. Which instance a launch belongs to is an -NVTX question, not a name question — `docs/kernel_identity.md`. +**Three op names repeat** (`rms_norm` ×2, `act_quant` ×2, `moe_router` ×2): one +kernel name in a trace is one op here, or the node goes unpaired and the kernel +files as unmodeled. Which instance a launch belongs to is an NVTX question, never +a name question — `docs/kernel_identity.md`. | id | operator | kernel class | FLOPs | bytes | dtype | t (µs) | bound | |---|---|---|---|---|---|---|---| @@ -1095,20 +946,17 @@ NVTX question, not a name question — `docs/kernel_identity.md`. **Σ per layer: 5.6 GF, 874.9 MB, 0.211 ms.** -**Twenty kernels, and two of them cost anything.** `.17` alone is **88 % of the -layer's bytes and 76 % of its time**; `.7`, `.3`, `.9` and `.19` are most of the -remainder; the other **15 nodes sit at the 2 µs launch floor** — 0.030 ms per layer -of pure launch, 14 % of it. Those 15 rows are why §4.1's launch facet exists, and -folding any of them into the GEMM it precedes would report this layer as more -memory-bound than it is. - -Two folds are deliberate and named, because they are the places a reader will -expect a row and not find one. **SwiGLU is inside `.17`** and inside `mlp_gate_up` -on the dense layers: `silu_and_mul` is already one of `mlp_gate_up`'s needles in -`deviation._OP_RULES`, so a separate node would be a prediction with no observed -kernel to pair against. **The residual adds are inside `.1` and `.11`**: vLLM runs -`RMSNorm.forward(x, residual)` as one `fused_add_rms_norm` kernel, so a separate -residual node would predict a launch that never happens. +**Twenty kernels, two of which cost anything.** `.17` is **88 % of the layer's +bytes and 76 % of its time**; `.7`, `.3`, `.9` and `.19` are most of the rest; the +other **15 sit at the 2 µs launch floor** — 0.030 ms per layer, 14 % of it. That is +where §4.1's launch facet comes from, and folding any of them into the GEMM it +precedes would report the layer as more memory-bound than it is. + +Two folds are deliberate, because they are where a reader will expect a row and +not find one. **SwiGLU is inside `.17`** (and inside `mlp_gate_up` on the dense +layers): `silu_and_mul` is already `mlp_gate_up`'s needle, so a separate node +would have no kernel to pair against. **The residual adds are inside `.1` and +`.11`**: vLLM runs `RMSNorm.forward(x, residual)` as one `fused_add_rms_norm`. ### A.2 — Archetype `Ls,f`, 18 layers (full indexer + MoE) @@ -1119,14 +967,11 @@ residual node would predict a launch that never happens. | .6a | `attn_index_proj` | GEMM ×3 — `wq_b` (2048→4096), `wk` (6144→128), `weights_proj` (6144→32), **replicated** | 599.8 MF | 19.937 MB | **BF16** | 4.15 | memory | | .6b | `attn_index_score` | 32 heads score the whole history + top-2048 | 2,147.5 MF | 33.563 MB | **BF16 math, FP8 keys** | 6.99 | memory | -`.6a` is BF16 because the indexer is named in `modules_to_not_convert` — at FP8 it -would price at 10.0 MB, and this is the layer type whose cost grows with context. - -**`.6b` carries two dtypes and they answer different questions.** Its *bytes* are -governed by how the keys are stored (fp8, under the vendor recipe's fp8 cache); -its *FLOPs* by what the indexer computes in, which is the projection's bf16. At -decode the distinction is invisible — the node is memory-bound at every context — -and at prefill it is the difference between 1.2 % and 2.2 % of the step. +`.6a` is BF16 because the indexer is named in `modules_to_not_convert`; at FP8 it +would price at 10.0 MB. **`.6b` carries two dtypes answering different questions** — +its *bytes* follow how the keys are stored (fp8), its *FLOPs* what the indexer +computes in (bf16). Invisible at decode, where the node is memory-bound at every +context; at prefill it is the difference between 1.2 % and 2.2 % of the step. **`.6b` is also the only node in the model that grows with S**, and it does not stay small: @@ -1173,20 +1018,14 @@ for the sequence, not for the verify rows. **Σ per stage: 18.0 GF, 1,268.2 MB, 0.297 ms — ×5 = 6.34 GB, 1.483 ms.** -Three things in that table are the whole §3.3 argument: - -- **M.3–M.22 is a full MoE block.** The MTP module carries its own 256-expert - `mlp.experts.*` bank in the checkpoint, so 69 % of a draft stage's bytes are - expert weights it re-reads every stage. There is no saturation to help: 32 rows - wakes ~163 experts, five times over. -- **M.1, M.2 and M.23 are BF16**, all named in `modules_to_not_convert`, and - together they are **31 %** of the stage. `eh_proj` is replicated per rank (Q10); - the vocabulary projection is shared with the backbone and gets no cheaper for - being a draft. -- **What is absent:** no `attn_index_proj`, no `attn_index_score`. The MTP block - has no indexer tensors in the weight map, which is - `index_share_for_mtp_iteration: true` made visible — the draft inherits the - top-2048 selection the backbone already paid for. +Three things there are the whole §3.3 argument. **M.3–M.22 is a full MoE block** — +the module carries its own 256-expert bank, so 69 % of a stage's bytes are expert +weights re-read every stage, with no saturation to help (32 rows wakes ~163 +experts, five times over). **M.1, M.2 and M.23 are BF16**, all in +`modules_to_not_convert`, and together **31 %** of the stage. And **what is absent** +— no `attn_index_proj`, no `attn_index_score` — is +`index_share_for_mtp_iteration: true` made visible: the draft inherits the +selection the backbone already paid for. ### A.5 — Node budget for the whole step diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 43dc760..e6e2295 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -65,24 +65,12 @@ BatchConfig, HardwareSpec, ShardingConfig, + _canon_dtype, distinct_experts, roofline, weight_bytes, ) - -def _canon(dtype: str) -> str: - """Canonical dtype name, so ``"e4m3"`` and ``"fp8"`` answer the same question.""" - d = dtype.lower() - if d in ("fp8", "e4m3", "e5m2"): - return "fp8" - if d in ("bf16", "fp16", "float16", "half"): - return "fp16" - if d in ("fp32", "float32", "tf32"): - return "fp32" - return d - - FULL_INDEXER = "full" SHARED_INDEXER = "shared" DENSE_MLP = "dense" @@ -250,29 +238,6 @@ def kv_entry_bytes(spec: GlmMoeDsaModelSpec) -> float: return latent + rope -def effective_kv_tokens(spec: GlmMoeDsaModelSpec, kv_len: int) -> int: - """KV positions the attention *core* reads — bounded by the indexer's top-k. - - Every GLM layer runs DSA, so the core reads at most ``index_topk`` selected - positions regardless of how long the context grows. This is why attention is - flat in context on this architecture and the term that still grows is the - indexer scan, not the core. - """ - if kv_len <= 0: - return 0 - return min(kv_len, spec.index_topk) - - -def index_candidates(spec: GlmMoeDsaModelSpec, kv_len: int) -> int: - """Positions the indexer must score — the whole (uncompressed) history. - - GLM does not compress before selecting, so the indexer scores every past - token. This is the only term in the attention path that grows with context, - and it is paid on ``full`` layers alone. - """ - return max(0, kv_len) - - def _capped_prefix_sum(context: int, tokens: int, cap: int) -> float: """``sum(min(context + i, cap) for i in 1..tokens)`` in closed form. @@ -552,7 +517,7 @@ def add_act_quant(op: str, elems: float, gemm_op: str) -> None: checkpoint there is nothing to quantise and the kernel does not exist, which is the sort of difference a single model-wide dtype cannot express. """ - if _canon(spec.dtype_for(gemm_op, wd)) != "fp8": + if _canon_dtype(spec.dtype_for(gemm_op, wd)) != "fp8": return # Read the bf16 activation, write the fp8 one plus its scales. add(op, 2.0 * rows * elems, From 99512c777ad33726baac43bb81037a52adca663d Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 15:22:25 -0700 Subject: [PATCH 11/35] Fix lint, and the MTP break-even the review caught MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ruff UP038 on the isinstance tuple — the rest of the file already used X | Y, this one was missed. The review's substantive catch: break-even alpha was computed as (1.70-1)/D = 0.140, which follows BatchConfig.tokens_per_step's 1 + D*alpha. That is right for independent draws and wrong for speculative decoding, where the verifier accepts a prefix — expected accepted length is sum(alpha^i), not 1 + D*alpha. At D=5 the linear form overstates accepted tokens by 1.8x at alpha=0.5, and real break-even is 0.415, not 0.140. §3.3 now gives both rows and says which to read; propagated to §4.2, §4.3 and the open-questions table. Not changing BatchConfig — that convention is shared with every family. Also from the review: renumbered §4.4 to §4.3 (§4.3 was deleted as a duplicate and the gap was left), stated the node arithmetic explicitly (1,614 = 1,591 + 23, 1,706 = 1,591 + 115, and 192 rows = B x (1+D)), moved G11 into the gap table instead of leaving it as prose after it, promoted the expert-parallel question to Q1 and the prefill-tile question to Q3, and noted the reproduce commands are branch-relative. Declined the suggestion to add an ep_imbalance uncertainty range — it would mean inventing measurements we do not have, and A8 already says it is declared rather than fitted. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 83 ++++++++++++++++++++----------------- gitm/planner/glm_graph.py | 2 +- 2 files changed, 46 insertions(+), 39 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 90a7901..78cce84 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -8,7 +8,8 @@ Built from the model repos' own files — `config.json` and shape. **No traces.** Every number is a roofline floor at vendor peak: a lower bound on time, not a target. -Reproduce any figure here: +Reproduce any figure here (against this branch — the planner is actively changing, +so check `git rev-parse HEAD` matches if a number disagrees): ```bash gitm plan glm-5.2-fp8 --gpu H200 --batch 32 --kv-len 8192 --tp 8 --ep 8 @@ -444,6 +445,12 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): | verify, 192 rows | 1,591 | 106.4 GB | 26.652 ms | | **MTP step total** | **1,706** | **112.8 GB** | **28.135 ms** | +The node counts reconcile as `1,614 = 1,591 + 23` and `1,706 = 1,591 + 115`: the +backbone is the same 1,591 nodes either way, and what changes is the draft region +— one stage (23 nodes) at D=0, five (115) at D=5. Nothing is double-counted. +"192 rows" is `B × (1 + D)` = 32 × 6: the verify pass is the backbone at 1+D rows, +not a different batch size. + **Cost ratio 1.70× for up to 6 tokens.** Where the extra 44.2 GB goes: - **Verify, +37.8 GB**, almost all one line: the expert union saturates, so 6× the @@ -456,15 +463,27 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): **~86 % of the price of speculation is the MoE expert bank** — charged because more rows and stages touch more experts, not because more work is done per token. -**Break-even** needs α > `(1.70 − 1)/5 = 0.140`, against 1,933 tok/s with MTP off: +**Break-even, and a caveat that moves it by 3×.** `BatchConfig.tokens_per_step` +counts accepted tokens as `1 + D·α` — the repo-wide convention, used by every +family. That is right for independent draws and wrong for speculative decoding, +where the verifier accepts a *prefix*: token *k* only counts if 1…*k*−1 were also +accepted, so expected accepted length is `Σ αⁱ` for *i* = 0…D. The difference is +not small at D=5. + +| α | 0.0 | 0.5 | 0.7 | 0.9 | break-even | +| --- | --- | --- | --- | --- | --- | +| linear `1+Dα` (what the graph reports) | 1,137 | 3,981 | 5,118 | **6,256** | α > 0.140 | +| prefix chain `Σ αⁱ` (what a verifier does) | 1,137 | 2,239 | 3,345 | **5,329** | **α > 0.415** | -| α | 0.0 | 0.5 | 0.7 | 0.9 | -| ---- | ----- | ----- | ----- | ----- | -| tok/s| 1,137 | 3,981 | 5,118 | **6,256** | +against 1,933 tok/s with MTP off. **Read the second row.** The first is what +`--spec-tokens` prints today, and it overstates throughput by up to 1.8× at α=0.5; +the honest break-even on this model is nearer **0.42 than 0.14**. Fixing the +convention is a change to shared `BatchConfig` semantics affecting every family, +so it is stated here rather than made here. Z.ai claims the GLM-5.2 MTP layer raises accepted length up to 20 % over its -predecessor, which puts the operating point well past break-even — but **α is a -serving observable this graph does not predict** (§6.2, C4). +predecessor, which likely clears even the chained bar — but **α is a serving +observable this graph does not predict** (§6.2, C4). ⚑ **All of this assumes the step is memory-bound.** At B ≤ 8 it is not (§4.1), and in the launch regime the draft's 115 extra launches are pure cost against a step @@ -504,7 +523,7 @@ fraction is never used to move a row across a boundary. Two notations on purpose: §4.1 is a **node** table (decode concentrates cost, so the question is which nodes own the step), §4.2 a **region** table (for prefill and MTP the question is what flips the label, which needs a flip column, not a -ranking). +ranking). §4.3 reverse-indexes that column. ### 4.1 Decode as a node table — B=32, S=8192, TP8/EP8, FP8, per rank @@ -588,13 +607,13 @@ at TP8/EP8, FP8. | **MTP** | draft `eh_proj` ×5 | memory | `[12288,6144]` BF16, **replicated per rank** | **BF16** — in `modules_to_not_convert` | whether it is TP-sharded | | **MTP** | verify attention | **memory, unchanged** | 0.43 GB — read **per sequence, not per row**; 1+D rows share one block table | FP8 KV | seq length; explicitly *not* D | | **MTP** | accept/reject + KV rollback | launch + **host sync** | tiny tensors, but a data-dependent host-visible seq length (S7) | n/a | pointer rewind vs memmove | -| **MTP** | **whole MTP step** | ⚑ **memory above B≈16, launch below — and the two regimes disagree about whether MTP helps** | **1.70×** cost for ≤6 tokens at B=32; break-even α = **0.140** | mixed | **graph capture**; batch; α; D | +| **MTP** | **whole MTP step** | ⚑ **memory above B≈16, launch below — and the two regimes disagree about whether MTP helps** | **1.70×** cost for ≤6 tokens at B=32; break-even α = **0.415** on a prefix chain (§3.3) | mixed | **graph capture**; batch; α; D | **Hardware sensitivity:** nothing flips between H200 SXM and H20 on the compute rows — but H20's much lower FP8 peak moves every prefill projection further into compute-bound, and its bandwidth moves the decode floor directly. -### 4.4 Flip-variable index +### 4.3 Flip-variable index §4.2's flip column, reverse-indexed to the eight variables that move more than one row, with the magnitude each is worth: @@ -608,7 +627,7 @@ row, with the magnitude each is worth: | ⚑ **CUDA-graph capture** | 69 % of the B=1 floor, 85 % at eager 5 µs. Decides whether MTP is a 3× win or a net loss | | ⚑ **EP vs TP** | the a2a is 45 % of prefill under EP8, but the per-rank bank is **8× smaller** — a trade, not a cost, since the bank is 85 % of decode DRAM | | ⚑ **Precision, and KV dtype** | 1.79× on the decode floor and **10.7 → 5.4 H200s** for weights; 55 GB vs 100 GB of KV per rank at 1M | -| **D and acceptance α** | 1.70× cost at D=5, break-even α = 0.140, 1,137 → 6,256 tok/s across α | +| **D and acceptance α** | 1.70× cost at D=5; break-even α = **0.415** on a prefix chain, 2,239 → 5,329 tok/s across α (§3.3 — the linear convention the graph prints says 0.140, and overstates by up to 1.8×) | Two more move single rows and are named where they appear: **absorbed-vs-unabsorbed MLA** (±2× on `attn_kv_b` + `attn_out_proj`, Q1) and **expert imbalance** (skew @@ -787,23 +806,9 @@ gaps below are narrower because of it. | **G7** | **An MTP chain D stages deep, each with its own vocabulary projection** | The graph emitted **one** draft block and **one** `lm_head` for what the vendor recipe runs **five** deep. `lm_head` is 19 % of the draft's bytes, so a D-deep chain was understated by ~5× on its largest term | a stage loop in `predict_glm_graph` driven by `BatchConfig.speculative_tokens`, with `mtp_eh_proj` and an `lm_head` per stage; `--spec-tokens` on the CLI | **yes** | | **G8** | **A graph that is only its GEMMs, priced against a bound it cannot express** | The family emitted 16 nodes per layer where a layer lowers to ~20 kernels, and the seven missing ones were all pointwise: the norms, the dynamic fp8 activation scaling, the fused gating, the prologue gather and the epilogue's all-gather. Every one is a rounding error in bytes and **a full kernel launch in time** — so at B=1 the graph reported a step as memory-bound that is 69 % launches. A roofline with a launch bound and a graph with no launches in it cannot both be right | Emit them. `_pointwise`, `add_rms_norm` (with the residual fused in, as vLLM runs it), `add_act_quant` gated on the consuming GEMM actually being fp8, plus `embed_tokens` / `rms_norm` / `logits_all_gather` around the stack. Node names constrained by G9 | **yes** | | **G9** | **Op names a capture can actually pair against** | G8's new nodes needed names, and `deviation.classify_op` is a *name guess* (`docs/kernel_identity.md`): a name it cannot classify leaves the predicted node permanently unmatched **and** the real kernel filed as unmodeled — two errors in opposite directions, in the diff the family exists to support. Three norm sites are one kernel name; `silu_and_mul` was already claimed by `mlp_gate_up`; `moe_align`/`topk_softmax` were already claimed by `moe_router`, a decision the dense-MoE and hybrid families depend on | Follow the canonical names rather than redefine them: one `rms_norm` op for all three sites, SwiGLU folded back into the GEMM that owns its needle, gating emitted as a second `moe_router` instance. Then `_OP_RULES` gains only what is genuinely new and unclaimed — `rms_norm`, `act_quant`, `embed_tokens`, `moe_permute`/`moe_combine`, `attn_index_proj`, `attn_kv_b`, `mtp_eh_proj`. A test asserts every op the graph emits resolves | **yes** | +| **G11** | **An intervention vocabulary that can name the expert term** | `kernels/library.yaml` scopes every lever with `applies_to_kernels`, drawn from a canonical op list that is `qkv_proj · attn_score_value · attn_out_proj · mlp_gate_up · mlp_down · lm_head`. **`moe_routed` and `moe_shared` are not in it**, so the two entries meant to target expert traffic scope to `[mlp_gate_up, mlp_down]` — true of a dense FFN, false of either MoE family. §5 rank 5 aims levers at **74 % of a decode step** through tooling that cannot match it | Add the two ops to the vocabulary and re-scope those entries. **Pre-existing and not GLM-specific** — `moe_graph.py` emits the same names, so DeepSeek-V4 has it identically | **no** — the fix is a shared-vocabulary change and should land where both families' coverage can be checked at once | | **G10** | **Which adjacent nodes may overlap and which may not** | `Graph.total_pred_s` is a sum, not a DAG. GLM puts both kinds of serialisation in one step — the draft chain is genuinely serial, the 158 collectives are not — and **both appear as the same positive residual today**. §6.1 is unanswerable without telling them apart | *Not shipped.* It is a cross-family IR change and it is already sequenced on the roadmap. What this note adds is the requirement. `expected_stream_id=1` is set on collectives, but note that **nothing reads it today** — `optimizer/monitor.py` tests overlap using the *observed* kernel's stream, so the predicted field is carried by the IR and consumed by no one. It is a hook for the invariant in `docs/invariants.md` §3, not a wiring of it | **no — deliberately** | -**G11 — found, not fixed: the intervention library has no vocabulary for the -expert term.** `gitm/kernels/library.yaml` scopes each lever with -`applies_to_kernels`, drawn from a canonical op list (`_decode_step_ops`) that is -`qkv_proj · attn_score_value · attn_out_proj · mlp_gate_up · mlp_down · lm_head`. -**`moe_routed` and `moe_shared` are not in it**, and the two entries that mean to -target expert traffic scope themselves to `[mlp_gate_up, mlp_down]` with the -comment "the routed-expert GEMMs" — which is true of a dense FFN and not of this -graph. So §5 rank 5, the lever list on the node that is **74 % of a decode step**, -cannot be matched by the tooling that is supposed to act on it. - -This is **pre-existing and not GLM-specific** — `moe_graph.py` emits the same op -names, so DeepSeek-V4 has it identically — which is why it is reported here rather -than fixed on this branch: the fix is a change to a shared vocabulary, and it -should land where both families' coverage can be checked at once. - ### 7.2 The one that needed more than a table row **G2 must not ship as a copy of another family's prefill path.** Aliasing @@ -845,17 +850,19 @@ graph change. The commands at the top of this note regenerate any of them. | # | Question | What it changes | How to resolve | |---|---|---|---| -| **Q1** | Does the engine run **absorbed** MLA at decode? | drops `attn_kv_b` and **doubles** `attn_out_proj`'s input width (16384→32768). ±2× on the #4 and #8 lines | serving image / C6 | -| **Q2** | Is the decode step **CUDA-graph captured**? | at B=1 the difference between 1.24 ms and 3.30 ms per step, and it decides whether MTP is a 3× win or a net loss | engine config + `--cuda-graph-trace=node` | -| **Q3** | Is the **router GEMM** fp32, or only its accumulation? | **11.0 % of prefill.** At bf16 the row shrinks ~15× | the engine's MoE gate implementation | -| **Q4** | Is the grouped GEMM **device-sized or host-sized**? | decides **rank 2 *or* rank 3** — the fork in §5.2 | trace D2H count, or the kernel source | -| **Q5** | Is the **EP dispatch** bf16 or fp8? | **half of 105.7 GB** at prefill — the largest single recoverable number here | serving image / C6 | -| **Q6** | Is the **IndexShare selection** passed device-side? | 21 syncs/step if not (S9) | trace D2H attribution | -| **Q7** | Chunked prefill on, at what chunk size? | **up to 14.9× on prefill bytes** | engine launch args (C6) | -| **Q8** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is 0.140; the range 0.5→0.9 is 3,981→6,256 tok/s | engine metrics (C4) — **not predictable from a config** | -| **Q9** | Are the **index keys** cached in fp8 or bf16? | 2× on 90.2 GB/step at 1M context | C6 / trace | -| **Q10** | Is `eh_proj` **TP-sharded** in the MTP block? | 151 MB → 19 MB per rank per draft stage | serving image | -| **Q11** | Does the prefill attention kernel read the selected KV **once per request** or once per query tile? | up to 64× on the prefill core's bytes — the graph takes the optimistic floor | `dram__bytes_read.sum` on the prefill core | +| **Q1** | Is expert parallelism actually on? | **Rank 1 exists or it does not.** `--enable-expert-parallel` is absent from the vendor recipe; without it the `moe_all_to_all` rows disappear (44 % of prefill) and the per-rank expert bank doubles instead. Not a refinement — a different graph | engine launch args (C6); capture C5 | +| **Q2** | Does the engine run **absorbed** MLA at decode? | drops `attn_kv_b` and **doubles** `attn_out_proj`'s input width (16384→32768). ±2× on the #4 and #8 lines | serving image / C6 | +| **Q3** | Does the prefill attention kernel read the selected KV **once per request**, or once per query tile? | **up to 64× on the prefill core's bytes.** The graph takes the optimistic floor (A9), so a tiled kernel would move the prefill roofline conclusion, not just a row | `dram__bytes_read.sum` on the prefill core | +| **Q4** | Is the decode step **CUDA-graph captured**? | at B=1 the difference between 1.24 ms and 3.30 ms per step, and it decides whether MTP is a 3× win or a net loss | engine config + `--cuda-graph-trace=node` | +| **Q5** | Is the **router GEMM** fp32, or only its accumulation? | **11.0 % of prefill.** At bf16 the row shrinks ~15× | the engine's MoE gate implementation | +| **Q6** | Is the grouped GEMM **device-sized or host-sized**? | decides **rank 2 *or* rank 3** — the fork in §5.2 | trace D2H count, or the kernel source | +| **Q7** | Is the **EP dispatch** bf16 or fp8? | **half of 105.7 GB** at prefill — the largest single recoverable number here | serving image / C6 | +| **Q8** | Is the **IndexShare selection** passed device-side? | 21 syncs/step if not (S9) | trace D2H attribution | +| **Q9** | Chunked prefill on, at what chunk size? | **up to 14.9× on prefill bytes** | engine launch args (C6) | +| **Q10** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is **0.415** on a prefix chain (0.140 under the linear convention the graph prints — §3.3); 0.5→0.9 is 2,239→5,329 tok/s | engine metrics (C4) — **not predictable from a config** | +| **Q11** | Are the **index keys** cached in fp8 or bf16? | 2× on 90.2 GB/step at 1M context | C6 / trace | +| **Q12** | Is `eh_proj` **TP-sharded** in the MTP block? | 151 MB → 19 MB per rank per draft stage | serving image | + ### 8.2 Assumptions in force @@ -869,7 +876,7 @@ graph change. The commands at the top of this note regenerate any of them. | **A6** | The serving path uses a grouped GEMM, not a per-expert loop | Architecture rule; the reference implementation is the *semantics*, not the execution | a per-expert kernel launch pattern in the trace | | **A7** | 158 collectives per step (2 per layer × 79) | TP convention, now modelled explicitly (G6) | NCCL kernel count per step | | **A8** | `ep_imbalance = 1.0` | **Declared, not fitted** — it is trace-calibrated by design and there are no traces | any measured skew | -| **A9** | The prefill attention core streams the selected cache **once per request** | An optimistic floor (Q11). A tiled kernel re-reads per query block | `dram__bytes_read.sum` on the prefill core | +| **A9** | The prefill attention core streams the selected cache **once per request** | An optimistic floor (Q3). A tiled kernel re-reads per query block | `dram__bytes_read.sum` on the prefill core | | **A10** | The exact kernel names, everywhere | `confidence: none` throughout. The *class* is justified; the implementation is not knowable without the serving image | — | --- diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index e6e2295..47f6dd1 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -1021,7 +1021,7 @@ def _op_dtype_overrides( found: dict[str, str] = {} skipped = q.get("modules_to_not_convert") or q.get("ignored_layers") or [] - if isinstance(skipped, (list, tuple)) and weight_dtype != model_dtype: + if isinstance(skipped, list | tuple) and weight_dtype != model_dtype: for tensor in skipped: name = str(tensor).lower() for needle, op in _UNQUANTISED_OPS: From 2a010fb6f031fc1dbe2b247937c497c90e05d85e Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 15:25:34 -0700 Subject: [PATCH 12/35] Warn in gitm plan that a speculative token rate is a ceiling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Best point in the second review: the note documents that tokens_per_step's 1 + D*alpha overstates a speculative step, then ships a CLI that prints the overstated number with no label. Anyone running `gitm plan --spec-tokens 5` and not reading the note gets 0.140 as break-even when it is 0.415. The plan table now says so when D > 0, and quantifies it when an acceptance rate is given (new --acceptance-rate, report-only — it cannot move the step floor). Still not changing BatchConfig: the convention is shared with every family. Doc consistency, also from the review: - §4.2's prefill attention row asserted "whole cache once per request" as fact while A9 called it an optimistic floor. The row and the §2.1 diagram now carry the caveat, so a tiled-kernel measurement reads as a known branch and not an anomaly. - Labelled the verify row "backbone only" — 1,591 excludes the draft, and the reviewer read it twice as though it might not. - The headline 1,933 tok/s carries an [A8] marker; ep_imbalance = 1.0 propagates through every MoE byte term and was silent on the summary. - §6 gets G10 as a stated precondition: the classification rule needs serial-vs-parallelisable, and nothing reads expected_stream_id today. - Chunk size: gave the derivable rule (bank costs 95.7 GB x ceil(P/C), so bytes scale as 1/C) instead of inventing a recommended value. Declined: pinning a commit hash at doc-generation time (the note is hand-written, the branch is the pin), an ep_imbalance uncertainty range (would mean inventing measurements), and a prompt-injection caveat on the vendor's tool-call flags (quoted to make the hardware assumption auditable, not as a production recipe). Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 25 +++++++++++++++++++------ gitm/planner/registry.py | 33 ++++++++++++++++++++++++++++++++- tests/test_glm_graph.py | 28 ++++++++++++++++++++++++++++ 3 files changed, 79 insertions(+), 7 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 78cce84..bae4251 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -224,6 +224,7 @@ flowchart TD IX -- "57 layers" --> RE["reuse the group's selection — NO KERNEL"] IS --> ATT RE --> ATT{"attention core over ≤2048 selected keys
FLOPs capped · BYTES ARE NOT"} + ATT -.->|"⚠ bytes = whole cache once per REQUEST;
a tiled kernel re-reads per query block (A9/Q3)"| ATT ATT --> AR1{{"o_proj → all_reduce — 174 MB, BANDWIDTH-bound"}} AR1 --> G["router GEMM FP32 → fused gating → top-8 of 256
DATA-DEPENDENT SHAPE"] G --> A2A{{"EP dispatch all-to-all — 1.39 GB/layer, the largest single term"}} @@ -429,9 +430,11 @@ change kind. At P = 8,192 in one chunk: | bytes | **422 GB** | 517 GB | 1,088 GB | **6,313 GB** | | floor | 264 ms | 281 ms | 396 ms | **1,543 ms** | -**14.9× the bytes for identical FLOPs** — the bank is read per *chunk*. A small -`--max-num-batched-tokens`, chosen to protect decode latency, is paid for here at -a rate no per-token cost model shows. +**14.9× the bytes for identical FLOPs** — the bank is read per *chunk*. The rule is +derivable rather than tuned: the expert bank costs `95.7 GB × ceil(P/C)` per prompt +whatever C is, so prefill bytes scale as `1/C` until C is small enough that the +per-chunk activation terms stop mattering. Pick `--max-num-batched-tokens` as large +as decode latency tolerates; there is no prefill-side reason to make it small. ### 3.3 MTP — the whole-step economics @@ -442,7 +445,7 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): | vanilla decode (D=0) | 1,614 | 68.60 GB | 16.551 ms | | — of which the draft | 23 | 1.27 GB | 0.297 ms | | draft chain, D=5 | 115 | **6.34 GB** | 1.483 ms | -| verify, 192 rows | 1,591 | 106.4 GB | 26.652 ms | +| verify, 192 rows (backbone only) | 1,591 | 106.4 GB | 26.652 ms | | **MTP step total** | **1,706** | **112.8 GB** | **28.135 ms** | The node counts reconcile as `1,614 = 1,591 + 23` and `1,706 = 1,591 + 115`: the @@ -543,7 +546,11 @@ what they cost. | 9 more nodes at the launch floor | <5 MB | — | 79 each | 0.158 each | launch | 1.0 % each | | `attn_index_score` | 33.6 MB | 64.0 | 21 | 0.147 | memory | 0.9 % ← the only term that grows with S | | `lm_head` · `mtp_eh_proj` · `logits_all_gather` | 239.5 / 152.2 / 17.3 MB | — | 2 / 1 / 1 | 0.100 / 0.032 / 0.019 | memory | 0.9 % total | -| **1,614 nodes** | **68.60 GB** | | | **16.551** | | **1,933 tok/s** | +| **1,614 nodes** | **68.60 GB** | | | **16.551** | | **1,933 tok/s** [A8] | + +[A8] every MoE byte term here assumes `ep_imbalance = 1.0`; real skew moves less +weight traffic but lengthens the grouped-GEMM tail, and it is trace-calibrated by +design rather than predicted. | facet | nodes | Σ ms | share | | | --- | ---: | ---: | ---: | --- | @@ -597,7 +604,7 @@ at TP8/EP8, FP8. | **Pre** | `all_reduce` ×158 | **comm — BANDWIDTH** | 174 MB each, 27.5 GB/pass = 30.5 ms | BF16 payload | P; below P≈256 flips to latency | | **Pre** | projections (`q_a`,`o_proj`,`q_b`,`kv_a`,`kv_b`) | **compute** | AI 1,403 / 963 / 488 vs the fp8 ridge 412 | FP8 e4m3 | prompt length — below P≈512 memory-bound | | **Pre** | indexer scan ×21 | **compute** | 5.77 TF against 22 MB of keys — `O(P·C + P²/2)` × 32 heads, and **2.2 % of the step**. **The quadratic lives here, not in the core** | **BF16** arithmetic, fp8 keys — two dtypes, one node | P **and** C | -| **Pre** | attention core ×79 | compute, **linear in C** | FLOPs capped at 2,048 keys/query; **bytes are the whole cache once per request** | FP8 KV | ⚑ **`index_topk`**; C. Not P² | +| **Pre** | attention core ×79 | compute, **linear in C** | FLOPs capped at 2,048 keys/query; bytes are the whole cache **once per request — an optimistic floor** (A9/Q3): a tiled kernel re-reads per query block, up to C/2048 more | FP8 KV | ⚑ **`index_topk`**; C; ⚠ **per-request vs per-tile** | | **Pre** | permute / combine | memory | 8.5 GB/layer-pass on the `8P`-row expanded tensor, near-zero FLOPs | BF16 activations | top-k; **expert imbalance** | | **Pre** | *everything else* — embed gather, norms, `lm_head`, gating | memory or launch | each under 1.5 %; `lm_head` reads 239.5 MB for **one row per request** | BF16; FP32 top-k | none of them flips | | **Pre** | **whole prefill pass** | **memory** | **AI 281 vs ridge 412**; 66 % memory / 34 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | @@ -686,6 +693,12 @@ Assume the Nsight Systems / CUPTI trace arrives tomorrow. ### 6.1 The classification rule — *unexpected ≠ recoverable* +**A precondition, not a footnote.** Steps (1) and (2) below require telling a +serial gap from a parallelisable one, and today the graph cannot: `total_pred_s` +is a sum, and `expected_stream_id` is written but read by nothing (§7.1 G10). Until +a capture carries stream assignment, ranks 1 and 7 are judged from the timeline by +hand rather than by the rule. That is a gate on §6, not just roadmap work. + Three questions, in order; the first that answers decides it. **(1) Is there a producer→consumer edge across the gap?** Yes → **architectural**; the §2/§3 graphs exist precisely to answer this without guessing. **(2) Does the diff --git a/gitm/planner/registry.py b/gitm/planner/registry.py index 3c40b88..45fe8b0 100644 --- a/gitm/planner/registry.py +++ b/gitm/planner/registry.py @@ -124,6 +124,9 @@ def add_plan_arguments(ap: argparse.ArgumentParser) -> argparse.ArgumentParser: ap.add_argument("--spec-tokens", type=int, default=0, help="Speculative (MTP) draft tokens per step. Adds a D-deep " "draft chain and makes the backbone a 1+D-row verify.") + ap.add_argument("--acceptance-rate", type=float, default=0.0, + help="Fraction of drafted tokens the verifier keeps. Only " + "affects the reported token rate, never the step floor.") ap.add_argument("--tp", type=int, default=1, help="Tensor-parallel size.") ap.add_argument("--ep", type=int, default=1, help="Expert-parallel size.") ap.add_argument("--dp", type=int, default=1, help="Data-parallel size.") @@ -261,6 +264,31 @@ def _render_table(g, hw: HardwareSpec, spec, family: str, note: str) -> str: if any(b for b in bounds.values() if len(b) > 1): out.append(" * this op's instances do not share a bound — the label is " "the majority one") + if g.batch.speculative_tokens > 0: + # ``tokens_per_step`` counts accepted tokens as ``1 + D*alpha``. A verifier + # accepts a *prefix*, so the real expectation is ``sum(alpha**i)`` — smaller + # for every alpha < 1, by up to 1.8x at D=5, alpha=0.5. The convention is + # shared with every family and not this module's to change, but a rate + # printed from it should not be read as achievable. + d = g.batch.speculative_tokens + a = g.batch.acceptance_rate + linear = 1.0 + d * a + chain = sum(a ** i for i in range(d + 1)) + out.append( + f" ! speculative step (D={d}): rates above assume accepted tokens = " + f"1+D*alpha. A verifier accepts a prefix, so the expectation is " + f"sum(alpha^i)" + ) + if a > 0 and linear > chain: + out.append( + f" at alpha={a:g} that is {linear:.2f} vs {chain:.2f} tokens/step " + f"— the printed rate is {linear / chain:.2f}x optimistic" + ) + else: + out.append( + " set --acceptance-rate to compare the two; break-even is " + "higher than the linear form implies" + ) if g.has_unpriced_collectives: out.append(" ! collectives unpriced — this SKU has no interconnect bandwidth " "in the catalogue") @@ -328,7 +356,8 @@ def main(argv: list[str] | None = None) -> int: for b in sizes: g = _predict(spec, family, hw, BatchConfig(batch=b, kv_cache_len=args.kv_len, - speculative_tokens=args.spec_tokens), + speculative_tokens=args.spec_tokens, + acceptance_rate=args.acceptance_rate), sharding) cb = sum(1 for n in g.nodes if n.prediction.bound == "compute") print(f" {b:7d} {g.total_pred_s * 1e3:9.3f} " @@ -338,6 +367,7 @@ def main(argv: list[str] | None = None) -> int: batch = BatchConfig( batch=args.batch, kv_cache_len=args.kv_len, speculative_tokens=args.spec_tokens, + acceptance_rate=args.acceptance_rate, prefill_tokens=args.prefill_tokens, prefill_context=args.prefill_context, prefill_requests=args.prefill_requests, ) @@ -355,6 +385,7 @@ def main(argv: list[str] | None = None) -> int: "sharding": {"tp": args.tp, "ep": args.ep, "dp": args.dp}, "batch": {"batch": args.batch, "kv_cache_len": args.kv_len, "speculative_tokens": args.spec_tokens, + "acceptance_rate": args.acceptance_rate, "prefill_tokens": args.prefill_tokens, "prefill_context": args.prefill_context, "prefill_requests": args.prefill_requests}, diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index b54b7d6..f6475da 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -20,6 +20,8 @@ from __future__ import annotations +import contextlib +import io import os import tempfile from dataclasses import replace @@ -517,6 +519,32 @@ def test_act_quant_exists_only_where_a_gemm_is_actually_fp8(): assert len(_ops(predict_glm_graph(fp8, batch=batch), "act_quant")) == 2 * blocks +def test_plan_warns_that_a_speculative_rate_is_a_ceiling(): + """``tokens_per_step`` counts ``1 + D*alpha``; a verifier accepts a prefix. + + The convention is shared with every family and not this branch's to change, so + the rate is printed as-is — but a number that is up to 1.8x optimistic at D=5 + must not leave the CLI unlabelled, or the design note's caveat protects only + the readers who found the design note. + """ + from gitm.planner.registry import main + + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + main(["glm-5.2-fp8", "--gpu", "H200", "--batch", "32", "--kv-len", "8192", + "--tp", "8", "--ep", "8", "--spec-tokens", "5", + "--acceptance-rate", "0.5"]) + out = buf.getvalue() + assert "speculative step (D=5)" in out + assert "1.78x optimistic" in out + + # A non-speculative step says nothing, because nothing is being approximated. + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + main(["glm-5.2-fp8", "--gpu", "H200", "--batch", "32", "--kv-len", "8192"]) + assert "speculative step" not in buf.getvalue() + + def test_detect_family_routes_glm_before_sparse_moe(): """Both families carry index_topk + n_routed_experts; model_type must win.""" assert detect_family(GLM_CONFIG) == "glm_moe_dsa" From 7af8184b40e944ede431997948edadb773947a01 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 15:26:38 -0700 Subject: [PATCH 13/35] Condense the attn_kv_b needle comment Six lines of why the entry used to be absent, where three lines of why it is safe now will do. Kept the load-bearing half: absorbed MLA launches no such kernel, so these needles match nothing there. Co-Authored-By: Claude Opus 5 --- gitm/optimizer/deviation.py | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/gitm/optimizer/deviation.py b/gitm/optimizer/deviation.py index 527b996..8b24e03 100644 --- a/gitm/optimizer/deviation.py +++ b/gitm/optimizer/deviation.py @@ -161,12 +161,9 @@ "attn_q_a": ("q_a_proj", "q_lora", "q_down"), "attn_q_b": ("q_b_proj", "q_up"), "attn_kv_a": ("kv_a_proj", "kv_lora", "kv_down", "compress_kv"), - # `kv_b_proj` was absent here while the only MLA families modelled the - # *absorbed* decode form, where W^UK folds into the query and W^UV into the - # output projection and no such kernel is launched. The GLM-5.2 graph models - # it unabsorbed, so the kernel exists and has a node to land on. The entry is - # safe either way: an absorbed deployment launches nothing these needles - # match, so it stays absent rather than mis-attributing. + # Unabsorbed MLA only: the absorbed decode form folds W^UK into the query and + # W^UV into the output and launches no such kernel, so these needles match + # nothing there and the op stays absent rather than mis-attributing. "attn_kv_b": ("kv_b_proj", "kv_up", "w_uk", "w_uv"), "qkv_proj": ("qkv",), "attn_out_proj": ("o_proj", "out_proj", "attn_out"), From 4d4b3942b91073bb67885f477278eda8d2af3791 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 15:30:00 -0700 Subject: [PATCH 14/35] Bound Q12, name the chunking assumption, mark the unusable break-even row MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bounding the prefill-tile question reversed my own call from last round. I promoted it to Q3 on the reviewer's "64x error" framing without computing it. The prefill attention core is 0.10% of prefill bytes, so even a 128-row tiling takes the step from 422 GB to 448 GB — 1.1x, and it does not move the prefill conclusion. Demoted back to last, now with the number attached so nobody promotes it again on the multiplier alone. The good catch this round: the vendor recipe does not set --max-num-batched-tokens, so the 264.0 ms prefill floor silently assumed one 8,192-token chunk. Said so, added the 4 x 2,048 row (707 GB, 319 ms), and pointed at C6 to confirm what the engine actually uses. Also: struck the linear break-even row so it cannot be skimmed as usable (the CLI now warns too, and the row is kept only so the discrepancy is recognisable in that output); footnoted why alpha=0 is the same in both columns; gave S1 a forward reference to the §5.2 fork it resolves; fixed the ep_imbalance note to state the direction — over-predicted traffic means the throughput figures are conservative, not optimistic; and noted in G10 that expected_stream_id defaults to 0, which is indistinguishable from an explicit compute-stream assignment, so whoever wires the invariant should make it int | None first. Gemini's claimed _OP_RULES ordering bug is not one: embed_tokens is line 158, lm_head 176. It read the diff hunks rather than the file, and test_every_needle_is_reachable covers exactly this. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 71 +++++++++++++++++++++---------------- 1 file changed, 41 insertions(+), 30 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index bae4251..8022d72 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -224,7 +224,7 @@ flowchart TD IX -- "57 layers" --> RE["reuse the group's selection — NO KERNEL"] IS --> ATT RE --> ATT{"attention core over ≤2048 selected keys
FLOPs capped · BYTES ARE NOT"} - ATT -.->|"⚠ bytes = whole cache once per REQUEST;
a tiled kernel re-reads per query block (A9/Q3)"| ATT + ATT -.->|"⚠ bytes = whole cache once per REQUEST;
a tiled kernel re-reads per block (A9/Q12) — bounded at 1.1x"| ATT ATT --> AR1{{"o_proj → all_reduce — 174 MB, BANDWIDTH-bound"}} AR1 --> G["router GEMM FP32 → fused gating → top-8 of 256
DATA-DEPENDENT SHAPE"] G --> A2A{{"EP dispatch all-to-all — 1.39 GB/layer, the largest single term"}} @@ -408,6 +408,12 @@ change kind. At P = 8,192 in one chunk: - **The epilogue inverts:** `lm_head` reads 239.5 MB to produce one row per *request*, where at decode every row is a last position. +**The chunking is an assumption, and the vendor recipe does not pin it.** Nothing +in that `vllm serve` line sets `--max-num-batched-tokens`, so the figures below — +which assume the whole 8,192-token prompt arrives as **one chunk** — are the +best case. Confirm the engine's actual value (C6); the same prompt at a 2,048 +default costs **707 GB and 319 ms**, not 422 GB and 264 ms. + | Prefill, P=8192, C=0, TP8/EP8, per rank | value | | --- | --- | | predicted floor | **264.0 ms** for the chunk (31.0 k tok/s) | @@ -425,10 +431,10 @@ change kind. At P = 8,192 in one chunk: **Chunk size multiplies the whole MoE term.** The same 8,192 tokens: -| chunking | 1 × 8,192 | 2 × 4,096 | 8 × 1,024 | 64 × 128 | -| --- | --- | --- | --- | --- | -| bytes | **422 GB** | 517 GB | 1,088 GB | **6,313 GB** | -| floor | 264 ms | 281 ms | 396 ms | **1,543 ms** | +| chunking | 1 × 8,192 | 2 × 4,096 | 4 × 2,048 | 8 × 1,024 | 64 × 128 | +| --- | --- | --- | --- | --- | --- | +| bytes | **422 GB** | 517 GB | 707 GB | 1,088 GB | **6,313 GB** | +| floor | 264 ms | 281 ms | 319 ms | 396 ms | **1,543 ms** | **14.9× the bytes for identical FLOPs** — the bank is read per *chunk*. The rule is derivable rather than tuned: the expert bank costs `95.7 GB × ceil(P/C)` per prompt @@ -473,16 +479,20 @@ where the verifier accepts a *prefix*: token *k* only counts if 1…*k*−1 were accepted, so expected accepted length is `Σ αⁱ` for *i* = 0…D. The difference is not small at D=5. -| α | 0.0 | 0.5 | 0.7 | 0.9 | break-even | +| α | 0.0 † | 0.5 | 0.7 | 0.9 | break-even | | --- | --- | --- | --- | --- | --- | -| linear `1+Dα` (what the graph reports) | 1,137 | 3,981 | 5,118 | **6,256** | α > 0.140 | -| prefix chain `Σ αⁱ` (what a verifier does) | 1,137 | 2,239 | 3,345 | **5,329** | **α > 0.415** | +| ~~linear `1+Dα`~~ — **do not use** | 1,137 | ~~3,981~~ | ~~5,118~~ | ~~6,256~~ | ~~α > 0.140~~ | +| **prefix chain `Σ αⁱ`** — what a verifier does | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.415** | + +against 1,933 tok/s with MTP off. † at α=0 the two agree by construction (one +accepted token either way), and the 1,137 is the *cost* of drafting for nothing — +it is below the MTP-off baseline, which is the point of the column. -against 1,933 tok/s with MTP off. **Read the second row.** The first is what -`--spec-tokens` prints today, and it overstates throughput by up to 1.8× at α=0.5; -the honest break-even on this model is nearer **0.42 than 0.14**. Fixing the -convention is a change to shared `BatchConfig` semantics affecting every family, -so it is stated here rather than made here. +**The struck row is what `gitm plan --spec-tokens` prints today**, and it +overstates throughput by up to 1.8×. It is shown only so the discrepancy is +recognisable in CLI output; the CLI itself now says so. Fixing the convention +means changing shared `BatchConfig` semantics for every family, so it is stated +here rather than made here. Z.ai claims the GLM-5.2 MTP layer raises accepted length up to 20 % over its predecessor, which likely clears even the chained bar — but **α is a serving @@ -496,7 +506,7 @@ that was never moving bytes. The sign of the MTP decision flips with batch. | # | Where | Kind | conf | Trace signature if real | | ------ | --------------------------- | ----------------------------------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------ | -| **S1** | after the gating kernel | host readback of the expert histogram to size the grouped GEMM | **low** | **76 D2H per decoded token** — fatal for graph capture | +| **S1** | after the gating kernel | host readback of the expert histogram to size the grouped GEMM | **low** | **76 D2H per decoded token** — fatal for graph capture. One count resolves **both** rank 2 and rank 3 (§5.2) | | S2 | around each all-reduce | stream-to-stream event wait | medium | 158 event pairs/step; at 688 kB **the gap *is* the cost** | | S3 | around each EP all-to-all | dispatch/combine barrier | medium | 76 more, and **none on the 3 dense layers**; an imbalanced rank stalls every other one | | S4 | sampling / detokenisation | D2H of sampled ids every step | **high** | one D2H + host round-trip per step; unavoidable, but its *placement* decides overlap | @@ -548,9 +558,10 @@ what they cost. | `lm_head` · `mtp_eh_proj` · `logits_all_gather` | 239.5 / 152.2 / 17.3 MB | — | 2 / 1 / 1 | 0.100 / 0.032 / 0.019 | memory | 0.9 % total | | **1,614 nodes** | **68.60 GB** | | | **16.551** | | **1,933 tok/s** [A8] | -[A8] every MoE byte term here assumes `ep_imbalance = 1.0`; real skew moves less -weight traffic but lengthens the grouped-GEMM tail, and it is trace-calibrated by -design rather than predicted. +[A8] every MoE byte term assumes `ep_imbalance = 1.0`. Real skew touches *fewer* +distinct experts, so the prediction over-states traffic and therefore over-states +time: **the throughput figures are conservative**, not optimistic. What skew adds +instead is grouped-GEMM tail latency, which this graph does not model at all. | facet | nodes | Σ ms | share | | | --- | ---: | ---: | ---: | --- | @@ -604,7 +615,7 @@ at TP8/EP8, FP8. | **Pre** | `all_reduce` ×158 | **comm — BANDWIDTH** | 174 MB each, 27.5 GB/pass = 30.5 ms | BF16 payload | P; below P≈256 flips to latency | | **Pre** | projections (`q_a`,`o_proj`,`q_b`,`kv_a`,`kv_b`) | **compute** | AI 1,403 / 963 / 488 vs the fp8 ridge 412 | FP8 e4m3 | prompt length — below P≈512 memory-bound | | **Pre** | indexer scan ×21 | **compute** | 5.77 TF against 22 MB of keys — `O(P·C + P²/2)` × 32 heads, and **2.2 % of the step**. **The quadratic lives here, not in the core** | **BF16** arithmetic, fp8 keys — two dtypes, one node | P **and** C | -| **Pre** | attention core ×79 | compute, **linear in C** | FLOPs capped at 2,048 keys/query; bytes are the whole cache **once per request — an optimistic floor** (A9/Q3): a tiled kernel re-reads per query block, up to C/2048 more | FP8 KV | ⚑ **`index_topk`**; C; ⚠ **per-request vs per-tile** | +| **Pre** | attention core ×79 | compute, **linear in C** | FLOPs capped at 2,048 keys/query; bytes are the whole cache **once per request — an optimistic floor** (A9/Q12), but a bounded one: this node is 0.10 % of prefill bytes, so the worst tiling is 1.1× on the step | FP8 KV | ⚑ **`index_topk`**; C; ⚠ **per-request vs per-tile** | | **Pre** | permute / combine | memory | 8.5 GB/layer-pass on the `8P`-row expanded tensor, near-zero FLOPs | BF16 activations | top-k; **expert imbalance** | | **Pre** | *everything else* — embed gather, norms, `lm_head`, gating | memory or launch | each under 1.5 %; `lm_head` reads 239.5 MB for **one row per request** | BF16; FP32 top-k | none of them flips | | **Pre** | **whole prefill pass** | **memory** | **AI 281 vs ridge 412**; 66 % memory / 34 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | @@ -820,7 +831,7 @@ gaps below are narrower because of it. | **G8** | **A graph that is only its GEMMs, priced against a bound it cannot express** | The family emitted 16 nodes per layer where a layer lowers to ~20 kernels, and the seven missing ones were all pointwise: the norms, the dynamic fp8 activation scaling, the fused gating, the prologue gather and the epilogue's all-gather. Every one is a rounding error in bytes and **a full kernel launch in time** — so at B=1 the graph reported a step as memory-bound that is 69 % launches. A roofline with a launch bound and a graph with no launches in it cannot both be right | Emit them. `_pointwise`, `add_rms_norm` (with the residual fused in, as vLLM runs it), `add_act_quant` gated on the consuming GEMM actually being fp8, plus `embed_tokens` / `rms_norm` / `logits_all_gather` around the stack. Node names constrained by G9 | **yes** | | **G9** | **Op names a capture can actually pair against** | G8's new nodes needed names, and `deviation.classify_op` is a *name guess* (`docs/kernel_identity.md`): a name it cannot classify leaves the predicted node permanently unmatched **and** the real kernel filed as unmodeled — two errors in opposite directions, in the diff the family exists to support. Three norm sites are one kernel name; `silu_and_mul` was already claimed by `mlp_gate_up`; `moe_align`/`topk_softmax` were already claimed by `moe_router`, a decision the dense-MoE and hybrid families depend on | Follow the canonical names rather than redefine them: one `rms_norm` op for all three sites, SwiGLU folded back into the GEMM that owns its needle, gating emitted as a second `moe_router` instance. Then `_OP_RULES` gains only what is genuinely new and unclaimed — `rms_norm`, `act_quant`, `embed_tokens`, `moe_permute`/`moe_combine`, `attn_index_proj`, `attn_kv_b`, `mtp_eh_proj`. A test asserts every op the graph emits resolves | **yes** | | **G11** | **An intervention vocabulary that can name the expert term** | `kernels/library.yaml` scopes every lever with `applies_to_kernels`, drawn from a canonical op list that is `qkv_proj · attn_score_value · attn_out_proj · mlp_gate_up · mlp_down · lm_head`. **`moe_routed` and `moe_shared` are not in it**, so the two entries meant to target expert traffic scope to `[mlp_gate_up, mlp_down]` — true of a dense FFN, false of either MoE family. §5 rank 5 aims levers at **74 % of a decode step** through tooling that cannot match it | Add the two ops to the vocabulary and re-scope those entries. **Pre-existing and not GLM-specific** — `moe_graph.py` emits the same names, so DeepSeek-V4 has it identically | **no** — the fix is a shared-vocabulary change and should land where both families' coverage can be checked at once | -| **G10** | **Which adjacent nodes may overlap and which may not** | `Graph.total_pred_s` is a sum, not a DAG. GLM puts both kinds of serialisation in one step — the draft chain is genuinely serial, the 158 collectives are not — and **both appear as the same positive residual today**. §6.1 is unanswerable without telling them apart | *Not shipped.* It is a cross-family IR change and it is already sequenced on the roadmap. What this note adds is the requirement. `expected_stream_id=1` is set on collectives, but note that **nothing reads it today** — `optimizer/monitor.py` tests overlap using the *observed* kernel's stream, so the predicted field is carried by the IR and consumed by no one. It is a hook for the invariant in `docs/invariants.md` §3, not a wiring of it | **no — deliberately** | +| **G10** | **Which adjacent nodes may overlap and which may not** | `Graph.total_pred_s` is a sum, not a DAG. GLM puts both kinds of serialisation in one step — the draft chain is genuinely serial, the 158 collectives are not — and **both appear as the same positive residual today**. §6.1 is unanswerable without telling them apart | *Not shipped.* It is a cross-family IR change and it is already sequenced on the roadmap. What this note adds is the requirement. `expected_stream_id=1` is set on collectives, but note that **nothing reads it today** — `optimizer/monitor.py` tests overlap using the *observed* kernel's stream, so the predicted field is carried by the IR and consumed by no one. It is a hook for the invariant in `docs/invariants.md` §3, not a wiring of it. **And the field defaults to `0`, which is indistinguishable from an explicit "compute stream"** — whoever wires the invariant should make it `int | None` first, or every pre-GLM family silently claims stream 0 | **no — deliberately** | ### 7.2 The one that needed more than a table row @@ -865,16 +876,16 @@ graph change. The commands at the top of this note regenerate any of them. |---|---|---|---| | **Q1** | Is expert parallelism actually on? | **Rank 1 exists or it does not.** `--enable-expert-parallel` is absent from the vendor recipe; without it the `moe_all_to_all` rows disappear (44 % of prefill) and the per-rank expert bank doubles instead. Not a refinement — a different graph | engine launch args (C6); capture C5 | | **Q2** | Does the engine run **absorbed** MLA at decode? | drops `attn_kv_b` and **doubles** `attn_out_proj`'s input width (16384→32768). ±2× on the #4 and #8 lines | serving image / C6 | -| **Q3** | Does the prefill attention kernel read the selected KV **once per request**, or once per query tile? | **up to 64× on the prefill core's bytes.** The graph takes the optimistic floor (A9), so a tiled kernel would move the prefill roofline conclusion, not just a row | `dram__bytes_read.sum` on the prefill core | -| **Q4** | Is the decode step **CUDA-graph captured**? | at B=1 the difference between 1.24 ms and 3.30 ms per step, and it decides whether MTP is a 3× win or a net loss | engine config + `--cuda-graph-trace=node` | -| **Q5** | Is the **router GEMM** fp32, or only its accumulation? | **11.0 % of prefill.** At bf16 the row shrinks ~15× | the engine's MoE gate implementation | -| **Q6** | Is the grouped GEMM **device-sized or host-sized**? | decides **rank 2 *or* rank 3** — the fork in §5.2 | trace D2H count, or the kernel source | -| **Q7** | Is the **EP dispatch** bf16 or fp8? | **half of 105.7 GB** at prefill — the largest single recoverable number here | serving image / C6 | -| **Q8** | Is the **IndexShare selection** passed device-side? | 21 syncs/step if not (S9) | trace D2H attribution | -| **Q9** | Chunked prefill on, at what chunk size? | **up to 14.9× on prefill bytes** | engine launch args (C6) | -| **Q10** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is **0.415** on a prefix chain (0.140 under the linear convention the graph prints — §3.3); 0.5→0.9 is 2,239→5,329 tok/s | engine metrics (C4) — **not predictable from a config** | -| **Q11** | Are the **index keys** cached in fp8 or bf16? | 2× on 90.2 GB/step at 1M context | C6 / trace | -| **Q12** | Is `eh_proj` **TP-sharded** in the MTP block? | 151 MB → 19 MB per rank per draft stage | serving image | +| **Q3** | Is the decode step **CUDA-graph captured**? | at B=1 the difference between 1.24 ms and 3.30 ms per step, and it decides whether MTP is a 3× win or a net loss | engine config + `--cuda-graph-trace=node` | +| **Q4** | Is the **router GEMM** fp32, or only its accumulation? | **11.0 % of prefill.** At bf16 the row shrinks ~15× | the engine's MoE gate implementation | +| **Q5** | Is the grouped GEMM **device-sized or host-sized**? | decides **rank 2 *or* rank 3** — the fork in §5.2 | trace D2H count, or the kernel source | +| **Q6** | Is the **EP dispatch** bf16 or fp8? | **half of 105.7 GB** at prefill — the largest single recoverable number here | serving image / C6 | +| **Q7** | Is the **IndexShare selection** passed device-side? | 21 syncs/step if not (S9) | trace D2H attribution | +| **Q8** | Chunked prefill on, at what chunk size? | **up to 14.9× on prefill bytes** | engine launch args (C6) | +| **Q9** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is **0.415** on a prefix chain (0.140 under the linear convention the graph prints — §3.3); 0.5→0.9 is 2,239→5,329 tok/s | engine metrics (C4) — **not predictable from a config** | +| **Q10** | Are the **index keys** cached in fp8 or bf16? | 2× on 90.2 GB/step at 1M context | C6 / trace | +| **Q11** | Is `eh_proj` **TP-sharded** in the MTP block? | 151 MB → 19 MB per rank per draft stage | serving image | +| **Q12** | Does the prefill attention kernel read the selected KV once per request, or once per query tile? | Up to 64× on that node — but **the node is 0.10 % of prefill bytes**, so even a 128-row tiling takes the step from 422 GB to 448 GB. **1.1×, and it does not move the prefill conclusion.** Listed last because it is bounded, not because it is small | `dram__bytes_read.sum` on the prefill core | ### 8.2 Assumptions in force @@ -889,7 +900,7 @@ graph change. The commands at the top of this note regenerate any of them. | **A6** | The serving path uses a grouped GEMM, not a per-expert loop | Architecture rule; the reference implementation is the *semantics*, not the execution | a per-expert kernel launch pattern in the trace | | **A7** | 158 collectives per step (2 per layer × 79) | TP convention, now modelled explicitly (G6) | NCCL kernel count per step | | **A8** | `ep_imbalance = 1.0` | **Declared, not fitted** — it is trace-calibrated by design and there are no traces | any measured skew | -| **A9** | The prefill attention core streams the selected cache **once per request** | An optimistic floor (Q3). A tiled kernel re-reads per query block | `dram__bytes_read.sum` on the prefill core | +| **A9** | The prefill attention core streams the selected cache **once per request** | An optimistic floor (Q12), and a *bounded* one: the node is 0.10 % of prefill bytes, so the worst tiling costs 1.1× on the step | `dram__bytes_read.sum` on the prefill core | | **A10** | The exact kernel names, everywhere | `confidence: none` throughout. The *class* is justified; the implementation is not knowable without the serving image | — | --- From e11ee7b6d628f5f65458262a4854b18649059ec6 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 17:43:13 -0700 Subject: [PATCH 15/35] Price embed_tokens at its own declared precision MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit embed_tokens mapped onto the lm_head op, so the untied halves were priced together. Right for GLM-5.2, where both are in modules_to_not_convert, and silently wrong for any checkpoint that quantised one and not the other — dtype_for("embed_tokens") answered fp8 on a model that explicitly does not convert it. The node itself was bf16 via act_dtype, so nothing was mispriced today; the question just returned a wrong answer. Now its own op: the footprint prices the two tensors separately, the gather reads at the table's width and writes at the activation width, and the fp8 catalogue entry declares it. Footprints unchanged (755.9 / 1508.1 GB) because both halves are bf16 here; a test pins that quantising the embedding moves the footprint by the size of the table rather than by nothing. Also relabelled the §3.3 MTP rows — three reviews read 1,591 vs 1,614 as ambiguous, so "vanilla decode (D=0), draft stage included" now says it on the row instead of only in the note below, and the baseline is marked as carrying no acceptance convention (tokens_per_step degenerates to batch at D=0). gemini-review is failing on a 503 from Google's API, not on this branch — the action posted the error itself. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 17 ++++++++------- gitm/planner/glm_graph.py | 29 ++++++++++++++++---------- gitm/planner/models/glm-5.2-fp8.yaml | 1 + tests/test_glm_graph.py | 31 ++++++++++++++++++++++++++++ 4 files changed, 60 insertions(+), 18 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 8022d72..08e4a1f 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -448,15 +448,17 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): | pass | nodes | bytes | floor | | ---------------------- | --------- | ------------ | ------------- | -| vanilla decode (D=0) | 1,614 | 68.60 GB | 16.551 ms | -| — of which the draft | 23 | 1.27 GB | 0.297 ms | -| draft chain, D=5 | 115 | **6.34 GB** | 1.483 ms | +| vanilla decode (D=0), **draft stage included** | 1,614 | 68.60 GB | 16.551 ms | +| — of which that one draft stage | 23 | 1.27 GB | 0.297 ms | +| draft chain, D=5 (5 stages) | 115 | **6.34 GB** | 1.483 ms | | verify, 192 rows (backbone only) | 1,591 | 106.4 GB | 26.652 ms | | **MTP step total** | **1,706** | **112.8 GB** | **28.135 ms** | -The node counts reconcile as `1,614 = 1,591 + 23` and `1,706 = 1,591 + 115`: the -backbone is the same 1,591 nodes either way, and what changes is the draft region -— one stage (23 nodes) at D=0, five (115) at D=5. Nothing is double-counted. +**The MTP module is always present** — it is a block in the checkpoint, not an +option — so "vanilla decode" already contains one draft stage. The backbone (the +78 transformer layers plus prologue and epilogue) is **1,591 nodes in every +column**; only the draft region changes. Hence `1,614 = 1,591 + 23` and +`1,706 = 1,591 + 115`, and nothing is double-counted. "192 rows" is `B × (1 + D)` = 32 × 6: the verify pass is the backbone at 1+D rows, not a different batch size. @@ -484,7 +486,8 @@ not small at D=5. | ~~linear `1+Dα`~~ — **do not use** | 1,137 | ~~3,981~~ | ~~5,118~~ | ~~6,256~~ | ~~α > 0.140~~ | | **prefix chain `Σ αⁱ`** — what a verifier does | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.415** | -against 1,933 tok/s with MTP off. † at α=0 the two agree by construction (one +against 1,933 tok/s with MTP off — a baseline that carries *no* acceptance +convention, since `tokens_per_step` degenerates to `batch` at D=0. † at α=0 the two agree by construction (one accepted token either way), and the 1,137 is the *cost* of drafting for nothing — it is below the MTP-off baseline, which is the point of the column. diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 47f6dd1..16949e8 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -336,6 +336,7 @@ def model_weight_bytes( rw = weight_bytes(spec.dtype_for("moe_router", spec.weight_dtype)) iw = weight_bytes(spec.dtype_for("attn_index_proj", spec.weight_dtype)) lw = weight_bytes(spec.dtype_for("lm_head", spec.weight_dtype)) + tw = weight_bytes(spec.dtype_for("embed_tokens", spec.weight_dtype)) h = spec.hidden inter = spec.moe_intermediate_size @@ -368,11 +369,11 @@ def model_weight_bytes( + h * spec.index_n_heads ) - # Untied input embedding and vocabulary projection. Both stay wide on the FP8 - # checkpoint (``embed_tokens`` and ``lm_head`` are in ``modules_to_not_convert``), - # so they are priced at their own width rather than the backbone's — 1.9 GB of - # the resident footprint that an fp8 read would halve on paper and not on disk. - embed = 2.0 * spec.vocab * h * lw / tp + # Untied input embedding and vocabulary projection: two tensors, priced + # separately because the checkpoint names them separately and could quantise + # one without the other. Both stay wide on GLM-5.2-FP8, which is 1.9 GB of + # resident footprint an fp8 read would halve on paper and not on disk. + embed = spec.vocab * h * (tw + lw) / tp return ( experts @@ -856,14 +857,20 @@ def predict_glm_graph( aw = weight_bytes(spec.act_dtype) rows = float(batch.positions_per_step + batch.prefill_tokens) - # Prologue: one gather from the untied input embedding. No FLOPs, and the - # bytes are the rows it touches, not the 1.9 GB table — an index_select reads - # what it selects. + # Prologue: one gather from the untied input embedding. No FLOPs, and the bytes + # are the rows it touches, not the 1.9 GB table — an index_select reads what it + # selects. It reads at the *table's* width and writes at the activation width; + # those are the same on GLM-5.2, and would not be on a checkpoint that + # quantised the embedding. + embed_dtype = spec.dtype_for("embed_tokens", spec.weight_dtype) g.nodes.append( PredictedNode( "embed_tokens", None, - roofline("embed_tokens", 0.0, rows * spec.hidden * aw, hw, - spec.act_dtype, serial_launches=1), + roofline( + "embed_tokens", 0.0, + rows * spec.hidden * (weight_bytes(embed_dtype) + aw), hw, + embed_dtype, serial_launches=1, + ), ) ) @@ -995,7 +1002,7 @@ def is_glm_moe_dsa_config(cfg: dict[str, Any]) -> bool: #: have nothing to price. _UNQUANTISED_OPS: tuple[tuple[str, str], ...] = ( ("lm_head", "lm_head"), - ("embed_tokens", "lm_head"), # untied, but priced together in the epilogue + ("embed_tokens", "embed_tokens"), ("eh_proj", "mtp_eh_proj"), ("indexer", "attn_index_proj"), ("indexers_proj", "attn_index_proj"), diff --git a/gitm/planner/models/glm-5.2-fp8.yaml b/gitm/planner/models/glm-5.2-fp8.yaml index 5e4d64a..93ad2f6 100644 --- a/gitm/planner/models/glm-5.2-fp8.yaml +++ b/gitm/planner/models/glm-5.2-fp8.yaml @@ -34,6 +34,7 @@ spec: # and lm_head at half of 154,880 x 6,144. op_dtype_overrides: - [attn_index_proj, bf16] + - [embed_tokens, bf16] - [lm_head, bf16] - [mtp_eh_proj, bf16] - [moe_router, fp32] diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index f6475da..f4aa36f 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -184,6 +184,37 @@ def test_fp8_checkpoint_reads_what_the_quantiser_skipped(): assert spec.dtype_for("moe_router", spec.weight_dtype) == "fp32" +def test_embed_tokens_carries_its_own_declared_precision(): + """The untied halves are two tensors and the checkpoint names them separately. + + ``embed_tokens`` used to map onto the ``lm_head`` op, which priced the pair + together. That is right for GLM-5.2, where both are in + ``modules_to_not_convert`` — and silently wrong for any checkpoint that + quantised one and not the other, with ``dtype_for("embed_tokens")`` answering + fp8 on a model that explicitly does not convert it. + """ + fp8 = load_spec("glm-5.2-fp8") + assert fp8.dtype_for("embed_tokens", fp8.weight_dtype) == "bf16" + assert fp8.dtype_for("lm_head", fp8.weight_dtype) == "bf16" + + # The gather reads the table, so the node runs at the table's width. + node = [n for n in predict_glm_graph(fp8).nodes if n.op == "embed_tokens"][0] + assert node.prediction.dtype == "bf16" + + # And the override is load-bearing: quantising the embedding must move the + # footprint by the size of the table, not by nothing. + quantised = replace( + fp8, + op_dtype_overrides=tuple( + o for o in fp8.op_dtype_overrides if o[0] != "embed_tokens" + ), + ) + table = fp8.vocab * fp8.hidden # one byte per element saved at fp8 + assert model_weight_bytes(fp8) - model_weight_bytes(quantised) == pytest.approx( + table, rel=0.01 + ) + + def test_fp8_footprint_matches_published_checkpoint(): """The same shape arithmetic, checked against a second published precision. From e1ad9af92da8fc227fdfe211f3ea28dc1bad891a Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 17:45:51 -0700 Subject: [PATCH 16/35] Stop counting an indexer the MTP block does not carry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit model_weight_bytes counted n_full_indexer_layers + num_nextn_predict_layers indexers, so the MTP block got one whether or not it recomputes the index. With index_share_for_mtp_iteration it does not — it reuses the main selection and carries no indexer tensors, which is what the weight map shows and what _emit_layer already honoured by emitting no indexer node for it. So the graph emitted 21 indexers and the footprint counted 22. Only 18.7 MB bf16 on a 1.5 TB model, but it contradicted the weight-map evidence §1 rests on, and the two numbers are computed independently so nothing caught it. Now gated on the flag, and symmetric: with sharing off the graph emits 22 and the footprint counts 22. Tested as an identity rather than a constant — the emitted node count must equal the footprint's indexer count either way round, and the footprint must move by exactly one indexer between the two readings. Also marked the §4.1 and §3.3 summary tables [EP]: they price EP8, which the vendor recipe does not ask for. Q1 said so; the tables didn't, and they are what gets read. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 10 +++++++++- gitm/planner/glm_graph.py | 10 +++++++++- tests/test_glm_graph.py | 29 +++++++++++++++++++++++++++++ 3 files changed, 47 insertions(+), 2 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 08e4a1f..abb1882 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -454,6 +454,8 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): | verify, 192 rows (backbone only) | 1,591 | 106.4 GB | 26.652 ms | | **MTP step total** | **1,706** | **112.8 GB** | **28.135 ms** | +All four rows price EP8 — see [EP] under §4.1. + **The MTP module is always present** — it is a block in the checkpoint, not an option — so "vanilla decode" already contains one draft stage. The backbone (the 78 transformer layers plus prologue and epilogue) is **1,591 nodes in every @@ -559,7 +561,13 @@ what they cost. | 9 more nodes at the launch floor | <5 MB | — | 79 each | 0.158 each | launch | 1.0 % each | | `attn_index_score` | 33.6 MB | 64.0 | 21 | 0.147 | memory | 0.9 % ← the only term that grows with S | | `lm_head` · `mtp_eh_proj` · `logits_all_gather` | 239.5 / 152.2 / 17.3 MB | — | 2 / 1 / 1 | 0.100 / 0.032 / 0.019 | memory | 0.9 % total | -| **1,614 nodes** | **68.60 GB** | | | **16.551** | | **1,933 tok/s** [A8] | +| **1,614 nodes** | **68.60 GB** | | | **16.551** | | **1,933 tok/s** [A8][EP] | + +[EP] this table prices **EP8**, which the vendor recipe does not ask for — it sets +no `--enable-expert-parallel`. Under TP8-only the `moe_all_to_all` row disappears +and the per-rank expert bank doubles instead: a different graph, not a corrected +one (Q1, capture C5). Every EP-dependent figure in §3 and §4 is conditional on +that flag. [A8] every MoE byte term assumes `ep_imbalance = 1.0`. Real skew touches *fewer* distinct experts, so the prediction over-states traffic and therefore over-states diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 16949e8..0919415 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -342,7 +342,15 @@ def model_weight_bytes( n_sparse = spec.n_sparse_mlp_layers + spec.num_nextn_predict_layers n_dense = spec.n_layers - spec.n_sparse_mlp_layers - n_full_idx = spec.n_full_indexer_layers + spec.num_nextn_predict_layers + # The MTP block counts here only if it recomputes the index. With + # ``index_share_for_mtp_iteration`` it reuses the main model's selection and + # carries no indexer tensors — which is exactly what the weight map shows, and + # what ``_emit_layer`` already honours by emitting no indexer node for it. + # Counting it anyway put one indexer's weights (18.7 MB bf16) in the footprint + # that the checkpoint does not contain, and contradicted the graph beside it. + n_full_idx = spec.n_full_indexer_layers + ( + 0 if spec.index_share_for_mtp_iteration else spec.num_nextn_predict_layers + ) n_attn = spec.n_layers + spec.num_nextn_predict_layers experts = n_sparse * spec.n_routed_experts * 3 * h * inter * ew / es diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index f4aa36f..6de5f89 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -215,6 +215,35 @@ def test_embed_tokens_carries_its_own_declared_precision(): ) +@pytest.mark.parametrize("shares", [True, False]) +def test_footprint_counts_the_indexers_the_graph_emits(shares): + """The graph and the footprint must agree on how many indexers exist. + + They are computed independently — one walks layers emitting nodes, the other + sums shapes — so they can disagree silently. They did: the footprint counted + an indexer for the MTP block while the graph, correctly, emitted none for it, + because ``index_share_for_mtp_iteration`` means it reuses the main selection + and carries no indexer tensors. 18.7 MB of weights the checkpoint does not + have, and a contradiction of the weight-map evidence the note rests on. + + Asserted as an identity rather than a constant, so it holds either way round. + """ + spec = replace(_spec(), index_share_for_mtp_iteration=shares) + emitted = len(_ops(predict_glm_graph(spec), "attn_index_proj")) + expected = spec.n_full_indexer_layers + (0 if shares else spec.num_nextn_predict_layers) + assert emitted == expected + + # And the footprint moves by exactly one indexer between the two readings. + one = ( + spec.q_lora_rank * spec.index_n_heads * spec.index_head_dim + + spec.hidden * spec.index_head_dim + + spec.hidden * spec.index_n_heads + ) * 2 # bf16 + shared_spec = replace(spec, index_share_for_mtp_iteration=True) + own_spec = replace(spec, index_share_for_mtp_iteration=False) + assert model_weight_bytes(own_spec) - model_weight_bytes(shared_spec) == pytest.approx(one) + + def test_fp8_footprint_matches_published_checkpoint(): """The same shape arithmetic, checked against a second published precision. From 093621d4e3a7e50413d75d9b7cd6c30d9d451121 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 17:49:48 -0700 Subject: [PATCH 17/35] Make the two most-misread tables derive themselves MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Neither of this round's bug reports is one — both are arithmetic slips — but the break-even table has now been misread twice and the MTP node table four times, which is the tables' problem rather than the readers'. Break-even: the review divided 5,329 tok/s by the 1,933 baseline *rate*, got 2.76, and compared it to Sum a^i = 4.686. The step is 1.70x longer, so 2.76 x 1.70 = 4.69 and the numbers were right. The table now carries the accepted-tokens row and the arithmetic (32 x Sum a^i / 28.135 ms), so the rate is derivable from the page instead of trusted. MTP nodes: 1,614 = 1,591 + 23 and 1,706 = 1,591 + 115 were stated in prose under the table and misread anyway. Split into backbone / +draft / = nodes columns; the sums are now visible rather than asserted. Also dropped a dead needle: "indexers_proj" in _UNQUANTISED_OPS never matched, "indexer" is a substring of it and is tested first. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 29 ++++++++++++++++------------- gitm/planner/glm_graph.py | 3 +-- 2 files changed, 17 insertions(+), 15 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index abb1882..52db791 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -446,21 +446,20 @@ as decode latency tolerates; there is no prefill-side reason to make it small. At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): -| pass | nodes | bytes | floor | -| ---------------------- | --------- | ------------ | ------------- | -| vanilla decode (D=0), **draft stage included** | 1,614 | 68.60 GB | 16.551 ms | -| — of which that one draft stage | 23 | 1.27 GB | 0.297 ms | -| draft chain, D=5 (5 stages) | 115 | **6.34 GB** | 1.483 ms | -| verify, 192 rows (backbone only) | 1,591 | 106.4 GB | 26.652 ms | -| **MTP step total** | **1,706** | **112.8 GB** | **28.135 ms** | +| pass | backbone | + draft | = nodes | bytes | floor | +| --- | ---: | ---: | ---: | --- | --- | +| vanilla decode (D=0) | 1,591 | 23 (1 stage) | **1,614** | 68.60 GB | 16.551 ms | +| MTP step (D=5) | 1,591 | 115 (5 stages) | **1,706** | 112.8 GB | **28.135 ms** | +| — the verify pass alone | 1,591 | — | 1,591 | 106.4 GB | 26.652 ms | +| — the draft chain alone | — | 115 | 115 | 6.34 GB | 1.483 ms | All four rows price EP8 — see [EP] under §4.1. -**The MTP module is always present** — it is a block in the checkpoint, not an -option — so "vanilla decode" already contains one draft stage. The backbone (the -78 transformer layers plus prologue and epilogue) is **1,591 nodes in every -column**; only the draft region changes. Hence `1,614 = 1,591 + 23` and -`1,706 = 1,591 + 115`, and nothing is double-counted. +The backbone column is the 78 transformer layers plus prologue and epilogue, and +it is **the same 1,591 nodes in every row** — verify is that backbone at 1+D rows, +not a second graph. Only the draft region changes, and the MTP module is always +present (it is a block in the checkpoint, not an option), which is why even the +D=0 row carries one stage. "192 rows" is `B × (1 + D)` = 32 × 6: the verify pass is the backbone at 1+D rows, not a different batch size. @@ -485,8 +484,12 @@ not small at D=5. | α | 0.0 † | 0.5 | 0.7 | 0.9 | break-even | | --- | --- | --- | --- | --- | --- | +| accepted tokens/step, `Σ αⁱ` (i=0…5) | 1.000 | 1.969 | 2.941 | 4.686 | — | +| **tok/s** = `32 × Σ αⁱ ÷ 28.135 ms` | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.415** | | ~~linear `1+Dα`~~ — **do not use** | 1,137 | ~~3,981~~ | ~~5,118~~ | ~~6,256~~ | ~~α > 0.140~~ | -| **prefix chain `Σ αⁱ`** — what a verifier does | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.415** | + +Divide by the **MTP step** (28.135 ms), not by the baseline rate: the step is +1.70× longer, so a rate ratio against 1,933 tok/s is not an accepted-token count. against 1,933 tok/s with MTP off — a baseline that carries *no* acceptance convention, since `tokens_per_step` degenerates to `batch` at D=0. † at α=0 the two agree by construction (one diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 0919415..5747277 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -1012,8 +1012,7 @@ def is_glm_moe_dsa_config(cfg: dict[str, Any]) -> bool: ("lm_head", "lm_head"), ("embed_tokens", "embed_tokens"), ("eh_proj", "mtp_eh_proj"), - ("indexer", "attn_index_proj"), - ("indexers_proj", "attn_index_proj"), + ("indexer", "attn_index_proj"), # subsumes indexers_proj, indexer.k_norm ("mlp.gate", "moe_router"), ) From c7130a728a082f48a7df449ab016c1ea383db255 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 17:53:46 -0700 Subject: [PATCH 18/35] Say that act_dtype is inherited, and scope the draft-gap row MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both of this round's reports are wrong, but the first has a real cause. Gemini read glm-5.2-fp8.yaml, saw weight/expert/kv all fp8 and no act_dtype, and concluded the norms run fp8 — contradicting the note. They do not: act_dtype is bf16, inherited from glm-5.2 through `extends`, and every norm/permute/scatter node is bf16. But `extends` makes an inherited value invisible, and three fp8 lines invite exactly that inference. Added a comment saying what is inherited and why: weights are fp8 on disk, activations are bf16 in flight and quantised per-GEMM at run time. Break-even solves to 0.4147 against the exact 28.135/16.551 ratio, so 0.415 stands. Also scoped §5.3's draft-gap row, which was a fair reading even if the bug was not: it covers the draft chain only. Verify is one backbone forward with no cross-stage dependency, so its collectives are as overlappable as any other step's and a gap around them is not excused by that section. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 4 ++++ gitm/planner/models/glm-5.2-fp8.yaml | 7 +++++++ 2 files changed, 11 insertions(+) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 52db791..122dc74 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -710,6 +710,10 @@ KV cache replicated across TP ranks** (one shared MLA latent cannot be split), a **the indexer scan growing with context** (IndexShare already cut it 3.7×; the remainder is the cost of selecting from an uncompressed history). +The draft-gap row covers the **draft chain only**. The verify pass is one backbone +forward with no cross-stage dependency, so its collectives are as overlappable as +any other step's (rank 7) — a gap around them is not excused by this section. + --- ## 6. Validation plan diff --git a/gitm/planner/models/glm-5.2-fp8.yaml b/gitm/planner/models/glm-5.2-fp8.yaml index 93ad2f6..69a4575 100644 --- a/gitm/planner/models/glm-5.2-fp8.yaml +++ b/gitm/planner/models/glm-5.2-fp8.yaml @@ -28,6 +28,13 @@ spec: # the two halves separately for exactly this reason. kv_dtype: fp8 + # NOT overridden, and deliberately so: act_dtype stays bf16, inherited from + # `glm-5.2`. Weights are fp8 on disk; activations are bf16 in flight and get + # quantised per-GEMM at run time (activation_scheme "dynamic"), which is the + # act_quant node. Every norm, permute and scatter therefore runs bf16 here. + # Listed as a comment because `extends` makes an inherited value invisible, + # and three fp8 lines above it invite the wrong inference. + # From quantization_config.modules_to_not_convert, plus the router's own dtype # from the base config. Without these the indexer — the one attention node # whose cost grows with context — is priced at half its real weight traffic, From 1de831cb83439dcc0733799a003de5bed0770480 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 17:57:41 -0700 Subject: [PATCH 19/35] Fix a fragment I left behind, and make the byte column add up MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructuring the break-even table last commit orphaned the sentence that used to follow it — the paragraph started "against 1,933 tok/s with MTP off" with nothing before it, and the footnote marker ended up mid-line. Rewritten as a whole sentence with the footnote on its own. The MTP byte column rounded 106.4 + 6.34 to 112.8, which does not add up on the page. Exact figures are 106.43 + 6.34 = 112.77, and the extra over vanilla is 44.17 GB. In a note where every number is meant to be checkable a 0.06 GB gap is worth closing rather than explaining. Not taking the embedding-sharding suggestion: vLLM's VocabParallelEmbedding shards the input embedding by vocabulary, it does not replicate it, and both sibling families already price it as vocab * h / tp. Replicating it here would put this family out of step with moe_graph and hybrid_graph for a claim vLLM does not make. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 17 +++++++++-------- .../importers/mixed_dump/nsys_2024_min.sqlite | Bin 32768 -> 32768 bytes .../mixed_dump/torch_trace_min.json.gz | Bin 950 -> 929 bytes tests/fixtures/importers/nsys_2023_min.sqlite | Bin 32768 -> 32768 bytes tests/fixtures/importers/nsys_2024_min.sqlite | Bin 32768 -> 32768 bytes tests/fixtures/importers/nsys_2025_min.sqlite | Bin 32768 -> 32768 bytes tests/fixtures/importers/parity_nsys.sqlite | Bin 32768 -> 32768 bytes .../importers/torch_trace_min.json.gz | Bin 950 -> 929 bytes 8 files changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 122dc74..3da7f3e 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -449,8 +449,8 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): | pass | backbone | + draft | = nodes | bytes | floor | | --- | ---: | ---: | ---: | --- | --- | | vanilla decode (D=0) | 1,591 | 23 (1 stage) | **1,614** | 68.60 GB | 16.551 ms | -| MTP step (D=5) | 1,591 | 115 (5 stages) | **1,706** | 112.8 GB | **28.135 ms** | -| — the verify pass alone | 1,591 | — | 1,591 | 106.4 GB | 26.652 ms | +| MTP step (D=5) | 1,591 | 115 (5 stages) | **1,706** | 112.77 GB | **28.135 ms** | +| — the verify pass alone | 1,591 | — | 1,591 | 106.43 GB | 26.652 ms | | — the draft chain alone | — | 115 | 115 | 6.34 GB | 1.483 ms | All four rows price EP8 — see [EP] under §4.1. @@ -463,7 +463,7 @@ D=0 row carries one stage. "192 rows" is `B × (1 + D)` = 32 × 6: the verify pass is the backbone at 1+D rows, not a different batch size. -**Cost ratio 1.70× for up to 6 tokens.** Where the extra 44.2 GB goes: +**Cost ratio 1.70× for up to 6 tokens.** Where the extra 44.17 GB goes: - **Verify, +37.8 GB**, almost all one line: the expert union saturates, so 6× the rows costs 1.57× the expert bytes (163 → 256 distinct). KV read does **not** @@ -489,12 +489,13 @@ not small at D=5. | ~~linear `1+Dα`~~ — **do not use** | 1,137 | ~~3,981~~ | ~~5,118~~ | ~~6,256~~ | ~~α > 0.140~~ | Divide by the **MTP step** (28.135 ms), not by the baseline rate: the step is -1.70× longer, so a rate ratio against 1,933 tok/s is not an accepted-token count. +1.70× longer, so a rate ratio against the 1,933 tok/s baseline is not an +accepted-token count. That baseline carries *no* acceptance convention of its own +— at D=0 `tokens_per_step` degenerates to `batch`. -against 1,933 tok/s with MTP off — a baseline that carries *no* acceptance -convention, since `tokens_per_step` degenerates to `batch` at D=0. † at α=0 the two agree by construction (one -accepted token either way), and the 1,137 is the *cost* of drafting for nothing — -it is below the MTP-off baseline, which is the point of the column. +† At α=0 both formulas give exactly one accepted token, so the two rows agree by +construction. The 1,137 is the cost of drafting for nothing, and it sits **below** +the 1,933 MTP-off baseline — which is the point of keeping the column. **The struck row is what `gitm plan --spec-tokens` prints today**, and it overstates throughput by up to 1.8×. It is shown only so the discrepancy is diff --git a/tests/fixtures/importers/mixed_dump/nsys_2024_min.sqlite b/tests/fixtures/importers/mixed_dump/nsys_2024_min.sqlite index 20d880cf6ab630deff631eed6fec3743cac00021..dff4aa3132294f8cc955105f82c1c9622c8ec106 100644 GIT binary patch delta 15 WcmZo@U}|V!nvlfQrnxa?PCWoD0|nRs delta 15 WcmZo@U}|V!nvlda_2kBsIrRW9eFldB diff --git a/tests/fixtures/importers/mixed_dump/torch_trace_min.json.gz b/tests/fixtures/importers/mixed_dump/torch_trace_min.json.gz index 647757e9670daeea308fdc80dd8ec9328041e00b..8f15e5bd9f2119396c19c2a00f3118cd5a1ecf1b 100644 GIT binary patch delta 19 YcmdnSzL1?uzMF#q1epGBl%LKF04ziVjQ{`u delta 40 vcmZ3;zKva8zMF&NVNhfi)BlqEqU4PDlA^@q)cD-YJiV;q{5*z@T+^8WD*X Date: Wed, 2 Sep 2026 18:01:32 -0700 Subject: [PATCH 20/35] Print the KV formula so it reproduces its own number MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The block wrote 78 x (512*1 + 64*2) + 21 x 128*1 = 52,618, but that expression evaluates to 52,608. The 10-byte gap is the fp8 block scale: weight_bytes("fp8") is 1.000244, not 1, which §7.0 credits the planner for and this formula then dropped. Anyone reproducing the number by hand got a different one. Formula now shows the factor. Not taking the kv_dtype suggestion: spec_from_hf_config infers kv_dtype from the checkpoint's own dtype on purpose. The FP8 catalogue entry sets fp8 because the vendor's serve command does, and its provenance lists it under `estimated` for exactly that reason. Reading a serving decision out of a config.json that does not contain one is the inference the reader is supposed to be able to reject. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 3da7f3e..625e5ea 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -167,8 +167,9 @@ per layer per token, elements: full-indexer layers, additionally + 128 ← the cached index key whole model, bytes per token of context (fp8 latent, bf16 rope key + index key): - 78 × (512·1 + 64·2) + 21 × 128·1 = 52,618 B/token - bf16 throughout: 95,232 B/token + an fp8 weight costs 1.000244 B, not 1 B — the 128×128 block scale (§7.0) + 78 × (512×1.000244 + 64×2) + 21 × 128×1.000244 = 52,618 B/token + bf16 throughout: 95,232 B/token ``` | Context | fp8 KV | bf16 KV | From d9b23514f40bffe90f7b80e5975d71f8726056d2 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 18:04:30 -0700 Subject: [PATCH 21/35] Show the per-stage node arithmetic and the baseline row MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Neither point is a bug; both are things the tables asserted instead of showing. A draft stage is 23 nodes: the 20 of a shared-indexer MoE layer plus rms_norm, mtp_eh_proj and lm_head. The no-indexer fact is already priced into the 20 — that is why it is 20 and not 22 — and every stage is the same block invoked again, so 5 x 23 = 115 is exact. Verified: all five stages emit 23. Appendix A.4 now also says 23 emitted nodes rather than leaving the reader to notice M.24 is a sync point and not a node. The break-even table now carries the MTP-off row (32 / 16.551 ms) beside the MTP rows, so the comparison is on the page instead of in the prose. Each row divides its own tokens by its own step time; break-even is where they meet, at Sum a^i = 28.135/16.551 = 1.700. gemini-review is red on a Google API 503 for the second time; nothing on the branch. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 625e5ea..8b68733 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -456,6 +456,11 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): All four rows price EP8 — see [EP] under §4.1. +A stage is **23 nodes**: the 20 of a shared-indexer MoE layer (A.1) plus +`rms_norm`, `mtp_eh_proj` and `lm_head` (A.4). It carries no indexer — that is +already why it is 20 and not 22 — and every stage is the same block invoked again, +so `5 × 23 = 115` is exact rather than approximate. + The backbone column is the 78 transformer layers plus prologue and epilogue, and it is **the same 1,591 nodes in every row** — verify is that backbone at 1+D rows, not a second graph. Only the draft region changes, and the MTP module is always @@ -485,11 +490,14 @@ not small at D=5. | α | 0.0 † | 0.5 | 0.7 | 0.9 | break-even | | --- | --- | --- | --- | --- | --- | -| accepted tokens/step, `Σ αⁱ` (i=0…5) | 1.000 | 1.969 | 2.941 | 4.686 | — | +| accepted tokens/step, `Σ αⁱ` (i=0…5) | 1.000 | 1.969 | 2.941 | 4.686 | **1.700** | | **tok/s** = `32 × Σ αⁱ ÷ 28.135 ms` | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.415** | +| MTP off, for comparison = `32 ÷ 16.551 ms` | 1,933 | 1,933 | 1,933 | 1,933 | — | | ~~linear `1+Dα`~~ — **do not use** | 1,137 | ~~3,981~~ | ~~5,118~~ | ~~6,256~~ | ~~α > 0.140~~ | -Divide by the **MTP step** (28.135 ms), not by the baseline rate: the step is +Each row divides its own token count by its own step time, which is what makes +the two comparable: break-even is where they meet, at `Σ αⁱ = 28.135/16.551 = +1.700`. Divide by the **MTP step** (28.135 ms), not by the baseline rate: the step is 1.70× longer, so a rate ratio against the 1,933 tok/s baseline is not an accepted-token count. That baseline carries *no* acceptance convention of its own — at D=0 `tokens_per_step` degenerates to `batch`. @@ -1067,6 +1075,10 @@ for the sequence, not for the verify rows. | M.23 | `lm_head` | GEMM, vocab-sharded `[6144→19360]` | 7,612.7 MF | **239.528 MB** | **BF16** | 49.90 | memory | | M.24 | `argmax` + D2H | sampling + host round-trip | — | small | BF16 | — | **sync (S6)** | +**23 emitted nodes** — M.1, M.2, the 20 of M.3–M.22, and M.23. M.24 is a +synchronization point, not a graph node, which is why the stage counts 23 and not +24 in §3.3. + **Σ per stage: 18.0 GF, 1,268.2 MB, 0.297 ms — ×5 = 6.34 GB, 1.483 ms.** Three things there are the whole §3.3 argument. **M.3–M.22 is a full MoE block** — From db811503512b7bc0375aaab5586179de68a52259 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 18:11:10 -0700 Subject: [PATCH 22/35] Don't run the draft head when nothing is being drafted num_nextn_predict_layers says the MTP block is in the checkpoint; it does not say the engine executes it. Drafting happens only under a speculative config, so at D=0 there are zero drafted tokens and there should be zero stages. The graph forced one via max(1, speculative_tokens), charging a pure decode step 0.3 ms of drafting that a server without --speculative-config never does. Gated on speculative_tokens > 0. The block's weights stay resident either way -- model_weight_bytes still counts them -- which is the distinction, and a test now pins both halves of it. This also removes the number six reviews have read as a contradiction: the "1,614 vs 1,591" only existed because D=0 was carrying a phantom draft stage. A vanilla decode step is 1,591 nodes flat. Numbers that move: decode 16.551 -> 16.254 ms, 1,933 -> 1,969 tok/s, 68.60 -> 67.34 GB. MTP ratio 1.70 -> 1.73 and break-even alpha 0.415 -> 0.426, both because the baseline got cheaper while the D=5 step did not. The 1M indexer share goes 53.4 -> 53.9%. Footprint unchanged at 755.9 GB. One test had to change meaning rather than value: the indexer identity compared emitted nodes against resident weights, and those legitimately diverge at D=0 now. It runs at D=1, where the stage actually launches. Not taking Gemini's max(1, sh.tp) suggestion: _emit_collective is a module-level function and computes tp in its own scope. Removing it would be a NameError, not a simplification. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 133 ++++++++++++++++++------------------ gitm/planner/glm_graph.py | 15 +++- tests/test_glm_graph.py | 48 ++++++++++--- 3 files changed, 114 insertions(+), 82 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 8b68733..ef99c84 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -323,7 +323,7 @@ pricing a tower it never read. flowchart LR subgraph V["VANILLA DECODE"] direction TB - v1["1 row per seq"] --> v2["78 layers + 1 MTP
1,614 nodes"] --> v3["lm_head"] --> v4["sample"] + v1["1 row per seq"] --> v2["78 layers
1,591 nodes"] --> v3["lm_head"] --> v4["sample"] v4 --> v5(("1 token")) end subgraph D["DRAFT — 5 SERIAL stages, 115 nodes"] @@ -449,32 +449,28 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): | pass | backbone | + draft | = nodes | bytes | floor | | --- | ---: | ---: | ---: | --- | --- | -| vanilla decode (D=0) | 1,591 | 23 (1 stage) | **1,614** | 68.60 GB | 16.551 ms | -| MTP step (D=5) | 1,591 | 115 (5 stages) | **1,706** | 112.77 GB | **28.135 ms** | +| vanilla decode (D=0) | 1,591 | 0 | **1,591** | 67.34 GB | 16.254 ms | +| MTP step (D=5) | 1,591 | 115 (5 × 23) | **1,706** | 112.77 GB | **28.135 ms** | | — the verify pass alone | 1,591 | — | 1,591 | 106.43 GB | 26.652 ms | | — the draft chain alone | — | 115 | 115 | 6.34 GB | 1.483 ms | -All four rows price EP8 — see [EP] under §4.1. +The backbone is the same **1,591 nodes in every row** — the 78 transformer layers +plus prologue and epilogue. Verify is that backbone at 1+D rows, not a second +graph. A draft stage is **23 nodes**: the 20 of a shared-indexer MoE layer (A.1) +plus `rms_norm`, `mtp_eh_proj` and `lm_head` (A.4), identical every stage, so +`5 × 23 = 115` is exact. -A stage is **23 nodes**: the 20 of a shared-indexer MoE layer (A.1) plus -`rms_norm`, `mtp_eh_proj` and `lm_head` (A.4). It carries no indexer — that is -already why it is 20 and not 22 — and every stage is the same block invoked again, -so `5 × 23 = 115` is exact rather than approximate. +**At D=0 the draft head does not run at all.** `num_nextn_predict_layers: 1` says +the block is in the checkpoint; it does not say the engine executes it. Without a +speculative config nothing is drafted, so no stage is emitted — the weights stay +resident and none of their kernels launch. -The backbone column is the 78 transformer layers plus prologue and epilogue, and -it is **the same 1,591 nodes in every row** — verify is that backbone at 1+D rows, -not a second graph. Only the draft region changes, and the MTP module is always -present (it is a block in the checkpoint, not an option), which is why even the -D=0 row carries one stage. -"192 rows" is `B × (1 + D)` = 32 × 6: the verify pass is the backbone at 1+D rows, -not a different batch size. +**Cost ratio 1.73× for up to 6 tokens.** Where the extra 45.43 GB goes: -**Cost ratio 1.70× for up to 6 tokens.** Where the extra 44.17 GB goes: - -- **Verify, +37.8 GB**, almost all one line: the expert union saturates, so 6× the +- **Verify, +39.1 GB**, almost all one line: the expert union saturates, so 6× the rows costs 1.57× the expert bytes (163 → 256 distinct). KV read does **not** move — 0.43 GB either way, read per *sequence* — and neither does `lm_head`. -- **The draft chain, +5.07 GB.** Each of 5 stages draws on a full 256-expert bank, +- **The draft chain, +6.34 GB**, all of it new work. Each of 5 stages draws on a full 256-expert bank, linear in D with no saturation to help, so **the draft is 5.3 % of the MTP step where a dense-draft model's would be 1–2 %**. @@ -490,21 +486,21 @@ not small at D=5. | α | 0.0 † | 0.5 | 0.7 | 0.9 | break-even | | --- | --- | --- | --- | --- | --- | -| accepted tokens/step, `Σ αⁱ` (i=0…5) | 1.000 | 1.969 | 2.941 | 4.686 | **1.700** | -| **tok/s** = `32 × Σ αⁱ ÷ 28.135 ms` | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.415** | -| MTP off, for comparison = `32 ÷ 16.551 ms` | 1,933 | 1,933 | 1,933 | 1,933 | — | -| ~~linear `1+Dα`~~ — **do not use** | 1,137 | ~~3,981~~ | ~~5,118~~ | ~~6,256~~ | ~~α > 0.140~~ | +| accepted tokens/step, `Σ αⁱ` (i=0…5) | 1.000 | 1.969 | 2.941 | 4.686 | **1.731** | +| **tok/s** = `32 × Σ αⁱ ÷ 28.135 ms` | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.426** | +| MTP off, for comparison = `32 ÷ 16.254 ms` | 1,969 | 1,969 | 1,969 | 1,969 | — | +| ~~linear `1+Dα`~~ — **do not use** | 1,137 | ~~3,981~~ | ~~5,118~~ | ~~6,256~~ | ~~α > 0.146~~ | Each row divides its own token count by its own step time, which is what makes -the two comparable: break-even is where they meet, at `Σ αⁱ = 28.135/16.551 = -1.700`. Divide by the **MTP step** (28.135 ms), not by the baseline rate: the step is -1.70× longer, so a rate ratio against the 1,933 tok/s baseline is not an +the two comparable: break-even is where they meet, at `Σ αⁱ = 28.135/16.254 = +1.731`. Divide by the **MTP step** (28.135 ms), not by the baseline rate: the step is +1.70× longer, so a rate ratio against the 1,969 tok/s baseline is not an accepted-token count. That baseline carries *no* acceptance convention of its own — at D=0 `tokens_per_step` degenerates to `batch`. † At α=0 both formulas give exactly one accepted token, so the two rows agree by construction. The 1,137 is the cost of drafting for nothing, and it sits **below** -the 1,933 MTP-off baseline — which is the point of keeping the column. +the 1,969 MTP-off baseline — which is the point of keeping the column. **The struck row is what `gitm plan --spec-tokens` prints today**, and it overstates throughput by up to 1.8×. It is shown only so the discrepancy is @@ -534,7 +530,7 @@ that was never moving bytes. The sign of the MTP decision flips with batch. | S8 | KV rollback | discard rejected rows | **low** on mechanism | pointer rewind (free) or real memmove (not free) — the trace tells you which | | S9 | indexer selection handoff | the `full` layer's top-k must be visible to its 3 `shared` layers | **low** | if it round-trips the host, IndexShare costs a sync it should not — **21 per step** | -**S5** is the decode-specific one worth chasing: ~1,614 kernels in ~16 ms, then +**S5** is the decode-specific one worth chasing: ~1,591 kernels in ~16 ms, then control returns to a Python scheduler — if the scheduler is slower than the step, no kernel-level work matters. **S7** is the one that breaks CUDA graphs: MTP adds a per-step, host-visible, data-dependent sequence length, so a capturing stack needs @@ -564,17 +560,17 @@ what they cost. | node | bytes | AI | ×N | Σ ms | bound | share | | --- | ---: | ---: | ---: | ---: | --- | ---: | -| `moe_routed` | 771.3 MB | 3.1 | 76 | **12.213** | memory | **73.8 %** | -| `attn_score_value` | 42.0 MB | 12.8 | 79 | 0.690 | memory | 4.2 % | -| `moe_all_to_all` | 5.5 MB | — | 76 | 0.465 | comm | 2.8 % | -| `rms_norm` | 1.6 MB | 0.8 | 160 | 0.322 | launch | 1.9 % | -| `act_quant` | 0.6 MB | 0.7 | 158 | 0.316 | launch | 1.9 % | -| `moe_router` (GEMM + gating) | 3.4 MB | 15.0 | 152 | 0.304 | launch | 1.8 % | -| `attn_q_a` · `attn_out_proj` | 13.1 MB each | 61.4 | 79 each | 0.216 each | memory | 1.3 % each | -| 9 more nodes at the launch floor | <5 MB | — | 79 each | 0.158 each | launch | 1.0 % each | +| `moe_routed` | 771.3 MB | 3.1 | 75 | **12.052** | memory | **74.1 %** | +| `attn_score_value` | 42.0 MB | 12.8 | 78 | 0.682 | memory | 4.2 % | +| `moe_all_to_all` | 5.5 MB | — | 75 | 0.459 | comm | 2.8 % | +| `rms_norm` | 1.6 MB | 0.8 | 157 | 0.314 | launch | 1.9 % | +| `act_quant` | 0.6 MB | 0.7 | 156 | 0.312 | launch | 1.9 % | +| `moe_router` (GEMM + gating) | 3.4 MB | 15.0 | 150 | 0.300 | launch | 1.8 % | +| `attn_q_a` · `attn_out_proj` | 13.1 MB each | 61.4 | 78 each | 0.213 each | memory | 1.3 % each | +| 9 more nodes at the launch floor | <5 MB | — | 78 each | 0.156 each | launch | 1.0 % each | | `attn_index_score` | 33.6 MB | 64.0 | 21 | 0.147 | memory | 0.9 % ← the only term that grows with S | -| `lm_head` · `mtp_eh_proj` · `logits_all_gather` | 239.5 / 152.2 / 17.3 MB | — | 2 / 1 / 1 | 0.100 / 0.032 / 0.019 | memory | 0.9 % total | -| **1,614 nodes** | **68.60 GB** | | | **16.551** | | **1,933 tok/s** [A8][EP] | +| `lm_head` · `logits_all_gather` | 239.5 / 17.3 MB | — | 1 / 1 | 0.050 / 0.019 | memory | 0.4 % total | +| **1,591 nodes** | **67.34 GB** | | | **16.254** | | **1,969 tok/s** [A8][EP] | [EP] this table prices **EP8**, which the vendor recipe does not ask for — it sets no `--enable-expert-parallel`. Under TP8-only the `moe_all_to_all` row disappears @@ -589,8 +585,8 @@ instead is grouped-GEMM tail latency, which this graph does not model at all. | facet | nodes | Σ ms | share | | | --- | ---: | ---: | ---: | --- | -| memory | 441 | 14.203 | 85.8 % | five node types | -| launch | 1,173 | 2.348 | 14.2 % | 73 % of all nodes, a seventh of the time | +| memory | 434 | 13.940 | 85.8 % | five node types | +| launch | 1,157 | 2.314 | 14.2 % | 73 % of all nodes, a seventh of the time | | compute | 0 | 0.000 | 0.0 % | the entire roofline claim, one row | **A MoE layer is 20 kernels and two of them cost anything** (Appendix A.1). Two @@ -600,9 +596,9 @@ is the only term that grows with S, which does not stay small: | context S | `attn_index_score` | share of step | step floor | | --------- | ------------------ | ------------- | ---------- | -| 8,192 | 0.147 ms | 0.9 % | 16.551 ms | -| 131,072 | 2.349 ms | 12.5 % | 18.753 ms | -| **1,048,576** | **18.795 ms** | **53.4 %** | 35.199 ms | +| 8,192 | 0.147 ms | 0.9 % | 16.254 ms | +| 131,072 | 2.349 ms | 12.7 % | 18.457 ms | +| **1,048,576** | **18.795 ms** | **53.9 %** | 34.902 ms | **At 1M the indexer scan is the largest node in the step** — and it is the node IndexShare already cut 3.7×. "Flat in context" is true of the attention *core* and @@ -612,16 +608,16 @@ false of the step. | B | floor | tok/s | launch nodes | launch time | compute nodes | | --- | ---------- | ----- | ------------ | ----------- | ------------- | -| 1 | 3.946 ms | 253 | 1,353 | 2.708 ms = **69 %** | 0 | -| 4 | 5.633 ms | 710 | 1,352 | 2.706 ms = 48 % | 0 | -| 16 | 11.289 ms | 1,417 | 1,173 | 2.348 ms = 21 % | 0 | -| 32 | 16.551 ms | 1,933 | 1,173 | 2.348 ms = 14 % | 0 | -| 64 | 22.400 ms | 2,857 | 1,097 | 2.196 ms = 10 % | 76 | -| 128 | 27.682 ms | 4,624 | 939 | 1.880 ms = 7 % | 76 | -| 256 | 34.510 ms | 7,418 | 705 | 1.412 ms = 4 % | 79 | +| 1 | 3.813 ms | 262 | 1,335 | 2.670 ms = **70 %** | 0 | +| 4 | 5.478 ms | 730 | 1,334 | 2.668 ms = 49 % | 0 | +| 16 | 11.061 ms | 1,447 | 1,157 | 2.314 ms = 21 % | 0 | +| 32 | 16.254 ms | 1,969 | 1,157 | 2.314 ms = 14 % | 0 | +| 64 | 22.028 ms | 2,905 | 1,082 | 2.164 ms = 10 % | 75 | +| 128 | 27.245 ms | 4,698 | 926 | 1.852 ms = 7 % | 75 | +| 256 | 33.977 ms | 7,534 | 695 | 1.390 ms = 4 % | 76 | **Below B≈16 the step is launch-bound in memory-bound clothes.** At B=1 that -69 % already assumes CUDA-graph replay; at the eager 5 µs it is **85 %**, and the +70 % already assumes CUDA-graph replay; at the eager 5 µs it is **85 %**, and the whole low-batch analysis changes sign (A4, §5 rank 3). @@ -649,7 +645,7 @@ at TP8/EP8, FP8. | **MTP** | draft `eh_proj` ×5 | memory | `[12288,6144]` BF16, **replicated per rank** | **BF16** — in `modules_to_not_convert` | whether it is TP-sharded | | **MTP** | verify attention | **memory, unchanged** | 0.43 GB — read **per sequence, not per row**; 1+D rows share one block table | FP8 KV | seq length; explicitly *not* D | | **MTP** | accept/reject + KV rollback | launch + **host sync** | tiny tensors, but a data-dependent host-visible seq length (S7) | n/a | pointer rewind vs memmove | -| **MTP** | **whole MTP step** | ⚑ **memory above B≈16, launch below — and the two regimes disagree about whether MTP helps** | **1.70×** cost for ≤6 tokens at B=32; break-even α = **0.415** on a prefix chain (§3.3) | mixed | **graph capture**; batch; α; D | +| **MTP** | **whole MTP step** | ⚑ **memory above B≈16, launch below — and the two regimes disagree about whether MTP helps** | **1.73×** cost for ≤6 tokens at B=32; break-even α = **0.426** on a prefix chain (§3.3) | mixed | **graph capture**; batch; α; D | **Hardware sensitivity:** nothing flips between H200 SXM and H20 on the compute rows — but H20's much lower FP8 peak moves every prefill projection further into @@ -663,13 +659,13 @@ row, with the magnitude each is worth: | Flip variable | Direction and magnitude | | --- | --- | | **Decode batch B** | expert bytes are **sub-linear**: 8 distinct experts at B=1, 163 at 32, 252 at 128. 253 → 7,418 tok/s across 1→256, and the whole-step label goes launch → memory at B≈16 | -| **Sequence length S** | moves the indexer scan and **nothing else**: 0.9 % of the step at 8K → 12.5 % at 128K → **53.4 % at 1M** | +| **Sequence length S** | moves the indexer scan and **nothing else**: 0.9 % of the step at 8K → 12.7 % at 128K → **53.9 % at 1M** | | **Prompt length P** | every prefill projection (memory→compute above P≈512), the router, the indexer's quadratic, all-reduce (latency→bandwidth above P≈256) | | ⚑ **Chunk size** | **14.9×** on prefill bytes across 1→64 chunks, for identical FLOPs | | ⚑ **CUDA-graph capture** | 69 % of the B=1 floor, 85 % at eager 5 µs. Decides whether MTP is a 3× win or a net loss | | ⚑ **EP vs TP** | the a2a is 45 % of prefill under EP8, but the per-rank bank is **8× smaller** — a trade, not a cost, since the bank is 85 % of decode DRAM | | ⚑ **Precision, and KV dtype** | 1.79× on the decode floor and **10.7 → 5.4 H200s** for weights; 55 GB vs 100 GB of KV per rank at 1M | -| **D and acceptance α** | 1.70× cost at D=5; break-even α = **0.415** on a prefix chain, 2,239 → 5,329 tok/s across α (§3.3 — the linear convention the graph prints says 0.140, and overstates by up to 1.8×) | +| **D and acceptance α** | 1.73× cost at D=5; break-even α = **0.426** on a prefix chain, 2,239 → 5,329 tok/s across α (§3.3 — the linear convention the graph prints says 0.146, and overstates by up to 1.8×) | Two more move single rows and are named where they appear: **absorbed-vs-unabsorbed MLA** (±2× on `attn_kv_b` + `attn_out_proj`, Q1) and **expert imbalance** (skew @@ -688,7 +684,7 @@ graph**, to be confirmed against a capture. | **2** | **Grouped-GEMM group sizing (the fork)** | either **76 D2H per token** (no graph capture possible) **or** fixed-capacity padding (**all 256 experts read every step**) | The only data-dependent shape in the graph is the fused gating kernel. A stack does one or the other — see §5.2 | `cuda_api_sum` D2H count per decode step | 0 D2H **and** MoE bytes that track `distinct_experts(B)` → a device-side path, nothing to recover | | **3** | ⚑ **CUDA-graph capture at low batch** | **63 % of the B=1 floor is launch overhead** (81 % at eager 5 µs) | 1,033 nodes × 2 µs = 2.07 ms against a 1.24 ms memory term | `cuda_api_sum` launch count with `--cuda-graph-trace=node`; expect **1** `cudaGraphLaunch` | already captured → this rank is worth nothing, and rank 2's fork is already resolved to "padded" | | **4** | **The indexer scan at long context** | at 1M it is **54 % of the step**, and IndexShare's 3.7× is already banked | 90.2 GB/step of index keys at S=1M. Levers: fp8 index keys (2×), temporal reuse across steps, a smaller `index_topk_freq` group | `dram__bytes_read.sum` on the scan vs 90.2 GB; whether keys are fp8 | keys already fp8 and no temporal reuse available → architectural | -| **5** | **`moe_routed` — the memory-bound heart** | 73.8 % of decode; **EPLB placement and the grouped-GEMM backend are the levers** | Weight traffic scales with *distinct* experts woken, not with FLOPs. Good placement cuts per-rank distinct traffic; DeepGEMM cuts the constant | measured per-rank expert traffic vs `distinct_experts`; **the real EP imbalance** | measured traffic already at the union prediction with balance ≈1.0 → the bank is the bank | +| **5** | **`moe_routed` — the memory-bound heart** | 74.1 % of decode; **EPLB placement and the grouped-GEMM backend are the levers** | Weight traffic scales with *distinct* experts woken, not with FLOPs. Good placement cuts per-rank distinct traffic; DeepGEMM cuts the constant | measured per-rank expert traffic vs `distinct_experts`; **the real EP imbalance** | measured traffic already at the union prediction with balance ≈1.0 → the bank is the bank | | **6** | ⚑ **fp32 router at prefill** | **11.0 % of prefill** rests on the reading that the router *GEMM* runs in fp32 | 26 GF/layer at 67 TF/s. If only the softmax/top-k accumulates in fp32 and the GEMM is bf16, this row shrinks ~15× | the engine's router implementation; `cuda_gpu_kern_sum` dtype of the gate GEMM | the GEMM is genuinely fp32 → architectural, and the row stays | | **7** | **Decode collective placement** | 158 all-reduces + 76 all-to-alls + 1 logits all-gather per step on the compute stream, **latency-bound**, with idle SMs around them | 688 kB payloads: the gap *is* the cost. Nothing prevents other layers' work overlapping | NCCL ranges and stream ids on the timeline (S2/S3) | already on a separate stream with overlap → nothing to recover | | **8** | **Chunk size** | a decode-latency-protecting `--max-num-batched-tokens` can cost **14.9× the prefill bytes** | the expert bank is read per chunk | total prefill MoE DRAM ÷ 95.7 GB — the quotient **is** the chunk count | quotient ≈ 1 → prefill is already unchunked | @@ -755,7 +751,7 @@ Two worked examples, because the rule is easy to agree with and hard to apply: layers' work overlapping. Q2: *yes* — stream assignment. **Recoverable** (rank 7). **The trap runs in both directions.** The accept/reject readback will look alarming -and is architectural. The 1,614 kernel launches are entirely *expected* from §3 and +and is architectural. The 1,591 kernel launches are entirely *expected* from §3 and are the largest recoverable item at low batch. **Neither surprise nor familiarity is evidence.** @@ -808,7 +804,7 @@ Key: **R:** recoverable → the rank it feeds · **A:** architectural, do not ch |---|---|---|---|---| | **0** | **Launch args, as text** — not a measurement | C6 | chunk size, graph capture, KV dtype, D, TP/EP, absorbed MLA | Resolves or reframes **ranks 1, 2, 3, 6, 8 before a timeline is opened** | | **1** | `cuda_api_sum` — **D2H count per decode step** | C1 | **0** in the MoE region | **R:** 76/token → host-resolved group sizes, no graph capture → **rank 2**. **R:** 0 but MoE bytes flat in B → padded capacity → **rank 3** instead (§5.2). **A:** none | -| **2** | `cuda_api_sum` — **launches per step**, needs `--cuda-graph-trace=node` | C1 | **1** `cudaGraphLaunch` | **R:** ~1,614 individual launches + CPU gaps below B≈16 → **rank 3**. **F1:** count wildly off 1,614 → the lowering in §3 is wrong by an order of magnitude | +| **2** | `cuda_api_sum` — **launches per step**, needs `--cuda-graph-trace=node` | C1 | **1** `cudaGraphLaunch` | **R:** ~1,591 individual launches + CPU gaps below B≈16 → **rank 3**. **F1:** count wildly off 1,591 → the lowering in §3 is wrong by an order of magnitude | | **3** | `dram__bytes_read.sum` — **MoE region** | C1, C3 | 1.26 GB/layer/rank = **95.7 GB/pass**; **≈85 %** of decode | **R:** prefill total ÷ 95.7 GB > 1 → chunked re-read, and the quotient **is** the chunk count → **rank 8**. **R:** flat in B → padded capacity → **rank 3**. **F2:** much lower → L2 residency, and the central claim of both phases is wrong | | **4** | `dram__bytes_read.sum` — **indexer region**, swept in S | C2 | 33.6 MB/layer at 8K → **90.2 GB/step at 1M**, on **21 layers only** | **R:** keys read on 78 layers → IndexShare is not being honoured, a hidden 3.7× → **rank 4**. **R:** 2× the prediction → keys are bf16 where fp8 would do. **A:** growth on 21 layers is the architecture | | **5** | **NCCL kernel duration vs payload** | C1, C3, C5 | prefill **∝ payload** (~900 GB/s, 117 ms a2a); decode **flat, ~2 µs × 235** | **R:** prefill a2a at bf16 payload → fp8 dispatch → **rank 1**. **R:** decode collectives on the compute stream with idle SMs → **rank 7**. **A:** the ring latency floor | @@ -910,7 +906,7 @@ graph change. The commands at the top of this note regenerate any of them. | **Q6** | Is the **EP dispatch** bf16 or fp8? | **half of 105.7 GB** at prefill — the largest single recoverable number here | serving image / C6 | | **Q7** | Is the **IndexShare selection** passed device-side? | 21 syncs/step if not (S9) | trace D2H attribution | | **Q8** | Chunked prefill on, at what chunk size? | **up to 14.9× on prefill bytes** | engine launch args (C6) | -| **Q9** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is **0.415** on a prefix chain (0.140 under the linear convention the graph prints — §3.3); 0.5→0.9 is 2,239→5,329 tok/s | engine metrics (C4) — **not predictable from a config** | +| **Q9** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is **0.426** on a prefix chain (0.146 under the linear convention the graph prints — §3.3); 0.5→0.9 is 2,239→5,329 tok/s | engine metrics (C4) — **not predictable from a config** | | **Q10** | Are the **index keys** cached in fp8 or bf16? | 2× on 90.2 GB/step at 1M context | C6 / trace | | **Q11** | Is `eh_proj` **TP-sharded** in the MTP block? | 151 MB → 19 MB per rank per draft stage | serving image | | **Q12** | Does the prefill attention kernel read the selected KV once per request, or once per query tile? | Up to 64× on that node — but **the node is 0.10 % of prefill bytes**, so even a 128-row tiling takes the step from 422 GB to 448 GB. **1.1×, and it does not move the prefill conclusion.** Listed last because it is bounded, not because it is small | `dram__bytes_read.sum` on the prefill core | @@ -1038,8 +1034,8 @@ stay small: | context S | `.6b` bytes/layer | `.6b` time/layer | Σ over 21 layers | share of step | | --------- | ----------------- | ---------------- | ---------------- | ------------- | | 8,192 | 33.6 MB | 6.99 µs | 0.147 ms | 0.9 % | -| 131,072 | 537.0 MB | 0.112 ms | 2.349 ms | 12.5 % | -| 1,048,576 | **4,296.0 MB** | **0.895 ms** | **18.795 ms** | **53.4 %** | +| 131,072 | 537.0 MB | 0.112 ms | 2.349 ms | 12.7 % | +| 1,048,576 | **4,296.0 MB** | **0.895 ms** | **18.795 ms** | **53.9 %** | **Σ per layer at S=8192: 8.3 GF, 928.4 MB, 0.222 ms** — 5 % more than `Ls,sh`, and that 5 % is the whole price of IndexShare's 21-of-78 schedule at short context. @@ -1095,14 +1091,15 @@ selection the backbone already paid for. | region | ×N | nodes each | Σ nodes | Σ ms | share | | ------ | -- | ---------- | ------- | ---- | ----- | | prologue + epilogue | 1 | 4 | 4 | 0.073 | 0.4 % | -| `Ld,f` dense layers | 3 | 17 | 51 | 0.155 | 0.9 % | -| `Ls,f` full-indexer MoE | 18 | 22 | 396 | 3.999 | 24.2 % | -| `Ls,sh` shared-indexer MoE | 57 | 20 | 1,140 | 12.028 | 72.7 % | -| `Lmtp` draft (D=1) | 1 | 23 | 23 | 0.297 | 1.8 % | -| **total** | | | **1,614** | **16.551** | | - -At D=5 the draft region becomes 5 × 23 = 115 nodes and 1.483 ms, and the backbone -runs at 192 rows instead of 32 — **1,706 nodes, 28.135 ms.** +| `Ld,f` dense layers | 3 | 17 | 51 | 0.155 | 1.0 % | +| `Ls,f` full-indexer MoE | 18 | 22 | 396 | 3.999 | 24.6 % | +| `Ls,sh` shared-indexer MoE | 57 | 20 | 1,140 | 12.028 | 74.0 % | + +| **total** | | | **1,591** | **16.254** | | + +At D=5 a draft region of 5 × 23 = 115 nodes and 1.483 ms appears, and the backbone +runs at 192 rows instead of 32 — **1,706 nodes, 28.135 ms.** At D=0 there is no +draft region at all. The prologue/epilogue rows are 4 nodes and 0.4 % of the step, and two of them are on the critical path between the last layer and the sample: `lm_head` re-reads diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 5747277..26b0b1b 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -944,10 +944,19 @@ def add_lm_head(rows: float, layer: int | None) -> None: # It is not a smaller copy of the model. The block carries a full # ``mlp.experts.*`` bank, so its cost is expert weight traffic paid ``D`` times # over, not arithmetic. - if spec.num_nextn_predict_layers > 0 and batch.positions_per_step > 0: + # ``num_nextn_predict_layers`` says the block exists in the checkpoint; it does + # not say the engine runs it. Drafting happens only under a speculative config, + # so at D=0 there are zero drafted tokens and zero stages — the module's weights + # stay resident (``model_weight_bytes`` counts them) and none of its kernels + # launch. Forcing one stage here charged a pure decode step 0.3 ms of drafting + # that a server without ``--speculative-config`` never does. + if ( + spec.num_nextn_predict_layers > 0 + and batch.speculative_tokens > 0 + and batch.positions_per_step > 0 + ): draft_batch = replace(batch, prefill_tokens=0, speculative_tokens=0) - stages = max(1, batch.speculative_tokens) - for stage in range(stages): + for stage in range(batch.speculative_tokens): _emit_layer( g, spec, hw, spec.n_layers + stage, batch=draft_batch, sh=sh, diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index 6de5f89..24fee99 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -111,7 +111,7 @@ def test_indexshare_shared_layers_emit_no_indexer(): # One per full-indexer layer, and the MTP head shares (no indexer node). assert n_proj == n_score == spec.n_full_indexer_layers == 21 # Every layer still runs the attention core over the selected positions. - assert len(_ops(g, "attn_score_value")) == spec.n_layers + spec.num_nextn_predict_layers + assert len(_ops(g, "attn_score_value")) == spec.n_layers def test_mla_kv_traffic_uses_shared_latent_not_heads(): @@ -137,9 +137,7 @@ def test_dense_prefix_runs_ffn_not_mixture(): # Two per sparse block: the h->256 GEMM, then the fused gating kernel that # scores and selects. Same op name because the trace cannot tell them apart # (see MOE_LAYER_NODES), so the count is doubled, not the node list. - assert len(_ops(g, "moe_router")) == 2 * ( - spec.n_sparse_mlp_layers + spec.num_nextn_predict_layers - ) + assert len(_ops(g, "moe_router")) == 2 * spec.n_sparse_mlp_layers def test_precision_is_bf16_no_fp4_leak(): @@ -229,7 +227,12 @@ def test_footprint_counts_the_indexers_the_graph_emits(shares): Asserted as an identity rather than a constant, so it holds either way round. """ spec = replace(_spec(), index_share_for_mtp_iteration=shares) - emitted = len(_ops(predict_glm_graph(spec), "attn_index_proj")) + # With a draft stage actually running — at D=0 the block's weights are + # resident but none of its kernels launch, so node count and footprint + # legitimately diverge there and the identity is about the stage that runs. + g = predict_glm_graph(spec, batch=BatchConfig(batch=1, kv_cache_len=4096, + speculative_tokens=1)) + emitted = len(_ops(g, "attn_index_proj")) expected = spec.n_full_indexer_layers + (0 if shares else spec.num_nextn_predict_layers) assert emitted == expected @@ -357,7 +360,7 @@ def test_two_collectives_per_layer_not_one(): spec, batch=BatchConfig(batch=1, kv_cache_len=4096), sharding=ShardingConfig(tp=8), ) - n_blocks = spec.n_layers + spec.num_nextn_predict_layers + n_blocks = spec.n_layers # no draft stage without --spec-tokens assert len(_ops(g, "tp_all_reduce_attn")) == n_blocks assert len(_ops(g, "tp_all_reduce_mlp")) == n_blocks @@ -375,9 +378,7 @@ def test_dense_layers_dispatch_no_experts(): spec, batch=BatchConfig(batch=1, kv_cache_len=4096), sharding=ShardingConfig(tp=8, ep=8), ) - assert len(_ops(g, "moe_all_to_all")) == ( - spec.n_sparse_mlp_layers + spec.num_nextn_predict_layers - ) + assert len(_ops(g, "moe_all_to_all")) == spec.n_sparse_mlp_layers assert not [n for n in g.nodes if n.op == "moe_all_to_all" and n.layer < 3] @@ -454,6 +455,32 @@ def test_indexer_scans_with_every_index_head(): assert scan.prediction.bytes == pytest.approx(pairs * spec.index_head_dim * 2) +def test_no_draft_head_without_a_speculative_config(): + """``num_nextn_predict_layers`` says the block exists, not that it runs. + + Drafting happens only under a speculative config. At D=0 nothing is drafted, + so no stage is emitted — the weights stay resident (``model_weight_bytes`` + still counts them) and none of their kernels launch. Emitting one anyway + charged a pure decode step 0.3 ms of drafting a server without + ``--speculative-config`` never does. + """ + spec = _spec() + batch = BatchConfig(batch=32, kv_cache_len=8192) + g = predict_glm_graph(spec, batch=batch) + assert not [n for n in g.nodes if n.layer is not None and n.layer >= spec.n_layers] + assert len(_ops(g, "lm_head")) == 1 # the epilogue only + assert len(_ops(g, "attn_q_a")) == spec.n_layers + + # The weights are still resident either way — that is the distinction. + assert model_weight_bytes(spec) > model_weight_bytes( + replace(spec, num_nextn_predict_layers=0) + ) + + # One stage per drafted token once a speculative config exists. + g5 = predict_glm_graph(spec, batch=replace(batch, speculative_tokens=5)) + assert len(_ops(g5, "mtp_eh_proj")) == 5 + + def test_a_pure_prefill_step_runs_no_draft_head(): """The draft proposes continuations; a prefill chunk has nothing to continue. @@ -575,8 +602,7 @@ def test_act_quant_exists_only_where_a_gemm_is_actually_fp8(): assert not _ops(predict_glm_graph(bf16, batch=batch), "act_quant") # Two per block: one ahead of the attention GEMMs, one ahead of the FFN — # the two groups of fp8 GEMMs, with bf16 work in between. - blocks = fp8.n_layers + fp8.num_nextn_predict_layers - assert len(_ops(predict_glm_graph(fp8, batch=batch), "act_quant")) == 2 * blocks + assert len(_ops(predict_glm_graph(fp8, batch=batch), "act_quant")) == 2 * fp8.n_layers def test_plan_warns_that_a_speculative_rate_is_a_ceiling(): From 00d60a66166664e39d4e691436e4442463e91c0d Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 18:14:32 -0700 Subject: [PATCH 23/35] Prefill is comm-bound under EP8, not memory-bound MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The whole-pass AI of 281 divided FLOPs by all 422 GB of bytes — 289 GB of HBM traffic plus 133 GB of NVLink payload — and then compared the result against a ridge derived from HBM bandwidth. Two pools, two bandwidths, one denominator; the comparison was not sound. Against HBM alone the AI is 410 against an fp8 ridge of 412. Prefill is balanced on that axis to within half a percent, not comfortably memory-bound as the note claimed. And the floor is set elsewhere entirely: 148 of 264 ms is interconnect time, so the label is comm. The row-level table already said this — the EP all-to-all is 44.5% of the step — while the summary two sections earlier said memory-bound. The summary was the wrong one. Also worth stating plainly: this reverses under TP8-only, where the wire term largely disappears and the pass goes back to HBM-bound. That is the same EP on/off fork Q1 and capture C5 turn on, and it now decides a bound label rather than just a row. Break-even table: gave the closed form (1-a^6)/(1-a) and moved the reason the linear formula is wrong onto the struck row itself — it assumes independent acceptance where a verifier takes a prefix. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 39 ++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index ef99c84..aa60064 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -418,17 +418,30 @@ default costs **707 GB and 319 ms**, not 422 GB and 264 ms. | Prefill, P=8192, C=0, TP8/EP8, per rank | value | | --- | --- | | predicted floor | **264.0 ms** for the chunk (31.0 k tok/s) | -| bytes moved | **422.0 GB** | | FLOPs | **118.4 TF** | -| whole-pass AI | **281** — below the fp8 ridge of 412, so **memory-bound overall** | -| facets | compute 666 n / 89.3 ms / 33.8 % · memory 848 n / 174.6 ms / 66.1 % · launch 77 n / 0.2 ms | - -> ⚠ **Dense intuition says prefill is compute-bound. Here it is not**, for two -> structural reasons: 256 experts × top-8 reads the whole bank per layer -> regardless of P, and expert parallelism turns the dispatch into a wire-bound -> all-to-all. `confidence: high for the arithmetic, medium for the conclusion` — -> the soft link is that essentially all 256 experts are hit, which assumes routing -> is not pathologically concentrated (A5). +| bytes — **HBM** | **289.0 GB** | +| bytes — **interconnect** | **133.2 GB** | +| AI against HBM | **410**, against an fp8 ridge of **412** — *at* the ridge, not below it | +| where the time goes | **wire 148.0 ms (56 %)** · HBM + compute 116.0 ms (44 %) | + +**Two byte pools, and only one of them answers to the HBM ridge.** An earlier +version of this table divided FLOPs by *all* 422 GB and reported AI 281, which +mixed 133 GB of NVLink payload into a denominator the ridge derives from HBM +bandwidth. Against HBM alone the pass sits at AI 410 — balanced on that axis to +within half a percent — and the thing actually setting the floor is the wire. + +> ⚠ **Dense intuition says prefill is compute-bound. Under EP8 it is +> communication-bound**, and the two claims are not close: 56 % of the predicted +> floor is interconnect time. On the HBM axis the pass is balanced (AI 410 vs +> ridge 412), so neither "compute-bound" nor "memory-bound" is the right label for +> it — the label is *comm*. Two structural reasons: 256 experts × top-8 reads the +> whole bank per layer regardless of P, and expert parallelism turns the dispatch +> into an all-to-all that is 105.7 GB per pass. +> +> `confidence: high for the arithmetic, medium for the conclusion` — the soft +> links are that essentially all 256 experts are hit (A5) and that EP is on at all +> (Q1). Under TP8-only the wire term largely disappears and the pass reverts to +> HBM-bound; that is the same fork §4.2 and capture C5 turn on. **Chunk size multiplies the whole MoE term.** The same 8,192 tokens: @@ -486,10 +499,10 @@ not small at D=5. | α | 0.0 † | 0.5 | 0.7 | 0.9 | break-even | | --- | --- | --- | --- | --- | --- | -| accepted tokens/step, `Σ αⁱ` (i=0…5) | 1.000 | 1.969 | 2.941 | 4.686 | **1.731** | +| accepted tokens/step, `Σ αⁱ` = `(1−α⁶)/(1−α)` | 1.000 | 1.969 | 2.941 | 4.686 | **1.731** | | **tok/s** = `32 × Σ αⁱ ÷ 28.135 ms` | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.426** | | MTP off, for comparison = `32 ÷ 16.254 ms` | 1,969 | 1,969 | 1,969 | 1,969 | — | -| ~~linear `1+Dα`~~ — **do not use** | 1,137 | ~~3,981~~ | ~~5,118~~ | ~~6,256~~ | ~~α > 0.146~~ | +| ~~linear `1+Dα`~~ — **do not use**: assumes *independent* acceptance, but a verifier takes a **prefix** — token *k* only counts if 1…*k*−1 also passed | 1,137 | ~~3,981~~ | ~~5,118~~ | ~~6,256~~ | ~~α > 0.146~~ | Each row divides its own token count by its own step time, which is what makes the two comparable: break-even is where they meet, at `Σ αⁱ = 28.135/16.254 = @@ -638,7 +651,7 @@ at TP8/EP8, FP8. | **Pre** | attention core ×79 | compute, **linear in C** | FLOPs capped at 2,048 keys/query; bytes are the whole cache **once per request — an optimistic floor** (A9/Q12), but a bounded one: this node is 0.10 % of prefill bytes, so the worst tiling is 1.1× on the step | FP8 KV | ⚑ **`index_topk`**; C; ⚠ **per-request vs per-tile** | | **Pre** | permute / combine | memory | 8.5 GB/layer-pass on the `8P`-row expanded tensor, near-zero FLOPs | BF16 activations | top-k; **expert imbalance** | | **Pre** | *everything else* — embed gather, norms, `lm_head`, gating | memory or launch | each under 1.5 %; `lm_head` reads 239.5 MB for **one row per request** | BF16; FP32 top-k | none of them flips | -| **Pre** | **whole prefill pass** | **memory** | **AI 281 vs ridge 412**; 66 % memory / 34 % compute | mixed FP8/BF16/FP32 | prompt length; **chunk size**; EP degree | +| **Pre** | **whole prefill pass** | ⚑ **comm** under EP8 | **56 % of the floor is wire**. On the HBM axis alone AI 410 vs ridge 412 — balanced, not memory-bound | mixed FP8/BF16/FP32 | **EP on/off** (Q1); prompt length; chunk size | | **MTP** | verify expert GEMMs | **memory** | 256 distinct experts at 192 rows vs 163 at 32 — **+57 %/layer, ~85 % of MTP's cost** | FP8 block-scaled | `B(1+D)` vs E=256; **D**; imbalance | | **MTP** | draft expert GEMMs ×D | **memory** | the MTP block carries a **full 256-expert bank**; 5 stages × 163 experts, **linear in D, no saturation** | FP8 block-scaled | **D**; batch | | **MTP** | draft `lm_head` ×5 | memory | 239.5 MB × 5 = **1.20 GB = 19 % of the draft's bytes** | **BF16** | sharded sampling; draft vocab | From 860ef62e9a7e5aab966fd9f567578d9291d998ef Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 18:16:14 -0700 Subject: [PATCH 24/35] A skipped norm does not imply a skipped projection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "indexer" needle in _UNQUANTISED_OPS also matched indexer.k_norm, so a norm appearing in modules_to_not_convert marked the whole indexer bf16. Right for GLM-5.2, where indexers_proj is listed too, and wrong in general: every fp8 scheme leaves norms wide, so a norm in that list carries no information about the projection's width. A checkpoint that skipped only the norm and quantised the GEMM would have been read backwards. Needles now name the projections — indexers_proj, indexer.wq_b, indexer.wk, indexer.weights_proj. The catalogue entry is unaffected (it declares the override directly), and a test pins both directions: a norm alone leaves the projection fp8, naming the projection makes it bf16. Same class as the embed_tokens fix earlier in this PR — a mapping that happened to give the right answer for this checkpoint while answering the wrong question. Co-Authored-By: Claude Opus 5 --- gitm/planner/glm_graph.py | 10 +++++++++- tests/test_glm_graph.py | 25 +++++++++++++++++++++++++ 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/gitm/planner/glm_graph.py b/gitm/planner/glm_graph.py index 26b0b1b..3064bb9 100644 --- a/gitm/planner/glm_graph.py +++ b/gitm/planner/glm_graph.py @@ -1021,7 +1021,15 @@ def is_glm_moe_dsa_config(cfg: dict[str, Any]) -> bool: ("lm_head", "lm_head"), ("embed_tokens", "embed_tokens"), ("eh_proj", "mtp_eh_proj"), - ("indexer", "attn_index_proj"), # subsumes indexers_proj, indexer.k_norm + # The indexer's *projections*, named specifically. A bare "indexer" needle also + # matches ``indexer.k_norm``, and a norm carries no information about the + # projection's width — every fp8 scheme leaves norms wide, so a checkpoint that + # skipped only the norm and quantised ``indexers_proj`` would be read as + # leaving the whole indexer bf16. + ("indexers_proj", "attn_index_proj"), + ("indexer.wq_b", "attn_index_proj"), + ("indexer.wk", "attn_index_proj"), + ("indexer.weights_proj", "attn_index_proj"), ("mlp.gate", "moe_router"), ) diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index 24fee99..98b8abb 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -166,6 +166,7 @@ def test_fp8_checkpoint_reads_what_the_quantiser_skipped(): "modules_to_not_convert": [ "model.layers.0.input_layernorm", "model.layers.47.mlp.gate.e_score_correction_bias", + "model.layers.74.self_attn.indexers_proj", "model.layers.74.self_attn.indexer.k_norm", "model.layers.78.eh_proj", "lm_head", @@ -182,6 +183,30 @@ def test_fp8_checkpoint_reads_what_the_quantiser_skipped(): assert spec.dtype_for("moe_router", spec.weight_dtype) == "fp32" +def test_a_skipped_norm_does_not_imply_a_skipped_projection(): + """``indexer.k_norm`` in the not-convert list says nothing about the GEMM. + + Every fp8 scheme leaves norms wide, so a norm appearing there carries no + information. A bare ``indexer`` needle matched it and marked the whole indexer + bf16 — right for GLM-5.2, where ``indexers_proj`` is listed too, and wrong for + any checkpoint that skipped only the norm. + """ + cfg = dict(GLM_CONFIG) + cfg["quantization_config"] = { + "quant_method": "fp8", + "modules_to_not_convert": ["model.layers.74.self_attn.indexer.k_norm"], + } + spec = spec_from_hf_config(cfg) + assert spec.dtype_for("attn_index_proj", spec.weight_dtype) == "fp8" + + # Name the projection and it is honoured. + cfg["quantization_config"]["modules_to_not_convert"].append( + "model.layers.74.self_attn.indexers_proj" + ) + spec = spec_from_hf_config(cfg) + assert spec.dtype_for("attn_index_proj", spec.weight_dtype) == "bf16" + + def test_embed_tokens_carries_its_own_declared_precision(): """The untied halves are two tensors and the checkpoint names them separately. From a6691024048e2fb1fb53e7ece55b598361d48be0 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 18:20:49 -0700 Subject: [PATCH 25/35] Count accepted tokens as a prefix chain, in BatchConfig MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The note carried a warning about its own tool's output, which was the wrong place to fix it. tokens_per_step computed 1 + D*alpha -- the answer for independent draws -- where a verifier walks the draft in order and stops at the first rejection. The expectation is sum(alpha^i) for i=0..D. At D=5, alpha=0.5 the linear form claims 3.5 accepted tokens against a real 1.97. It is shared with every family and it was wrong for all of them: any verifier accepts a prefix. Tree attention, which verifies several candidate continuations at once, would need its own term and does not have one here -- said so in the docstring rather than leaving it implied. Safe to change: tokens_per_step has one behavioural consumer and one test pin, and no graph uses it to compute a floor. It is a reporting quantity. Two things fell out while fixing it. The plan table's rate line never used tokens_per_step at all -- it printed batch/step, so --acceptance-rate did nothing and the CLI was reporting the alpha=0 floor. It now reports the accepted rate, and prints 1,137 / 2,239 / 5,329 at alpha 0 / 0.5 / 0.9, matching §3.3 exactly. And my earlier CLI warning was itself wrong: it said the rate assumed 1 + D*alpha when the rate assumed no acceptance at all. Replaced with one that fires only when no acceptance rate is given. Doc: the struck "do not use" row is gone, since there is nothing left to warn about. Also named the indexer's actual tensors in the precision table (wq_b, wk, weights_proj alongside indexers_proj), and marked Lmtp in the archetype table as running only under a speculative config -- 3 + 18 + 57 = 78 transformer blocks, with the 79th conditional. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 62 ++++++++++++++++++------------------- gitm/planner/registry.py | 38 +++++++++-------------- gitm/planner/roofline.py | 17 +++++++++- tests/test_glm_graph.py | 34 +++++++++++++------- tests/test_moe_graph.py | 4 ++- 5 files changed, 86 insertions(+), 69 deletions(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index aa60064..78d52bf 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -134,7 +134,12 @@ is byte-for-byte identical between them. | `Ld,f` | **3** | MLA+DSA | **full** | dense 12288 | 576 + 128 | 2× all-reduce | | `Ls,f` | **18**| MLA+DSA | **full** | MoE 256×2048 | 576 + 128 | 2× all-reduce (+2× a2a under EP) | | `Ls,sh` | **57**| MLA+DSA | shared | MoE 256×2048 | 576 | 2× all-reduce (+2× a2a under EP) | -| `Lmtp` | **1** | MLA+DSA | shared | MoE 256×2048 | 576 | 2× all-reduce, ×D stages | +| `Lmtp` | **1**, ×D | MLA+DSA | shared | MoE 256×2048 | 576 | 2× all-reduce per stage | + +`Ld,f` and `Ls,f` are both full-indexer archetypes — 3 + 18 = **21**, matching §1. +The transformer stack is 3 + 18 + 57 = **78**; `Lmtp` is a 79th block that exists +in the checkpoint and runs **only under a speculative config**, once per drafted +token (§3.3). The two schedules do not line up — a 3-layer dense prefix, a 3-layer `full` prefix, then an IndexShare period of 4 — so no single modulo rule reproduces @@ -190,7 +195,7 @@ B200s and `--max-num-seqs 32` for full context. | `q_a`/`q_b`/`kv_a`/`kv_b`, **`o_proj`**, dense FFN | **FP8 e4m3**, 128×128 block | absent from `modules_to_not_convert` | | routed experts, shared expert | **FP8 e4m3**, 128×128 block | absent from `modules_to_not_convert` | | `lm_head`, `embed_tokens` | **BF16** | named in `modules_to_not_convert` | -| **lightning indexer** (`indexers_proj`, `k_norm`) | **BF16** | named in `modules_to_not_convert` | +| **lightning indexer** — `indexers_proj`, `wq_b`, `wk`, `weights_proj` (+ `k_norm`) | **BF16** | named in `modules_to_not_convert` | | MTP `eh_proj`, `enorm`, `hnorm` | **BF16** | named in `modules_to_not_convert` | | MoE router (`mlp.gate` + `e_score_correction_bias`) | **FP32** | `moe_router_dtype: "float32"`, base config | | all norms | **BF16** | named in `modules_to_not_convert` | @@ -490,36 +495,28 @@ resident and none of their kernels launch. **~86 % of the price of speculation is the MoE expert bank** — charged because more rows and stages touch more experts, not because more work is done per token. -**Break-even, and a caveat that moves it by 3×.** `BatchConfig.tokens_per_step` -counts accepted tokens as `1 + D·α` — the repo-wide convention, used by every -family. That is right for independent draws and wrong for speculative decoding, -where the verifier accepts a *prefix*: token *k* only counts if 1…*k*−1 were also -accepted, so expected accepted length is `Σ αⁱ` for *i* = 0…D. The difference is -not small at D=5. +**Break-even.** Accepted tokens follow a **prefix chain**: the verifier walks the +draft in order and stops at the first rejection, so token *k* counts only if +1…*k*−1 did. The expectation is `Σ αⁱ = (1−α^(D+1))/(1−α)`, not `1 + D·α` — that +would be the answer for *independent* draws, and at D=5 it overstates accepted +tokens by 1.8× at α=0.5. -| α | 0.0 † | 0.5 | 0.7 | 0.9 | break-even | +| α | 0.0 | 0.5 | 0.7 | 0.9 | break-even | | --- | --- | --- | --- | --- | --- | -| accepted tokens/step, `Σ αⁱ` = `(1−α⁶)/(1−α)` | 1.000 | 1.969 | 2.941 | 4.686 | **1.731** | +| accepted tokens/step, `(1−α⁶)/(1−α)` | 1.000 | 1.969 | 2.941 | 4.686 | **1.731** | | **tok/s** = `32 × Σ αⁱ ÷ 28.135 ms` | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.426** | | MTP off, for comparison = `32 ÷ 16.254 ms` | 1,969 | 1,969 | 1,969 | 1,969 | — | -| ~~linear `1+Dα`~~ — **do not use**: assumes *independent* acceptance, but a verifier takes a **prefix** — token *k* only counts if 1…*k*−1 also passed | 1,137 | ~~3,981~~ | ~~5,118~~ | ~~6,256~~ | ~~α > 0.146~~ | Each row divides its own token count by its own step time, which is what makes -the two comparable: break-even is where they meet, at `Σ αⁱ = 28.135/16.254 = -1.731`. Divide by the **MTP step** (28.135 ms), not by the baseline rate: the step is -1.70× longer, so a rate ratio against the 1,969 tok/s baseline is not an -accepted-token count. That baseline carries *no* acceptance convention of its own -— at D=0 `tokens_per_step` degenerates to `batch`. - -† At α=0 both formulas give exactly one accepted token, so the two rows agree by -construction. The 1,137 is the cost of drafting for nothing, and it sits **below** -the 1,969 MTP-off baseline — which is the point of keeping the column. - -**The struck row is what `gitm plan --spec-tokens` prints today**, and it -overstates throughput by up to 1.8×. It is shown only so the discrepancy is -recognisable in CLI output; the CLI itself now says so. Fixing the convention -means changing shared `BatchConfig` semantics for every family, so it is stated -here rather than made here. +them comparable: break-even is where they meet, at `Σ αⁱ = 28.135/16.254 = 1.731`. + +`BatchConfig.tokens_per_step` computes the chain, so +`gitm plan --spec-tokens 5 --acceptance-rate 0.5` prints the 2,239 above. It used +to compute `1 + D·α` — an earlier revision of this note carried a warning about +its own tool's output, which was the wrong place to fix it. The linear form was +shared with every family and wrong for all of them, since any verifier accepts a +prefix; a tree-attention scheme verifying several candidate continuations at once +would need its own term and does not have one here. Z.ai claims the GLM-5.2 MTP layer raises accepted length up to 20 % over its predecessor, which likely clears even the chained bar — but **α is a serving @@ -678,7 +675,7 @@ row, with the magnitude each is worth: | ⚑ **CUDA-graph capture** | 69 % of the B=1 floor, 85 % at eager 5 µs. Decides whether MTP is a 3× win or a net loss | | ⚑ **EP vs TP** | the a2a is 45 % of prefill under EP8, but the per-rank bank is **8× smaller** — a trade, not a cost, since the bank is 85 % of decode DRAM | | ⚑ **Precision, and KV dtype** | 1.79× on the decode floor and **10.7 → 5.4 H200s** for weights; 55 GB vs 100 GB of KV per rank at 1M | -| **D and acceptance α** | 1.73× cost at D=5; break-even α = **0.426** on a prefix chain, 2,239 → 5,329 tok/s across α (§3.3 — the linear convention the graph prints says 0.146, and overstates by up to 1.8×) | +| **D and acceptance α** | 1.73× cost at D=5; break-even α = **0.426**, 2,239 → 5,329 tok/s across α (§3.3) | Two more move single rows and are named where they appear: **absorbed-vs-unabsorbed MLA** (±2× on `attn_kv_b` + `attn_out_proj`, Q1) and **expert imbalance** (skew @@ -897,8 +894,10 @@ Five things, in the order they re-rank the tables. `git log` has the file list. 5. **The MTP chain is D stages deep**, each with its own vocabulary projection (G7), driven by `--spec-tokens`. -Two supporting fixes outside the family: an fp32 peak for modern SKUs (G4), and -`gitm plan` keeping the launch bound and pricing the ridge per dtype (G5). +Three supporting fixes outside the family: an fp32 peak for modern SKUs (G4), +`gitm plan` keeping the launch bound and pricing the ridge per dtype (G5), and +`BatchConfig.tokens_per_step` counting accepted tokens as a prefix chain rather +than `1 + D·α` — shared with every family and wrong for all of them. **Deleted:** three committed JSON node dumps, 29k lines that went stale on every graph change. The commands at the top of this note regenerate any of them. @@ -919,7 +918,7 @@ graph change. The commands at the top of this note regenerate any of them. | **Q6** | Is the **EP dispatch** bf16 or fp8? | **half of 105.7 GB** at prefill — the largest single recoverable number here | serving image / C6 | | **Q7** | Is the **IndexShare selection** passed device-side? | 21 syncs/step if not (S9) | trace D2H attribution | | **Q8** | Chunked prefill on, at what chunk size? | **up to 14.9× on prefill bytes** | engine launch args (C6) | -| **Q9** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is **0.426** on a prefix chain (0.146 under the linear convention the graph prints — §3.3); 0.5→0.9 is 2,239→5,329 tok/s | engine metrics (C4) — **not predictable from a config** | +| **Q9** | **α**, the MTP acceptance rate, in production | the entire MTP decision. Break-even is **0.426**; 0.5→0.9 is 2,239→5,329 tok/s | engine metrics (C4) — **not predictable from a config** | | **Q10** | Are the **index keys** cached in fp8 or bf16? | 2× on 90.2 GB/step at 1M context | C6 / trace | | **Q11** | Is `eh_proj` **TP-sharded** in the MTP block? | 151 MB → 19 MB per rank per draft stage | serving image | | **Q12** | Does the prefill attention kernel read the selected KV once per request, or once per query tile? | Up to 64× on that node — but **the node is 0.10 % of prefill bytes**, so even a 128-row tiling takes the step from 422 GB to 448 GB. **1.1×, and it does not move the prefill conclusion.** Listed last because it is bounded, not because it is small | `dram__bytes_read.sum` on the prefill core | @@ -968,7 +967,8 @@ KV per rank on top of a 96 GB weight share. ## Appendix A — Predicted node tables Trace-day reference for §3. **B=32, S=8192, TP8/EP8, FP8 weights and KV, per -rank.** The 79 blocks are `Ld,f` ×3 + `Ls,f` ×18 + `Ls,sh` ×57 + `Lmtp` ×1; A.2–A.4 +rank.** The 78 transformer blocks are `Ld,f` ×3 + `Ls,f` ×18 + `Ls,sh` ×57, plus +`Lmtp` ×D when drafting is on; A.2–A.4 are **deltas** from A.1, since everything unlisted is byte-for-byte identical. Two columns are omitted rather than repeated. Every node runs on the compute diff --git a/gitm/planner/registry.py b/gitm/planner/registry.py index 45fe8b0..98ab880 100644 --- a/gitm/planner/registry.py +++ b/gitm/planner/registry.py @@ -256,7 +256,15 @@ def _render_table(g, hw: HardwareSpec, spec, family: str, note: str) -> str: f"{g.batch.prefill_tokens / total:,.0f} tok/s " f"prefilling {g.batch.prefill_tokens:,} tokens" if g.batch.is_prefill - else f"{g.batch.batch / total:,.0f} tok/s at batch {g.batch.batch}" + # ``tokens_per_step`` is the accepted-token count: the batch on a + # plain decode step, and the prefix-chain expectation once drafting is + # on. Reporting ``batch / total`` there would price D drafts and then + # credit none of them. + else f"{g.batch.tokens_per_step / total:,.0f} tok/s at batch " + f"{g.batch.batch}" + + (f", D={g.batch.speculative_tokens} " + f"alpha={g.batch.acceptance_rate:g}" + if g.batch.speculative_tokens > 0 else "") ), f" {len(g.nodes)} nodes, {n_compute} compute-bound, " f"{n_launch} launch-bound", @@ -264,31 +272,13 @@ def _render_table(g, hw: HardwareSpec, spec, family: str, note: str) -> str: if any(b for b in bounds.values() if len(b) > 1): out.append(" * this op's instances do not share a bound — the label is " "the majority one") - if g.batch.speculative_tokens > 0: - # ``tokens_per_step`` counts accepted tokens as ``1 + D*alpha``. A verifier - # accepts a *prefix*, so the real expectation is ``sum(alpha**i)`` — smaller - # for every alpha < 1, by up to 1.8x at D=5, alpha=0.5. The convention is - # shared with every family and not this module's to change, but a rate - # printed from it should not be read as achievable. - d = g.batch.speculative_tokens - a = g.batch.acceptance_rate - linear = 1.0 + d * a - chain = sum(a ** i for i in range(d + 1)) + if g.batch.speculative_tokens > 0 and g.batch.acceptance_rate <= 0: + # A speculative step with no acceptance rate given prices the work and + # reports one accepted token, which is the floor rather than the outcome. out.append( - f" ! speculative step (D={d}): rates above assume accepted tokens = " - f"1+D*alpha. A verifier accepts a prefix, so the expectation is " - f"sum(alpha^i)" + f" ! speculative step (D={g.batch.speculative_tokens}) with no " + "--acceptance-rate: the rate above assumes every draft is rejected" ) - if a > 0 and linear > chain: - out.append( - f" at alpha={a:g} that is {linear:.2f} vs {chain:.2f} tokens/step " - f"— the printed rate is {linear / chain:.2f}x optimistic" - ) - else: - out.append( - " set --acceptance-rate to compare the two; break-even is " - "higher than the linear form implies" - ) if g.has_unpriced_collectives: out.append(" ! collectives unpriced — this SKU has no interconnect bandwidth " "in the catalogue") diff --git a/gitm/planner/roofline.py b/gitm/planner/roofline.py index f5d1a41..f440e54 100644 --- a/gitm/planner/roofline.py +++ b/gitm/planner/roofline.py @@ -591,8 +591,23 @@ def tokens_per_step(self) -> float: Always at least ``batch``: the non-speculative token is verified, not drafted, so it is never rejected. + + The speculative term is a **prefix chain**, not a product. A verifier + walks the draft in order and stops at the first rejection, so draft token + *k* is kept only if 1…*k*-1 were also kept: the expectation is + ``sum(alpha**i for i in 0..D)``, not ``1 + D*alpha``. The two are far + apart where it matters — at D=5, alpha=0.5 the linear form claims 3.5 + accepted tokens against a real 1.97, overstating throughput 1.8x and + putting break-even at less than a third of its true value. + + This models a single-chain verifier (EAGLE/MTP-style), which is what every + family here drafts with. A tree-attention scheme that verifies several + candidate continuations at once accepts more than one chain and would need + its own term. """ - return self.batch * (1.0 + max(0, self.speculative_tokens) * self.acceptance_rate) + d = max(0, self.speculative_tokens) + a = self.acceptance_rate + return self.batch * sum(a ** i for i in range(d + 1)) @dataclass(frozen=True) diff --git a/tests/test_glm_graph.py b/tests/test_glm_graph.py index 98b8abb..188d492 100644 --- a/tests/test_glm_graph.py +++ b/tests/test_glm_graph.py @@ -630,26 +630,36 @@ def test_act_quant_exists_only_where_a_gemm_is_actually_fp8(): assert len(_ops(predict_glm_graph(fp8, batch=batch), "act_quant")) == 2 * fp8.n_layers -def test_plan_warns_that_a_speculative_rate_is_a_ceiling(): - """``tokens_per_step`` counts ``1 + D*alpha``; a verifier accepts a prefix. +def test_accepted_tokens_follow_a_prefix_chain(): + """A verifier stops at the first rejection, so acceptance compounds. - The convention is shared with every family and not this branch's to change, so - the rate is printed as-is — but a number that is up to 1.8x optimistic at D=5 - must not leave the CLI unlabelled, or the design note's caveat protects only - the readers who found the design note. + ``1 + D*alpha`` is the independent-draws answer and overstates throughput by + up to 1.8x at D=5; the CLI reported it, and the design note had to carry a + warning about its own output. Fixed at the source instead. """ + b = BatchConfig(batch=32, speculative_tokens=5, acceptance_rate=0.5) + chain = sum(0.5 ** i for i in range(6)) + assert b.tokens_per_step == pytest.approx(32 * chain) + assert b.tokens_per_step < 32 * (1 + 5 * 0.5) # strictly under the linear form + + # No speculation, no chain: the term degenerates to the batch. + assert BatchConfig(batch=32).tokens_per_step == 32 + # Perfect acceptance keeps every drafted token. + assert BatchConfig( + batch=1, speculative_tokens=5, acceptance_rate=1.0 + ).tokens_per_step == pytest.approx(6) + + +def test_plan_flags_a_speculative_step_with_no_acceptance_rate(): + """Priced without an acceptance rate, the reported rate is a floor.""" from gitm.planner.registry import main buf = io.StringIO() with contextlib.redirect_stdout(buf): main(["glm-5.2-fp8", "--gpu", "H200", "--batch", "32", "--kv-len", "8192", - "--tp", "8", "--ep", "8", "--spec-tokens", "5", - "--acceptance-rate", "0.5"]) - out = buf.getvalue() - assert "speculative step (D=5)" in out - assert "1.78x optimistic" in out + "--tp", "8", "--ep", "8", "--spec-tokens", "5"]) + assert "no --acceptance-rate" in buf.getvalue() - # A non-speculative step says nothing, because nothing is being approximated. buf = io.StringIO() with contextlib.redirect_stdout(buf): main(["glm-5.2-fp8", "--gpu", "H200", "--batch", "32", "--kv-len", "8192"]) diff --git a/tests/test_moe_graph.py b/tests/test_moe_graph.py index c922623..f8c3d3b 100644 --- a/tests/test_moe_graph.py +++ b/tests/test_moe_graph.py @@ -406,7 +406,9 @@ def test_tokens_per_step_accounts_for_acceptance(): """Drafts are paid for always and counted only when kept.""" b = BatchConfig(batch=4, speculative_tokens=3, acceptance_rate=0.5) assert b.positions_per_step == 16 # all drafted work is computed - assert b.tokens_per_step == pytest.approx(4 * (1 + 3 * 0.5)) + # A prefix chain, not 1 + D*alpha: the verifier stops at the first rejection, + # so token k counts only if 1..k-1 did. + assert b.tokens_per_step == pytest.approx(4 * (1 + 0.5 + 0.25 + 0.125)) # ── the observed side lines up with the predicted side ────────────────────── From 2db5f215f29ef51868528666126dfc892e08a353 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 18:22:32 -0700 Subject: [PATCH 26/35] Show where a draft stage's 0.297 ms goes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second review to reconstruct the draft chain from the expert bank alone and find time missing. Their 0.804 ms is exactly the moe_routed term (0.803); the other 0.680 ms is lm_head, mtp_eh_proj and the attention block, itemised in A.4 the whole time. Twice now A.4 has not been reached from §3.3, so the split is in §3.3: 54% expert bank, 17% vocabulary projection, 11% eh_proj, 18% the rest. That also makes the section's own point better than the prose did — the draft is not just an expert-bank cost, and the two BF16 tensors it drags along are more than a quarter of it. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 78d52bf..99107fe 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -488,7 +488,12 @@ resident and none of their kernels launch. - **Verify, +39.1 GB**, almost all one line: the expert union saturates, so 6× the rows costs 1.57× the expert bytes (163 → 256 distinct). KV read does **not** move — 0.43 GB either way, read per *sequence* — and neither does `lm_head`. -- **The draft chain, +6.34 GB**, all of it new work. Each of 5 stages draws on a full 256-expert bank, +- **The draft chain, +6.34 GB**, all of it new work. A stage is **0.297 ms** and + the chain is 5 of them: `moe_routed` 0.161 ms, `lm_head` 0.050, `mtp_eh_proj` + 0.032, the rest of the attention block 0.054. The expert bank is **54 %** of a stage, + not all of it — the vocabulary projection (17 %) and the `[12288→6144]` fusion + (11 %) are another 28 % together, and neither gets cheaper for being a draft. + Itemised per node in **Appendix A.4**. Each of 5 stages draws on a full 256-expert bank, linear in D with no saturation to help, so **the draft is 5.3 % of the MTP step where a dense-draft model's would be 1–2 %**. From 1e73a95b24efb26aa19be327cba06194dc8e2728 Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 18:24:16 -0700 Subject: [PATCH 27/35] Name the crossover rate, and say how the embedding shards MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three clarity items, no behaviour change. The break-even cell gave alpha > 0.426 without saying what it crosses; it now says the tok/s row meets 1,969 there, which is the MTP-off rate a row below. Seventh reading of the verify row as though its node count should scale with rows. Verify is the backbone at 6x the rows and the same node count — more work per kernel, not more kernels — which is also why its bytes move and its nodes do not. Said on the row. Gemini asked how embed_tokens shards, and the note only ever stated it for lm_head. The code divides both by tp because vLLM builds the input table as VocabParallelEmbedding, split by vocabulary the same way the output projection is, and both sibling families already price it that way. Now stated in §3.1 rather than left to be inferred from the code — this is the third time it has been asked. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 99107fe..409d142 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -387,6 +387,12 @@ The step does not begin at layer 0 or end at layer 77. | E1 | `lm_head` | GEMM (BF16), tall-skinny | `[32,6144] × [6144,19360]` | 7.61 GF | **239.5 MB** | memory | | E2 | `logits_all_gather` | collective (all-gather) | `[32,19360] → [32,154880]` fp32 | 0 F | **17.3 MB** | memory | +**Both vocabulary tensors shard, not just `lm_head`.** vLLM builds the input +embedding as `VocabParallelEmbedding`, split by vocabulary across TP ranks the +same way the output projection is, so `model_weight_bytes` divides both by `tp` — +as the dense-MoE and hybrid families already do. A framework that replicated the +input table instead would hold `tp × 1.9 GB / 2` more per node. + **D0 reads what it selects, not the table** — 393 kB at 32 rows; the resident 1.9 GB matters for the fit math, not the step. **E0 runs over `logits_rows`**, so at prefill it is one row per prompt and not the chunk. **E2 exists because E1 is @@ -470,6 +476,10 @@ At B=32, S=8192, D=5 (the vendor recipe's `num_speculative_tokens`): | vanilla decode (D=0) | 1,591 | 0 | **1,591** | 67.34 GB | 16.254 ms | | MTP step (D=5) | 1,591 | 115 (5 × 23) | **1,706** | 112.77 GB | **28.135 ms** | | — the verify pass alone | 1,591 | — | 1,591 | 106.43 GB | 26.652 ms | + +Verify is the backbone at **6× the rows and the same node count** — more work per +kernel, not more kernels. That is why its bytes rise (106.43 vs 67.34 GB) while +its node count does not move. | — the draft chain alone | — | 115 | 115 | 6.34 GB | 1.483 ms | The backbone is the same **1,591 nodes in every row** — the 78 transformer layers @@ -509,7 +519,7 @@ tokens by 1.8× at α=0.5. | α | 0.0 | 0.5 | 0.7 | 0.9 | break-even | | --- | --- | --- | --- | --- | --- | | accepted tokens/step, `(1−α⁶)/(1−α)` | 1.000 | 1.969 | 2.941 | 4.686 | **1.731** | -| **tok/s** = `32 × Σ αⁱ ÷ 28.135 ms` | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.426** | +| **tok/s** = `32 × Σ αⁱ ÷ 28.135 ms` | 1,137 | **2,239** | **3,345** | **5,329** | **α > 0.426**, where this row crosses 1,969 | | MTP off, for comparison = `32 ÷ 16.254 ms` | 1,969 | 1,969 | 1,969 | 1,969 | — | Each row divides its own token count by its own step time, which is what makes From 3a1e191d68bb9be0fa3f985992840ae56866019b Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 18:26:28 -0700 Subject: [PATCH 28/35] Name the fp8 constant the KV formula borrows weight_bytes("fp8") is 1.000244 because of the 128x128 block scales stored with fp8 weights. An fp8 KV cache carries a per-token or per-tensor scale instead, so using the weight-side constant for cache bytes is the wrong one -- 52,618 against a truer 52,608. Left the number alone: 0.02% moves nothing, and changing kv_bytes_per_token would mean a cache-specific byte constant that no other family has and that no measurement here could distinguish. Named in the formula block so it is a known approximation rather than an unexamined one. Co-Authored-By: Claude Opus 5 --- docs/glm-5.2/DESIGN-NOTE.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/glm-5.2/DESIGN-NOTE.md b/docs/glm-5.2/DESIGN-NOTE.md index 409d142..b6ed2ac 100644 --- a/docs/glm-5.2/DESIGN-NOTE.md +++ b/docs/glm-5.2/DESIGN-NOTE.md @@ -175,6 +175,12 @@ whole model, bytes per token of context (fp8 latent, bf16 rope key + index key): an fp8 weight costs 1.000244 B, not 1 B — the 128×128 block scale (§7.0) 78 × (512×1.000244 + 64×2) + 21 × 128×1.000244 = 52,618 B/token bf16 throughout: 95,232 B/token + + ⚠ 1.000244 is a *weight*-side constant borrowed for cache bytes. An fp8 KV + cache carries a per-token or per-tensor scale, not 128×128 blocks, so the + true figure is nearer 78×640 + 21×128 = 52,608. The 10-byte gap is 0.02% + and moves nothing; it is named because the constant is the wrong one, not + because the number is. ``` | Context | fp8 KV | bf16 KV | From 8db324e2b49b1bbfd79f9c5b4b088d008d3464ed Mon Sep 17 00:00:00 2001 From: Nicholas Lawrence Date: Wed, 2 Sep 2026 18:53:44 -0700 Subject: [PATCH 29/35] CI: stop failing PRs on review-bot outages, and show the bot the code Two problems, both in the review workflows rather than in any branch. Failing on upstream errors. claude-review exits 1 on any HTTPError, so an expired key or a 429 blocks the PR; gemini-review's action exits non-zero on Gemini's 503, which happened five times on #104. An advisory reviewer that cannot reach its API has found nothing -- it has not found a problem. Both now retry the transient classes (429, 5xx, network, timeout) with backoff and then pass with a ::warning:: rather than a red check. 4xx that will fail identically on retry -- auth, billing, malformed -- are not retried, so a dead key costs one call, not three. lint and pytest stay blocking; they are the checks that say something about the code. The bot was never shown the code. The diff was filtered to '*.py' '*.sh' '*.md' '*.yaml' '*.yml' in one git diff, which orders by path, then truncated to 75KB. On #104 that is 88KB of docs/ ahead of 132KB of gitm/, so every one of ~19 reviews saw .github and part of DESIGN-NOTE.md and never a line of Python -- which is why they all opened "this is a documentation-only diff" and why every finding was about prose. Code paths are now diffed first and prose appended, so truncation drops prose. Also told the model the diff may be truncated, since several reviews inferred things from what was absent. Co-Authored-By: Claude Opus 5 --- .github/workflows/claude-review.yml | 51 ++++++++++++++++++++------ .github/workflows/gemini-pr-review.yml | 5 +++ 2 files changed, 45 insertions(+), 11 deletions(-) diff --git a/.github/workflows/claude-review.yml b/.github/workflows/claude-review.yml index e6c8a00..35bb735 100644 --- a/.github/workflows/claude-review.yml +++ b/.github/workflows/claude-review.yml @@ -16,11 +16,17 @@ jobs: with: fetch-depth: 0 + # Code first, prose second. The diff is truncated below when it is large, + # and `git diff` orders by path — so a PR that touches `docs/` and `gitm/` + # used to spend its whole budget on documentation and never show the + # reviewer a line of Python. Every review of PR #104 opened by calling it + # "a documentation-only diff" while 2k lines of planner changes sat past + # the cut. Emitting code first means truncation drops prose instead. - name: Get PR diff run: | - git diff origin/${{ github.base_ref }}...HEAD \ - -- '*.py' '*.sh' '*.md' '*.yaml' '*.yml' \ - > pr_diff.txt + BASE="origin/${{ github.base_ref }}" + git diff "$BASE"...HEAD -- '*.py' '*.sh' '*.yaml' '*.yml' > pr_diff.txt + git diff "$BASE"...HEAD -- '*.md' >> pr_diff.txt echo "Diff size: $(wc -c < pr_diff.txt) bytes" - name: Run Claude Review @@ -44,7 +50,7 @@ jobs: fi python3 << 'EOF' - import json, os, subprocess, sys, urllib.request + import json, os, subprocess, sys, time, urllib.error, urllib.request with open("pr_diff.txt") as f: diff = f.read() @@ -64,6 +70,8 @@ jobs: "**⚡ Performance** — unnecessary CPU↔GPU transfers, missed parallelism\n" "**📊 Reproducibility** — seed handling, non-determinism risks\n" "**💡 Suggestions** — missing error handling, untested edge cases\n\n" + "The diff may be truncated; review what is present and do not " + "infer anything from what is missing.\n\n" f"```diff\n{diff}\n```" ) }] @@ -80,13 +88,34 @@ jobs: } ) - try: - with urllib.request.urlopen(req) as resp: - data = json.load(resp) - comment = data["content"][0]["text"] - except urllib.error.HTTPError as e: - print(f"API error {e.code}: {e.read().decode()}", file=sys.stderr) - sys.exit(1) + # Retry the transient classes only. 429 and 5xx are load; 4xx otherwise + # (auth, billing, malformed) will fail identically on every attempt, so + # retrying them just burns runner minutes. + comment = None + reason = None + for attempt in range(3): + try: + with urllib.request.urlopen(req, timeout=180) as resp: + comment = json.load(resp)["content"][0]["text"] + break + except urllib.error.HTTPError as e: + reason = f"HTTP {e.code}: {e.read().decode()[:300]}" + if e.code not in (429, 500, 502, 503, 504) or attempt == 2: + break + time.sleep(5 * (attempt + 1)) + except (urllib.error.URLError, TimeoutError, KeyError, IndexError) as e: + reason = f"{type(e).__name__}: {e}" + if attempt == 2: + break + time.sleep(5 * (attempt + 1)) + + # An advisory reviewer that cannot reach its API has found nothing; it + # has not found a problem. Failing the check here blocks a PR on someone + # else's outage or on an expired key, which is not a signal about the + # code. Say so in the log and pass. + if comment is None: + print(f"::warning::Claude review skipped — {reason}", file=sys.stderr) + sys.exit(0) body = f"## 🤖 Claude Code Review\n\n{comment}" subprocess.run([ diff --git a/.github/workflows/gemini-pr-review.yml b/.github/workflows/gemini-pr-review.yml index 2576518..0c8cd1d 100644 --- a/.github/workflows/gemini-pr-review.yml +++ b/.github/workflows/gemini-pr-review.yml @@ -18,7 +18,12 @@ jobs: with: fetch-depth: 0 # Fetches full history so git diff can calculate cleanly + # Advisory, so an upstream outage must not block a PR. This action exits + # non-zero when the Gemini API answers 503 "experiencing high demand", + # which it did five times on PR #104 alone — a red check that says nothing + # about the branch. The review still posts when the API answers. - name: Run Gemini Code Review + continue-on-error: true uses: sshnaidm/gemini-code-review-action@v2 with: gemini-key: ${{ secrets.GEMINI_API_KEY }} From 22364f8096ed834e11c1cf048550ea14d52ea6fa Mon Sep 17 00:00:00 2001 From: Adit Chawdhary <25533953+aditchawdhary@users.noreply.github.com> Date: Fri, 4 Sep 2026 02:07:40 -0700 Subject: [PATCH 30/35] Delete tests/fixtures/importers/mixed_dump/nsys_2024_min.sqlite --- .../importers/mixed_dump/nsys_2024_min.sqlite | Bin 32768 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 tests/fixtures/importers/mixed_dump/nsys_2024_min.sqlite diff --git a/tests/fixtures/importers/mixed_dump/nsys_2024_min.sqlite b/tests/fixtures/importers/mixed_dump/nsys_2024_min.sqlite deleted file mode 100644 index dff4aa3132294f8cc955105f82c1c9622c8ec106..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 32768 zcmeI5e{36P8OPtd_Y&tE`|h0hSIgSooUFC;qjNHo6_a&w6O)?!Xo%geDL<~ReU804 zcCas5QUR*FY^_Ea)2Oy03UmUb!aoc&VB(jG1Y)dN0Tsc7gqTFBI##x>8!$Rnc+cmH zb8*}CAEbbulM>(izR&x8p7--U&q?paQlcLk&M2~&$`_{Mig*_i5JjOrQA7yQf00e-* zEfOf%O(vJmN0kyvd|y@`jKoHy(da}}8VScj##TfBcsLXbi?PtQ;jm~N6TP!?p_s|% z#8`NDtV1lyMYSarm3Tpsaz!yR8Ve7G$2-KNJex_#qwy)ZVybmWiTq4X(an63`g+&8 zb<|`zs1aF-55$$Y{%JA7o}JMg_&!-@a<#WprLMA8EHpkCj!BWx5A2Wz z$9Ar0)*F;p42oW}xZ=tr^@uyfCGQ3+-=TKKO>3k5kFSID+6>#mV1&gT-b`Dt1AY%uU03SpBg6rvWaH>5QZ9_b&OxZawz5$0Xl zXm+ay$-a3-UREK+LSiVhY&XPYQ89KT^3(IndyEVh*3xB$hwr^1!=do_Xn6Q~47Cyb zt_(H1)x$6;PsMYSSvgsLvMfDqis^hoF?1PNskuE_$Ru~`+1OETP8geemYOw__hs{m z{mW*T+smex+c%j{icrZUqUm@+PL9Y^`NI5?FEKe!9w|2%tO=wbPiBe=c?OTkg;<)@ z=2ER1OnrMOlWW5Us)WnA5mgGA++-wKT)LqRJHON|daI8*acn#?5*nWnhr$yb;<6{L zdNvn06XF;7qx^P$1NR2^BKK|X5$+(D=RU^m;O^l%xwXUqE+7B|fB+Bx0zd!=00AHX z1b_e#_LTQwAC1&T|Ha6ddjOSx4XKx%=HAMY;tmolA(zhOOMO=TeE`fF8o&AA`i&A9fX zU!n*5wupBcRa8H|qwd%O#gJsIhq3jT&S{u8+*QKE0c1N}=aTECTY2 zZM7gWAwXKF8F(WsC;^0cl{p(&^3 zi1V)FOryqyA+fX=b%-a&x${o*F~+Qs@q0W*72?V9Wv}Dcml;;$B9J}0y;da`b9^zt zG-zbV-#Lvcs-IBWk=@IfD*4j7!Kgv9!QoCi?p2t2jf$ELesO(`j{T3P9nNW{u0lp! zXVf8{>|gWRe|MQ-D*58M&8R_EWLIMLuLKyZaZ&f^J+&%>X?tNWL+f1D8db=C+Se!T zVTGYIE+lt7Ym6$yQ}drs+i#m@P_>fWD|Cp>x0vQXE#iOcmE=DEH29lesg9>)q5nrd!MHV-NIUm6#Zy`tTyLp(LTa=O_wjo;QQsoAJP zJT-lfX?l4PU#(VBQ?1Iw{-*B;_@9+Z5^Y8m)z8F0(?TzPtCBCRf>DEHLzBO+>Ao@i z5511m?B#28Y_Feg@=oKw>y^YAb%-b1j~Lr)i}-JPC0UIcWJR{4e%lWO{N_?6iI!Rw zX~6bKFMdPk(r8p6*t469KJ)yZT1mJ@h5wR;U%Z6>QmG`7HmVRm{KrN9gd4wH z$(L5js6n!U-`2-x#_&bGj!5>Z=l?o>0r404v-~ss7sxGKKmZ5;0U!VbfB+Bx0zd!= z00AHX1c1PQfqekFYUWJ@Qyid+ooCa;t+MX|zx)551twRBK#qB`ux z6?=-Jgy_@4eaBnc@kK;5+_XBvs#7STqSemuDkZtY7mPZAuVxoRO;y1@zK2tvb)BGahFOXkw0RbQY1b_e#00KY&2mk>f z00e*l5C8(Jo`9Ri?A@cgA_I|-80zx-JEOZt0-XW>;5Ijf8?Ha1{{5fgFOc*9s$U)i z2n2ut5C8%|00;m9AOHk_01yBIKmZ8blmI#ZZ(0|gnfz9vT+@+oW z&msQYO$A^O2mk>f00e*l5C8%|00;m9AOHk_01#MR1lp;(b@g{s`U&6l Date: Fri, 4 Sep 2026 02:08:17 -0700 Subject: [PATCH 31/35] Delete tests/fixtures/importers/torch_trace_min.json.gz --- tests/fixtures/importers/torch_trace_min.json.gz | Bin 929 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 tests/fixtures/importers/torch_trace_min.json.gz diff --git a/tests/fixtures/importers/torch_trace_min.json.gz b/tests/fixtures/importers/torch_trace_min.json.gz deleted file mode 100644 index 8f15e5bd9f2119396c19c2a00f3118cd5a1ecf1b..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 929 zcmV;S177?eiwFP!00002|J7PiPunmM{wr0V0r~D6C*h5$(?A+~*rW+*LQzOesc4%j zaX}3Azt72v>U8b0rM7T+vaiIx+;_h3vz^zWD$`7Ue3Elj4bKNZUWfB^CUtALD2q&1 zRnqiNbpN5YUN_B2uG^qnX*sSR+dTE<`_<^`VsHVBoqoHy=BJ!}`Y>FrPW1o$9^s}Q zAxl-=ew5{0PHV&4^Vw2OQ$6cg&Stf}@k$TI&Y14y%d+V_p&6W~wMBKV_cCshsJYe4bR&hyc=Au!#U#O#?I|KV*1N;IErD!#my8-uzsh`>nb7mtgq~Z4o*z^JEJ3fzi7MvFqL?K2%StbS zG|!Fq7reUy_Ju$2VexD1b@G2!+oeC`Ch%YtcyZ90x0m2fjbj=5FW$U9#M1`yuLXQc_Q=o(>-$5UH1AWK=rP@6%$x~4B!9Q|#JDqkhx|xR@(H5uiq{wX;27whFz&8- zebYery|pLKx^FT1q91yHl1~9SJKV#_f4V0TIg@%Ad%U%$2%YIXVt$|GQ%ue@9)Z7f z&lz`iV@r4pTYC`~zN)}9BUGnq%?C{FUZh@7cC z3O%NK9)UBRN0G-{dmeLVcefP3i@qN(mWduDTfCU>f8AIHm#P>U-z>B)?}9^6zk5;g z?ejnV*UPz)DlW~hXiVqOhI7F|zL|jwTSve)e8WKW^=%8($hvR3Vq5YpwvMUb-L6$q zZMn9#G+yD>7pvV~Z5-J*ai%9O7A@_m Date: Fri, 4 Sep 2026 02:09:20 -0700 Subject: [PATCH 32/35] Delete tests/fixtures/importers/nsys_2025_min.sqlite --- tests/fixtures/importers/nsys_2025_min.sqlite | Bin 32768 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 tests/fixtures/importers/nsys_2025_min.sqlite diff --git a/tests/fixtures/importers/nsys_2025_min.sqlite b/tests/fixtures/importers/nsys_2025_min.sqlite deleted file mode 100644 index 346229a6eac0191176235b51e3da8ee238236ebb..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 32768 zcmeI5e{36P8OPtd_Y&tE`|h0hSIgSooUFC;qjNHo6_a&w6O)?!Xo%geDL<~ReU804 zcCas5QUR*FY^_Ea)2Oy03UmUb!aoc&VB(jG1Y)dN0Tsc7gqTFBI##x>8!$Rnc+cmH zb8*}CAEbbulM>(izR&x8p7--U&q?paQlcLk&M2~&$`_{Mig*_i5JjOrQA7yQf00e-* zEfOf%O(vJmN0kyvd|y@`jKoHy(da}}8VScj##TfBcsLXbi?PtQ;jm~N6TP!?p_s|% z#8`NDtV1lyMYSarm3Tpsaz!yR8Ve7G$2-KNJex_#qwy)ZVybmWiTq4X(an63`g+&8 zb<|`zs1aF-55$$Y{%JA7o}JMg_&!-@a<#WprLMA8EHpkCj!BWx5A2Wz z$9Ar0)*F;p42oW}xZ=tr^@uyfCGQ3+-=TKKO>3k5kFSID+6>#mV1&gT-b`Dt1AY%uU03SpBg6rvWaH>5QZ9_b&OxZawz5$0Xl zXm+ay$-a3-UREK+LSiVhY&XPYQ89KT^3(IndyEVh*3xB$hwr^1!=do_Xn6Q~47Cyb zt_(H1)x$6;PsMYSSvgsLvMfDqis^hoF?1PNskuE_$Ru~`+1OETP8geemYOw__hs{m z{mW*T+smex+c%j{icrZUqUm@+PL9Y^`NI5?FEKe!9w|2%tO=wbPiBe=c?OTkg;<)@ z=2ER1OnrMOlWW5Us)WnA5mgGA++-wKT)LqRJHON|daI8*acn#?5*nWnhr$yb;<6{L zdNvn06XF;7qx^P$1NR2^BKK|X5$+(D=RU^m;O^l%xwXUqE+7B|fB+Bx0zd!=00AHX z1b_e#_LTQwAC1&T|Ha6ddjOSx4XKx%=HAMY;tmolA(zhOOMO=TeE`fF8o&AA`i&A9fX zU!n*5wupBcRa8H|qwd%O#gJsIhq3jT&S{u8+*QKE0c1N}=aTECTY2 zZM7gWAwXKF8F(WsC;^0cl{p(&^3 zi1V)FOryqyA+fX=b%-a&x${o*F~+Qs@q0W*72?V9Wv}Dcml;;$B9J}0y;da`b9^zt zG-zbV-#Lvcs-IBWk=@IfD*4j7!Kgv9!QoCi?p2t2jf$ELesO(`j{T3P9nNW{u0lp! zXVf8{>|gWRe|MQ-D*58M&8R_EWLIMLuLKyZaZ&f^J+&%>X?tNWL+f1D8db=C+Se!T zVTGYIE+lt7Ym6$yQ}drs+i#m@P_>fWD|Cp>x0vQXE#iOcmE=DEH29lesg9>)q5nrd!MHV-NIUm6#Zy`tTyLp(LTa=O_wjo;QQsoAJP zJT-lfX?l4PU#(VBQ?1Iw{-*B;_@9+Z5^Y8m)z8F0(?TzPtCBCRf>DEHLzBO+>Ao@i z5511m?B#28Y_Feg@=oKw>y^YAb%-b1j~Lr)i}-JPC0UIcWJR{4e%lWO{N_?6iI!Rw zX~6bKFMdPk(r8p6*t469KJ)yZT1mJ@h5wR;U%Z6>QmG`7HmVRm{KrN9gd4wH z$(L5js6n!U-`2-x#_&bGj!5>Z=l?o>0r404v-~ss7sxGKKmZ5;0U!VbfB+Bx0zd!= z00AHX1c1PQfqekFYUWJ@Qyid+ooCa;t+MX|zx)551twRBK#qB`ux z6?=-Jgy_@4eaBnc@kK;5+_XBvs#7STqSemuDkZtY7mPZAuVxoRO;y1@zK2tvb)BGahFOXkw0RbQY1b_e#00KY&2mk>f z00e*l5C8(Jo`9Ri?A@cgA_I|-80zx-JEOZt0-XW>;5Ijf8?Ha1{{5fgFOc*9s$U)i z2n2ut5C8%|00;m9AOHk_01yBIKmZ8blmI#ZZ(0|gn@7>(B*{_}d z&msQYO$A^O2mk>f00e*l5C8%|00;m9AOHk_01#MR1lp;(b@g{s`U&6l Date: Fri, 4 Sep 2026 02:11:51 -0700 Subject: [PATCH 33/35] Delete tests/fixtures/importers/mixed_dump/torch_trace_min.json.gz --- .../importers/mixed_dump/torch_trace_min.json.gz | Bin 929 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 tests/fixtures/importers/mixed_dump/torch_trace_min.json.gz diff --git a/tests/fixtures/importers/mixed_dump/torch_trace_min.json.gz b/tests/fixtures/importers/mixed_dump/torch_trace_min.json.gz deleted file mode 100644 index 8f15e5bd9f2119396c19c2a00f3118cd5a1ecf1b..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 929 zcmV;S177?eiwFP!00002|J7PiPunmM{wr0V0r~D6C*h5$(?A+~*rW+*LQzOesc4%j zaX}3Azt72v>U8b0rM7T+vaiIx+;_h3vz^zWD$`7Ue3Elj4bKNZUWfB^CUtALD2q&1 zRnqiNbpN5YUN_B2uG^qnX*sSR+dTE<`_<^`VsHVBoqoHy=BJ!}`Y>FrPW1o$9^s}Q zAxl-=ew5{0PHV&4^Vw2OQ$6cg&Stf}@k$TI&Y14y%d+V_p&6W~wMBKV_cCshsJYe4bR&hyc=Au!#U#O#?I|KV*1N;IErD!#my8-uzsh`>nb7mtgq~Z4o*z^JEJ3fzi7MvFqL?K2%StbS zG|!Fq7reUy_Ju$2VexD1b@G2!+oeC`Ch%YtcyZ90x0m2fjbj=5FW$U9#M1`yuLXQc_Q=o(>-$5UH1AWK=rP@6%$x~4B!9Q|#JDqkhx|xR@(H5uiq{wX;27whFz&8- zebYery|pLKx^FT1q91yHl1~9SJKV#_f4V0TIg@%Ad%U%$2%YIXVt$|GQ%ue@9)Z7f z&lz`iV@r4pTYC`~zN)}9BUGnq%?C{FUZh@7cC z3O%NK9)UBRN0G-{dmeLVcefP3i@qN(mWduDTfCU>f8AIHm#P>U-z>B)?}9^6zk5;g z?ejnV*UPz)DlW~hXiVqOhI7F|zL|jwTSve)e8WKW^=%8($hvR3Vq5YpwvMUb-L6$q zZMn9#G+yD>7pvV~Z5-J*ai%9O7A@_m Date: Fri, 4 Sep 2026 02:13:20 -0700 Subject: [PATCH 34/35] Delete tests/fixtures/importers/parity_nsys.sqlite --- tests/fixtures/importers/parity_nsys.sqlite | Bin 32768 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 tests/fixtures/importers/parity_nsys.sqlite diff --git a/tests/fixtures/importers/parity_nsys.sqlite b/tests/fixtures/importers/parity_nsys.sqlite deleted file mode 100644 index af444b72a75e3a0dc97843c6ca6a9731a1e05012..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 32768 zcmeI)-*3}K00;0pyRDtD&LA^ie41>P#!@5zbpr|tXXe9}2lTHkN@5oZsF zLEnwoEpnMqN>&&ngh=A0idPyG5t&L8Vox*i)1netd{LPdrAkpI+Uwb0zVI#?#ea&P zNFqQ00uX=z1Rwwb2%L?;56VpG#tph3xjQ{>>&-R8c1-KLO=qof|DB{-YZ|(3Fk8R7 zF@De0Wea|XnU-y=8Osz8K5CVPJRGN-Fmi*)8HE3zy2B5Ak6Y;lR9TYYc*Ffclvb;l(=(-VnRerg zs`JQQbt5<3&*Uy7&1DOdURBwF+uJ|x;nT0Dlv1ljpOxc-)$TWJ)6r|TxnQpt@}CcFv@3m!@W^9CvWz=D8*(!&!1|xZZs`p<9%jHiTqKsq~6(?b$H($bbGuL z->Q?DBitPZQ7$WIGalmI!0$XrM_V;cw)4sDN%BPFo!-!UlxZEOGtJ}l2il99D)PP7 zt{d=9o%e^qXmT#r8HpL`_uk}Q%vG&yQ>8RNPY>mIEVQD) zA9T%5IEhHPPkNG_osU_gY1Z{-o2?t|+|)H*n@ieTqWvx25Fh{n2tWV=5P$##AOHaf zKmY;|_{;?gl1#33dB5-M6lfZC$LoJt`;};KwBN-J0Rj+!00bZa0SG_<0uX=z1Rwx` zGZvVWugJ2h(!z02RaKr{5u5jol$1~^6$+wURH83lN#>CxN#w~2o1<4$(NeUVl#|~Y z^hd_{}#!vk47A0TgkB>(;)Yj?%N z1JN(4{YAuv009U<00Izz00bZa0SG_<0uX?}`4G^k5h;X}|39)}?Z*HB From a872584eba5a6177436a27d7259aeed71e0d42c9 Mon Sep 17 00:00:00 2001 From: Adit Chawdhary <25533953+aditchawdhary@users.noreply.github.com> Date: Fri, 4 Sep 2026 02:15:20 -0700 Subject: [PATCH 35/35] Delete tests/fixtures/importers/nsys_2023_min.sqlite --- tests/fixtures/importers/nsys_2023_min.sqlite | Bin 32768 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 tests/fixtures/importers/nsys_2023_min.sqlite diff --git a/tests/fixtures/importers/nsys_2023_min.sqlite b/tests/fixtures/importers/nsys_2023_min.sqlite deleted file mode 100644 index 90773521d1b68d95cf46fe6a338ceb44548ea104..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 32768 zcmeI5e{36P8OPtd_Y&tE`|h0hSIgSooUFC;qjOR#D<{ZvB*SprO zqo&G1jmS!TAg;vqPm>`Ky;t<+s-g$t*;&nj?-O+MCo+LKB1Gm=qa(|Cls5 zzH>#h-k`ixY9SVtf*hY- z-jm1|3UW5CkZoVqT_aVT&n05>GqUd4VBk9(!X{TJL@iivN^2xM(my_Vqcv+I%)7GD z>{btwee;UEtU`)~#8770ZivaEV(dudXXcmp7#S|CrOOPD+;>xkL*a?h@bHZoY9shv z8EST`hhb8lj_0PbamrXCXZ!w<~p^`~N)A53w9FeE6G%ax$`lpy3?7#Yu{5d8 zrCK$Z`u0#J*M<#L372ytsuVK0sYtT8bVD0v-`5~;;D zijQJ2xU|Gd*Y*^HN^Ds*t^|WQ@`~Vq91I?i6H302IY@jjqfnXC8`!4$>JU@ee4IGj zCKC;+0w*=o)gu+VdbW1;lvh`7cXe-_>j_BNU{E6y462xW%g(lWD?Zipe>b}k@xSMv<)7dS{5apiH*goZpK_5GQZ}0U!VbfB+Bx0zlw@hJe^@!Pu*Q)HAbAM_#MrH04G|-J4D@jtqK%L?V#yXnMuY8Xecq zFSxEBXY6UTNTLzN4x1rks|e z&MhaHMvV(YVrenz5KoTt7o6tfj9DY&_jrsd#FOL8UdOMmFs#N!AbWIYtx7KD_+o%* z(8!R#a~f4tKcTcEyO%Ll@}+fyQG;ZI!<}^8r!e&z6*U|D;`$mL`ybCZoHI;ag^ak) zs6#y2zvi|7?h3; z3PWjJNbY*p7*&X;=0BgY-!a3WY9+Z>=n$K4G0lHk#Q)YS$z@a_D{4OEZ~mr$uUGQL z<1}ihen!&Gb9?bSdL?;1juk32)!dqF9#ZhXG%h52MY~alcxrm(OtWVOzpYnNvr&b3 zYWg12^ztIUR;{F_T9t?WP2UmlKP#0a+KeiypUHuygoFTuh$rElleWMxe!W^r>7d?3b=Zw7 z_7p`4(Wiy`Pqeh-ONeN=X?28Er%*!2DIqY^(q?)}#5C=sDs#x|YoYEjVU4@RM_vbD z_56?dvxt9{{{#7e3kU!KAOHk_01yBIKmZ5;0U!VbfB+C!-2?<|(!L^PlSSLy>6Xtj zJgxSyG;4Dq+uBPlj|x=9ME!omZ%(v)rhNXV`9;KEB){MS0zd!=00AHX1b_e#00KY& z2mk>f00dS&0XL1=dq#If1|lIb)aCbgMt6?{Is^W}?QRM;+;~L&`#;5BBf00e*l5C8%|00;m9AOHk_01&t(0doG|v@Se1L;kVCF71E5bo;xvYz}Ph($4?q z5r6)c0x$>!fB+Bx0zd!=00AHX1b_e#00KY&2&^sw?Nr^m`g