diff --git a/docs/designs/ptoas_persistent_simt_fragment_plan.md b/docs/designs/ptoas_persistent_simt_fragment_plan.md new file mode 100644 index 0000000000..0863b1bc0f --- /dev/null +++ b/docs/designs/ptoas_persistent_simt_fragment_plan.md @@ -0,0 +1,617 @@ +# PTOAS SIMT 寄存器驻留 fragment 方案 + +## 背景 + +RMSNorm 中存在重复的 UB 读取。权重 `W` 已从 GM 搬入 UB，但每个 token 进入 `SimtVF` 时仍需将 `w_ub` 读入寄存器；以 `batch/N_CORES = 64` 为例，每核重复 64 次。目标是在同一 kernel 的多个 `SimtVF` 之间驻留该数据，将 `UB→VRF` 读取从 64 次降为 1 次。 + +本方案采用**显式 fragment**： + +- 前端不新增 API。`alloc_fragment` 声明于 `SimtVF` 外即视为跨 `SimtVF` 驻留的 fragment。 +- `pto.persistent` 标记 allocation 是跨 SIMT section 的持久化载体。`residentElements` 由 init section 中确定完成初始化的 element 定义，不由 allocation 的物理 padding 或后续 consumer 的访问集合定义。 +- `residentElements` 中每个 element 都分配稳定 slot，并作为一个完整对象穿过所有受 init section 支配的 SIMT section；consumer 访问不用于裁剪。 +- 不设只读限制，须支持 read-modify-write（如寄存器上的 reduce 累加）。 +- fragment 在不同 `SimtVF` 中的 layout 一致性由 TileLang layout inference 保证。 +- PTOAS 侧使用已有的 `pto.keep` / `pto.resume` 表达跨 `SimtVF` 的寄存器保存与恢复。 + +处理链路为： + +```text +TileLang TIR → TileLang PTO codegen → PTODSL Python → PTODSL lowering → PTO IR → PTOAS +``` + +寄存器驻留优化在 PTOAS 中完成，输入为 codegen 产出的、SIMT 代码仍包裹在内联 section 中、尚未 outline 的 PTO IR。 + +## 输入表示与前置约束 + +物化 pass 的输入保持「SIMT 代码包裹在内联 section 中、尚未 outline」的形态，采用以下跨层约定： + +1. **PTODSL 侧：保留 SIMT 内联形态、推迟 outline。** `with pto.simt(...)` emit `pto.section.simt`，block 退出时不生成 `pto.simt_entry + pto.simt_launch`。outline 统一在 PTOAS 完成。 + +2. **PTO IR：使用 SIMT 内联 region op。** `pto.section.simt<<>>` 与 `pto.section.cube` / `pto.section.vector` 保持同类 section 表示，并在 op 头部携带静态三维 launch 参数。当前 section region 保持单顶层 block，但该 block 内可以包含 `scf.if` / `scf.for` 等 nested structured control flow。`pto-outline-simt-sections` 在物化后将其转换为 `simt_entry + simt_launch`。 + +3. **PTOAS 侧：承接 simt 作用域外的 persistent alloca。** persistent fragment 的 `llvm.alloca` 位于 simt 作用域外的 kernel 函数体，且并非真实内存（只有被 keep/resume 替换后才成立）。物化 pass 之前该 alloca 不得被当作栈内存 lower；物化未触发时按 fail-fast 报错（见「资源预算、跨层契约与错误处理」）。 + +4. **物化 pass 内部：outline 前的 unroll 与 keep/resume 位置。** 现有 keep/resume verifier（`PTOValidateVPTOIR.cpp`）要求 keep 紧邻 `func.return`。`pto-unroll-simt-for`（`Passes.td`）已扩展为同时处理带 `pto.simt_entry` 的函数和 outline 前的 `pto.section.simt`，且不会影响普通函数中 section 外的循环；keep/resume 在 outline 前物化，outline 后自然落在 entry 的入口/出口，从而复用现有 verifier。 + +## Outline 前置条件 + +- outline 前，fragment 的外层定义、完整 init 和其余 SIMT section 位于同一函数内，对象校验、slot 分配、物化和 cleanup 均可在一个 `func.func` 内完成。 +- outline 后每次 `simt_launch` 是一次不透明调用，fragment 的完整 allocation、初始化 section 和其余 carry section 的关系均不可见，PTOAS 无法可靠完成对象级物化。 +- keep/resume 在 outline 前生成、outline 后自然落到各 `simt_entry` 的入口与出口，满足现有 verifier 对位置的要求。 + +## 输入 IR 形态 + +内联 SIMT section 的实际 op 名为 `pto.section.simt`。 + +fragment 在最终 TIR 中是一块 `local` buffer（`T.allocate([32], "float32", "local")`）。TileLang PTO codegen 将其 emit 为 kernel body 作用域（所有 `with pto.simt` 块之外）的 `pto.alloc_buffer`；PTODSL lowering 将其转换为带 `pto.persistent` 属性的 `llvm.alloca`。该属性用于识别 def/use。示意输入： + +```mlir +func.func @main_kernel(...) attributes {pto.entry} { + %w_frag = llvm.alloca %c32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + + // init section：UB → fragment + pto.section.simt<<<128, 1, 1>>> { + %w0 = pto.load %w_ub[%idx0] : !pto.ptr -> f32 + %p0 = llvm.getelementptr %w_frag[%off0] : (!llvm.ptr, i64) -> !llvm.ptr, f32 + llvm.store %w0, %p0 : f32, !llvm.ptr + // ... 同样定义本次需要驻留的其它 element + } + + scf.for %t = %c0 to %tokens step %c1 { + pto.section.simt<<<128, 1, 1>>> { + %p0 = llvm.getelementptr %w_frag[%off0] : (!llvm.ptr, i64) -> !llvm.ptr, f32 + %w0 = llvm.load %p0 : !llvm.ptr -> f32 + %y = arith.mulf %x, %w0 : f32 + // 可选 read-modify-write： + // llvm.store %updated, %p0 : f32, !llvm.ptr + } + } + return +} +``` + +PTOAS 把 `%w_frag` 认成 persistent fragment 的依据： + +- defining op 是带 `pto.persistent` 的 `llvm.alloca`； +- 定义点在 `pto.section.simt` 外，支配所有使用它的 section； +- 至少一个 use 在 `pto.section.simt` 内； +- allocation 边界、init 中确定初始化的 element 和访问 offset 静态可解析。 + +## `pto.persistent` 的来源 + +`pto.persistent` 由 fragment 的词法作用域表达，PTODSL 前端合成，PTOAS 消费并校验。规则「`alloc_buffer` 在 SIMT region 外即视为跨 VF 驻留」在前端求值。 + +**DSL 表面。** fragment 写在其自然作用域：跨 VF 复用的 fragment 位于所有 `with pto.simt` 块之外，section-local 的临时 fragment 位于块内。以 `kernel.py` 为例： + +```python +w_frag = pto.alloc_buffer((32,), pto.f32) # SIMT region 外 → persistent +with pto.simt(128, 1, 1): # init：UB → w_frag + for i in pto.static_range(0, 16): + scalar.store(scalar.load(...), w_frag, i * 2) + +with pto.for_(0, 64, step=1) as t: + with pto.simt(128, 1, 1): # consume + x_frag = pto.alloc_buffer((32,), pto.f32) # SIMT region 内 → 非 persistent + sum_sq = pto.alloc_buffer((1,), pto.f32) # 同上 + ... + scalar.store(... * scalar.load(w_frag, i_2 * 2, contiguous=2), ...) +``` + +**PTODSL lowering。** `pto.alloc_buffer(...)` 在 trace 时读取所处 subkernel 作用域（`ptodsl/_ops.py`）：位于 `with pto.simt(...)` 外则 persistent，emit 出的 `llvm.alloca` 附加 `pto.persistent`（`UnitAttr`）；位于块内则为普通 local alloca。无需新增前端 API 或显式 `persistent=True` 参数。 + +**PTOAS。** 消费 `pto.persistent`，并校验对象级契约：定义点在 region 外且支配所有 use、allocation shape 静态、init section 至少确定初始化一个 element、其余 carry section 由 init 支配且 launch 维度一致、所有访问均属于 `residentElements`，且访问 offset 静态并位于 allocation 范围内。不满足则 fail-fast（见「资源预算、跨层契约与错误处理」）。PTOAS 不自行发现 persistent fragment。 + +## Persistent 对象语义 + +`pto.persistent` 表示整个 `llvm.alloca` 是 persistent fragment 的存储载体，但 allocation 的物理 element count 不直接等于驻留 slot 数。物化 pass 遵循以下契约： + +1. allocation 的 element type 和静态 element count 定义合法地址范围；init section 中确定完成初始化的 element 集合定义 `residentElements`。 +2. init section 是访问该 allocation 且支配其它所有访问 section 的唯一 section，不要求它直接位于 kernel entry block。当前规则中，直接位于 init section 顶层 block 中的 store 视为确定初始化；每个 resident element 的首次访问必须是 store。 +3. init 未写入的 padding element 不占 slot，也不需要伪造初值。例如 allocation 经 layout inference 为 32 个 `f32`，init 只确定写入 element 0～19，则 `residentElements = {0..19}`，slot 数为 20。 +4. `residentElements` 一经 init 确定便保持不变。consumer 是否访问某个 resident element 不参与裁剪；任一访问落在集合外均报错。 +5. 所有受 init 支配的其它 SIMT section 都 carry 完整 `residentElements`：入口 resume 全部 resident element，出口 keep 全部 resident element。无本地访问的 section 也不能跳过。 +6. 不做跨 section per-element liveness 或末次使用分析。read-modify-write 只改变 element 的值，不改变 `residentElements` 和 slot 集合。 +7. 多个 persistent allocation 使用函数级互不重叠的 slot；每个对象可以有自己的 init section，已 active 的对象必须穿过其激活点之后的其它对象 init section。 + +## 输出 IR 形态 + +keep/resume 承载跨 section 的寄存器状态。示意输出： + +```mlir +func.func @main_kernel(...) attributes {pto.entry} { + pto.section.simt<<<128, 1, 1>>> { // init + %w0 = pto.load %w_ub[%idx0] : !pto.ptr -> f32 + pto.keep %w0 {slot = 0 : i64} : f32 + // ... keep 完整 residentElements + } + + scf.for %t = %c0 to %tokens step %c1 { + pto.section.simt<<<128, 1, 1>>> { // consume + %w0 = pto.resume {slot = 0 : i64} : f32 + // ... resume 完整 residentElements + %y = arith.mulf %x, %w0 : f32 + pto.keep %w0 {slot = 0 : i64} : f32 // 出口 re-keep + // ... re-keep 完整 residentElements + } + } + return +} +``` + +read-modify-write 场景下 resume 为携带值、keep 为更新后的值： + +```mlir +pto.section.simt<<<128, 1, 1>>> { + %acc0 = pto.resume {slot = 0 : i64} : f32 + %acc1 = arith.addf %acc0, %partial : f32 + pto.keep %acc1 {slot = 0 : i64} : f32 +} +``` + +出口 re-keep 是完整驻留对象的一部分：resume 之后 body 可能占用其它 SIMT 寄存器、覆盖这些物理 slot，因此 init 之后的每个 SIMT section 都必须在入口 resume、出口 keep 完整 `residentElements`。即使 section 不访问该 fragment，也要用 `resume → keep` 携带该集合；不进行末次使用分析，最后一个 section 同样 re-keep。 + +outline pass 再把每个 section 转成 `func.func {pto.simt_entry}` + `pto.simt_launch`，keep 落在 return 前、resume 落在入口，正好满足现有 verifier。 + +## Pass 设计 + +- 分析实现：`lib/PTO/Transforms/SIMTPersistentFragmentAnalysis.cpp` / `SIMTPersistentFragmentAnalysis.h` +- Analysis-only pass：`lib/PTO/Transforms/PTOAnalyzeSIMTPersistentFragment.cpp`，pass 名 `pto-analyze-simt-persistent-fragment` +- C++：`lib/PTO/Transforms/PTOMaterializeSIMTPersistentFragment.cpp` +- Pass 名：`pto-materialize-simt-persistent-fragment` +- 作用域：`func::FuncOp`（内联 section 在 kernel 函数内，无需跨函数分析） +- 构造：`createPTOMaterializeSIMTPersistentFragmentPass()` + +### 1. 收集 SIMT section + +遍历 kernel 函数中的全部 `pto.section.simt`，按函数 walk 顺序记录父函数、位置、线程维度和 body region。该列表是函数级计划的一部分，不能只收集访问 persistent alloca 的 section；已 outline 为 `func.func {pto.simt_entry}` 的函数不在本 pass 的处理范围内。 + +### 2. 定位并校验 persistent fragment + +顺 SSA 定位带 `pto.persistent` 的 `llvm.alloca`，并校验结构前提： + +- 定义点在 `pto.section.simt` 外，且支配所有 use； +- 至少一个 use 在 `pto.section.simt` 内； +- 访问该 fragment 且支配其它所有访问 section 的唯一 section 是 init section，并由其中确定完成初始化的 element 建立非空 `residentElements`； +- init section 必须支配其余 carry section，同一 fragment 的 carry section 使用相同 launch 维度； +- init 之后的访问必须属于 `residentElements`；allocation 中未被 init 确定初始化的 padding element 不参与驻留； +- 若同时在非 SIMT 代码中被读写，报错，避免跨 section 寄存器状态与普通内存语义混用。 + +持久化由 `pto.persistent` 表达，PTOAS 不从「被多个 section 使用」等结构特征反推持久化；上述校验用于确认属性的前提成立，不成立则 fail-fast。 + +RMSNorm 权重驻留的典型形态：kernel 作用域 alloca `w_frag` → init section store → 循环体 consume section load。 + +### 3. 解析 allocation、residentElements、访问与 slot + +对每个 fragment 解析 section 内的 load/store： + +- 支持 alloca offset 0 直接访问、常量 GEP、可折叠为常量的 offset； +- 向量访问（`vector<2xf32>` / `vector<4xf32>`）按 lane 展开为连续 scalar element 访问； +- 每个访问须有静态 element offset；动态 offset 直接报错，不做静默回退。 + +slot 分配由 init-defined `residentElements` 决定。先按 init section 的程序顺序确认每个 element 的首次访问为 store，再按 element offset 升序建立 `(fragment, element_offset) → slot` 映射。consumer 的访问集合不能缩减该映射；allocation 中未被 init 写入且不被访问的 padding 不占 slot。`i1/i8/i16/i32/f16/bf16/f32` 每个 resident element 占 1 个 slot；`i64` 每个 resident element 占 2 个 slot且起始 slot 偶数对齐。vector load/store 只是多个标量 element 的访问形式，不改变已确定的 resident set 和 slot 编号。当前实现按 alloca 出现顺序、resident element offset 升序分配。 + +**向量访问需特别处理**：生成的 IR 中 fragment 常以 `vector<2xf32>` 经字节 GEP（`getelementptr ... i8`，index 已乘 element 字节数）访问。对「同一 alloca、字节偏移寻址、标量与向量混用」这种形态，标准 mem2reg/SROA 无法提升。此处需按静态字节 offset 做定制切分（SROA 风格），将每个 offset 映射到确定的 slot，而非依赖现成 mem2reg。此为实现中风险最高的一处。 + +### 4. 边界插 resume/keep + 通用 mem2reg + +直接在 load/store 上手动替换只对 write-once-read-many 成立，read-modify-write 会得到错误结果。采用的做法是在 section 边界插入 seed/save，再交由通用 mem2reg 穿线： + +- **init section**：不生成 resume；出口对全部 `residentElements` 生成 `pto.keep`。 +- **每个 carry section 入口**：对全部 resident slot 插入 `pto.resume`。本地访问的 element 用 resume 结果初始化 section-local proxy；未访问的 resident element 直接把 resume 结果传给出口 keep。 +- **每个 carry section 出口**：对全部 `residentElements` 生成 `pto.keep`。本地更新过的 element 保存更新值，未访问的 resident element 原样 re-keep。 +- 对 section 中本地访问的 element 创建 scalar proxy，将原 load/store 的地址改写到对应 proxy；未访问的 element 直接连接 resume 与 keep。 +- 对每个 proxy 运行 mem2reg：入口 `store resume` 成为初始 SSA def，body 读-改-写成为正常 SSA 链，出口 `load` 取到链尾值并由 keep 保存，proxy 的全部 load/store 被提升。 +- `pto.resume` / `pto.keep` 读写的是 slot payload 而非 alloca，mem2reg 不会触及——resume 成为链入口、keep 取到链出口。只读场景下 resume→keep 为自移动（可消除），读-改-写场景下 resume 为携带值、keep 为更新值，二者均正确。 + +物化按函数级 `sections` 顺序执行。每个 section 的入口操作统一插到 body 开头：先生成 proxy 使用的 count=1 常量，再生成完整 resume group，之后创建全部 proxy 和 seed store；所有 proxy 就绪后，按 section body 的 operation 顺序统一改写原访问。出口先为所有本地 proxy 生成最终 load，再统一生成按 slot 排序的连续 keep group。每个 proxy 单独调用 `tryToPromoteMemorySlots`，避免“多个 allocator 中只提升一部分”被误判为整体成功。 + +除 init section 外，不根据本地首次访问决定是否 resume；每个 carry section 一律 resume/keep 完整 `residentElements`。当前 persistent load/store 直接位于 section 顶层 block，proxy 的定义和使用保持线性。nested structured control flow 可以执行与 fragment 无关的计算，也可以使用顶层 fragment load 产生的普通 SSA value，但不能在 nested region 内访问或更新 fragment。条件写、循环携带 fragment 状态以及相应的路径合流由 Phase 4 处理；它们不改变边界 slot 集合，只改变每个 slot 在出口处的 reaching definition。 + +### 5. cleanup + +mem2reg 提升后，persistent alloca 的派生 GEP 已无最终 load/store user。物化 pass 从 alloca result 出发递归检查并按叶子到根删除死 GEP；若仍存在非 GEP user 或 GEP 仍有 live use，则直接报错。最后确认 alloca result 无 use 后显式删除 `llvm.alloca {pto.persistent}`。proxy 使用的 count=1 常量在无 use 时同时删除，不依赖 outline 前额外运行 DCE。 + +### 6. outline + +cleanup 后 section 中仅剩 slot-only 的 keep/resume。将每个 `pto.section.simt` outline 成 `func.func {pto.simt_entry}` + `pto.simt_launch`：init section → `@init_wfrag`，其余 section → 对应 SIMT helper。`@init_wfrag` 一次性 keep 完整 slot 集合，每个 helper 入口 resume、出口 re-keep，在调用链上形成 `keep →(resume…re-keep)×N`，中间不再回 UB 重读。 + +### 7. 两个 pass 的内部流程 + +`pto-analyze-simt-persistent-fragment` 和 `pto-materialize-simt-persistent-fragment` 共享同一个 `SIMTPersistentFragmentAnalysis`。前者是只读的诊断/checkpoint，后者消费不可变 plan 并修改 IR；analysis-only pass 不是 plan 的传输载体，materialize 即使在 pipeline 中没有显式经过 analysis-only pass，也会通过 `getAnalysis` 按需构造相同结果。 + +以下图中 pass 名仅作为标题，节点表示各 pass 实际完成的分析或 IR 处理。 + +#### 7.1 Analysis pass：构造 persistent fragment plan + +```text +func::FuncOp / inline PTO IR + | + v +收集全部 pto.section.simt 与 pto.persistent llvm.alloca + | + v +检查 kernel-entry、词法作用域和 alloca 支配关系 + | + v +沿 alloca → GEP → load/store use graph 追踪指针 + | + v +折叠累计 byte offset，归一化 element offset 与 vector lane + | + v +按访问和 dominance 确定 init section、residentElements、carry sections + | + v +校验访问类型、边界、对齐、launch 维度和 resident 集合 + | + v +按 fragment/element 顺序分配函数级 keep/resume slot + | + v +按 section、operation、lane 顺序将归一化访问写入 +`ResidentElementPlan.accesses`（`AccessLane`） + | + v +缓存完整 immutable plan + | + v +只读 checkpoint 保留 analysis，IR 不发生修改 +``` + +#### 7.2 Materialize pass：将 plan 物化为 slot traffic + +```text +func::FuncOp / inline PTO IR + | + v +取得或按需构造 immutable persistent fragment plan + | + v +按函数 section 顺序建立 transform worklist + | + v +把每个 fragment 的完整 residentElements 绑定到 init/carry section + | + v +预先校验 section 归属、element 顺序、resident set 完整性、 + access ownership 和 lane 连续性 + | + v +在 carry section 入口插入完整 resume group + | + v +为本地访问 element 创建 scalar proxy + carry section 用 resume seed，init section 由已验证的首次 store 定义 + | + v +按原 operation 顺序改写 scalar load/store + 并将 vector access 拆成 lane 级 proxy 读写后重建结果 + | + v +在 section 出口读取 proxy 最终值，按 slot 顺序生成完整 keep group + | + v +逐 proxy 执行 mem2reg，删除无用的临时 count 常量 + | + v +删除 dead GEP tree 与原 persistent alloca + | + v +得到 slot-only SIMT sections，交给后续 outline pass +``` + +处理边界如下： + +- **analysis 构造阶段**只读取 IR。任何一个 alloca、访问、init、resident set、slot 或 lane 校验失败，均不发布部分 plan；`isValid()` 为 false。 +- **analysis-only pass**只强制构造并验证 analysis，成功后 `markAllAnalysesPreserved()`，不插入或删除任何操作。 +- **materialize preflight**先为全部 section 建立 worklist，校验 access/lane 唯一分配、fragment/element 顺序和完整 resident set；preflight 失败时不应留下部分 resume/keep。 +- **materialize body**才创建 proxy、改写 load/store、执行 mem2reg 和生成 keep/resume；cleanup 完成后删除原 persistent alloca 及死 GEP。 +- `pto-outline-simt-sections` 不属于上述 materialize pass 的内部步骤，而是消费 slot-only section 的后续独立 pass。 + +### 8. 分析 plan 到物化工作列表 + +analysis 对 `func::FuncOp` 发布的稳定结果是 `SIMTPersistentFragmentAnalysis` 中缓存的 `PersistentMaterializationPlan`。analysis-only pass 只强制构造和校验该结果；materialize 通过 `getAnalysis()` 取得同一个不可变 plan，不依赖 analysis-only pass 作为数据传输载体。随后 `buildPersistentTransformWorklist()` 将按 fragment/element 组织的 plan 转成按 section/operation 组织的临时物化工作列表；该转换只建立引用、局部访问集合和改写映射，不修改 IR。 + +```text +SIMTPersistentFragmentAnalysis (MLIR analysis cache) +└── plan: optional + ├── sections[]: SectionSimtOp // function walk order + └── fragments[]: PersistentFragmentAnalysis // alloca walk order + ├── allocaOp + ├── initSection + ├── carrySections[] + └── residentElements[] // element offset order + └── ResidentElementPlan + ├── elementOffset + ├── slot + └── accesses[]: AccessLane + ├── op + └── laneIndex +``` + +顺序和所有权是 plan 的一部分契约：`sections` 按函数 walk 顺序保存，`fragments` 按 persistent alloca walk 顺序保存，`residentElements` 按 element offset 升序保存，`AccessLane` 按 section、operation 和 lane 顺序挂在所属 element 下。`elementOffset` 和 `slot` 由父级 `ResidentElementPlan` 提供，`AccessLane` 只保存原始访问 operation 和 vector lane 序号。 + +materialize 的 preflight 数据结构为： + +```text +PersistentMaterializationPlan +fragment -> resident element -> access lane + | + | buildPersistentTransformWorklist() + v +PersistentTransformWorklist +section -> element work item +section -> operation -> lane rewrite +``` + +```text +PersistentTransformWorklist +└── sections[]: PersistentSectionWorklist // 与 plan.sections 顺序一致 + ├── section: SectionSimtOp + ├── elements[]: PersistentElementWorkItem // active fragment/element 顺序 + │ ├── fragment* // 引用 immutable plan + │ ├── residentElement* // 引用 immutable plan + │ └── accesses[]: AccessLane // 仅保留当前 section 的访问 + └── laneRewritesByAccess: Operation* -> PersistentLaneRewrite[] + └── PersistentLaneRewrite + ├── laneIndex // 原 scalar/vector access 的 lane + └── elementIndex // 当前 elements[] 的下标 +``` + +转换分为三步： + +1. 为 `plan.sections` 中的每个 section 创建一个 `PersistentSectionWorklist`，保持函数 walk 顺序；没有 active fragment 的 section 也保留为空 worklist。 +2. 对每个 fragment，把完整 `residentElements` 依次加入其 init 和全部 carry section。`PersistentElementWorkItem` 直接引用原 fragment/element，并从 `ResidentElementPlan::accesses` 中筛出位于当前 section 的访问；即使本 section 没有访问某个 resident element，该 element 仍在 `elements[]` 中，用于生成完整 resume/keep 集合。 +3. preflight 按 section 校验 element 顺序、完整 resident set、访问归属和 lane 唯一性，再把 element-centric 的 `ResidentElementPlan -> AccessLane` 反向整理为 operation-centric 的 `Operation -> (laneIndex, elementIndex)`。该映射使同一个 vector load/store 可以一次完成所有 lane 的 scalar proxy 改写。跨 section 共享的 `assignedAccessLanes` 记录每个 `(operation, lane)` 至多分配一次，随后与 plan 对照确认没有遗漏；该集合不进入最终 worklist。 + +`materializeSection()` 随后创建与 `sectionWorklist.elements` 等长且下标平行的临时 rewrite state： + +```text +rewrites[]: PersistentElementRewrite +├── proxy // 有本地访问时创建的 scalar alloca +└── resumeValue // carry section 中该 element 的入口值 +``` + +`laneRewritesByAccess` 中的 `elementIndex` 用来取得 `rewrites[elementIndex].proxy`；它只是当前 section 的临时数组下标，不是原 fragment 的 `elementOffset`，也不是 persistent `slot`。`ResidentElementPlan::slot` 由 analysis 按函数范围分配并跨 section 保持稳定，同一个 element 的 `pto.resume` 和 `pto.keep` 均直接使用该 slot。对没有本地访问的 carry element，不创建 proxy，直接将 `resumeValue` 作为同 slot 的 keep payload。 + +分析成功由 `isValid()` 表示，materialize 只通过 `getPlan()` 以 `const` 方式读取该结果；analysis 失败时不发布部分 plan。没有 persistent alloca 时发布合法的空 plan；存在 persistent alloca 时，只有完整校验成功才发布非空 plan。 + +`FragmentShape`、pointer-use graph 的中间记录和 normalized access discovery 只在 analysis 内部用于类型、offset、bounds、resident set 和 lane 校验，plan 发布后销毁。`PersistentTransformWorklist`、section-local work item、lane rewrite 和 element rewrite state 都只属于 materialize，先完成全局 preflight 再修改 IR，物化结束后销毁，不进入 analysis cache。 + +## 流水接入位置 + +```text +PTODSL lowering + → PTO IR（含内联 pto.section.simt，保留 structured control flow） + → canonicalizer / SCCP / CSE + → unroll section 内标注的循环 + → canonicalizer / SCCP / CSE + → pto-analyze-simt-persistent-fragment（显式 analysis checkpoint） + → pto-materialize-simt-persistent-fragment + → outline pto.section.simt 成 func.func {pto.simt_entry} + → SCF-to-CF（在 late outline 之后） + → 现有 VPTO emission / LLVM lowering +``` + +流水顺序为“保留 structured control flow → persistent analysis/materialize → late outline → SCF-to-CF”。当前 SCF-to-CF 位于 LLVM emission pipeline 的 outline 之后。persistent pass 不依赖 section 的显式 CFG；`pto.section.simt` 的单顶层 block 可以容纳 nested structured control flow，RMSNorm 不需要多顶层 block section。 + +unroll 须在物化之前执行。slot 数由 init 确定写入的 `residentElements` 决定，不依赖后续访问 footprint。以 `threads=256`、layout capacity 为 32 的 `d=5120` RMSNorm 为例，init 只确定写入 lane-local element 0～19，因此分配 20 个 slot；allocation 中 element 20～31 是 layout padding，不参与驻留。init/consumer 的 load/store 位于静态循环中，不展开就无法把各访问归一到确定 element offset，也无法建立 resident set。展开并折叠 `%i` 后，物化 pass 才能把各访问连接到稳定 slot。`pto-unroll-simt-for` 同时处理 outline 前的 `pto.section.simt` 和 outline 后的 `pto.simt_entry`，只展开显式标注 `{pto.unroll = "full"}` 的循环。 + +## 当前支持范围与限制 + +当前实现已完成内联 SIMT 表示、persistent fragment 分析、init/resident set 与函数级 slot 分配、scalar/vector keep/resume 物化、section-local proxy mem2reg 和 late outline 前 cleanup。当前支持标量以及一维 `vector<2xT>` / `vector<4xT>` 访问。 + +当前 `pto.section.simt` region 具有一个顶层 block。该 block 可以包含具有自身 region/block 的 `scf.if`、`scf.for`；fragment-transparent 的 nested structured control flow 由当前 pass 和 late outline 保留。 + +当前 persistent fragment 的 pointer/GEP/load/store 必须直接位于 section 顶层 block。顶层 fragment load 产生的普通 SSA value 可以进入 nested region，nested region 内与 fragment 无关的计算、分支和循环不受此限制。persistent access 位于 nested `scf.if` / `scf.for` 中时，由 Phase 4 处理。section 可以嵌套在外层 `scf.if`、`scf.for` 或普通 CFG block 中，init section 必须支配该 fragment 的全部访问 section。section region 的多顶层 block CFG 不在本方案范围内。 + +RMSNorm 的 reduce-all 控制流位于 `SimtVF` 对应的 section 顶层 block 内，不访问 persistent weight fragment；weight fragment 的 load/store 仍位于顶层 block。该形态属于当前实现范围。 + +### 当前非目标 + +- 不采用 persistent UB buffer 自动 peel 方案。 +- 不在 PTOAS 里推断 TileLang fragment layout。 +- 不新增 TileLang 前端 API。 +- 输出为标量 keep/resume，不引入 `keep_range` / `resume_range`。 +- 不处理动态 offset fragment。 +- 不做跨复杂 CFG 的通用 mem2reg。 +- 不处理 section region 的多顶层 block CFG。 +- 不处理 persistent pointer/GEP/load/store 位于 nested `scf.if` / `scf.for` 中的条件访问、循环携带状态或多出口状态；fragment-transparent 的 nested structured control flow 已支持，路径相关 fragment 状态属于 Phase 4。 +- 不根据 consumer 访问做 per-element liveness、末次使用分析或 resident set 裁剪。 +- 不做优雅回退（超预算即 fail-fast）。 +- 不处理已经 outline 后才出现的 persistent 推断。 + +## 资源预算、跨层契约与错误处理 + +`R4..R126` 共 123 个 slot。`M` 由所有 persistent allocation 的 `residentElements` 决定：每个 resident element 的 `slot_width(element_type)` 加上 64 位 element 的对齐 padding。consumer 实际访问较少不能降低 `M`，但 init 未写入的 allocation padding 不计入 `M`。理论上需满足 `M + body 峰值 + 余量 ≤ 123` 才执行变换，但 body 峰值取决于寄存器压力，需接近寄存器分配才能得到准确值，TIR 与 PTO 均无法准确估计。当前实现只按 resident set 的 `M` 判定，超出预算即报错。 + +**回退语义。** codegen 后 consume section 只保留对 `w_frag` 的访问，原 UB 地址表达式已删除，PTOAS 无法恢复 UB load。因此契约违反统一采用 **fail-fast**： + +- fragment 定义点不支配 section use → 报错； +- fragment element 数量不可知 → 报错； +- init section 没有确定初始化任何 element，或 resident element 首次访问为 load → 报错； +- init 之后访问不属于 `residentElements` 的 element → 报错； +- 找不到唯一支配全部访问 section 的 init，或 carry section launch 维度不一致 → 报错； +- 访问 offset 不是编译期常量 → 报错； +- 访问类型无法拆成 keep/resume 支持的标量 → 报错； +- slot 数超预算 → 报错； +- persistent access 位于 nested region，或 section region 含多个顶层 block → 报错； +- fragment 同时被 SIMT 和非 SIMT 代码读写 → 报错或明确跳过。 + +### Layout 一致性 + +PTO 侧以 alloca 的 element type、element count 和 element offset 定义合法布局，以 init-defined `residentElements` 定义驻留对象。resident `element_offset = k` 在所有 section 中始终映射到同一 slot；consumer 可以只访问对象子集，边界 resume/keep 仍覆盖全部 resident element。各 section 的访问 offset 集合可以不同。 + +TileLang layout inference 负责保证同一 fragment 在各 `SimtVF` 中具有一致的 lane-to-element 映射。PTOAS 负责可验证的结构约束：init section 确定建立非空 `residentElements`；所有访问类型、对齐和 offset 均落在同一 allocation 内；init 后访问只能落在 resident set 内；carry section 的 `<<>>` launch 参数与 init section 一致。违反这些约束时直接报错，避免不同 lane 拓扑或错误 offset 产生静默数值错误。 + +### Keep/Resume 语义 + +不新增底层硬件语义，复用现有 `pto.keep` / `pto.resume`（`include/PTO/IR/VPTOOps.td`）： + +- `pto.resume {slot = N}`：从固定寄存器 slot 取回一个元素； +- `pto.keep %v {slot = N}`：把一个元素存到固定寄存器 slot； +- slot → 物理寄存器的映射由 PTOAS 现有 lowering 负责（`slot N → R{4+N}`）； +- BiSheng 继续负责普通寄存器分配，PTOAS 只显式表达跨 VF 的 live-in/live-out slot。 + +当前实现直接生成多条标量 keep/resume；`keep_range` / `resume_range` 不在本方案范围内。 + +### 跨层对齐约定 + +| 项目 | 约定 | +|---|---| +| 内联 SIMT section | PTODSL emit `pto.section.simt` 并推迟 outline；PTOAS 在物化后执行 late outline | +| persistent 判定 | PTODSL 依 `alloc_buffer` 作用域合成 `pto.persistent`；PTOAS 消费并校验，不自行发现 | +| 对象边界 | allocation shape 定义合法地址范围；init 中确定完成初始化的 element 定义固定 `residentElements` | +| 初始化 | 唯一支配全部访问 section 的 init 建立非空 resident set；其它访问不得扩展 resident set | +| layout 一致性 | TileLang layout inference 保证 lane-to-element 映射；PTOAS 检查 shape、offset bounds 和 launch 维度 | +| fragment 大小 | PTOAS 需要静态得到 allocation 边界和 init 写入 offset，并按 `residentElements` 计算 slot 预算 | +| 访问约束 | 当前实现要求访问 offset 静态可解析；consumer 可以只访问对象子集 | +| 只读限制 | 不设只读限制，必须支持 read-modify-write | +| outline 顺序 | keep/resume 物化必须在 section outline 之前 | +| SCF-to-CF 时机 | 保留 structured control flow 完成 persistent analysis/materialize，late outline 后再做 SCF-to-CF | +| section 控制流 | region 保持一个顶层 block；fragment-transparent 的 nested `scf.if` / `scf.for` 已支持；persistent access 必须在顶层 block | +| 路径状态合流 | nested 分支/循环中的 fragment state 通过 path-sensitive SSA/block argument 合流；init 使用 definite-assignment；section 保持单一外层出口 | + +## Phase 4：路径相关 fragment 状态 + +`pto.section.simt` 的 region 保持一个顶层 block。顶层 block 可以包含 `scf.if`、`scf.for` 等 nested structured control flow。当前实现要求 persistent fragment 的 pointer/GEP/load/store 直接位于该顶层 block；nested region 可以使用这些访问产生的普通 SSA value，但不能访问或更新 fragment。section 可以嵌套在函数外层的 `scf.if`、`scf.for` 或普通 CFG block 中，前提是 init section 支配该 fragment 的全部访问 section。 + +RMSNorm 的 reduce-all 控制流属于 fragment-transparent nested structured control flow，persistent weight 的访问仍位于 section 顶层 block；现有 section 表示和 pipeline 顺序适用。 + +Phase 4 支持 persistent access 位于 nested `scf.if` / `scf.for` 中时的路径状态分析和 SSA 合流。section 外层仍保持单顶层 block、单一外层出口；显式多顶层 block CFG 不在本方案范围内。 + +### 支持范围 + +| 场景 | persistent element 语义 | +|---|---| +| carry section 中单分支 `scf.if` | true 分支写入新值；false 路径保留 section 入口的 resume 值 | +| carry section 中完整 `scf.if/else` | 两个分支可以分别写入，也可以只有一个分支写入；merge 后选择对应路径的 reaching definition | +| 分支内 read-modify-write | 分支读取 resume 或前序定义，计算并写回；未执行该分支时保留旧值 | +| 嵌套 `scf.if` | 按每层 merge 关系合并 element 状态，出口得到单一 SSA 值 | +| init section 中条件初始化 | 每条可达路径都定义全部 `residentElements`；任一路径缺失定义即拒绝 | +| `scf.for` 循环携带值 | 循环入口使用 resume 或循环前定义，backedge 携带上一迭代值，循环退出值进入 keep | + +循环控制值可以是动态值，但 persistent element offset 必须静态可解析。循环在 analysis 前完整 unroll 后，展开产生的顶层访问按当前规则处理；未展开且访问位于循环 region 内的情况由 Phase 4 处理。 + +### 实现方案 + +1. **递归访问发现。** 在 section 顶层 block 内递归遍历已支持的 `scf.if`、`scf.for` region。每个访问继续归一化为 `ResidentElementPlan` / `AccessLane`；控制流位置和路径信息作为 analysis 内部状态，不改变发布 plan 的 element、slot、lane 语义。nested section、`scf.while`、未知 `RegionBranchOpInterface` 和不可归约 CFG 直接报错。 +2. **proxy 表达 fragment 状态。** carry section 在入口生成完整 resume group，并将每个 resume 值写入对应 scalar proxy；init section 由经过校验的初始化 store 建立 proxy 的初始定义。nested region 内的 fragment load/store 改写为 proxy load/store，普通 SSA 结果保持原有数据流。section 出口从 proxy 读取最终值并生成完整 keep group。 +3. **路径状态合流。** branch merge 使用 block argument/PHI 表达 reaching definition，loop backedge 携带上一迭代值。未执行写入的路径显式传递入口 resume 值，不使用 poison 或 undef。 +4. **late outline 后提升 proxy。** 含 nested SCF 的 section 先完成 persistent analysis/materialize，再 outline 为 SIMT helper；SCF-to-CF 在 helper 内执行，随后运行 proxy cleanup 和 mem2reg。简单线性 section 沿用 outline 前的 cleanup。 +5. **definite assignment。** carry section 的 resume 为所有 resident element 提供入口定义。init section 使用前向数据流，在每个 block 记录所有到达路径均已定义的 element 集合，merge 使用 predecessor 集合的交集。每个出口必须定义完整 `residentElements`。 +6. **边界 slot 保持不变。** 每个 carry section 的入口 resume 和出口 keep 都覆盖完整 `residentElements`，slot、类型和顺序与 analysis plan 一致；不根据控制流或 consumer 访问裁剪 resident set。 + +### 诊断边界 + +- persistent access 位于 section 顶层 block 之外； +- init 任一可达路径未完整定义 resident set； +- fragment 访问使用无法静态归一化的动态 element offset； +- branch predecessor 无法补充 block argument 或 branch operand； +- `scf.while`、异常式退出、未知 `RegionBranchOpInterface` 或不可归约 CFG； +- proxy cleanup 后仍存在 proxy alloca 或 persistent fragment load/store。 + +### 验收条件 + +- 单分支、双分支、嵌套 `scf.if` 的条件写和 read-modify-write 均得到正确 reaching definition； +- 未执行写入的路径保留 resume 旧值； +- init 的所有路径完整定义 resident set 时通过，缺失任一 element 时失败； +- `scf.for` 的 entry、backedge 和 exit 值正确； +- outline、SCF-to-CF、proxy cleanup 后通过 VPTO verifier 和 LLVM emission； +- resident set、slot 编号和 keep/resume 顺序保持 analysis plan 的定义。 + +Phase 4 的 pipeline 为： + +```text +保留 structured SCF + -> SIMT unroll / SCCP / canonicalizer / CSE + -> persistent analysis + materialize + -> late outline + -> SCF-to-CF（outlined helper） + -> persistent proxy mem2reg + cleanup + -> VPTO validation / lowering +``` + +## 路线图 + +Phase 0～3 是当前实现基线，Phase 4 是路径相关 persistent fragment 状态扩展。 + +| 阶段 | 目标 | 范围 | 产出 | +|---|---|---|---| +| Phase 0 | 打通内联 section + 晚 outline | PTODSL emit 内联 SIMT section；PTOAS 新增 `pto.section.simt` op + outline pass | 前置依赖落地 | +| Phase 1 | outline 前 persistent fragment 物化 | 内联 section、单顶层 block（允许 fragment-transparent nested SCF）、标量 alloca、常量 offset、标量 keep/resume | pass + lit 测试 | +| Phase 2 | 向量访问拆标量 | 字节偏移寻址下的 `vector<2/4xf32>` 定制切分 | 覆盖 RMSNorm 的 vectorized 访问 | +| Phase 3 | slot 预算与工程化 | i64 对齐、超预算诊断、错误边界和端到端验证 | 工程化 | +| Phase 4 | nested access 的路径相关 fragment 状态 | 条件读写、SSA 合流、循环携带值、init definite-assignment；保持单一外层出口 | nested SCF/SSA 扩展 + 完整控制流 lit | + +## 测试计划 + +lit 测试覆盖： + +1. **init + 只读 consume**：kernel 作用域定义 fragment，init section store，循环内 consume section load。期望：init 出口生成 keep，consume 入口生成 resume、出口 re-keep，原 load/store 被删除。 +2. **read-modify-write**：consume 内 load fragment、算新值、store 回。期望：resume 作为 SSA 链初值，keep 用更新后的值，不要求只读。 +3. **多元素 fragment**：`[4 x f32]`，init 确定写入 offset 0/1/2/3。期望：生成连续 slot，每个 resident offset 对应稳定 slot，keep/resume 数量和编号符合预期。 +4. **consumer 子集访问**：init 建立 resident set `{0,1,2,3}`，consumer 只访问 element 1。期望：边界仍 carry 全部 4 个 resident element，不根据 consumer 裁剪。 +5. **无本地访问的中间 section**：init 与 consumer 之间插入不访问 fragment 的 SIMT section。期望：中间 section 仍生成完整 resume/keep group。 +6. **allocation padding**：`[4 x f32]` 的 init 只确定写入 element 0～2，后续也只访问 0～2。期望：resident set 为 `{0,1,2}`，只分配 3 个 slot。 +7. **访问非 resident element 报错**：init 只确定写入 element 0～2，后续访问 element 3。期望：在物化前报告 element 3 不属于 resident set。 +8. **向量访问拆标量**：`vector<2xf32>` / `vector<4xf32>` 经字节 GEP 访问 fragment。期望：向量 load 由多个 slot assemble，向量 store 拆成多个 slot 更新，keep/resume 仍是标量。 +9. **section 内局部 fragment 不处理**：fragment 定义在 section 内。期望：不生成 keep/resume，保持普通局部 fragment 语义。 +10. **非法动态 offset 报错**：persistent fragment 用动态 GEP offset。期望：报错，提示 offset 必须静态可解析。 +11. **launch 维度不一致报错**：init 与后续 carry section 的 SIMT 维度不同。期望：在生成 keep/resume 前报错。 +12. **外层控制流中的 init**：init 和 consumer 位于同一个动态 `scf.if` 分支，且 init 支配 consumer。期望：允许 init 嵌套在外层控制流中。 +13. **不存在统一 init**：两个互斥分支分别访问并初始化同一 fragment，没有任何 section 支配全部访问。期望：报告找不到唯一 dominating init。 +14. **多个 fragment 与不同 init 点**：section 0 初始化 fragment A，section 1 carry A 并初始化 fragment B，section 2 使用 A/B。期望：各 section 包含正确的 active resident element，空本地访问仍保留，并按全局 slot 排序。 +15. **fragment-transparent nested SCF（当前支持）**：单顶层 block 的 init/carry section 中插入不访问 fragment 的 `scf.if` / `scf.for`，并让顶层 fragment load 的普通 SSA 结果进入 nested region。期望：keep/resume 正常生成，nested structured control flow 保持，原 persistent load/store 被删除。 +16. **nested persistent access 边界（当前负例）**：把 persistent load/store 放入 `scf.if` / `scf.for` region。期望：当前 pass 在 materialize 前报告 access 必须直接位于 section 顶层 block。 +17. **Phase 4 条件状态合流**：carry section 的单分支写、双分支写和 read-modify-write。期望：merge 后得到正确的 reaching definition，未执行写入的路径保留 resume 值。 +18. **Phase 4 初始化与出口**：init 的某条路径缺失 resident element 定义时失败；所有合法路径生成完整且顺序一致的 slot keep。 + +## 验证与性能记录 + +以下内容是当前实现和 benchmark 环境的验证快照，不改变前述规范性契约。 + +### 当前验证状态 + +- `ptoas` 构建通过。 +- 标量三元素 init/consume 输入可完成 keep/resume 物化、late outline 和 VPTO pipeline。 +- analysis-only、persistent materialize、inline section outline/unroll 相关的 30 个定向 lit 测试通过，包含 fragment-transparent nested SCF 正向和 nested persistent access 负向用例。全量 `check-pto` 当前为 752 个通过、128 个失败；失败均集中在现有 PTODSL daemon 环境的 `ImportError: cannot import name 'ir' from 'mlir'`，不作为本 pass 的回归结论。 +- RMSNorm `d=4096, batch=4096` inline 输入可完成 persistent materialization、late outline 和 `--emit-vpto`；输出中不再包含 `pto.persistent`。 + +### 性能结果 + +端到端性能使用 TileLang RMSNorm PTO 后端生成代码作为优化前基线，persistent fragment 版本只把每个 token 重复执行的 W UB→寄存器读取提取到独立 init SIMT section，并通过 keep/resume 在后续 token SIMT section 间保持权重。 + +测试口径如下： + +- 数据类型为 FP32，`batch=4096`，grid 为 64； +- 在同一张 Ascend NPU、同一套 CANN/PTOAS 构建上测试； +- 使用 TileLang `do_bench` 的 `msprof` backend，warmup 30 次、repeat 20 次，并在采样间执行 L2 cache flush； +- 有效带宽按算法逻辑访存量 `2 * batch * d * sizeof(fp32) + d * sizeof(fp32) + batch * sizeof(fp32)` 计算，分别对应 X 读取、Y 写出、W 读取和 RSTD 写出； +- 正确性阈值为 `max_y_diff < 1e-3` 且 `max_rstd_diff < 1e-3`。 + +基线使用 `tilelang-ds/examples/ascend/example_rmsnorm.py` 的 +`run_regression_perf`，通过 `tilelang.compile(..., target="pto")` 生成未做 +persistent fragment 物化的 kernel。三个 hidden size 分别在独立进程中执行，避免 +前一个 shape 的编译缓存或设备状态影响后续结果。测试使用本地 +`/home/qukelin/projects/PTOAS/build/tools/ptoas/ptoas`（`ptoas 0.51`）和 CANN +9.1；每个 shape 均完成 kernel 编译、warmup 和 `msprof` 采样。 + +下表统一列出未驻留基线和两种 persistent fragment 初始化路径。单元格格式为 +“延迟 / 有效带宽”。 + +| Shape | resident slots | PTO backend 基线 | UB → fragment | GM → fragment | +|---:|---:|---:|---:|---:| +| `d=4096` | 32 | 80.6 us / 1665 GB/s | 70.625 us / 1900.9 GB/s | 70.342 us / 1908.5 GB/s | +| `d=5120` | 20 | 84.5 us / 1986 GB/s | 79.264 us / 2117.1 GB/s | 79.111 us / 2121.2 GB/s | +| `d=7168` | 28 | 127.3 us / 1845 GB/s | 117.471 us / 1999.9 GB/s | 119.344 us / 1968.5 GB/s | + +| Shape | `max_y_diff` | `max_rstd_diff` | 正确性 | +|---:|---:|---:|:---:| +| `d=4096` | `1.907e-06` | `1.788e-07` | 通过 | +| `d=5120` | `2.861e-06` | `2.384e-07` | 通过 | +| `d=7168` | `2.861e-06` | `1.788e-07` | 通过 | + +三个 shape 均通过正确性检查。按上表测量值计算，UB → fragment 的延迟降低 +6.2%～12.4%、有效带宽提升 6.6%～14.2%；GM → fragment 的延迟降低 +6.2%～12.7%、有效带宽提升 6.7%～14.6%。 diff --git a/include/PTO/Transforms/Passes.h b/include/PTO/Transforms/Passes.h index 41beb89380..4075952302 100644 --- a/include/PTO/Transforms/Passes.h +++ b/include/PTO/Transforms/Passes.h @@ -92,6 +92,9 @@ std::unique_ptr createPTOFusionRegionGenPass(); LogicalResult validateIntToPtrUses(func::FuncOp func); std::unique_ptr createPTOUnrollSIMTForPass(); +std::unique_ptr createPTOAnalyzeSIMTPersistentFragmentPass(); +std::unique_ptr createPTOMaterializeSIMTPersistentFragmentPass(); +std::unique_ptr createPTOOutlineSIMTSectionsPass(); std::unique_ptr createPTOInferVPTOVecScopePass(); std::unique_ptr createVPTOExpandWrapperOpsPass(); std::unique_ptr createPTOVPTOPtrBoundaryPass(); diff --git a/include/PTO/Transforms/Passes.td b/include/PTO/Transforms/Passes.td index 93e83bab71..f5a4eae946 100644 --- a/include/PTO/Transforms/Passes.td +++ b/include/PTO/Transforms/Passes.td @@ -813,10 +813,12 @@ def PTOResolveBufferSelect : Pass<"pto-resolve-buffer-select", "ModuleOp"> { def PTOUnrollSIMTFor : Pass<"pto-unroll-simt-for", "func::FuncOp"> { let summary = - "Unroll explicitly annotated scf.for loops in pto.simt_entry functions"; + "Unroll explicitly annotated scf.for loops in SIMT contexts"; let description = [{ - Walks functions with `pto.simt_entry` attribute and fully unrolls `scf.for` - loops that carry the `{pto.unroll = "full"}` attribute. + Walks functions with the `pto.simt_entry` attribute or inline + `pto.section.simt` regions and fully unrolls `scf.for` loops that carry the + `{pto.unroll = "full"}` attribute. Annotated loops outside an inline SIMT + section in an ordinary function are left unchanged. Only loops with constant lower bound, upper bound, and positive step are unrolled. The annotation is opt-in: loops without the attribute are @@ -838,6 +840,70 @@ def PTOUnrollSIMTFor : Pass<"pto-unroll-simt-for", "func::FuncOp"> { ]; } +def PTOAnalyzeSIMTPersistentFragment + : Pass<"pto-analyze-simt-persistent-fragment", "func::FuncOp"> { + let summary = "Analyze SIMT persistent fragments without changing IR"; + let description = [{ + Validates persistent fragment definitions, access graphs, init/carry + sections, resident elements, and keep/resume slot assignments. This pass + is analysis-only; it does not insert or remove operations. + }]; + let constructor = + "mlir::pto::createPTOAnalyzeSIMTPersistentFragmentPass()"; + let dependentDialects = [ + "mlir::func::FuncDialect", + "mlir::LLVM::LLVMDialect", + "mlir::pto::PTODialect" + ]; +} + +def PTOMaterializeSIMTPersistentFragment + : Pass<"pto-materialize-simt-persistent-fragment", "func::FuncOp"> { + let summary = + "Materialize SIMT persistent fragments as keep/resume slot traffic"; + let description = [{ + Consumes `llvm.alloca` values marked with `pto.persistent` in a kernel body + before inline SIMT sections are outlined. The first implementation handles + single-block `pto.section.simt` bodies with scalar and fixed-width vector + LLVM load/store accesses through statically-resolved offsets. Vector + accesses are split into scalar lanes before the section-local state is + rewritten into `pto.resume` at entry and `pto.keep` at exit. + + Unsupported persistent fragment shapes fail fast because the allocation is + only a lexical carrier for register-resident state, not real stack memory. + }]; + let constructor = + "mlir::pto::createPTOMaterializeSIMTPersistentFragmentPass()"; + let dependentDialects = [ + "mlir::func::FuncDialect", + "mlir::LLVM::LLVMDialect", + "mlir::pto::PTODialect", + "mlir::arith::ArithDialect" + ]; +} + +def PTOOutlineSIMTSections + : Pass<"pto-outline-simt-sections", "ModuleOp"> { + let summary = + "Outline inline pto.section.simt regions into pto.simt_entry functions"; + let description = [{ + Converts each inline `pto.section.simt` region into a private helper + function marked with `pto.simt_entry`, captures external SSA values as + helper arguments, and replaces the original section with a + `pto.simt_launch` using the section launch dimensions. + + This is the late outline step used by TileLang/PTODSL style frontends that + keep SIMT code inline until PTOAS has had a chance to materialize + persistent SIMT fragments. + }]; + let constructor = "mlir::pto::createPTOOutlineSIMTSectionsPass()"; + let dependentDialects = [ + "mlir::func::FuncDialect", + "mlir::pto::PTODialect", + "mlir::arith::ArithDialect" + ]; +} + def PTOInferVPTOVecScope : Pass<"pto-infer-vpto-vecscope", "func::FuncOp"> { let summary = diff --git a/lib/PTO/IR/PTO.cpp b/lib/PTO/IR/PTO.cpp index d693255f2f..21633849bd 100644 --- a/lib/PTO/IR/PTO.cpp +++ b/lib/PTO/IR/PTO.cpp @@ -17696,18 +17696,6 @@ static LogicalResult verifyUniqueKeepGroupSlots(KeepOp current, return success(); } -static LogicalResult verifySimtKeepResumeCommon(Operation *op, int64_t slot) { - func::FuncOp func = getParentFunc(op); - if (!func || !func->hasAttr(pto::kPTOSimtEntryAttrName)) - return op->emitOpError("must appear inside a function marked with '") - << pto::kPTOSimtEntryAttrName << "'"; - if (slot < 0 || slot >= kSimtKeepResumeSlotLimit) { - return op->emitOpError("requires slot in range [0, ") - << (kSimtKeepResumeSlotLimit - 1) << "]"; - } - return success(); -} - static bool isSupportedSimtKeepResumeType(Type type) { if (auto intType = dyn_cast(type)) return intType.getWidth() <= 64; @@ -17728,6 +17716,17 @@ static LogicalResult verifyInsideSimtExecutionScope(Operation *op) { return success(); } +static LogicalResult verifySimtKeepResumeCommon(Operation *op, int64_t slot) { + if (!isInsideSimtExecutionScope(op)) + return op->emitOpError("must appear inside a function marked with '") + << pto::kPTOSimtEntryAttrName << "' or inside pto.section.simt"; + if (slot < 0 || slot >= kSimtKeepResumeSlotLimit) { + return op->emitOpError("requires slot in range [0, ") + << (kSimtKeepResumeSlotLimit - 1) << "]"; + } + return success(); +} + LogicalResult SyncthreadsOp::verify() { return verifyInsideSimtExecutionScope(getOperation()); } @@ -17778,12 +17777,23 @@ LogicalResult KeepOp::verify() { return failure(); Block *block = getOperation()->getBlock(); - Operation *terminator = block ? block->getTerminator() : nullptr; - if (!terminator || !isa(terminator)) + bool insideSection = + getOperation()->getParentOfType() != nullptr; + Operation *lastPayloadOp = nullptr; + if (insideSection) { + if (!block->empty()) + lastPayloadOp = &block->back(); + } else { + Operation *terminator = block->getTerminator(); + if (isa(terminator)) + lastPayloadOp = terminator->getPrevNode(); + } + if (!lastPayloadOp) return emitOpError( - "must be placed in the SIMT epilogue before func.return"); + "must be placed in the SIMT epilogue before func.return or the end " + "of pto.section.simt"); - Operation *cur = terminator->getPrevNode(); + Operation *cur = lastPayloadOp; while (cur && isa(cur)) cur = cur->getPrevNode(); Operation *lastKeep = cur; @@ -17791,7 +17801,7 @@ LogicalResult KeepOp::verify() { return emitOpError() << "must be placed in the SIMT epilogue before func.return; only " "'pto.syncthreads' may appear between the final 'pto.keep' group " - "and func.return"; + "and func.return or the end of pto.section.simt"; Operation *firstKeep = lastKeep; while (Operation *prev = firstKeep->getPrevNode()) { @@ -17802,7 +17812,8 @@ LogicalResult KeepOp::verify() { if (!isOpInRange(getOperation(), firstKeep, lastKeep)) return emitOpError() << "must be in the contiguous SIMT keep epilogue group immediately " - "before optional 'pto.syncthreads' and func.return"; + "before optional 'pto.syncthreads' and func.return or the end " + "of pto.section.simt"; if (failed(verifyUniqueKeepGroupSlots(*this, firstKeep, lastKeep))) return failure(); return success(); diff --git a/lib/PTO/Transforms/CMakeLists.txt b/lib/PTO/Transforms/CMakeLists.txt index c6cc3be548..c002b7e312 100644 --- a/lib/PTO/Transforms/CMakeLists.txt +++ b/lib/PTO/Transforms/CMakeLists.txt @@ -35,6 +35,10 @@ add_mlir_dialect_library(PTOTransforms VPTOBufferMaterialization.cpp PTOValidateVPTOIR.cpp PTOUnrollSIMTForPass.cpp + SIMTPersistentFragmentAnalysis.cpp + PTOAnalyzeSIMTPersistentFragment.cpp + PTOMaterializeSIMTPersistentFragment.cpp + PTOOutlineSIMTSections.cpp PTOInferVPTOVecScope.cpp InsertSync/PTOInsertSync.cpp diff --git a/lib/PTO/Transforms/PTOAnalyzeSIMTPersistentFragment.cpp b/lib/PTO/Transforms/PTOAnalyzeSIMTPersistentFragment.cpp new file mode 100644 index 0000000000..974cb29a3d --- /dev/null +++ b/lib/PTO/Transforms/PTOAnalyzeSIMTPersistentFragment.cpp @@ -0,0 +1,52 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +#include "PTO/Transforms/Passes.h" +#include "SIMTPersistentFragmentAnalysis.h" + +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Pass/Pass.h" + +namespace mlir { +namespace pto { +#define GEN_PASS_DEF_PTOANALYZESIMTPERSISTENTFRAGMENT +#include "PTO/Transforms/Passes.h.inc" +} // namespace pto +} // namespace mlir + +using namespace mlir; + +namespace { + +struct PTOAnalyzeSIMTPersistentFragmentPass + : public pto::impl::PTOAnalyzeSIMTPersistentFragmentBase< + PTOAnalyzeSIMTPersistentFragmentPass> { + using pto::impl::PTOAnalyzeSIMTPersistentFragmentBase< + PTOAnalyzeSIMTPersistentFragmentPass>:: + PTOAnalyzeSIMTPersistentFragmentBase; + + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (func.isExternal()) + return; + + const auto &analysis = getAnalysis(); + if (!analysis.isValid()) { + signalPassFailure(); + return; + } + + markAllAnalysesPreserved(); + } +}; + +} // namespace + +std::unique_ptr mlir::pto::createPTOAnalyzeSIMTPersistentFragmentPass() { + return std::make_unique(); +} diff --git a/lib/PTO/Transforms/PTOMaterializeSIMTPersistentFragment.cpp b/lib/PTO/Transforms/PTOMaterializeSIMTPersistentFragment.cpp new file mode 100644 index 0000000000..6aba8b1640 --- /dev/null +++ b/lib/PTO/Transforms/PTOMaterializeSIMTPersistentFragment.cpp @@ -0,0 +1,633 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- PTOMaterializeSIMTPersistentFragment.cpp ---------------------------===// +// +// Materialize lexically persistent SIMT fragments into pto.keep/pto.resume. +// +// The read-only discovery and planning lives in +// SIMTPersistentFragmentAnalysis. This pass only consumes the cached plan and +// performs section-local IR materialization. +// +//===----------------------------------------------------------------------===// + +#include "PTO/IR/PTO.h" +#include "PTO/Transforms/Passes.h" +#include "SIMTPersistentFragmentAnalysis.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/LLVMIR/LLVMDialect.h" +#include "mlir/IR/Dominance.h" +#include "mlir/Interfaces/DataLayoutInterfaces.h" +#include "mlir/Pass/Pass.h" +#include "mlir/Transforms/Mem2Reg.h" +#include "llvm/ADT/DenseMap.h" +#include "llvm/ADT/DenseSet.h" +#include "llvm/ADT/STLExtras.h" +#include "llvm/ADT/SmallVector.h" + +#include +#include + +namespace mlir { +namespace pto { +#define GEN_PASS_DEF_PTOMATERIALIZESIMTPERSISTENTFRAGMENT +#include "PTO/Transforms/Passes.h.inc" +} // namespace pto +} // namespace mlir + +using namespace mlir; + +namespace { + +using pto::AccessLane; +using pto::PersistentFragmentAnalysis; +using pto::PersistentMaterializationPlan; +using pto::ResidentElementPlan; + +struct PersistentElementRewrite { + LLVM::AllocaOp proxy; + Value resumeValue; +}; + +struct PersistentLaneRewrite { + // Scalar component index in the original access. Scalar accesses use lane 0. + unsigned laneIndex; + // Index into the section-local elements and parallel elementRewrites arrays. + unsigned elementIndex; +}; + +struct PersistentElementWorkItem { + const PersistentFragmentAnalysis *fragment = nullptr; + const ResidentElementPlan *residentElement = nullptr; + // Access lanes for this element that occur in the owning worklist section. + SmallVector accesses; +}; + +// A section-local, fully validated view of the immutable analysis plan. The +// worklist directly references fragment/element plans and owns only local lane +// bindings, so the transform does not rediscover them while mutating the body. +struct PersistentSectionWorklist { + explicit PersistentSectionWorklist(pto::SectionSimtOp section) + : section(section) {} + + pto::SectionSimtOp section; + SmallVector elements; + llvm::DenseMap> + laneRewritesByAccess; +}; + +struct PersistentTransformWorklist { + SmallVector sections; +}; + +// Redirect one scalar access to its section-local proxy. +static LogicalResult rewireScalarAccess(Operation *access, + LLVM::AllocaOp proxy) { + if (auto load = dyn_cast(access)) { + load.getAddrMutable().assign(proxy.getRes()); + return success(); + } + if (auto store = dyn_cast(access)) { + store.getAddrMutable().assign(proxy.getRes()); + return success(); + } + return access->emitOpError( + "expected a scalar llvm.load or llvm.store during persistent " + "fragment materialization"); +} + +// Return the scalar or vector value type carried by an LLVM memory access. +static FailureOr getPersistentAccessType(Operation *access) { + if (auto load = dyn_cast(access)) + return load.getRes().getType(); + if (auto store = dyn_cast(access)) + return store.getValue().getType(); + access->emitOpError( + "expected an llvm.load or llvm.store persistent fragment access"); + return failure(); +} + +// Validate and order each operation's lane-to-proxy rewrite before mutation. +static LogicalResult validateAccessRewritePlan( + llvm::DenseMap> + &laneRewritesByAccess) { + for (auto &[access, laneRewrites] : laneRewritesByAccess) { + FailureOr accessType = getPersistentAccessType(access); + if (failed(accessType)) + return failure(); + + llvm::sort(laneRewrites, [](const PersistentLaneRewrite &lhs, + const PersistentLaneRewrite &rhs) { + return lhs.laneIndex < rhs.laneIndex; + }); + + unsigned expectedLaneCount = 1; + if (auto vectorType = dyn_cast(*accessType)) + expectedLaneCount = static_cast(vectorType.getNumElements()); + + if (laneRewrites.size() != expectedLaneCount) { + return access->emitOpError() + << "persistent fragment access expected " << expectedLaneCount + << " scalar lane rewrite(s), got " << laneRewrites.size(); + } + for (auto [expectedLaneIndex, laneRewrite] : + llvm::enumerate(laneRewrites)) { + if (laneRewrite.laneIndex != expectedLaneIndex) { + return access->emitOpError() + << "persistent fragment access has non-contiguous lane " + "mapping at lane " + << expectedLaneIndex; + } + } + } + return success(); +} + +static bool +isFragmentActiveInSection(const PersistentFragmentAnalysis &fragment, + pto::SectionSimtOp section) { + if (fragment.initSection == section) + return true; + return llvm::any_of(fragment.carrySections, [&](pto::SectionSimtOp carry) { + return carry == section; + }); +} + +// Build and validate all section-local rewrite bindings before touching IR. +// The analysis result already fixes each fragment's init/carry lifetime. This +// check only verifies that the temporary worklist faithfully materializes that +// immutable result. +static LogicalResult +validateSectionWorklist(const PersistentMaterializationPlan &plan, + PersistentSectionWorklist §ionWorklist, + llvm::DenseMap> + &assignedAccessLanes) { + pto::SectionSimtOp section = sectionWorklist.section; + if (!section || !section.getBody().hasOneBlock()) { + return section ? section.emitOpError( + "persistent fragment transform requires a single-" + "block SIMT section") + : failure(); + } + + unsigned expectedElementIndex = 0; + for (const PersistentFragmentAnalysis &fragment : plan.fragments) { + if (!isFragmentActiveInSection(fragment, section)) + continue; + + for (const ResidentElementPlan &expectedElement : + fragment.residentElements) { + if (expectedElementIndex >= sectionWorklist.elements.size()) { + LLVM::AllocaOp allocaOp = fragment.allocaOp; + return allocaOp.emitOpError( + "persistent fragment section worklist is missing the complete " + "resident element set"); + } + + const PersistentElementWorkItem &element = + sectionWorklist.elements[expectedElementIndex]; + if (element.fragment != &fragment || + element.residentElement != &expectedElement) { + return section.emitOpError( + "persistent fragment section worklist does not preserve fragment " + "and resident element order"); + } + + const ResidentElementPlan &residentElement = *element.residentElement; + + for (const AccessLane &accessLane : element.accesses) { + if (!accessLane.op) { + LLVM::AllocaOp allocaOp = fragment.allocaOp; + return allocaOp.emitOpError( + "persistent fragment worklist contains a null access " + "operation"); + } + + bool isOwnedAccess = llvm::any_of( + residentElement.accesses, [&](const AccessLane &candidate) { + return candidate.op == accessLane.op && + candidate.laneIndex == accessLane.laneIndex; + }); + if (!isOwnedAccess) { + return accessLane.op->emitOpError( + "persistent fragment worklist access lane is owned by a " + "different resident element"); + } + + pto::SectionSimtOp accessSection = + accessLane.op->getParentOfType(); + if (accessSection != section || + accessLane.op->getBlock() != §ion.getBody().front()) { + return accessLane.op->emitOpError( + "persistent fragment access does not belong to its planned " + "SIMT section body"); + } + if (!assignedAccessLanes[accessLane.op] + .insert(accessLane.laneIndex) + .second) { + return accessLane.op->emitOpError( + "persistent fragment access lane was assigned to more than one " + "transform work item"); + } + // Invert the element-owned access into an operation-owned lane mapping. + // elementIndex selects this lane's scalar proxy from elementRewrites. + sectionWorklist.laneRewritesByAccess[accessLane.op].push_back( + {accessLane.laneIndex, expectedElementIndex}); + } + + ++expectedElementIndex; + } + } + + if (expectedElementIndex != sectionWorklist.elements.size()) { + return section.emitOpError( + "persistent fragment section worklist contains an unexpected " + "resident element"); + } + + return validateAccessRewritePlan(sectionWorklist.laneRewritesByAccess); +} + +// Construct the complete transform worklist. No operation insertion, erase, +// or operand replacement is allowed before this function succeeds. +static LogicalResult +buildPersistentTransformWorklist(const PersistentMaterializationPlan &plan, + PersistentTransformWorklist &worklist) { + worklist.sections.clear(); + worklist.sections.reserve(plan.sections.size()); + for (pto::SectionSimtOp section : plan.sections) + worklist.sections.emplace_back(section); + + llvm::DenseMap worklistBySection; + for (PersistentSectionWorklist §ionWorklist : worklist.sections) { + if (!sectionWorklist.section) + return failure(); + auto [it, inserted] = worklistBySection.try_emplace( + sectionWorklist.section.getOperation(), §ionWorklist); + (void)it; + if (!inserted) { + return sectionWorklist.section.emitOpError( + "persistent fragment section appears more than once in the plan"); + } + } + + // Append each fragment's complete resident set to its init and carry + // sections. The outer section vector remains in function walk order; + // appending fragments in alloca order preserves function-wide slot order. + for (const PersistentFragmentAnalysis &fragment : plan.fragments) { + LLVM::AllocaOp allocaOp = fragment.allocaOp; + if (!fragment.initSection) { + return allocaOp.emitOpError( + "persistent fragment has no init section in its analysis plan"); + } + + llvm::DenseSet addedSections; + auto addToSection = [&](pto::SectionSimtOp section) -> LogicalResult { + if (!addedSections.insert(section.getOperation()).second) { + return allocaOp.emitOpError( + "persistent fragment init/carry sections contain a duplicate"); + } + auto sectionIt = worklistBySection.find(section.getOperation()); + if (sectionIt == worklistBySection.end()) { + return allocaOp.emitOpError( + "persistent fragment init/carry section is not in the plan"); + } + + PersistentSectionWorklist §ionWorklist = *sectionIt->second; + for (const ResidentElementPlan &residentElement : + fragment.residentElements) { + SmallVector localAccesses; + for (const AccessLane &accessLane : residentElement.accesses) { + pto::SectionSimtOp accessSection = + accessLane.op + ? accessLane.op->getParentOfType() + : pto::SectionSimtOp(); + if (accessSection == section) + localAccesses.push_back(accessLane); + } + sectionWorklist.elements.push_back( + {&fragment, &residentElement, std::move(localAccesses)}); + } + return success(); + }; + + if (failed(addToSection(fragment.initSection))) + return failure(); + for (pto::SectionSimtOp section : fragment.carrySections) { + if (failed(addToSection(section))) + return failure(); + } + } + + llvm::DenseMap> assignedAccessLanes; + for (PersistentSectionWorklist §ionWorklist : worklist.sections) { + if (failed(validateSectionWorklist(plan, sectionWorklist, + assignedAccessLanes))) + return failure(); + } + + for (const PersistentFragmentAnalysis &fragment : plan.fragments) { + for (const ResidentElementPlan &element : fragment.residentElements) { + for (const AccessLane &accessLane : element.accesses) { + if (!accessLane.op) { + LLVM::AllocaOp allocaOp = fragment.allocaOp; + return allocaOp.emitOpError( + "persistent fragment analysis contains a null access " + "operation"); + } + auto assignedIt = assignedAccessLanes.find(accessLane.op); + if (assignedIt == assignedAccessLanes.end() || + !assignedIt->second.contains(accessLane.laneIndex)) { + accessLane.op->emitOpError( + "persistent fragment access lane was not assigned to a " + "transform work item"); + return failure(); + } + } + } + } + return success(); +} + +// Rewrite one analyzed access against its section-local scalar proxies. +static LogicalResult rewritePersistentAccess( + Operation *access, ArrayRef laneRewrites, + MutableArrayRef elementRewrites) { + FailureOr accessType = getPersistentAccessType(access); + if (failed(accessType)) + return failure(); + + if (!isa(*accessType)) { + assert(laneRewrites.size() == 1 && laneRewrites.front().laneIndex == 0 && + "scalar access must map to exactly one scalar lane"); + LLVM::AllocaOp proxy = + elementRewrites[laneRewrites.front().elementIndex].proxy; + assert(proxy && "locally accessed element must have a scalar proxy"); + return rewireScalarAccess(access, proxy); + } + + OpBuilder builder(access); + Location loc = access->getLoc(); + if (auto store = dyn_cast(access)) { + for (const PersistentLaneRewrite &laneRewrite : laneRewrites) { + LLVM::AllocaOp proxy = elementRewrites[laneRewrite.elementIndex].proxy; + assert(proxy && "vector store lane must have a scalar proxy"); + Value laneIndex = builder.create( + loc, laneRewrite.laneIndex, /*width=*/32); + Value laneValue = builder.create( + loc, store.getValue(), laneIndex); + builder.create(loc, laneValue, proxy.getRes()); + } + store.erase(); + return success(); + } + + auto load = cast(access); + auto vectorType = cast(*accessType); + Value rebuiltVector = builder.create(loc, vectorType); + for (const PersistentLaneRewrite &laneRewrite : laneRewrites) { + LLVM::AllocaOp proxy = elementRewrites[laneRewrite.elementIndex].proxy; + assert(proxy && "vector load lane must have a scalar proxy"); + Value laneValue = builder.create( + loc, vectorType.getElementType(), proxy.getRes()); + Value laneIndex = builder.create( + loc, laneRewrite.laneIndex, /*width=*/32); + rebuiltVector = builder.create( + loc, vectorType, rebuiltVector, laneValue, laneIndex); + } + load.getRes().replaceAllUsesWith(rebuiltVector); + load.erase(); + return success(); +} + +// Materialize all active persistent elements in one inline SIMT section. +static LogicalResult +materializeSection(const PersistentSectionWorklist §ionWorklist, + const DataLayout &dataLayout, DominanceInfo &dominance) { + const auto &elements = sectionWorklist.elements; + if (elements.empty()) + return success(); + + pto::SectionSimtOp section = sectionWorklist.section; + Block &body = section.getBody().front(); + SmallVector rewrites(elements.size()); + + // The section-local access/lane bindings were completely validated before + // the first section was modified. + const auto &laneRewritesByAccess = sectionWorklist.laneRewritesByAccess; + + OpBuilder entryBuilder(section.getContext()); + entryBuilder.setInsertionPointToStart(&body); + + bool hasLocalAccess = !laneRewritesByAccess.empty(); + Value proxyArraySize; + if (hasLocalAccess) { + proxyArraySize = entryBuilder.create(section.getLoc(), + 1, /*width=*/32); + } + + // Emit the complete resume prologue before proxy setup so the group remains + // contiguous. Elements initialized by this section have no incoming value. + for (auto [elementIndex, element] : llvm::enumerate(elements)) { + const PersistentFragmentAnalysis &fragment = *element.fragment; + const ResidentElementPlan &residentElement = *element.residentElement; + LLVM::AllocaOp allocaOp = fragment.allocaOp; + if (section == fragment.initSection) + continue; + + rewrites[elementIndex].resumeValue = + entryBuilder + .create(section.getLoc(), allocaOp.getElemType(), + static_cast(residentElement.slot)) + .getResult(); + } + + // A proxy gives generic mem2reg one scalar slot in the same region as all of + // its accesses. Carry sections seed it from resume; init sections rely on + // the previously validated first-store initialization. + for (auto [elementIndex, element] : llvm::enumerate(elements)) { + if (element.accesses.empty()) + continue; + + const PersistentFragmentAnalysis &fragment = *element.fragment; + LLVM::AllocaOp allocaOp = fragment.allocaOp; + Location elementLoc = element.accesses.front().op->getLoc(); + LLVM::AllocaOp proxy = entryBuilder.create( + elementLoc, allocaOp.getRes().getType(), allocaOp.getElemType(), + proxyArraySize); + rewrites[elementIndex].proxy = proxy; + + if (section != fragment.initSection) { + assert(rewrites[elementIndex].resumeValue && + "carry element must have a resume value"); + entryBuilder.create( + elementLoc, rewrites[elementIndex].resumeValue, proxy.getRes()); + } + } + + // Rewrite original accesses only after every section-local proxy exists. + // Block order gives vector scalarization a stable operation-level traversal. + size_t rewrittenAccessCount = 0; + for (Operation &op : llvm::make_early_inc_range(body)) { + auto accessIt = laneRewritesByAccess.find(&op); + if (accessIt == laneRewritesByAccess.end()) + continue; + if (failed(rewritePersistentAccess(&op, accessIt->second, rewrites))) + return failure(); + ++rewrittenAccessCount; + } + if (rewrittenAccessCount != laneRewritesByAccess.size()) { + return section.emitOpError( + "failed to rewrite every persistent fragment access in block order"); + } + + OpBuilder exitBuilder(section.getContext()); + exitBuilder.setInsertionPointToEnd(&body); + + // Compute every outgoing value before emitting keeps so the keep epilogue is + // one contiguous group. + SmallVector keepPayloads; + keepPayloads.reserve(elements.size()); + for (auto [elementIndex, element] : llvm::enumerate(elements)) { + const PersistentFragmentAnalysis &fragment = *element.fragment; + LLVM::AllocaOp allocaOp = fragment.allocaOp; + if (rewrites[elementIndex].proxy) { + keepPayloads.push_back( + exitBuilder + .create(section.getLoc(), allocaOp.getElemType(), + rewrites[elementIndex].proxy.getRes()) + .getResult()); + continue; + } + assert(rewrites[elementIndex].resumeValue && + "an element without local accesses must pass through resume"); + keepPayloads.push_back(rewrites[elementIndex].resumeValue); + } + + for (auto [element, payload] : llvm::zip_equal(elements, keepPayloads)) { + const ResidentElementPlan &residentElement = *element.residentElement; + exitBuilder.create( + section.getLoc(), payload, static_cast(residentElement.slot)); + } + + // Promote each proxy independently so success guarantees that this specific + // temporary allocation and all of its memory traffic were eliminated. + OpBuilder promotionBuilder(section.getContext()); + promotionBuilder.setInsertionPointToStart(&body); + for (auto [elementIndex, element] : llvm::enumerate(elements)) { + LLVM::AllocaOp proxy = rewrites[elementIndex].proxy; + if (!proxy) + continue; + SmallVector allocators{ + cast(proxy.getOperation())}; + if (failed(tryToPromoteMemorySlots(allocators, promotionBuilder, dataLayout, + dominance))) { + return section.emitOpError() + << "failed to promote scalar proxy for persistent fragment " + "element offset " + << element.residentElement->elementOffset; + } + } + + if (proxyArraySize && proxyArraySize.use_empty()) + proxyArraySize.getDefiningOp()->erase(); + return success(); +} + +// Remove the now-dead GEP tree rooted at a persistent alloca result. +static LogicalResult eraseDeadPersistentGEPs(Value pointer) { + SmallVector derivedPointers; + for (Operation *user : pointer.getUsers()) { + auto gep = dyn_cast(user); + if (!gep) { + return user->emitOpError( + "persistent fragment access remained after scalar proxy promotion"); + } + derivedPointers.push_back(gep); + } + + for (LLVM::GEPOp gep : derivedPointers) { + if (failed(eraseDeadPersistentGEPs(gep.getRes()))) + return failure(); + if (!gep.getRes().use_empty()) { + return gep.emitOpError( + "persistent fragment derived pointer still has live uses after " + "scalar proxy promotion"); + } + gep.erase(); + } + return success(); +} + +// Rewrite every planned section, then remove the original lexical carriers. +static LogicalResult +materializePersistentFragments(func::FuncOp func, DominanceInfo &dominance, + const PersistentMaterializationPlan &plan, + const PersistentTransformWorklist &worklist) { + DataLayout dataLayout = DataLayout::closest(func); + for (const PersistentSectionWorklist §ionWorklist : worklist.sections) { + if (failed(materializeSection(sectionWorklist, dataLayout, dominance))) + return failure(); + } + + for (const PersistentFragmentAnalysis &fragment : plan.fragments) { + LLVM::AllocaOp allocaOp = fragment.allocaOp; + if (failed(eraseDeadPersistentGEPs(allocaOp.getRes()))) + return failure(); + if (!allocaOp.getRes().use_empty()) { + return allocaOp.emitOpError( + "persistent fragment still has live uses after materialization"); + } + allocaOp.erase(); + } + return success(); +} + +struct PTOMaterializeSIMTPersistentFragmentPass + : public pto::impl::PTOMaterializeSIMTPersistentFragmentBase< + PTOMaterializeSIMTPersistentFragmentPass> { + using pto::impl::PTOMaterializeSIMTPersistentFragmentBase< + PTOMaterializeSIMTPersistentFragmentPass>:: + PTOMaterializeSIMTPersistentFragmentBase; + + void runOnOperation() override { + func::FuncOp func = getOperation(); + const auto &analysis = getAnalysis(); + if (!analysis.isValid()) { + signalPassFailure(); + return; + } + + const PersistentMaterializationPlan &plan = analysis.getPlan(); + if (plan.fragments.empty()) + return; + + // Convert the fragment/element-oriented analysis plan into a validated, + // section/operation-oriented worklist before any IR mutation. + PersistentTransformWorklist worklist; + if (failed(buildPersistentTransformWorklist(plan, worklist))) { + signalPassFailure(); + return; + } + DominanceInfo dominance(func); + if (failed( + materializePersistentFragments(func, dominance, plan, worklist))) { + signalPassFailure(); + return; + } + } +}; + +} // namespace + +std::unique_ptr +mlir::pto::createPTOMaterializeSIMTPersistentFragmentPass() { + return std::make_unique(); +} diff --git a/lib/PTO/Transforms/PTOOutlineSIMTSections.cpp b/lib/PTO/Transforms/PTOOutlineSIMTSections.cpp new file mode 100644 index 0000000000..90df3a527f --- /dev/null +++ b/lib/PTO/Transforms/PTOOutlineSIMTSections.cpp @@ -0,0 +1,281 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- PTOOutlineSIMTSections.cpp ----------------------------------------===// +// +// Outline inline pto.section.simt regions into pto.simt_entry helpers. +// +// PTODSL can keep SIMT code inline so PTOAS can still see values allocated in +// the outer kernel scope before materializing persistent SIMT fragments. This +// pass is the late outline step that restores the VPTO backend's existing +// simt_entry + simt_launch form. +// +//===----------------------------------------------------------------------===// + +#include "PTO/IR/PTO.h" +#include "PTO/Transforms/Passes.h" + +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/IR/BuiltinOps.h" +#include "mlir/IR/IRMapping.h" +#include "mlir/IR/Operation.h" +#include "mlir/Pass/Pass.h" +#include "llvm/ADT/DenseSet.h" +#include "llvm/ADT/SmallString.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/Twine.h" + +namespace mlir { +namespace pto { +#define GEN_PASS_DEF_PTOOUTLINESIMTSECTIONS +#include "PTO/Transforms/Passes.h.inc" +} // namespace pto +} // namespace mlir + +using namespace mlir; + +namespace { + +static bool isDefinedInside(Operation *scope, Value value) { + if (Operation *defOp = value.getDefiningOp()) + return scope->isAncestor(defOp); + + auto blockArg = dyn_cast(value); + if (!blockArg) + return false; + + Operation *owner = blockArg.getOwner()->getParentOp(); + return owner && scope->isAncestor(owner); +} + +static LogicalResult collectCaptures(pto::SectionSimtOp sectionOp, + SmallVectorImpl &captures) { + llvm::DenseSet seen; + Operation *scope = sectionOp.getOperation(); + + sectionOp.getBody().walk([&](Operation *op) { + for (Value operand : op->getOperands()) { + if (isDefinedInside(scope, operand)) + continue; + if (Operation *defOp = operand.getDefiningOp()) { + if (defOp->hasTrait()) + continue; + } + if (seen.insert(operand).second) + captures.push_back(operand); + } + }); + + return success(); +} + +static void cloneExternalConstants(pto::SectionSimtOp sectionOp, + OpBuilder &builder, IRMapping &mapping) { + llvm::DenseSet seen; + SmallVector constants; + Operation *scope = sectionOp.getOperation(); + + sectionOp.getBody().walk([&](Operation *op) { + for (Value operand : op->getOperands()) { + Operation *defOp = operand.getDefiningOp(); + if (!defOp || isDefinedInside(scope, operand) || + !defOp->hasTrait()) + continue; + if (seen.insert(defOp).second) + constants.push_back(defOp); + } + }); + + for (Operation *constant : constants) + builder.clone(*constant, mapping); +} + +static LogicalResult verifySectionCanBeOutlined(pto::SectionSimtOp sectionOp) { + func::FuncOp parentFunc = sectionOp->getParentOfType(); + if (!parentFunc) + return sectionOp.emitOpError("must be nested in a func.func"); + + if (parentFunc->hasAttr(pto::kPTOSimtEntryAttrName)) { + return sectionOp.emitOpError() + << "must not be nested in a function marked with '" + << pto::kPTOSimtEntryAttrName << "'"; + } + + if (!sectionOp.getBody().hasOneBlock()) + return sectionOp.emitOpError("requires a single-block body"); + + if (sectionOp.getBody().front().getNumArguments() != 0) + return sectionOp.emitOpError("does not support region block arguments"); + + bool hasNestedSection = false; + sectionOp.getBody().walk([&](pto::SectionSimtOp nested) { + if (nested != sectionOp) { + hasNestedSection = true; + return WalkResult::interrupt(); + } + return WalkResult::advance(); + }); + if (hasNestedSection) + return sectionOp.emitOpError("does not support nested pto.section.simt"); + + Operation *scope = sectionOp.getOperation(); + WalkResult escapeCheck = sectionOp.getBody().walk([&](Operation *op) { + for (Value result : op->getResults()) { + for (Operation *user : result.getUsers()) { + if (!scope->isAncestor(user)) + return WalkResult::interrupt(); + } + } + return WalkResult::advance(); + }); + if (escapeCheck.wasInterrupted()) { + return sectionOp.emitOpError( + "does not support values defined in pto.section.simt escaping the " + "section"); + } + + return success(); +} + +static std::string getUniqueHelperName(ModuleOp module, func::FuncOp parentFunc, + unsigned &outlineIndex) { + std::string parentName = parentFunc.getSymName().str(); + do { + std::string candidate = + (Twine(parentName) + "_simt_" + Twine(outlineIndex++)).str(); + if (!module.lookupSymbol(candidate)) + return candidate; + } while (true); +} + +static FailureOr getSimtThreadCount(pto::SectionSimtOp sectionOp) { + constexpr int64_t kMaxSimtThreads = 2048; + const int64_t dimX = sectionOp.getDimX(); + const int64_t dimY = sectionOp.getDimY(); + const int64_t dimZ = sectionOp.getDimZ(); + + int64_t threadCount = 1; + for (int64_t dimension : {dimX, dimY, dimZ}) { + if (dimension <= 0 || threadCount > kMaxSimtThreads / dimension) { + sectionOp.emitOpError() + << "SIMT launch dimensions must have a positive product no greater " + "than " + << kMaxSimtThreads << ", got (" << dimX << ", " << dimY << ", " + << dimZ << ")"; + return failure(); + } + threadCount *= dimension; + } + return threadCount; +} + +static func::FuncOp createOutlinedHelper(ModuleOp module, + pto::SectionSimtOp sectionOp, + StringRef helperName, + int64_t maxThreads, + ValueRange captures) { + MLIRContext *ctx = module.getContext(); + Location loc = sectionOp.getLoc(); + + SmallVector argTypes; + argTypes.reserve(captures.size()); + for (Value capture : captures) + argTypes.push_back(capture.getType()); + + OpBuilder moduleBuilder(module.getBodyRegion()); + moduleBuilder.setInsertionPointToEnd(&module.getBodyRegion().front()); + auto helper = moduleBuilder.create( + loc, helperName, FunctionType::get(ctx, argTypes, TypeRange{})); + helper.setPrivate(); + helper->setAttr(pto::kPTOSimtEntryAttrName, UnitAttr::get(ctx)); + + // The outlined helper is compiled independently from the outer launch. Keep + // its static lane count so the LLVM emitter can derive the correct SIMT + // register budget instead of falling back to the 1024-thread default. + helper->setAttr(pto::kPTOSimtMaxThreadsAttrName, + moduleBuilder.getI32IntegerAttr(maxThreads)); + + Block *entry = helper.addEntryBlock(); + IRMapping mapping; + for (auto [capture, arg] : llvm::zip(captures, entry->getArguments())) + mapping.map(capture, arg); + + OpBuilder bodyBuilder = OpBuilder::atBlockEnd(entry); + cloneExternalConstants(sectionOp, bodyBuilder, mapping); + for (Operation &op : sectionOp.getBody().front()) + bodyBuilder.clone(op, mapping); + bodyBuilder.create(loc); + + return helper; +} + +static void replaceSectionWithLaunch(pto::SectionSimtOp sectionOp, + StringRef helperName, + ValueRange captures) { + Location loc = sectionOp.getLoc(); + OpBuilder builder(sectionOp); + Value dimX = builder.create(loc, sectionOp.getDimX(), + /*width=*/32); + Value dimY = builder.create(loc, sectionOp.getDimY(), + /*width=*/32); + Value dimZ = builder.create(loc, sectionOp.getDimZ(), + /*width=*/32); + builder.create(loc, helperName, dimX, dimY, dimZ, + captures); + sectionOp.erase(); +} + +static LogicalResult outlineSection(ModuleOp module, + pto::SectionSimtOp sectionOp, + unsigned &outlineIndex) { + if (failed(verifySectionCanBeOutlined(sectionOp))) + return failure(); + + FailureOr maxThreads = getSimtThreadCount(sectionOp); + if (failed(maxThreads)) + return failure(); + + SmallVector captures; + if (failed(collectCaptures(sectionOp, captures))) + return failure(); + + func::FuncOp parentFunc = sectionOp->getParentOfType(); + std::string helperName = + getUniqueHelperName(module, parentFunc, outlineIndex); + createOutlinedHelper(module, sectionOp, helperName, *maxThreads, captures); + replaceSectionWithLaunch(sectionOp, helperName, captures); + return success(); +} + +struct PTOOutlineSIMTSectionsPass + : public pto::impl::PTOOutlineSIMTSectionsBase { + using pto::impl::PTOOutlineSIMTSectionsBase< + PTOOutlineSIMTSectionsPass>::PTOOutlineSIMTSectionsBase; + + void runOnOperation() override { + ModuleOp module = getOperation(); + SmallVector sections; + module.walk( + [&](pto::SectionSimtOp sectionOp) { sections.push_back(sectionOp); }); + + unsigned outlineIndex = 0; + for (pto::SectionSimtOp sectionOp : sections) { + if (failed(outlineSection(module, sectionOp, outlineIndex))) { + signalPassFailure(); + return; + } + } + } +}; + +} // namespace + +std::unique_ptr mlir::pto::createPTOOutlineSIMTSectionsPass() { + return std::make_unique(); +} diff --git a/lib/PTO/Transforms/PTOUnrollSIMTForPass.cpp b/lib/PTO/Transforms/PTOUnrollSIMTForPass.cpp index 7735520d3c..cc306c4100 100644 --- a/lib/PTO/Transforms/PTOUnrollSIMTForPass.cpp +++ b/lib/PTO/Transforms/PTOUnrollSIMTForPass.cpp @@ -8,15 +8,17 @@ //===- PTOUnrollSIMTForPass.cpp -------------------------------------------===// // -// Unroll explicitly annotated scf.for loops inside pto.simt_entry -// functions to eliminate divergent control flow before LLVM lowering. +// Unroll explicitly annotated scf.for loops inside SIMT contexts to eliminate +// divergent control flow before LLVM lowering. // // Only loops carrying the `{pto.unroll = "full"}` attribute are unrolled. -// The pass is gated to pto.simt_entry functions so it does not affect -// general-purpose loops. +// A SIMT context is either a pto.simt_entry function or an inline +// pto.section.simt region that has not been outlined yet. General-purpose +// loops outside these contexts are not affected. // //===----------------------------------------------------------------------===// +#include "PTO/IR/PTO.h" #include "PTO/Transforms/Passes.h" #include "mlir/Dialect/Func/IR/FuncOps.h" @@ -68,6 +70,22 @@ static bool isSIMTEntry(func::FuncOp func) { return func->hasAttr(pto::kPTOSimtEntryAttrName); } +/// Check whether a function contains an inline SIMT section. +static bool containsInlineSIMTSection(func::FuncOp func) { + WalkResult result = + func.walk([](pto::SectionSimtOp) { return WalkResult::interrupt(); }); + return result.wasInterrupted(); +} + +/// Check whether a loop is inside either supported SIMT representation. +static bool isInSIMTContext(scf::ForOp forOp) { + func::FuncOp func = forOp->getParentOfType(); + if (!func) + return false; + return isSIMTEntry(func) || + static_cast(forOp->getParentOfType()); +} + // --------------------------------------------------------------------------- // Rewrite pattern // --------------------------------------------------------------------------- @@ -79,9 +97,8 @@ struct UnrollSIMTForPattern : public OpRewritePattern { LogicalResult matchAndRewrite(scf::ForOp forOp, PatternRewriter &rewriter) const override { - // Only apply inside SIMT entry functions. - auto func = forOp->getParentOfType(); - if (!func || !isSIMTEntry(func)) + // Only apply inside an outlined or inline SIMT context. + if (!isInSIMTContext(forOp)) return failure(); // Only unroll loops with explicit {pto.unroll = "full"} annotation. @@ -126,7 +143,7 @@ struct PTOUnrollSIMTFor : public pto::impl::PTOUnrollSIMTForBase +#include +#include + +using namespace mlir; + +namespace mlir { +namespace pto { +namespace { + +constexpr llvm::StringLiteral kPersistentAttrName = "pto.persistent"; +constexpr int64_t kPersistentSlotLimit = 123; + +struct PointerWorkItem { + Value pointer; + int64_t byteOffset; +}; + +// Allocation shape needed while normalizing a single pointer use graph. The +// published fragment plan only needs the resulting resident elements and does +// not retain these validation-only values. +struct FragmentShape { + int64_t elementByteSize; + int64_t totalByteSize; +}; + +// These records exist only while the pointer use graph is being normalized. +// The published plan stores the resulting AccessLane values under their +// owning resident element instead of retaining this fragment-level index. +struct NormalizedPersistentAccess { + Operation *op; + pto::SectionSimtOp section; + int64_t elementOffset; + unsigned laneIndex; +}; + +struct PersistentAccessDiscovery { + SmallVector accesses; + llvm::DenseMap> accessIndices; +}; + +static bool isSupportedPersistentScalarType(Type type) { + if (auto intType = dyn_cast(type)) + return intType.getWidth() <= 64; + return type.isF16() || type.isBF16() || type.isF32(); +} + +static FailureOr getSignedInt64(const APInt &value, Operation *anchor, + StringRef description) { + if (!value.isSignedIntN(64)) { + anchor->emitOpError() << description << " does not fit in signed i64"; + return failure(); + } + return value.getSExtValue(); +} + +static FailureOr getConstantInt64(Value value, Operation *anchor, + StringRef description) { + APInt constant; + if (!matchPattern(value, m_ConstantInt(&constant))) { + anchor->emitOpError() << description + << " must be a compile-time integer constant"; + return failure(); + } + return getSignedInt64(constant, anchor, description); +} + +static FailureOr getFixedTypeByteSize(Type type, Operation *anchor, + StringRef description, + const DataLayout &dataLayout) { + llvm::TypeSize typeSize = dataLayout.getTypeSize(type); + if (typeSize.isScalable() || typeSize.getFixedValue() == 0 || + typeSize.getFixedValue() > + static_cast(std::numeric_limits::max())) { + anchor->emitOpError() << description + << " must have a non-zero fixed byte size, got " + << type; + return failure(); + } + return static_cast(typeSize.getFixedValue()); +} + +static FailureOr getFragmentShape(LLVM::AllocaOp allocaOp, + const DataLayout &dataLayout) { + Type elementType = allocaOp.getElemType(); + if (!isSupportedPersistentScalarType(elementType)) { + return allocaOp.emitOpError() + << "persistent SIMT fragment requires an integer scalar element " + "type up to 64 bits or f16/bf16/f32, got " + << elementType; + } + + FailureOr elementCount = getConstantInt64( + allocaOp.getArraySize(), allocaOp, "persistent fragment element count"); + if (failed(elementCount)) + return failure(); + if (*elementCount <= 0) { + return allocaOp.emitOpError( + "persistent SIMT fragment element count must be positive"); + } + + FailureOr elementByteSize = getFixedTypeByteSize( + elementType, allocaOp, "persistent fragment element type", dataLayout); + if (failed(elementByteSize)) + return failure(); + + int64_t totalByteSize; + if (llvm::MulOverflow(*elementCount, *elementByteSize, totalByteSize)) { + return allocaOp.emitOpError( + "persistent SIMT fragment byte size overflows signed i64"); + } + + return FragmentShape{*elementByteSize, totalByteSize}; +} + +static FailureOr getStaticGEPIndex(LLVM::GEPOp gep) { + LLVM::GEPIndicesAdaptor indices = gep.getIndices(); + if (indices.size() != 1) { + gep.emitOpError( + "persistent SIMT fragment currently requires llvm.getelementptr " + "to have exactly one index"); + return failure(); + } + + auto index = indices[0]; + if (auto constant = llvm::dyn_cast_if_present(index)) + return getSignedInt64(constant.getValue(), gep, "GEP index"); + + Value dynamicIndex = llvm::dyn_cast_if_present(index); + if (!dynamicIndex) { + gep.emitOpError("failed to decode persistent fragment GEP index"); + return failure(); + } + return getConstantInt64(dynamicIndex, gep, "persistent fragment GEP index"); +} + +static FailureOr getStaticGEPByteOffset(LLVM::GEPOp gep, + const DataLayout &dataLayout) { + FailureOr index = getStaticGEPIndex(gep); + if (failed(index)) + return failure(); + + FailureOr elementByteSize = getFixedTypeByteSize( + gep.getElemType(), gep, "GEP element type", dataLayout); + if (failed(elementByteSize)) + return failure(); + + int64_t byteOffset; + if (llvm::MulOverflow(*index, *elementByteSize, byteOffset)) { + gep.emitOpError("persistent fragment GEP byte offset overflows signed i64"); + return failure(); + } + return byteOffset; +} + +// Reject vector memory semantics that cannot be preserved by lane splitting. +static LogicalResult validateScalarizableVectorAccess(Operation *access) { + bool isVolatile = false; + LLVM::AtomicOrdering ordering = LLVM::AtomicOrdering::not_atomic; + if (auto load = dyn_cast(access)) { + isVolatile = load.getVolatile_(); + ordering = load.getOrdering(); + } else { + auto store = cast(access); + isVolatile = store.getVolatile_(); + ordering = store.getOrdering(); + } + if (isVolatile || ordering != LLVM::AtomicOrdering::not_atomic) { + return access->emitOpError( + "persistent SIMT fragment vector access cannot be volatile or " + "atomic"); + } + return success(); +} + +// Return the number of scalar element accesses represented by a load/store. +static FailureOr +getPersistentAccessLaneCount(Operation *access, Type accessType, + Type allocationElementType) { + Type scalarType = accessType; + unsigned laneCount = 1; + if (auto vectorType = dyn_cast(accessType)) { + if (vectorType.getRank() != 1 || vectorType.isScalable()) { + access->emitOpError( + "persistent SIMT fragment vector access must be fixed-length and " + "one-dimensional"); + return failure(); + } + int64_t vectorLaneCount = vectorType.getDimSize(0); + if (vectorLaneCount != 2 && vectorLaneCount != 4) { + access->emitOpError() + << "persistent SIMT fragment currently supports only 2- or 4-lane " + "vector accesses, got " + << vectorLaneCount << " lanes"; + return failure(); + } + scalarType = vectorType.getElementType(); + laneCount = static_cast(vectorLaneCount); + if (failed(validateScalarizableVectorAccess(access))) + return failure(); + } + + if (!isSupportedPersistentScalarType(scalarType)) { + return access->emitOpError() + << "persistent SIMT fragment access requires an integer scalar " + "element type up to 64 bits or f16/bf16/f32, got " + << scalarType; + } + if (scalarType != allocationElementType) { + return access->emitOpError() + << "persistent SIMT fragment access element type " << scalarType + << " must match alloca element type " << allocationElementType; + } + return laneCount; +} + +static LogicalResult recordAccess(Operation *access, Type accessType, + int64_t byteOffset, func::FuncOp parentFunc, + LLVM::AllocaOp allocaOp, + const FragmentShape &shape, + PersistentAccessDiscovery &discovery) { + pto::SectionSimtOp section = access->getParentOfType(); + if (!section) { + return access->emitOpError( + "persistent SIMT fragment access must be inside pto.section.simt"); + } + if (section->getParentOfType() != parentFunc) { + return access->emitOpError( + "persistent SIMT fragment access must remain in its defining " + "function"); + } + if (!section.getBody().hasOneBlock() || + access->getBlock() != §ion.getBody().front()) { + return access->emitOpError( + "persistent SIMT fragment access must be directly inside the " + "single top-level block of the pto.section.simt body"); + } + + Type elementType = allocaOp.getElemType(); + FailureOr laneCount = + getPersistentAccessLaneCount(access, accessType, elementType); + if (failed(laneCount)) + return failure(); + if (byteOffset < 0 || byteOffset % shape.elementByteSize != 0) { + return access->emitOpError() + << "persistent SIMT fragment byte offset " << byteOffset + << " must be non-negative and aligned to element byte size " + << shape.elementByteSize; + } + + int64_t accessByteSize; + if (llvm::MulOverflow(static_cast(*laneCount), shape.elementByteSize, + accessByteSize)) { + return access->emitOpError( + "persistent SIMT fragment access byte size overflows signed i64"); + } + int64_t accessEnd; + if (llvm::AddOverflow(byteOffset, accessByteSize, accessEnd) || + accessEnd > shape.totalByteSize) { + if (*laneCount == 1) { + return access->emitOpError() + << "persistent SIMT fragment access at byte offset " << byteOffset + << " exceeds allocation size " << shape.totalByteSize; + } + return access->emitOpError() + << "persistent SIMT fragment access at byte offset " << byteOffset + << " with " << *laneCount << " scalar lane(s) exceeds allocation " + << "size " << shape.totalByteSize; + } + + if (discovery.accessIndices.count(access)) { + return access->emitOpError( + "persistent SIMT fragment access was visited more than once"); + } + + int64_t firstElementOffset = byteOffset / shape.elementByteSize; + SmallVector accessIndices; + accessIndices.reserve(*laneCount); + for (unsigned laneIndex = 0; laneIndex < *laneCount; ++laneIndex) { + int64_t elementOffset; + if (llvm::AddOverflow(firstElementOffset, static_cast(laneIndex), + elementOffset)) { + return access->emitOpError( + "persistent SIMT fragment lane element offset overflows signed " + "i64"); + } + accessIndices.push_back(static_cast(discovery.accesses.size())); + discovery.accesses.push_back({access, section, elementOffset, laneIndex}); + } + discovery.accessIndices.try_emplace(access, std::move(accessIndices)); + return success(); +} + +// Compare the lane topology of two inline SIMT sections. +static bool haveSameLaunchDimensions(pto::SectionSimtOp lhs, + pto::SectionSimtOp rhs) { + return lhs.getDimX() == rhs.getDimX() && lhs.getDimY() == rhs.getDimY() && + lhs.getDimZ() == rhs.getDimZ(); +} + +// Find the unique fragment access section that dominates every other access. +static FailureOr +findInitSection(DominanceInfo &dominance, + const PersistentMaterializationPlan &plan, + const PersistentAccessDiscovery &discovery, + PersistentFragmentAnalysis &fragment) { + llvm::DenseSet accessedSectionSet; + for (const NormalizedPersistentAccess &access : discovery.accesses) { + if (!access.section) { + return fragment.allocaOp.emitOpError( + "persistent fragment access has no containing SIMT section"); + } + pto::SectionSimtOp accessSection = access.section; + accessedSectionSet.insert(accessSection.getOperation()); + } + + // Iterate the canonical section list to preserve function walk order without + // storing a long-lived operation-to-index map in the analysis result. + SmallVector accessedSections; + for (pto::SectionSimtOp section : plan.sections) { + if (accessedSectionSet.contains(section.getOperation())) + accessedSections.push_back(section); + } + if (accessedSections.size() != accessedSectionSet.size()) { + return fragment.allocaOp.emitOpError( + "failed to locate every persistent fragment access section in the " + "materialization plan"); + } + + pto::SectionSimtOp initSection; + for (pto::SectionSimtOp candidate : accessedSections) { + bool dominatesAllAccessSections = + llvm::all_of(accessedSections, [&](pto::SectionSimtOp section) { + if (section == candidate) + return true; + return dominance.dominates(candidate.getOperation(), + section.getOperation()); + }); + if (!dominatesAllAccessSections) + continue; + + if (initSection) { + fragment.allocaOp.emitOpError( + "persistent SIMT fragment has multiple init section candidates " + "that dominate every access section"); + return failure(); + } + initSection = candidate; + } + + if (!initSection) { + fragment.allocaOp.emitOpError( + "persistent SIMT fragment requires a unique init section that " + "dominates every access section"); + return failure(); + } + return initSection; +} + +// Build the resident set from elements first stored by the init section. +static LogicalResult +collectResidentElements(pto::SectionSimtOp initSection, + const PersistentAccessDiscovery &discovery, + PersistentFragmentAnalysis &fragment, + llvm::DenseSet &residentElementSet) { + for (Operation &op : initSection.getBody().front()) { + auto accessIt = discovery.accessIndices.find(&op); + if (accessIt == discovery.accessIndices.end()) + continue; + + for (unsigned accessIndex : accessIt->second) { + const NormalizedPersistentAccess &access = + discovery.accesses[accessIndex]; + if (!residentElementSet.insert(access.elementOffset).second) + continue; + + if (!isa(access.op)) { + return access.op->emitOpError() + << "persistent SIMT fragment element offset " + << access.elementOffset + << " must be initialized by a store before its first load in " + "the init section"; + } + fragment.residentElements.push_back( + {access.elementOffset, 0, /*accesses=*/{}}); + } + } + + if (fragment.residentElements.empty()) { + return initSection.emitOpError( + "persistent SIMT fragment init section must initialize at least one " + "element"); + } + llvm::sort(fragment.residentElements, [](const ResidentElementPlan &lhs, + const ResidentElementPlan &rhs) { + return lhs.elementOffset < rhs.elementOffset; + }); + return success(); +} + +// Reject accesses that would extend the init-defined resident set. +static LogicalResult +validateResidentAccesses(const llvm::DenseSet &residentElementSet, + const PersistentAccessDiscovery &discovery) { + for (const NormalizedPersistentAccess &access : discovery.accesses) { + if (!residentElementSet.contains(access.elementOffset)) { + return access.op->emitOpError() + << "persistent SIMT fragment element offset " + << access.elementOffset + << " is not initialized by the init section and is not part of " + "residentElements"; + } + } + return success(); +} + +// Validate the launch topology of every section that carries the fragment. +static LogicalResult +validateCarrySectionDimensions(pto::SectionSimtOp initSection, + DominanceInfo &dominance, + const PersistentMaterializationPlan &plan, + PersistentFragmentAnalysis &fragment) { + fragment.carrySections.clear(); + for (pto::SectionSimtOp section : plan.sections) { + if (section == initSection) + continue; + // Only sections dominated by init can carry initialized fragment state. + if (!dominance.dominates(initSection.getOperation(), + section.getOperation())) + continue; + if (!haveSameLaunchDimensions(initSection, section)) { + return section.emitOpError() + << "persistent SIMT fragment carry section launch dimensions (" + << section.getDimX() << ", " << section.getDimY() << ", " + << section.getDimZ() << ") must match init section dimensions (" + << initSection.getDimX() << ", " << initSection.getDimY() << ", " + << initSection.getDimZ() << ")"; + } + fragment.carrySections.push_back(section); + } + + return success(); +} + +// Identify the init section and validate its fixed resident set and carries. +static LogicalResult +analyzeResidentElements(DominanceInfo &dominance, + const PersistentMaterializationPlan &plan, + PersistentFragmentAnalysis &fragment, + const PersistentAccessDiscovery &discovery) { + FailureOr initSection = + findInitSection(dominance, plan, discovery, fragment); + if (failed(initSection)) + return failure(); + fragment.initSection = *initSection; + + llvm::DenseSet residentElementSet; + if (failed(collectResidentElements(*initSection, discovery, fragment, + residentElementSet))) + return failure(); + if (failed(validateResidentAccesses(residentElementSet, discovery))) + return failure(); + + return validateCarrySectionDimensions(*initSection, dominance, plan, + fragment); +} + +static FailureOr getPersistentSlotWidth(Type type, Operation *anchor) { + if (auto intType = dyn_cast(type)) { + if (intType.getWidth() <= 32) + return 1; + if (intType.getWidth() <= 64) + return 2; + } else if (type.isF16() || type.isBF16() || type.isF32()) { + return 1; + } + + anchor->emitOpError() << "persistent SIMT fragment type " << type + << " has no keep/resume slot-width mapping"; + return failure(); +} + +static LogicalResult +allocatePersistentSlots(func::FuncOp func, + PersistentMaterializationPlan &plan) { + int64_t nextSlot = 0; + for (PersistentFragmentAnalysis &fragment : plan.fragments) { + Type elementType = fragment.allocaOp.getElemType(); + FailureOr slotWidth = + getPersistentSlotWidth(elementType, fragment.allocaOp); + if (failed(slotWidth)) + return failure(); + + for (ResidentElementPlan &residentElement : fragment.residentElements) { + int64_t elementOffset = residentElement.elementOffset; + int64_t slot = nextSlot; + if (*slotWidth == 2 && slot % 2 != 0) + ++slot; + + if (slot > kPersistentSlotLimit - *slotWidth) { + return fragment.allocaOp.emitOpError() + << "persistent SIMT fragment slot allocation in function '" + << func.getSymName() << "' exceeds the " << kPersistentSlotLimit + << "-slot limit: element offset " << elementOffset << " of type " + << elementType << " needs " << *slotWidth + << " slot(s), next candidate slot is " << slot; + } + + residentElement.slot = static_cast(slot); + nextSlot = slot + *slotWidth; + } + } + + return success(); +} + +// Publish normalized accesses under their owning resident element. Iterating +// sections and block operations gives the final plan a stable +// section/order/lane traversal independent of pointer-use walk order. +static LogicalResult +materializeResidentAccessLanes(const PersistentMaterializationPlan &plan, + PersistentFragmentAnalysis &fragment, + const PersistentAccessDiscovery &discovery) { + size_t materializedAccessCount = 0; + for (pto::SectionSimtOp section : plan.sections) { + if (!section.getBody().hasOneBlock()) { + return section.emitOpError( + "persistent fragment access materialization requires a " + "single-block SIMT section"); + } + for (Operation &op : section.getBody().front()) { + auto accessIt = discovery.accessIndices.find(&op); + if (accessIt == discovery.accessIndices.end()) + continue; + + for (unsigned accessIndex : accessIt->second) { + if (accessIndex >= discovery.accesses.size()) { + return fragment.allocaOp.emitOpError( + "persistent fragment discovery has an invalid access index"); + } + const NormalizedPersistentAccess &access = + discovery.accesses[accessIndex]; + if (access.op != &op || access.section != section) { + return access.op->emitOpError( + "persistent fragment normalized access does not match its " + "section operation"); + } + + ResidentElementPlan *residentElement = + fragment.findResidentElement(access.elementOffset); + if (!residentElement) { + return access.op->emitOpError() + << "persistent fragment normalized access element offset " + << access.elementOffset << " is not resident"; + } + residentElement->accesses.push_back({access.op, access.laneIndex}); + ++materializedAccessCount; + } + } + } + + if (materializedAccessCount != discovery.accesses.size()) { + return fragment.allocaOp.emitOpError( + "failed to materialize every normalized persistent fragment access " + "under a resident element"); + } + return success(); +} + +static LogicalResult +analyzePersistentFragment(LLVM::AllocaOp allocaOp, DominanceInfo &dominance, + const PersistentMaterializationPlan &plan, + PersistentFragmentAnalysis &fragment, + PersistentAccessDiscovery &discovery) { + func::FuncOp parentFunc = allocaOp->getParentOfType(); + if (!parentFunc) + return allocaOp.emitOpError("must be nested in a func.func"); + + if (!isa(allocaOp->getAttr(kPersistentAttrName))) { + return allocaOp.emitOpError() + << "expects '" << kPersistentAttrName << "' to be a unit attribute"; + } + if (allocaOp->getParentOfType()) { + return allocaOp.emitOpError( + "persistent SIMT fragment must be defined outside pto.section.simt"); + } + if (parentFunc->hasAttr(pto::kPTOSimtEntryAttrName)) { + return allocaOp.emitOpError( + "persistent SIMT fragment must be defined before SIMT outlining"); + } + + DataLayout dataLayout = DataLayout::closest(allocaOp); + FailureOr shape = getFragmentShape(allocaOp, dataLayout); + if (failed(shape)) + return failure(); + + SmallVector pointerWorklist{{allocaOp.getRes(), 0}}; + llvm::DenseMap pointerOffsets; + + while (!pointerWorklist.empty()) { + PointerWorkItem item = pointerWorklist.pop_back_val(); + auto [offsetIt, inserted] = + pointerOffsets.try_emplace(item.pointer, item.byteOffset); + if (!inserted) { + if (offsetIt->second != item.byteOffset) { + return allocaOp.emitOpError( + "persistent fragment pointer has inconsistent byte offsets"); + } + continue; + } + + for (OpOperand &use : item.pointer.getUses()) { + Operation *user = use.getOwner(); + if (user->getParentOfType() != parentFunc) { + return user->emitOpError( + "persistent SIMT fragment pointer must remain in its defining " + "function"); + } + if (!dominance.dominates(allocaOp.getOperation(), user)) { + return user->emitOpError( + "persistent SIMT fragment definition must dominate every use"); + } + + if (auto gep = dyn_cast(user)) { + if (use.getOperandNumber() != 0) { + return gep.emitOpError( + "persistent SIMT fragment pointer must be the base of " + "llvm.getelementptr"); + } + + FailureOr gepByteOffset = + getStaticGEPByteOffset(gep, dataLayout); + if (failed(gepByteOffset)) + return failure(); + int64_t derivedByteOffset; + if (llvm::AddOverflow(item.byteOffset, *gepByteOffset, + derivedByteOffset)) { + return gep.emitOpError( + "persistent fragment cumulative byte offset overflows signed " + "i64"); + } + pointerWorklist.push_back({gep.getRes(), derivedByteOffset}); + continue; + } + + if (auto load = dyn_cast(user)) { + if (failed(recordAccess(load, load.getRes().getType(), item.byteOffset, + parentFunc, allocaOp, *shape, discovery))) + return failure(); + continue; + } + + if (auto store = dyn_cast(user)) { + if (use.getOperandNumber() != 1) { + return store.emitOpError( + "persistent SIMT fragment pointer must not be stored as a " + "value"); + } + if (failed(recordAccess(store, store.getValue().getType(), + item.byteOffset, parentFunc, allocaOp, *shape, + discovery))) + return failure(); + continue; + } + + return user->emitOpError() + << "unsupported use of persistent SIMT fragment pointer by '" + << user->getName() << "'"; + } + } + + if (discovery.accesses.empty()) { + return allocaOp.emitOpError( + "persistent SIMT fragment requires at least one llvm.load or " + "llvm.store inside pto.section.simt"); + } + + return analyzeResidentElements(dominance, plan, fragment, discovery); +} + +} // namespace + +SIMTPersistentFragmentAnalysis::SIMTPersistentFragmentAnalysis( + func::FuncOp func) { + SmallVector persistentAllocas; + func.walk([&](LLVM::AllocaOp allocaOp) { + if (allocaOp->hasAttr(kPersistentAttrName)) + persistentAllocas.push_back(allocaOp); + }); + + // A function without persistent allocations has a valid empty plan and does + // not need to satisfy the PTO kernel-entry constraint. + if (persistentAllocas.empty()) { + plan.emplace(); + return; + } + + if (!pto::isPTOEntryFunction(func)) { + persistentAllocas.front().emitOpError( + "persistent SIMT fragment must be defined in a PTO kernel entry"); + return; + } + + PersistentMaterializationPlan candidate; + func.walk([&](pto::SectionSimtOp section) { + candidate.sections.push_back(section); + }); + + candidate.fragments.reserve(persistentAllocas.size()); + SmallVector accessDiscoveries; + accessDiscoveries.reserve(persistentAllocas.size()); + DominanceInfo dominance(func); + for (LLVM::AllocaOp allocaOp : persistentAllocas) { + candidate.fragments.emplace_back(allocaOp); + accessDiscoveries.emplace_back(); + if (failed(analyzePersistentFragment(allocaOp, dominance, candidate, + candidate.fragments.back(), + accessDiscoveries.back()))) + return; + } + + if (failed(allocatePersistentSlots(func, candidate))) + return; + for (unsigned fragmentIndex = 0; + fragmentIndex < static_cast(candidate.fragments.size()); + ++fragmentIndex) { + if (failed(materializeResidentAccessLanes( + candidate, candidate.fragments[fragmentIndex], + accessDiscoveries[fragmentIndex]))) + return; + } + // Publish only a fully checked plan. No analysis result is visible to the + // materialization pass after an intermediate failure. + plan.emplace(std::move(candidate)); +} + +} // namespace pto +} // namespace mlir diff --git a/lib/PTO/Transforms/SIMTPersistentFragmentAnalysis.h b/lib/PTO/Transforms/SIMTPersistentFragmentAnalysis.h new file mode 100644 index 0000000000..a205f3af55 --- /dev/null +++ b/lib/PTO/Transforms/SIMTPersistentFragmentAnalysis.h @@ -0,0 +1,89 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +#ifndef PTO_LIB_PTO_TRANSFORMS_SIMTPERSISTENTFRAGMENTANALYSIS_H +#define PTO_LIB_PTO_TRANSFORMS_SIMTPERSISTENTFRAGMENTANALYSIS_H + +#include "PTO/IR/PTO.h" + +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/Dialect/LLVMIR/LLVMDialect.h" +#include "llvm/ADT/SmallVector.h" + +#include +#include +#include + +namespace mlir { +namespace pto { + +struct AccessLane { + Operation *op; + // Index of this scalar component in the original vector operation. Scalar + // accesses use lane 0. The owning ResidentElementPlan supplies the element + // offset. + unsigned laneIndex; +}; + +struct ResidentElementPlan { + int64_t elementOffset; + // Assigned by function-wide slot allocation before the plan is published. + unsigned slot = 0; + SmallVector accesses; +}; + +struct PersistentFragmentAnalysis { + explicit PersistentFragmentAnalysis(LLVM::AllocaOp allocaOp) + : allocaOp(allocaOp) {} + + LLVM::AllocaOp allocaOp; + // The unique section that initializes all resident elements. + SectionSimtOp initSection; + // All sections dominated by initSection that must carry the complete + // resident set. The sections are kept in function walk order and do not + // include initSection. + SmallVector carrySections; + // Elements initialized by the init section, sorted by element offset. + SmallVector residentElements; + + ResidentElementPlan *findResidentElement(int64_t elementOffset) { + for (ResidentElementPlan &element : residentElements) { + if (element.elementOffset == elementOffset) + return &element; + } + return nullptr; + } +}; + +struct PersistentMaterializationPlan { + // All inline SIMT sections in function walk order. + SmallVector sections; + // Persistent fragments in alloca walk order. + SmallVector fragments; +}; + +/// Cached read-only analysis consumed by persistent fragment materialization. +class SIMTPersistentFragmentAnalysis { +public: + explicit SIMTPersistentFragmentAnalysis(func::FuncOp func); + + bool isValid() const { return plan.has_value(); } + + const PersistentMaterializationPlan &getPlan() const { + assert(plan && "expected valid persistent fragment analysis"); + return *plan; + } + +private: + std::optional plan; +}; + +} // namespace pto +} // namespace mlir + +#endif // PTO_LIB_PTO_TRANSFORMS_SIMTPERSISTENTFRAGMENTANALYSIS_H diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_analysis.pto b/test/lit/vpto/materialize_simt_persistent_fragment_analysis.pto new file mode 100644 index 0000000000..f13353b022 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_analysis.pto @@ -0,0 +1,73 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-analyze-simt-persistent-fragment %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=ANALYZE +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-materialize-simt-persistent-fragment %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=MATERIALIZE + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_analysis(%dst: !pto.ptr) attributes {pto.entry} { + %c3_i32 = arith.constant 3 : i32 + %fragment = llvm.alloca %c3_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %zero = arith.constant 0.000000e+00 : f32 + %one = arith.constant 1.000000e+00 : f32 + %two = arith.constant 2.000000e+00 : f32 + llvm.store %zero, %fragment : f32, !llvm.ptr + %c4_i64 = arith.constant 4 : i64 + %element1 = llvm.getelementptr %fragment[%c4_i64] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %one, %element1 : f32, !llvm.ptr + %element2 = llvm.getelementptr %element1[%c4_i64] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %two, %element2 : f32, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %value0 = llvm.load %fragment : !llvm.ptr -> f32 + %c4_i64 = arith.constant 4 : i64 + %element1 = llvm.getelementptr %fragment[%c4_i64] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %value1 = llvm.load %element1 : !llvm.ptr -> f32 + %element2 = llvm.getelementptr %element1[%c4_i64] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %value2 = llvm.load %element2 : !llvm.ptr -> f32 + %value01 = arith.addf %value0, %value1 : f32 + %value = arith.addf %value01, %value2 : f32 + %c0 = arith.constant 0 : index + pto.store %value, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// ANALYZE-LABEL: func.func @persistent_fragment_analysis +// ANALYZE: %[[FRAGMENT:.*]] = llvm.alloca {{.*}} x f32 {pto.persistent} +// ANALYZE: pto.section.simt<<<32, 1, 1>>> { +// ANALYZE: llvm.store {{.*}}, %[[FRAGMENT]] : f32, !llvm.ptr +// ANALYZE: pto.section.simt<<<32, 1, 1>>> { +// ANALYZE: llvm.load %[[FRAGMENT]] : !llvm.ptr -> f32 +// ANALYZE-NOT: pto.resume +// ANALYZE-NOT: pto.keep + +// MATERIALIZE-LABEL: func.func @persistent_fragment_analysis +// MATERIALIZE-NOT: llvm.alloca +// MATERIALIZE: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32 +// MATERIALIZE: %[[ONE:.*]] = arith.constant 1.000000e+00 : f32 +// MATERIALIZE: %[[TWO:.*]] = arith.constant 2.000000e+00 : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: pto.keep %[[ZERO]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[ONE]] {slot = 1 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[TWO]] {slot = 2 : i64} : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: %[[RESUME0:.*]] = pto.resume {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: %[[RESUME1:.*]] = pto.resume {slot = 1 : i64} : f32 +// MATERIALIZE-NEXT: %[[RESUME2:.*]] = pto.resume {slot = 2 : i64} : f32 +// MATERIALIZE: %[[SUM01:.*]] = arith.addf %[[RESUME0]], %[[RESUME1]] : f32 +// MATERIALIZE: %[[SUM:.*]] = arith.addf %[[SUM01]], %[[RESUME2]] : f32 +// MATERIALIZE: pto.store %[[SUM]], +// MATERIALIZE: pto.keep %[[RESUME0]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[RESUME1]] {slot = 1 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[RESUME2]] {slot = 2 : i64} : f32 +// MATERIALIZE-NOT: llvm.getelementptr +// MATERIALIZE-NOT: llvm.load +// MATERIALIZE-NOT: llvm.store diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_consumer_subset.pto b/test/lit/vpto/materialize_simt_persistent_fragment_consumer_subset.pto new file mode 100644 index 0000000000..6efcaa7084 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_consumer_subset.pto @@ -0,0 +1,56 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-materialize-simt-persistent-fragment %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=MATERIALIZE + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_consumer_subset(%dst: !pto.ptr) attributes {pto.entry} { + %c3_i32 = arith.constant 3 : i32 + %fragment = llvm.alloca %c3_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %zero = arith.constant 0.000000e+00 : f32 + %one = arith.constant 1.000000e+00 : f32 + %two = arith.constant 2.000000e+00 : f32 + llvm.store %zero, %fragment : f32, !llvm.ptr + %c4_i64 = arith.constant 4 : i64 + %element1 = llvm.getelementptr %fragment[%c4_i64] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %one, %element1 : f32, !llvm.ptr + %element2 = llvm.getelementptr %element1[%c4_i64] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %two, %element2 : f32, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %c4_i64 = arith.constant 4 : i64 + %element1 = llvm.getelementptr %fragment[%c4_i64] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %value1 = llvm.load %element1 : !llvm.ptr -> f32 + %c0 = arith.constant 0 : index + pto.store %value1, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// MATERIALIZE-LABEL: func.func @persistent_fragment_consumer_subset +// MATERIALIZE-NOT: llvm.alloca +// MATERIALIZE: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32 +// MATERIALIZE: %[[ONE:.*]] = arith.constant 1.000000e+00 : f32 +// MATERIALIZE: %[[TWO:.*]] = arith.constant 2.000000e+00 : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: pto.keep %[[ZERO]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[ONE]] {slot = 1 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[TWO]] {slot = 2 : i64} : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: %[[RESUME0:.*]] = pto.resume {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: %[[RESUME1:.*]] = pto.resume {slot = 1 : i64} : f32 +// MATERIALIZE-NEXT: %[[RESUME2:.*]] = pto.resume {slot = 2 : i64} : f32 +// MATERIALIZE: pto.store %[[RESUME1]], +// MATERIALIZE: pto.keep %[[RESUME0]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[RESUME1]] {slot = 1 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[RESUME2]] {slot = 2 : i64} : f32 +// MATERIALIZE-NOT: llvm.getelementptr +// MATERIALIZE-NOT: llvm.load +// MATERIALIZE-NOT: llvm.store diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_dynamic_offset_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_dynamic_offset_invalid.pto new file mode 100644 index 0000000000..a65e98d389 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_dynamic_offset_invalid.pto @@ -0,0 +1,25 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_dynamic_offset(%index: i64, %dst: !pto.ptr) attributes {pto.entry} { + %c4_i32 = arith.constant 4 : i32 + %fragment = llvm.alloca %c4_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %element = llvm.getelementptr %fragment[%index] : (!llvm.ptr, i64) -> !llvm.ptr, f32 + %value = llvm.load %element : !llvm.ptr -> f32 + %c0 = arith.constant 0 : index + pto.store %value, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// CHECK: error: 'llvm.getelementptr' op persistent fragment GEP index must be a compile-time integer constant diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_escape_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_escape_invalid.pto new file mode 100644 index 0000000000..9356aa70ca --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_escape_invalid.pto @@ -0,0 +1,24 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_escape(%dst: !pto.ptr) attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %address = llvm.ptrtoint %fragment : !llvm.ptr to i64 + %c0 = arith.constant 0 : index + pto.store %address, %dst[%c0] : !pto.ptr, i64 + } + return + } +} + +// CHECK: error: 'llvm.ptrtoint' op unsupported use of persistent SIMT fragment pointer by 'llvm.ptrtoint' diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_i64_alignment_budget_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_i64_alignment_budget_invalid.pto new file mode 100644 index 0000000000..f5c12c0e89 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_i64_alignment_budget_invalid.pto @@ -0,0 +1,46 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_i64_alignment_budget_invalid(%dst: !pto.ptr) attributes {pto.entry} { + %c121_i32 = arith.constant 121 : i32 + %c1_i32 = arith.constant 1 : i32 + %fragment = llvm.alloca %c121_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + %wide_fragment = llvm.alloca %c1_i32 x i64 {pto.persistent} : (i32) -> !llvm.ptr + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c121 = arith.constant 121 : index + %zero = arith.constant 0.000000e+00 : f32 + %zero_i64 = arith.constant 0 : i64 + + pto.section.simt<<<32, 1, 1>>> { + "scf.for"(%c0, %c121, %c1) ({ + ^bb0(%i: index): + %offset = arith.index_cast %i : index to i64 + %element = llvm.getelementptr %fragment[%offset] : (!llvm.ptr, i64) -> !llvm.ptr, f32 + llvm.store %zero, %element : f32, !llvm.ptr + scf.yield + }) {pto.unroll = "full"} : (index, index, index) -> () + llvm.store %zero_i64, %wide_fragment : i64, !llvm.ptr + } + + pto.section.simt<<<32, 1, 1>>> { + %value = llvm.load %fragment : !llvm.ptr -> f32 + %wide = llvm.load %wide_fragment : !llvm.ptr -> i64 + %wide_i32 = arith.trunci %wide : i64 to i32 + %wide_f32 = arith.sitofp %wide_i32 : i32 to f32 + %result = arith.addf %value, %wide_f32 : f32 + pto.store %result, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// CHECK: error: 'llvm.alloca' op persistent SIMT fragment slot allocation in function 'persistent_fragment_i64_alignment_budget_invalid' exceeds the 123-slot limit: element offset 0 of type 'i64' needs 2 slot(s), next candidate slot is 122 diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_init_load_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_init_load_invalid.pto new file mode 100644 index 0000000000..9353cfa935 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_init_load_invalid.pto @@ -0,0 +1,24 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_init_load(%dst: !pto.ptr) attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %value = llvm.load %fragment : !llvm.ptr -> f32 + %c0 = arith.constant 0 : index + pto.store %value, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// CHECK: error: 'llvm.load' op persistent SIMT fragment element offset 0 must be initialized by a store before its first load in the init section diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_launch_dimensions_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_launch_dimensions_invalid.pto new file mode 100644 index 0000000000..ed6d43d8be --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_launch_dimensions_invalid.pto @@ -0,0 +1,28 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_launch_dimensions(%dst: !pto.ptr) attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %zero = arith.constant 0.000000e+00 : f32 + llvm.store %zero, %fragment : f32, !llvm.ptr + } + pto.section.simt<<<64, 1, 1>>> { + %one = arith.constant 1.000000e+00 : f32 + %c0 = arith.constant 0 : index + pto.store %one, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// CHECK: error: 'pto.section.simt' op persistent SIMT fragment carry section launch dimensions (64, 1, 1) must match init section dimensions (32, 1, 1) diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_nested_access_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_nested_access_invalid.pto new file mode 100644 index 0000000000..11083d6613 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_nested_access_invalid.pto @@ -0,0 +1,30 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_nested_access_invalid(%condition: i1, %dst: !pto.ptr) attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %one = arith.constant 1.000000e+00 : f32 + llvm.store %one, %fragment : f32, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + scf.if %condition { + %value = llvm.load %fragment : !llvm.ptr -> f32 + %c0 = arith.constant 0 : index + pto.store %value, %dst[%c0] : !pto.ptr, f32 + } + } + return + } +} + +// CHECK: error: 'llvm.load' op persistent SIMT fragment access must be directly inside the single top-level block of the pto.section.simt body diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_nested_init.pto b/test/lit/vpto/materialize_simt_persistent_fragment_nested_init.pto new file mode 100644 index 0000000000..c6b12ec6b9 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_nested_init.pto @@ -0,0 +1,30 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_nested_init(%condition: i32, %dst: !pto.ptr) attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %c0_i32 = arith.constant 0 : i32 + %condition_i1 = arith.cmpi ne, %condition, %c0_i32 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + scf.if %condition_i1 { + pto.section.simt<<<32, 1, 1>>> { + %one = arith.constant 1.000000e+00 : f32 + llvm.store %one, %fragment : f32, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %value = llvm.load %fragment : !llvm.ptr -> f32 + %c0 = arith.constant 0 : index + pto.store %value, %dst[%c0] : !pto.ptr, f32 + } + } + return + } +} diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_no_dominating_init_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_no_dominating_init_invalid.pto new file mode 100644 index 0000000000..094c4f105f --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_no_dominating_init_invalid.pto @@ -0,0 +1,32 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_no_dominating_init(%condition: i32) attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %c0_i32 = arith.constant 0 : i32 + %condition_i1 = arith.cmpi ne, %condition, %c0_i32 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + scf.if %condition_i1 { + pto.section.simt<<<32, 1, 1>>> { + %one = arith.constant 1.000000e+00 : f32 + llvm.store %one, %fragment : f32, !llvm.ptr + } + } else { + pto.section.simt<<<32, 1, 1>>> { + %two = arith.constant 2.000000e+00 : f32 + llvm.store %two, %fragment : f32, !llvm.ptr + } + } + return + } +} + +// CHECK: error: 'llvm.alloca' op persistent SIMT fragment requires a unique init section that dominates every access section diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_non_kernel_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_non_kernel_invalid.pto new file mode 100644 index 0000000000..71b2ab61ff --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_non_kernel_invalid.pto @@ -0,0 +1,27 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @kernel() attributes {pto.entry} { + return + } + + func.func private @helper() { + %c1_i32 = arith.constant 1 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %zero = arith.constant 0.000000e+00 : f32 + llvm.store %zero, %fragment : f32, !llvm.ptr + } + return + } +} + +// CHECK: error: 'llvm.alloca' op persistent SIMT fragment must be defined in a PTO kernel entry diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_non_resident_access_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_non_resident_access_invalid.pto new file mode 100644 index 0000000000..08aa5cc912 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_non_resident_access_invalid.pto @@ -0,0 +1,30 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_non_resident_access(%dst: !pto.ptr) attributes {pto.entry} { + %c2_i32 = arith.constant 2 : i32 + %fragment = llvm.alloca %c2_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + %c1_i64 = arith.constant 1 : i64 + %element1 = llvm.getelementptr %fragment[%c1_i64] : (!llvm.ptr, i64) -> !llvm.ptr, f32 + pto.section.simt<<<32, 1, 1>>> { + %zero = arith.constant 0.000000e+00 : f32 + llvm.store %zero, %fragment : f32, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %value = llvm.load %element1 : !llvm.ptr -> f32 + %c0 = arith.constant 0 : index + pto.store %value, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// CHECK: error: 'llvm.load' op persistent SIMT fragment element offset 1 is not initialized by the init section and is not part of residentElements diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_out_of_bounds_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_out_of_bounds_invalid.pto new file mode 100644 index 0000000000..2f144b4563 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_out_of_bounds_invalid.pto @@ -0,0 +1,26 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_out_of_bounds(%dst: !pto.ptr) attributes {pto.entry} { + %c2_i32 = arith.constant 2 : i32 + %fragment = llvm.alloca %c2_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %c2_i64 = arith.constant 2 : i64 + %element = llvm.getelementptr %fragment[%c2_i64] : (!llvm.ptr, i64) -> !llvm.ptr, f32 + %value = llvm.load %element : !llvm.ptr -> f32 + %c0 = arith.constant 0 : index + pto.store %value, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// CHECK: error: 'llvm.load' op persistent SIMT fragment access at byte offset 8 exceeds allocation size 8 diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_outside_access_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_outside_access_invalid.pto new file mode 100644 index 0000000000..a95635c667 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_outside_access_invalid.pto @@ -0,0 +1,21 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_outside_access() attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + %one = arith.constant 1.000000e+00 : f32 + llvm.store %one, %fragment : f32, !llvm.ptr + return + } +} + +// CHECK: error: 'llvm.store' op persistent SIMT fragment access must be inside pto.section.simt diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_read_modify_write.pto b/test/lit/vpto/materialize_simt_persistent_fragment_read_modify_write.pto new file mode 100644 index 0000000000..8a789f0ec3 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_read_modify_write.pto @@ -0,0 +1,44 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-materialize-simt-persistent-fragment %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=MATERIALIZE + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_read_modify_write(%dst: !pto.ptr) attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %one = arith.constant 1.000000e+00 : f32 + llvm.store %one, %fragment : f32, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %old = llvm.load %fragment : !llvm.ptr -> f32 + %two = arith.constant 2.000000e+00 : f32 + %updated = arith.addf %old, %two : f32 + llvm.store %updated, %fragment : f32, !llvm.ptr + %c0 = arith.constant 0 : index + pto.store %updated, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// MATERIALIZE-LABEL: func.func @persistent_fragment_read_modify_write +// MATERIALIZE-NOT: llvm.alloca +// MATERIALIZE: %[[ONE:.*]] = arith.constant 1.000000e+00 : f32 +// MATERIALIZE: %[[TWO:.*]] = arith.constant 2.000000e+00 : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: pto.keep %[[ONE]] {slot = 0 : i64} : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: %[[OLD:.*]] = pto.resume {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: %[[UPDATED:.*]] = arith.addf %[[OLD]], %[[TWO]] : f32 +// MATERIALIZE: pto.store %[[UPDATED]], +// MATERIALIZE: pto.keep %[[UPDATED]] {slot = 0 : i64} : f32 +// MATERIALIZE-NOT: llvm.getelementptr +// MATERIALIZE-NOT: llvm.load +// MATERIALIZE-NOT: llvm.store diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_scalar_vector_overlap.pto b/test/lit/vpto/materialize_simt_persistent_fragment_scalar_vector_overlap.pto new file mode 100644 index 0000000000..bbab8a3aa3 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_scalar_vector_overlap.pto @@ -0,0 +1,52 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-materialize-simt-persistent-fragment %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=MATERIALIZE + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_scalar_vector_overlap(%dst: !pto.ptr) attributes {pto.entry} { + %c2_i32 = arith.constant 2 : i32 + %fragment = llvm.alloca %c2_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %value = arith.constant dense<[1.000000e+00, 2.000000e+00]> : vector<2xf32> + llvm.store %value, %fragment : vector<2xf32>, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %c4_i64 = arith.constant 4 : i64 + %element1 = llvm.getelementptr %fragment[%c4_i64] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %scalar1 = llvm.load %element1 : !llvm.ptr -> f32 + %vector = llvm.load %fragment : !llvm.ptr -> vector<2xf32> + %c1_i32 = arith.constant 1 : i32 + %vector1 = llvm.extractelement %vector[%c1_i32 : i32] : vector<2xf32> + %sum = arith.addf %scalar1, %vector1 : f32 + %c0 = arith.constant 0 : index + pto.store %sum, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// MATERIALIZE-LABEL: func.func @persistent_fragment_scalar_vector_overlap +// MATERIALIZE-NOT: llvm.alloca +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE: pto.keep %{{.*}} {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %{{.*}} {slot = 1 : i64} : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: %[[RESUME0:.*]] = pto.resume {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: %[[RESUME1:.*]] = pto.resume {slot = 1 : i64} : f32 +// MATERIALIZE: %[[POISON:.*]] = llvm.mlir.poison : vector<2xf32> +// MATERIALIZE: %[[VECTOR0:.*]] = llvm.insertelement %[[RESUME0]], %[[POISON]][%{{.*}} : i32] : vector<2xf32> +// MATERIALIZE: %[[VECTOR1:.*]] = llvm.insertelement %[[RESUME1]], %[[VECTOR0]][%{{.*}} : i32] : vector<2xf32> +// MATERIALIZE: %[[LANE1:.*]] = llvm.extractelement %[[VECTOR1]][%{{.*}} : i32] : vector<2xf32> +// MATERIALIZE: %[[SUM:.*]] = arith.addf %[[RESUME1]], %[[LANE1]] : f32 +// MATERIALIZE: pto.store %[[SUM]], +// MATERIALIZE: pto.keep %[[RESUME0]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[RESUME1]] {slot = 1 : i64} : f32 +// MATERIALIZE-NOT: llvm.getelementptr +// MATERIALIZE-NOT: llvm.load +// MATERIALIZE-NOT: llvm.store diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_section_plan.pto b/test/lit/vpto/materialize_simt_persistent_fragment_section_plan.pto new file mode 100644 index 0000000000..724def5138 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_section_plan.pto @@ -0,0 +1,54 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-materialize-simt-persistent-fragment %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=MATERIALIZE + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_section_plan(%dst: !pto.ptr) attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %fragment0 = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + %fragment1 = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %one = arith.constant 1.000000e+00 : f32 + llvm.store %one, %fragment0 : f32, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %two = arith.constant 2.000000e+00 : f32 + llvm.store %two, %fragment1 : f32, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %value0 = llvm.load %fragment0 : !llvm.ptr -> f32 + %value1 = llvm.load %fragment1 : !llvm.ptr -> f32 + %result = arith.addf %value0, %value1 : f32 + %c0 = arith.constant 0 : index + pto.store %result, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// MATERIALIZE-LABEL: func.func @persistent_fragment_section_plan +// MATERIALIZE-NOT: llvm.alloca +// MATERIALIZE: %[[ONE:.*]] = arith.constant 1.000000e+00 : f32 +// MATERIALIZE: %[[TWO:.*]] = arith.constant 2.000000e+00 : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: pto.keep %[[ONE]] {slot = 0 : i64} : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: %[[CARRY0:.*]] = pto.resume {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[CARRY0]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[TWO]] {slot = 1 : i64} : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: %[[RESUME0:.*]] = pto.resume {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: %[[RESUME1:.*]] = pto.resume {slot = 1 : i64} : f32 +// MATERIALIZE: %[[RESULT:.*]] = arith.addf %[[RESUME0]], %[[RESUME1]] : f32 +// MATERIALIZE: pto.store %[[RESULT]], +// MATERIALIZE: pto.keep %[[RESUME0]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[RESUME1]] {slot = 1 : i64} : f32 +// MATERIALIZE-NOT: llvm.getelementptr +// MATERIALIZE-NOT: llvm.load +// MATERIALIZE-NOT: llvm.store diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_slot_budget_boundary.pto b/test/lit/vpto/materialize_simt_persistent_fragment_slot_budget_boundary.pto new file mode 100644 index 0000000000..77e968d710 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_slot_budget_boundary.pto @@ -0,0 +1,36 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_slot_budget_boundary(%dst: !pto.ptr) attributes {pto.entry} { + %c124_i32 = arith.constant 124 : i32 + %fragment = llvm.alloca %c124_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c123 = arith.constant 123 : index + %zero = arith.constant 0.000000e+00 : f32 + + pto.section.simt<<<32, 1, 1>>> { + "scf.for"(%c0, %c123, %c1) ({ + ^bb0(%i: index): + %offset = arith.index_cast %i : index to i64 + %element = llvm.getelementptr %fragment[%offset] : (!llvm.ptr, i64) -> !llvm.ptr, f32 + llvm.store %zero, %element : f32, !llvm.ptr + scf.yield + }) {pto.unroll = "full"} : (index, index, index) -> () + } + + pto.section.simt<<<32, 1, 1>>> { + %value = llvm.load %fragment : !llvm.ptr -> f32 + pto.store %value, %dst[%c0] : !pto.ptr, f32 + } + return + } +} diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_slot_budget_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_slot_budget_invalid.pto new file mode 100644 index 0000000000..26cf80f699 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_slot_budget_invalid.pto @@ -0,0 +1,38 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_slot_budget_invalid(%dst: !pto.ptr) attributes {pto.entry} { + %c124_i32 = arith.constant 124 : i32 + %fragment = llvm.alloca %c124_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c124 = arith.constant 124 : index + %zero = arith.constant 0.000000e+00 : f32 + + pto.section.simt<<<32, 1, 1>>> { + "scf.for"(%c0, %c124, %c1) ({ + ^bb0(%i: index): + %offset = arith.index_cast %i : index to i64 + %element = llvm.getelementptr %fragment[%offset] : (!llvm.ptr, i64) -> !llvm.ptr, f32 + llvm.store %zero, %element : f32, !llvm.ptr + scf.yield + }) {pto.unroll = "full"} : (index, index, index) -> () + } + + pto.section.simt<<<32, 1, 1>>> { + %value = llvm.load %fragment : !llvm.ptr -> f32 + pto.store %value, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// CHECK: error: 'llvm.alloca' op persistent SIMT fragment slot allocation in function 'persistent_fragment_slot_budget_invalid' exceeds the 123-slot limit: element offset 123 of type 'f32' needs 1 slot(s), next candidate slot is 123 diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_transparent_scf.pto b/test/lit/vpto/materialize_simt_persistent_fragment_transparent_scf.pto new file mode 100644 index 0000000000..f1b50b9211 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_transparent_scf.pto @@ -0,0 +1,60 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-materialize-simt-persistent-fragment %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=MATERIALIZE + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_transparent_scf(%condition: i1, %dst: !pto.ptr) attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %one = arith.constant 1.000000e+00 : f32 + scf.if %condition { + %before = arith.constant 2.000000e+00 : f32 + %c0 = arith.constant 0 : index + pto.store %before, %dst[%c0] : !pto.ptr, f32 + } + llvm.store %one, %fragment : f32, !llvm.ptr + scf.if %condition { + %after = arith.constant 3.000000e+00 : f32 + %c1 = arith.constant 1 : index + pto.store %after, %dst[%c1] : !pto.ptr, f32 + } + } + pto.section.simt<<<32, 1, 1>>> { + scf.if %condition { + %before = arith.constant 4.000000e+00 : f32 + %c2 = arith.constant 2 : index + pto.store %before, %dst[%c2] : !pto.ptr, f32 + } + %value = llvm.load %fragment : !llvm.ptr -> f32 + scf.if %condition { + %c3 = arith.constant 3 : index + pto.store %value, %dst[%c3] : !pto.ptr, f32 + } + } + return + } +} + +// MATERIALIZE-LABEL: func.func @persistent_fragment_transparent_scf( +// MATERIALIZE-SAME: %[[CONDITION:.*]]: i1, +// MATERIALIZE-NOT: llvm.alloca +// MATERIALIZE: %[[ONE:.*]] = arith.constant 1.000000e+00 : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE: scf.if %[[CONDITION]] +// MATERIALIZE: scf.if %[[CONDITION]] +// MATERIALIZE: pto.keep %[[ONE]] {slot = 0 : i64} : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: %[[RESUME:.*]] = pto.resume {slot = 0 : i64} : f32 +// MATERIALIZE: scf.if %[[CONDITION]] +// MATERIALIZE: scf.if %[[CONDITION]] +// MATERIALIZE: pto.store %[[RESUME]], +// MATERIALIZE: pto.keep %[[RESUME]] {slot = 0 : i64} : f32 +// MATERIALIZE-NOT: llvm.load +// MATERIALIZE-NOT: llvm.store diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_unaligned_offset_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_unaligned_offset_invalid.pto new file mode 100644 index 0000000000..9554c8ceab --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_unaligned_offset_invalid.pto @@ -0,0 +1,26 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_unaligned_offset(%dst: !pto.ptr) attributes {pto.entry} { + %c2_i32 = arith.constant 2 : i32 + %fragment = llvm.alloca %c2_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %c2_i64 = arith.constant 2 : i64 + %element = llvm.getelementptr %fragment[%c2_i64] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %value = llvm.load %element : !llvm.ptr -> f32 + %c0 = arith.constant 0 : index + pto.store %value, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// CHECK: error: 'llvm.load' op persistent SIMT fragment byte offset 2 must be non-negative and aligned to element byte size 4 diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_vector4_access.pto b/test/lit/vpto/materialize_simt_persistent_fragment_vector4_access.pto new file mode 100644 index 0000000000..81716223c8 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_vector4_access.pto @@ -0,0 +1,58 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-materialize-simt-persistent-fragment %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=MATERIALIZE + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_vector4_access(%dst: !pto.ptr) attributes {pto.entry} { + %c4_i32 = arith.constant 4 : i32 + %fragment = llvm.alloca %c4_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %value = arith.constant dense<[1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00]> : vector<4xf32> + llvm.store %value, %fragment : vector<4xf32>, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %value = llvm.load %fragment : !llvm.ptr -> vector<4xf32> + %c3_i32 = arith.constant 3 : i32 + %lane3 = llvm.extractelement %value[%c3_i32 : i32] : vector<4xf32> + %c0 = arith.constant 0 : index + pto.store %lane3, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// MATERIALIZE-LABEL: func.func @persistent_fragment_vector4_access +// MATERIALIZE-NOT: llvm.alloca +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE: %[[INIT0:.*]] = llvm.extractelement %{{.*}}[%{{.*}} : i32] : vector<4xf32> +// MATERIALIZE: %[[INIT1:.*]] = llvm.extractelement %{{.*}}[%{{.*}} : i32] : vector<4xf32> +// MATERIALIZE: %[[INIT2:.*]] = llvm.extractelement %{{.*}}[%{{.*}} : i32] : vector<4xf32> +// MATERIALIZE: %[[INIT3:.*]] = llvm.extractelement %{{.*}}[%{{.*}} : i32] : vector<4xf32> +// MATERIALIZE: pto.keep %[[INIT0]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[INIT1]] {slot = 1 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[INIT2]] {slot = 2 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[INIT3]] {slot = 3 : i64} : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: %[[RESUME0:.*]] = pto.resume {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: %[[RESUME1:.*]] = pto.resume {slot = 1 : i64} : f32 +// MATERIALIZE-NEXT: %[[RESUME2:.*]] = pto.resume {slot = 2 : i64} : f32 +// MATERIALIZE-NEXT: %[[RESUME3:.*]] = pto.resume {slot = 3 : i64} : f32 +// MATERIALIZE: %[[POISON:.*]] = llvm.mlir.poison : vector<4xf32> +// MATERIALIZE: %[[VECTOR0:.*]] = llvm.insertelement %[[RESUME0]], %[[POISON]][%{{.*}} : i32] : vector<4xf32> +// MATERIALIZE: %[[VECTOR1:.*]] = llvm.insertelement %[[RESUME1]], %[[VECTOR0]][%{{.*}} : i32] : vector<4xf32> +// MATERIALIZE: %[[VECTOR2:.*]] = llvm.insertelement %[[RESUME2]], %[[VECTOR1]][%{{.*}} : i32] : vector<4xf32> +// MATERIALIZE: %[[VECTOR3:.*]] = llvm.insertelement %[[RESUME3]], %[[VECTOR2]][%{{.*}} : i32] : vector<4xf32> +// MATERIALIZE: %[[LANE3:.*]] = llvm.extractelement %[[VECTOR3]][%{{.*}} : i32] : vector<4xf32> +// MATERIALIZE: pto.store %[[LANE3]], +// MATERIALIZE: pto.keep %[[RESUME0]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[RESUME1]] {slot = 1 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[RESUME2]] {slot = 2 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[RESUME3]] {slot = 3 : i64} : f32 +// MATERIALIZE-NOT: llvm.load +// MATERIALIZE-NOT: llvm.store diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_vector_access.pto b/test/lit/vpto/materialize_simt_persistent_fragment_vector_access.pto new file mode 100644 index 0000000000..9f9b2388f3 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_vector_access.pto @@ -0,0 +1,55 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-materialize-simt-persistent-fragment %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=MATERIALIZE + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_vector_access(%dst: !pto.ptr) attributes {pto.entry} { + %c2_i32 = arith.constant 2 : i32 + %fragment = llvm.alloca %c2_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %value = arith.constant dense<[1.000000e+00, 2.000000e+00]> : vector<2xf32> + llvm.store %value, %fragment : vector<2xf32>, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %value = llvm.load %fragment : !llvm.ptr -> vector<2xf32> + %c0_i32 = arith.constant 0 : i32 + %c1_i32 = arith.constant 1 : i32 + %lane0 = llvm.extractelement %value[%c0_i32 : i32] : vector<2xf32> + %lane1 = llvm.extractelement %value[%c1_i32 : i32] : vector<2xf32> + %sum = arith.addf %lane0, %lane1 : f32 + %c0 = arith.constant 0 : index + pto.store %sum, %dst[%c0] : !pto.ptr, f32 + } + return + } +} + +// MATERIALIZE-LABEL: func.func @persistent_fragment_vector_access +// MATERIALIZE-NOT: llvm.alloca +// MATERIALIZE: %[[INIT:.*]] = arith.constant dense<[1.000000e+00, 2.000000e+00]> : vector<2xf32> +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE: %[[INIT0:.*]] = llvm.extractelement %[[INIT]][%{{.*}} : i32] : vector<2xf32> +// MATERIALIZE: %[[INIT1:.*]] = llvm.extractelement %[[INIT]][%{{.*}} : i32] : vector<2xf32> +// MATERIALIZE: pto.keep %[[INIT0]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[INIT1]] {slot = 1 : i64} : f32 +// MATERIALIZE: pto.section.simt<<<32, 1, 1>>> { +// MATERIALIZE-NEXT: %[[RESUME0:.*]] = pto.resume {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: %[[RESUME1:.*]] = pto.resume {slot = 1 : i64} : f32 +// MATERIALIZE: %[[POISON:.*]] = llvm.mlir.poison : vector<2xf32> +// MATERIALIZE: %[[VECTOR0:.*]] = llvm.insertelement %[[RESUME0]], %[[POISON]][%{{.*}} : i32] : vector<2xf32> +// MATERIALIZE: %[[VECTOR1:.*]] = llvm.insertelement %[[RESUME1]], %[[VECTOR0]][%{{.*}} : i32] : vector<2xf32> +// MATERIALIZE: %[[LANE0:.*]] = llvm.extractelement %[[VECTOR1]][%{{.*}} : i32] : vector<2xf32> +// MATERIALIZE: %[[LANE1:.*]] = llvm.extractelement %[[VECTOR1]][%{{.*}} : i32] : vector<2xf32> +// MATERIALIZE: %[[SUM:.*]] = arith.addf %[[LANE0]], %[[LANE1]] : f32 +// MATERIALIZE: pto.store %[[SUM]], +// MATERIALIZE: pto.keep %[[RESUME0]] {slot = 0 : i64} : f32 +// MATERIALIZE-NEXT: pto.keep %[[RESUME1]] {slot = 1 : i64} : f32 +// MATERIALIZE-NOT: llvm.getelementptr +// MATERIALIZE-NOT: llvm.load +// MATERIALIZE-NOT: llvm.store diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_vector_non_resident_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_vector_non_resident_invalid.pto new file mode 100644 index 0000000000..312025c989 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_vector_non_resident_invalid.pto @@ -0,0 +1,27 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_vector_non_resident(%dst: !llvm.ptr) attributes {pto.entry} { + %c2_i32 = arith.constant 2 : i32 + %fragment = llvm.alloca %c2_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %zero = arith.constant 0.000000e+00 : f32 + llvm.store %zero, %fragment : f32, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %value = llvm.load %fragment : !llvm.ptr -> vector<2xf32> + llvm.store %value, %dst : vector<2xf32>, !llvm.ptr + } + return + } +} + +// CHECK: error: 'llvm.load' op persistent SIMT fragment element offset 1 is not initialized by the init section and is not part of residentElements diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_vector_out_of_bounds_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_vector_out_of_bounds_invalid.pto new file mode 100644 index 0000000000..cc30e61ad2 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_vector_out_of_bounds_invalid.pto @@ -0,0 +1,23 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_vector_out_of_bounds() attributes {pto.entry} { + %c1_i32 = arith.constant 1 : i32 + %fragment = llvm.alloca %c1_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %value = arith.constant dense<1.000000e+00> : vector<2xf32> + llvm.store %value, %fragment : vector<2xf32>, !llvm.ptr + } + return + } +} + +// CHECK: error: 'llvm.store' op persistent SIMT fragment access at byte offset 0 with 2 scalar lane(s) exceeds allocation size 4 diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_vector_volatile_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_vector_volatile_invalid.pto new file mode 100644 index 0000000000..53db0001f2 --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_vector_volatile_invalid.pto @@ -0,0 +1,27 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_vector_volatile_invalid(%dst: !llvm.ptr) attributes {pto.entry} { + %c2_i32 = arith.constant 2 : i32 + %fragment = llvm.alloca %c2_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %value = arith.constant dense<1.000000e+00> : vector<2xf32> + llvm.store %value, %fragment : vector<2xf32>, !llvm.ptr + } + pto.section.simt<<<32, 1, 1>>> { + %value = llvm.load volatile %fragment : !llvm.ptr -> vector<2xf32> + llvm.store %value, %dst : vector<2xf32>, !llvm.ptr + } + return + } +} + +// CHECK: error: 'llvm.load' op persistent SIMT fragment vector access cannot be volatile or atomic diff --git a/test/lit/vpto/materialize_simt_persistent_fragment_vector_width_invalid.pto b/test/lit/vpto/materialize_simt_persistent_fragment_vector_width_invalid.pto new file mode 100644 index 0000000000..ed51c4274f --- /dev/null +++ b/test/lit/vpto/materialize_simt_persistent_fragment_vector_width_invalid.pto @@ -0,0 +1,23 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @persistent_fragment_vector_width_invalid() attributes {pto.entry} { + %c3_i32 = arith.constant 3 : i32 + %fragment = llvm.alloca %c3_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.section.simt<<<32, 1, 1>>> { + %value = arith.constant dense<1.000000e+00> : vector<3xf32> + llvm.store %value, %fragment : vector<3xf32>, !llvm.ptr + } + return + } +} + +// CHECK: error: 'llvm.store' op persistent SIMT fragment currently supports only 2- or 4-lane vector accesses, got 3 lanes diff --git a/test/lit/vpto/outline_simt_sections.pto b/test/lit/vpto/outline_simt_sections.pto new file mode 100644 index 0000000000..3b0bd86974 --- /dev/null +++ b/test/lit/vpto/outline_simt_sections.pto @@ -0,0 +1,30 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-outline-simt-sections %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=OUTLINE + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @inline_simt_kernel(%dst: !pto.ptr) attributes {pto.entry} { + pto.section.simt<<<32, 2, 1>>> { + %c0 = arith.constant 0 : index + %tid = pto.get_tid_x : i32 + pto.store %tid, %dst[%c0] : !pto.ptr, i32 + } + return + } +} + +// OUTLINE-LABEL: func.func @inline_simt_kernel +// OUTLINE: arith.constant 32 : i32 +// OUTLINE: arith.constant 2 : i32 +// OUTLINE: arith.constant 1 : i32 +// OUTLINE: pto.simt_launch @inline_simt_kernel_simt_0<<<{{.*}}, {{.*}}, {{.*}}>>>(%arg0) : (!pto.ptr) -> () +// OUTLINE-NOT: pto.section.simt +// OUTLINE-LABEL: func.func private @inline_simt_kernel_simt_0(%arg0: !pto.ptr) attributes {pto.simt_entry, pto.simt_max_threads = 64 : i32} +// OUTLINE: pto.get_tid_x +// OUTLINE: pto.store {{.*}}, %arg0 diff --git a/test/lit/vpto/outline_simt_sections_invalid_threads.pto b/test/lit/vpto/outline_simt_sections_invalid_threads.pto new file mode 100644 index 0000000000..ab9a92c6eb --- /dev/null +++ b/test/lit/vpto/outline_simt_sections_invalid_threads.pto @@ -0,0 +1,22 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// the CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto %s -o /dev/null 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @inline_simt_kernel_invalid_threads(%dst: !pto.ptr) attributes {pto.entry} { + pto.section.simt<<<64, 64, 1>>> { + %c0 = arith.constant 0 : index + %tid = pto.get_tid_x : i32 + pto.store %tid, %dst[%c0] : !pto.ptr, i32 + } + return + } +} + +// CHECK: error: 'pto.section.simt' op SIMT launch dimensions must have a positive product no greater than 2048, got (64, 64, 1) diff --git a/test/lit/vpto/unroll_inline_simt_sections.pto b/test/lit/vpto/unroll_inline_simt_sections.pto new file mode 100644 index 0000000000..b711c480b0 --- /dev/null +++ b/test/lit/vpto/unroll_inline_simt_sections.pto @@ -0,0 +1,56 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=pto-unroll-simt-for %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=UNROLL + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @inline_simt_unroll_scope(%dst: !pto.ptr) attributes {pto.entry} { + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c2 = arith.constant 2 : index + %c4 = arith.constant 4 : index + + // An annotated loop outside a SIMT section must remain unchanged. + "scf.for"(%c0, %c2, %c1) ({ + ^bb0(%i: index): + %value = arith.index_cast %i : index to i32 + pto.store %value, %dst[%i] : !pto.ptr, i32 + scf.yield + }) {pto.unroll = "full"} : (index, index, index) -> () + + // An annotated loop inside an inline SIMT section must be fully unrolled. + pto.section.simt<<<32, 1, 1>>> { + "scf.for"(%c0, %c4, %c1) ({ + ^bb0(%i: index): + %value = arith.index_cast %i : index to i32 + pto.store %value, %dst[%i] : !pto.ptr, i32 + scf.yield + }) {pto.unroll = "full"} : (index, index, index) -> () + } + + // A loop without the annotation must remain inside the SIMT section. + pto.section.simt<<<32, 1, 1>>> { + scf.for %i = %c0 to %c2 step %c1 { + %value = arith.index_cast %i : index to i32 + pto.store %value, %dst[%i] : !pto.ptr, i32 + } + } + return + } +} + +// UNROLL-LABEL: func.func @inline_simt_unroll_scope +// UNROLL: scf.for +// UNROLL: pto.unroll = "full" +// UNROLL: pto.section.simt +// UNROLL-NOT: scf.for +// UNROLL-COUNT-4: pto.store +// UNROLL: pto.section.simt +// UNROLL: scf.for +// UNROLL-NOT: pto.unroll +// UNROLL: return diff --git a/test/vpto/cases/kernels/rmsnorm-fp32-4096-persistent-fragment-ptodsl/README.md b/test/vpto/cases/kernels/rmsnorm-fp32-4096-persistent-fragment-ptodsl/README.md new file mode 100644 index 0000000000..a3f7ad9937 --- /dev/null +++ b/test/vpto/cases/kernels/rmsnorm-fp32-4096-persistent-fragment-ptodsl/README.md @@ -0,0 +1,34 @@ +# RMSNorm Persistent SIMT Fragment PTODSL Case + +This directory stores a TileLang-generated PTODSL source variant used to +develop and validate persistent SIMT fragment materialization in PTOAS. + +## Files + +- `kernel.py`: RMSNorm fp32 4096x4096 PTODSL source adapted from a + TileLang PTO codegen dump. The weight fragment is allocated outside the + per-token SIMT section, initialized once from UB, and reused by subsequent + SIMT sections. + +## Purpose + +The case is a compile fixture for the optimization described in +`docs/designs/ptoas_persistent_simt_fragment_plan.md`. Current PTODSL marks the +SIMT-external fragment allocation with `pto.persistent` and preserves inline +SIMT sections. PTOAS materializes `pto.keep` / `pto.resume`, outlines the +sections, and compiles the resulting module through the VPTO pipeline. + +The kernel sets `ast_rewrite=False` for compatibility with the explicit PTODSL +source, but uses `pto.for_` for the token loop. The loop therefore remains an +`scf.for` in the pass input instead of being copied 64 times: one persistent +top-level `llvm.alloca`, one init section, and one inline carry section in the +loop body. The directory has no host launcher or reference check, so it remains +a compile-only fixture rather than a runnable kernel case. + +## Source + +The source was derived from: + +`~/.tilelang/cache/0.1.11_cpu_gitdd4f9aa0-aarch64/kernels/ab0771e55dcd836a25047f82c5e906a2b772be48c86efd19bd47e7c228b15b54/device_kernel.cu` + +and manually adapted into the target persistent-fragment shape. diff --git a/test/vpto/cases/kernels/rmsnorm-fp32-4096-persistent-fragment-ptodsl/kernel.py b/test/vpto/cases/kernels/rmsnorm-fp32-4096-persistent-fragment-ptodsl/kernel.py new file mode 100644 index 0000000000..6e6b85e4c5 --- /dev/null +++ b/test/vpto/cases/kernels/rmsnorm-fp32-4096-persistent-fragment-ptodsl/kernel.py @@ -0,0 +1,160 @@ +# Copyright (c) 2026 Huawei Technologies Co., Ltd. +# This program is free software, you can redistribute it and/or modify it under the terms and conditions of +# CANN Open Software License Agreement Version 2.0 (the "License"). +# Please refer to the License for details. You may not use this file except in compliance with the License. +# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +# See LICENSE in the root of the software repository for the full text of the License. + +from ptodsl import pto, scalar + + +@pto.jit( + name="main_kernel", + kernel_kind="vector", + target="a5", + mode="explicit", + ast_rewrite=False, +) +def main_kernel( + RSTD: pto.ptr(pto.f32, "gm"), + W: pto.ptr(pto.f32, "gm"), + X: pto.ptr(pto.f32, "gm"), + Y: pto.ptr(pto.f32, "gm"), + eps: pto.f32, +): + bx = pto.get_block_idx() + buf_dyn_shmem = pto.castptr(pto.const(0, dtype=pto.i64), pto.ptr(pto.i8, "ub")) + + pto.set_flag("V", "MTE2", event_id=0) + pto.set_flag("V", "MTE2", event_id=1) + pto.set_flag("MTE3", "V", event_id=0) + pto.set_flag("MTE3", "V", event_id=1) + + pto.mte_gm_ub( + W, + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + 0, + 16384, + nburst=(1, 16384, 16384), + ) + pto.set_flag("MTE2", "V", event_id=2) + pto.wait_flag("MTE2", "V", event_id=2) + + # Target shape for persistent SIMT fragment materialization: + # w_frag is allocated outside SIMT sections, initialized once from w_ub, and + # consumed by every per-token SIMT section below. + w_frag = pto.alloc_buffer((32,), pto.f32) + with pto.simt(128, 1, 1): + simtvf_tx = pto.get_tid_x() + for i in pto.static_range(0, 16): + scalar.store( + scalar.load( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + (i * 256) + (simtvf_tx * 2), + contiguous=2, + ), + w_frag, + i * 2, + ) + + with pto.for_(0, 64, step=1) as t: + pto.wait_flag("V", "MTE2", event_id=t & 1) + pto.mte_gm_ub( + pto.addptr(X, (t * 262144) + (bx * 4096)), + pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 4096) + 4224, + ), + 0, + 16384, + nburst=(1, 16384, 16384), + ) + pto.set_flag("MTE2", "V", event_id=t & 1) + pto.wait_flag("MTE3", "V", event_id=t & 1) + pto.wait_flag("MTE2", "V", event_id=t & 1) + + with pto.simt(128, 1, 1): + x_frag = pto.alloc_buffer((32,), pto.f32) + sum_sq = pto.alloc_buffer((1,), pto.f32) + simtvf_tx = pto.get_tid_x() + + for i in pto.static_range(0, 16): + scalar.store( + scalar.load( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((((t & 1) * 4096) + (i * 256)) + (simtvf_tx * 2)) + 4224, + contiguous=2, + ), + x_frag, + i * 2, + ) + + scalar.store(float.fromhex("0x0p+0"), sum_sq, 0) + for i_1 in pto.static_range(0, 32): + scalar.store( + scalar.load(sum_sq, 0) + + (scalar.load(x_frag, i_1) * scalar.load(x_frag, i_1)), + sum_sq, + 0, + ) + + scalar.store( + pto.simt_allreduce_sum( + scalar.load(sum_sq, 0), + threads=128, + scale=1, + thread_offset=0, + scratch=pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), 4096 + ), + ), + sum_sq, + 0, + ) + var = (scalar.load(sum_sq, 0) / float.fromhex("0x1p+12")) + eps + rstd_val = float.fromhex("0x1p+0") / pto.sqrt(var) + scalar.store( + rstd_val, + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 8) + 20608, + ) + + for i_2 in pto.static_range(0, 16): + scalar.store( + ( + scalar.load(x_frag, i_2 * 2, contiguous=2) + * pto.Vec(pto.f32, 2, init=rstd_val) + ) + * scalar.load(w_frag, i_2 * 2, contiguous=2), + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((((t & 1) * 4096) + (i_2 * 256)) + (simtvf_tx * 2)) + 12416, + ) + + pto.set_flag("V", "MTE3", event_id=t & 1) + pto.set_flag("V", "MTE2", event_id=t & 1) + pto.wait_flag("V", "MTE3", event_id=t & 1) + pto.mte_ub_gm( + pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 8) + 20608, + ), + pto.addptr(RSTD, (t * 64) + bx), + 4, + nburst=(1, 4, 4), + ) + pto.mte_ub_gm( + pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 4096) + 12416, + ), + pto.addptr(Y, (t * 262144) + (bx * 4096)), + 16384, + nburst=(1, 16384, 16384), + ) + pto.set_flag("MTE3", "V", event_id=t & 1) + + pto.wait_flag("V", "MTE2", event_id=0) + pto.wait_flag("V", "MTE2", event_id=1) + pto.wait_flag("MTE3", "V", event_id=0) + pto.wait_flag("MTE3", "V", event_id=1) diff --git a/test/vpto/cases/kernels/rmsnorm-fp32-5120-persistent-fragment-ptodsl/README.md b/test/vpto/cases/kernels/rmsnorm-fp32-5120-persistent-fragment-ptodsl/README.md new file mode 100644 index 0000000000..e693504d97 --- /dev/null +++ b/test/vpto/cases/kernels/rmsnorm-fp32-5120-persistent-fragment-ptodsl/README.md @@ -0,0 +1,16 @@ +# RMSNorm FP32 5120 Persistent Fragment Case + +The PTODSL source is in `kernel.py`. + +This PTODSL source is the persistent-fragment variant of the TileLang +RMSNorm kernel for `batch=4096` and `d=5120`. + +- The weight buffer is loaded from GM to UB once. +- A SIMT init section copies the 20 valid `f32` weight elements into the + persistent fragment. +- Each token SIMT section resumes and re-keeps those 20 elements. +- The remaining 12 elements of the 32-element physical fragment are layout + padding and do not consume persistent slots. + +The source uses the same UB offsets, `256` SIMT threads, and `64` token loop +as the corresponding TileLang-generated PTO kernel. diff --git a/test/vpto/cases/kernels/rmsnorm-fp32-5120-persistent-fragment-ptodsl/kernel.py b/test/vpto/cases/kernels/rmsnorm-fp32-5120-persistent-fragment-ptodsl/kernel.py new file mode 100644 index 0000000000..4efd148dd0 --- /dev/null +++ b/test/vpto/cases/kernels/rmsnorm-fp32-5120-persistent-fragment-ptodsl/kernel.py @@ -0,0 +1,160 @@ +# Copyright (c) 2026 Huawei Technologies Co., Ltd. +# This program is free software, you can redistribute it and/or modify it under the terms and conditions of +# CANN Open Software License Agreement Version 2.0 (the "License"). +# Please refer to the License for details. You may not use this file except in compliance with the License. +# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +# See LICENSE in the root of the software repository for the full text of the License. + +from ptodsl import pto, scalar + + +@pto.jit( + name="main_kernel", + kernel_kind="vector", + target="a5", + mode="explicit", + ast_rewrite=False, +) +def main_kernel( + RSTD: pto.ptr(pto.f32, "gm"), + W: pto.ptr(pto.f32, "gm"), + X: pto.ptr(pto.f32, "gm"), + Y: pto.ptr(pto.f32, "gm"), + eps: pto.f32, +): + bx = pto.get_block_idx() + buf_dyn_shmem = pto.castptr(pto.const(0, dtype=pto.i64), pto.ptr(pto.i8, "ub")) + + pto.set_flag("V", "MTE2", event_id=0) + pto.set_flag("V", "MTE2", event_id=1) + pto.set_flag("MTE3", "V", event_id=0) + pto.set_flag("MTE3", "V", event_id=1) + + pto.mte_gm_ub( + W, + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + 0, + 20480, + nburst=(1, 20480, 20480), + ) + pto.set_flag("MTE2", "V", event_id=2) + pto.wait_flag("MTE2", "V", event_id=2) + + # The physical fragment has 32 elements, while only 20 are valid for d=5120. + w_frag = pto.alloc_buffer((32,), pto.f32) + with pto.simt(256, 1, 1): + simtvf_tx = pto.get_tid_x() + for weight_i in pto.static_range(0, 10): + scalar.store( + scalar.load( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + (weight_i * 512) + (simtvf_tx * 2), + contiguous=2, + ), + w_frag, + weight_i * 2, + ) + + with pto.for_(0, 64, step=1) as t: + pto.wait_flag("V", "MTE2", event_id=t & 1) + pto.mte_gm_ub( + pto.addptr(X, (t * 327680) + (bx * 5120)), + pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 8192) + 5120, + ), + 0, + 20480, + nburst=(1, 20480, 20480), + ) + pto.set_flag("MTE2", "V", event_id=t & 1) + pto.wait_flag("MTE3", "V", event_id=t & 1) + pto.wait_flag("MTE2", "V", event_id=t & 1) + + with pto.simt(256, 1, 1): + x_frag = pto.alloc_buffer((32,), pto.f32) + sum_sq = pto.alloc_buffer((1,), pto.f32) + simtvf_tx = pto.get_tid_x() + + for input_i in pto.static_range(0, 16): + scalar.store( + scalar.load( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((((t & 1) * 8192) + (input_i * 512)) + (simtvf_tx * 2)) + + 5120, + contiguous=2, + ), + x_frag, + input_i * 2, + ) + + scalar.store(float.fromhex("0x0p+0"), sum_sq, 0) + for reduce_i in pto.static_range(0, 20): + scalar.store( + scalar.load(sum_sq, 0) + + (scalar.load(x_frag, reduce_i) * scalar.load(x_frag, reduce_i)), + sum_sq, + 0, + ) + + scalar.store( + pto.simt_allreduce_sum( + scalar.load(sum_sq, 0), + threads=256, + scale=1, + thread_offset=0, + scratch=pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), 37904 + ), + ), + sum_sq, + 0, + ) + var = (scalar.load(sum_sq, 0) / float.fromhex("0x1.4p+12")) + eps + rstd_val = float.fromhex("0x1p+0") / pto.sqrt(var) + scalar.store( + rstd_val, + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 8) + 37888, + ) + + for output_i in pto.static_range(0, 10): + scalar.store( + ( + scalar.load(x_frag, output_i * 2, contiguous=2) + * pto.Vec(pto.f32, 2, init=rstd_val) + ) + * scalar.load(w_frag, output_i * 2, contiguous=2), + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((((t & 1) * 8192) + (output_i * 512)) + (simtvf_tx * 2)) + + 21504, + ) + + pto.set_flag("V", "MTE3", event_id=t & 1) + pto.set_flag("V", "MTE2", event_id=t & 1) + pto.wait_flag("V", "MTE3", event_id=t & 1) + pto.mte_ub_gm( + pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 8) + 37888, + ), + pto.addptr(RSTD, (t * 64) + bx), + 4, + nburst=(1, 4, 4), + ) + pto.mte_ub_gm( + pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 8192) + 21504, + ), + pto.addptr(Y, (t * 327680) + (bx * 5120)), + 20480, + nburst=(1, 20480, 20480), + ) + pto.set_flag("MTE3", "V", event_id=t & 1) + + pto.wait_flag("V", "MTE2", event_id=0) + pto.wait_flag("V", "MTE2", event_id=1) + pto.wait_flag("MTE3", "V", event_id=0) + pto.wait_flag("MTE3", "V", event_id=1) diff --git a/test/vpto/cases/kernels/rmsnorm-fp32-7168-persistent-fragment-ptodsl/README.md b/test/vpto/cases/kernels/rmsnorm-fp32-7168-persistent-fragment-ptodsl/README.md new file mode 100644 index 0000000000..3ea191ec82 --- /dev/null +++ b/test/vpto/cases/kernels/rmsnorm-fp32-7168-persistent-fragment-ptodsl/README.md @@ -0,0 +1,16 @@ +# RMSNorm FP32 7168 Persistent Fragment Case + +The PTODSL source is in `kernel.py`. + +This PTODSL source is the persistent-fragment variant of the TileLang +RMSNorm kernel for `batch=4096` and `d=7168`. + +- The weight buffer is loaded from GM to UB once. +- A SIMT init section copies the 28 valid `f32` weight elements into the + persistent fragment. +- Each token SIMT section resumes and re-keeps those 28 elements. +- The remaining four elements of the 32-element physical fragment are layout + padding and do not consume persistent slots. + +The source uses the same UB offsets, `256` SIMT threads, and `64` token loop +as the corresponding TileLang-generated PTO kernel. diff --git a/test/vpto/cases/kernels/rmsnorm-fp32-7168-persistent-fragment-ptodsl/kernel.py b/test/vpto/cases/kernels/rmsnorm-fp32-7168-persistent-fragment-ptodsl/kernel.py new file mode 100644 index 0000000000..3bac53ba76 --- /dev/null +++ b/test/vpto/cases/kernels/rmsnorm-fp32-7168-persistent-fragment-ptodsl/kernel.py @@ -0,0 +1,160 @@ +# Copyright (c) 2026 Huawei Technologies Co., Ltd. +# This program is free software, you can redistribute it and/or modify it under the terms and conditions of +# CANN Open Software License Agreement Version 2.0 (the "License"). +# Please refer to the License for details. You may not use this file except in compliance with the License. +# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +# See LICENSE in the root of the software repository for the full text of the License. + +from ptodsl import pto, scalar + + +@pto.jit( + name="main_kernel", + kernel_kind="vector", + target="a5", + mode="explicit", + ast_rewrite=False, +) +def main_kernel( + RSTD: pto.ptr(pto.f32, "gm"), + W: pto.ptr(pto.f32, "gm"), + X: pto.ptr(pto.f32, "gm"), + Y: pto.ptr(pto.f32, "gm"), + eps: pto.f32, +): + bx = pto.get_block_idx() + buf_dyn_shmem = pto.castptr(pto.const(0, dtype=pto.i64), pto.ptr(pto.i8, "ub")) + + pto.set_flag("V", "MTE2", event_id=0) + pto.set_flag("V", "MTE2", event_id=1) + pto.set_flag("MTE3", "V", event_id=0) + pto.set_flag("MTE3", "V", event_id=1) + + pto.mte_gm_ub( + W, + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + 0, + 28672, + nburst=(1, 28672, 28672), + ) + pto.set_flag("MTE2", "V", event_id=2) + pto.wait_flag("MTE2", "V", event_id=2) + + # The physical fragment has 32 elements, while only 28 are valid for d=7168. + w_frag = pto.alloc_buffer((32,), pto.f32) + with pto.simt(256, 1, 1): + simtvf_tx = pto.get_tid_x() + for weight_i in pto.static_range(0, 14): + scalar.store( + scalar.load( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + (weight_i * 512) + (simtvf_tx * 2), + contiguous=2, + ), + w_frag, + weight_i * 2, + ) + + with pto.for_(0, 64, step=1) as t: + pto.wait_flag("V", "MTE2", event_id=t & 1) + pto.mte_gm_ub( + pto.addptr(X, (t * 458752) + (bx * 7168)), + pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 8192) + 7168, + ), + 0, + 28672, + nburst=(1, 28672, 28672), + ) + pto.set_flag("MTE2", "V", event_id=t & 1) + pto.wait_flag("MTE3", "V", event_id=t & 1) + pto.wait_flag("MTE2", "V", event_id=t & 1) + + with pto.simt(256, 1, 1): + x_frag = pto.alloc_buffer((32,), pto.f32) + sum_sq = pto.alloc_buffer((1,), pto.f32) + simtvf_tx = pto.get_tid_x() + + for input_i in pto.static_range(0, 16): + scalar.store( + scalar.load( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((((t & 1) * 8192) + (input_i * 512)) + (simtvf_tx * 2)) + + 7168, + contiguous=2, + ), + x_frag, + input_i * 2, + ) + + scalar.store(float.fromhex("0x0p+0"), sum_sq, 0) + for reduce_i in pto.static_range(0, 28): + scalar.store( + scalar.load(sum_sq, 0) + + (scalar.load(x_frag, reduce_i) * scalar.load(x_frag, reduce_i)), + sum_sq, + 0, + ) + + scalar.store( + pto.simt_allreduce_sum( + scalar.load(sum_sq, 0), + threads=256, + scale=1, + thread_offset=0, + scratch=pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), 39952 + ), + ), + sum_sq, + 0, + ) + var = (scalar.load(sum_sq, 0) / float.fromhex("0x1.cp+12")) + eps + rstd_val = float.fromhex("0x1p+0") / pto.sqrt(var) + scalar.store( + rstd_val, + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 8) + 39936, + ) + + for output_i in pto.static_range(0, 14): + scalar.store( + ( + scalar.load(x_frag, output_i * 2, contiguous=2) + * pto.Vec(pto.f32, 2, init=rstd_val) + ) + * scalar.load(w_frag, output_i * 2, contiguous=2), + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((((t & 1) * 8192) + (output_i * 512)) + (simtvf_tx * 2)) + + 23552, + ) + + pto.set_flag("V", "MTE3", event_id=t & 1) + pto.set_flag("V", "MTE2", event_id=t & 1) + pto.wait_flag("V", "MTE3", event_id=t & 1) + pto.mte_ub_gm( + pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 8192) + 23552, + ), + pto.addptr(Y, (t * 458752) + (bx * 7168)), + 28672, + nburst=(1, 28672, 28672), + ) + pto.mte_ub_gm( + pto.addptr( + pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), + ((t & 1) * 8) + 39936, + ), + pto.addptr(RSTD, (t * 64) + bx), + 4, + nburst=(1, 4, 4), + ) + pto.set_flag("MTE3", "V", event_id=t & 1) + + pto.wait_flag("V", "MTE2", event_id=0) + pto.wait_flag("V", "MTE2", event_id=1) + pto.wait_flag("MTE3", "V", event_id=0) + pto.wait_flag("MTE3", "V", event_id=1) diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/.gitignore b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/.gitignore new file mode 100644 index 0000000000..84c048a73c --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/.gitignore @@ -0,0 +1 @@ +/build/ diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/README.md b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/README.md new file mode 100644 index 0000000000..807228f924 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/README.md @@ -0,0 +1,110 @@ +# RMSNorm Persistent Fragment Sequence Reproducer + +This temporary, hardware-only directory preserves the source boundary of the +state/order-dependent failure seen when three different persistent-fragment +RMSNorm kernels run in one process. It is intentionally self-contained so it +can be removed as one directory after the root cause is understood. + +No TileLang checkout is required. No generated object, shared library, runner, +or compiler log is tracked; all build output goes to the ignored `build/` +directory (or to `BUILD_DIR` when overridden). + +## Source Layout + +- `kernels/d*/kernel.ptodsl.py`: TileLang-generated PTODSL source retained for + frontend comparison. It is not invoked by `build.sh`. +- `kernels/d*/kernel.pto`: textual PTO IR compiled by the current PTOAS. +- `kernels/d*/kernel_vpto.ll`: VPTO LLVM IR emitted from the corresponding + `kernel.pto` with the current PTOAS. +- `kernels/d*/launch.cpp`: AOT launch wrapper and dynamic-UB contract. +- `host/sequence_main.cpp`: one ACL process, one device, one stream, and the + ordered kernel launches. + +The launch contract is: + +| `d` | kernel symbol | grid | SIMT threads | dynamic UB bytes | +| ---: | --- | ---: | ---: | ---: | +| 4096 | `rmsnorm_d4096_kernel` | 64 | 128 | 82432 | +| 5120 | `rmsnorm_d5120_kernel` | 64 | 256 | 152576 | +| 7168 | `rmsnorm_d7168_kernel` | 64 | 256 | 160768 | + +These are the persistent-fragment UB sizes. Do not replace them with the +non-persistent baseline sizes (`82496`, `152640`, and `160832`). + +## Build + +From this directory: + +```bash +CANN_HOME=/path/to/cann \ +PTOAS_BIN=/path/to/PTOAS/build/tools/ptoas/ptoas \ +./build.sh +``` + +By default, `PTOAS_BIN` resolves to the `build/tools/ptoas/ptoas` binary in +this PTOAS checkout, and `CANN_HOME` resolves to the local CANN 9.1 T530 +installation used for the original reproduction. + +To keep every generated file outside the checkout, use the same `BUILD_DIR` +for both build and run: + +```bash +BUILD_DIR=/tmp/rmsnorm-persistent-sequence ./build.sh +BUILD_DIR=/tmp/rmsnorm-persistent-sequence \ + ASCEND_RT_VISIBLE_DEVICES=0 ACL_DEVICE_ID=0 \ + RMSNORM_SEQUENCE_REPEATS=1 ./run.sh +``` + +To regenerate the checked-in LLVM IR without producing device binaries: + +```bash +PTOAS_BIN=/home/qukelin/projects/PTOAS/build/tools/ptoas/ptoas +for d in 4096 5120 7168; do + "${PTOAS_BIN}" --pto-arch=a5 --pto-backend=vpto \ + --emit-vpto-llvm-ir "kernels/d${d}/kernel.pto" \ + -o "kernels/d${d}/kernel_vpto.ll" +done +``` + +## Run On Hardware + +The captured host defaults to two launches of 4096, two launches of 5120, and +one launch of 7168. This was the stronger state/order-dependent reproduction: + +```bash +ASCEND_RT_VISIBLE_DEVICES=0 ACL_DEVICE_ID=0 ./run.sh +``` + +To run exactly one launch of each shape in the same process: + +```bash +RMSNORM_SEQUENCE_REPEATS=1 ./run.sh +``` + +Use fresh processes for the single-kernel controls: + +```bash +RMSNORM_ONLY=4096 RMSNORM_SEQUENCE_REPEATS=1 ./run.sh +RMSNORM_ONLY=5120 RMSNORM_SEQUENCE_REPEATS=1 ./run.sh +RMSNORM_ONLY=7168 RMSNORM_SEQUENCE_REPEATS=1 ./run.sh +``` + +Each launch is followed by `aclrtSynchronizeStream`. The failing sequence has +reported runtime error `507035` and vector GM read error `355`; each shape has +also passed when launched alone. The failing symbol is state/order dependent +and is not necessarily the final 7168 kernel. A reproduced fault exits with a +nonzero status; a completed sequence prints sampled 7168 `rstd` values. + +On 2026-07-27, a one-launch sequence built with the current PTOAS checkout +reproduced the fault after these markers: + +```text +launch 4096 repeat=0 +launch 5120 repeat=0 +fault kernel_name=rmsnorm_d5120_kernel +runtime result = 507035 +errcode:(355) The address for VEC to read GM is out of bounds (exceeding 48 bits) +``` + +Run this reproducer on a real device because camodel does not preserve +persistent SIMT register state across SIMTVF exits. diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/build.sh b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/build.sh new file mode 100755 index 0000000000..6eb8e1a868 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/build.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +# Copyright (c) 2026 Huawei Technologies Co., Ltd. +# This program is free software, you can redistribute it and/or modify it under the terms and conditions of +# CANN Open Software License Agreement Version 2.0 (the "License"). +# Please refer to the License for details. You may not use this file except in compliance with the License. +# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +# See LICENSE in the root of the software repository for the full text of the License. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PTOAS_ROOT="$(cd "${SCRIPT_DIR}/../../../../.." && pwd)" +SOURCE_DIR="${SCRIPT_DIR}/kernels" +BUILD_DIR="${BUILD_DIR:-${SCRIPT_DIR}/build}" +CANN_HOME="${CANN_HOME:-/home/qukelin/tools/CANN_9.1/cann-9.1.T530}" +PTOAS_BIN="${PTOAS_BIN:-${PTOAS_ROOT}/build/tools/ptoas/ptoas}" +BISHENG_BIN="${BISHENG_BIN:-${CANN_HOME}/bin/bisheng}" + +die() { + echo "ERROR: $*" >&2 + exit 1 +} + +[[ -x "${PTOAS_BIN}" ]] || die "PTOAS_BIN is not executable: ${PTOAS_BIN}" +[[ -x "${BISHENG_BIN}" ]] || die "BISHENG_BIN is not executable: ${BISHENG_BIN}" +[[ -f "${CANN_HOME}/set_env.sh" ]] || die "missing CANN environment: ${CANN_HOME}/set_env.sh" + +set +u +source "${CANN_HOME}/set_env.sh" >/dev/null 2>&1 +set -u + +mkdir -p "${BUILD_DIR}" +BUILD_DIR="$(cd "${BUILD_DIR}" && pwd)" + +build_shape() { + local d="$1" + local source_dir="${SOURCE_DIR}/d${d}" + local output_dir="${BUILD_DIR}/d${d}" + + mkdir -p "${output_dir}" + echo "[build] d=${d}: PTOAS" + "${PTOAS_BIN}" \ + --pto-arch=a5 \ + --pto-backend=vpto \ + "${source_dir}/kernel.pto" \ + -o "${output_dir}/kernel.fatobj.o" \ + >"${output_dir}/ptoas.log" 2>&1 + + echo "[build] d=${d}: launch wrapper" + "${BISHENG_BIN}" \ + -c -fPIC -xcce -fenable-matrix --cce-aicore-enable-tl \ + -Xhost-start -Xhost-end \ + -mllvm -cce-aicore-stack-size=0x8000 \ + -mllvm -cce-aicore-function-stack-size=0x8000 \ + -mllvm -cce-aicore-record-overflow=true \ + -mllvm -cce-aicore-addr-transform \ + -mllvm -cce-aicore-dcci-insert-for-scalar=false \ + --cce-aicore-arch=dav-c310 \ + -DREGISTER_BASE -std=c++17 \ + -Wno-macro-redefined -Wno-ignored-attributes \ + -I "${CANN_HOME}/include" \ + -I "${CANN_HOME}/pkg_inc" \ + -I "${CANN_HOME}/pkg_inc/profiling" \ + -I "${CANN_HOME}/pkg_inc/runtime/runtime" \ + "${source_dir}/launch.cpp" \ + -o "${output_dir}/launch.o" \ + >"${output_dir}/bisheng-launch.log" 2>&1 + + echo "[build] d=${d}: shared library" + "${BISHENG_BIN}" \ + -fPIC -s -Wl,-z,relro -Wl,-z,now --cce-fatobj-link \ + -shared -Wl,-soname,libkernel.so \ + -L "${CANN_HOME}/lib64" \ + -Wl,-rpath,"${CANN_HOME}/lib64" \ + -o "${output_dir}/libkernel.so" \ + "${output_dir}/kernel.fatobj.o" \ + "${output_dir}/launch.o" \ + -Wl,--no-as-needed -lruntime \ + >"${output_dir}/bisheng-link.log" 2>&1 +} + +for d in 4096 5120 7168; do + build_shape "${d}" +done + +echo "[build] ACL sequence runner" +"${BISHENG_BIN}" \ + -xc++ -include stdint.h -include stddef.h -std=c++17 \ + "${SCRIPT_DIR}/host/sequence_main.cpp" \ + -I "${CANN_HOME}/include" \ + -L "${CANN_HOME}/lib64" \ + -Wl,-rpath,"${CANN_HOME}/lib64" \ + -Wl,--allow-shlib-undefined \ + -lruntime -lstdc++ -lascendcl -lm -ldl -lpthread \ + -o "${BUILD_DIR}/sequence_runner" \ + >"${BUILD_DIR}/bisheng-host.log" 2>&1 + +echo "Built source-only reproducer into ${BUILD_DIR}" diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/host/sequence_main.cpp b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/host/sequence_main.cpp new file mode 100644 index 0000000000..55172aa4cd --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/host/sequence_main.cpp @@ -0,0 +1,179 @@ +#include "acl/acl.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace { +using CallKernel = int (*)(void*, void*, void*, void*, float, void*); + +struct ShapeBuffers { + size_t d; + size_t elems; + std::vector x; + std::vector y; + std::vector w; + std::vector rstd; + void* xd = nullptr; + void* yd = nullptr; + void* wd = nullptr; + void* rd = nullptr; +}; + +void Fill(ShapeBuffers& b) { + for (size_t row = 0; row < 4096; ++row) { + for (size_t i = 0; i < b.d; ++i) { + const int centered = static_cast(i % 31) - 15; + b.x[row * b.d + i] = 0.25f + centered * 0.0078125f + (row % 7) * 0.015625f; + } + } + for (size_t i = 0; i < b.d; ++i) b.w[i] = 0.75f + (i % 17) * 0.03125f; +} + +float Rstd(const ShapeBuffers& b, size_t row) { + float sum = 0.0f; + for (size_t i = 0; i < b.d; ++i) { + const float v = b.x[row * b.d + i]; + sum += v * v; + } + return 1.0f / std::sqrt(sum / static_cast(b.d) + 1.0e-6f); +} + +} // namespace + +#define ACL_CHECK(expr) \ + do { \ + const aclError _ret = (expr); \ + if (_ret != ACL_SUCCESS) { \ + std::fprintf(stderr, "[ERROR] %s failed: %d (%s:%d)\n", #expr, static_cast(_ret), \ + __FILE__, __LINE__); \ + const char* recent = aclGetRecentErrMsg(); \ + if (recent != nullptr && recent[0] != '\0') std::fprintf(stderr, "%s\n", recent); \ + rc = 1; \ + goto cleanup; \ + } \ + } while (0) + +int main(int argc, char** argv) { + // Keep launch markers ordered with asynchronous ACL error output. + std::setbuf(stdout, nullptr); + if (argc < 4) { + std::fprintf(stderr, "usage: %s kernel4096.so kernel5120.so kernel7168.so\n", argv[0]); + return 2; + } + ShapeBuffers b4096{4096, 4096 * 4096, std::vector(4096 * 4096), + std::vector(4096 * 4096), std::vector(4096), + std::vector(4096), nullptr, nullptr, nullptr, nullptr}; + ShapeBuffers b5120{5120, 4096 * 5120, std::vector(4096 * 5120), + std::vector(4096 * 5120), std::vector(5120), + std::vector(4096), nullptr, nullptr, nullptr, nullptr}; + ShapeBuffers b7168{7168, 4096 * 7168, std::vector(4096 * 7168), + std::vector(4096 * 7168), std::vector(7168), + std::vector(4096), nullptr, nullptr, nullptr, nullptr}; + Fill(b4096); + Fill(b5120); + Fill(b7168); + + void* h0 = nullptr; + void* h1 = nullptr; + void* h2 = nullptr; + CallKernel k0 = nullptr; + CallKernel k1 = nullptr; + CallKernel k2 = nullptr; + aclrtStream stream = nullptr; + bool inited = false; + bool device_set = false; + int device_id = 0; + int rc = 0; + if (const char* env = std::getenv("ACL_DEVICE_ID")) device_id = std::atoi(env); + int repeats = 2; + if (const char* env = std::getenv("RMSNORM_SEQUENCE_REPEATS")) { + repeats = std::atoi(env); + if (repeats < 1) repeats = 1; + } + const char* only = std::getenv("RMSNORM_ONLY"); + + h0 = dlopen(argv[1], RTLD_NOW | RTLD_LOCAL); + h1 = dlopen(argv[2], RTLD_NOW | RTLD_LOCAL); + h2 = dlopen(argv[3], RTLD_NOW | RTLD_LOCAL); + if (h0 == nullptr || h1 == nullptr || h2 == nullptr) { + std::fprintf(stderr, "[ERROR] dlopen failed: %s\n", dlerror()); + return 1; + } + k0 = reinterpret_cast(dlsym(h0, "call")); + k1 = reinterpret_cast(dlsym(h1, "call")); + k2 = reinterpret_cast(dlsym(h2, "call")); + if (k0 == nullptr || k1 == nullptr || k2 == nullptr) { + std::fprintf(stderr, "[ERROR] dlsym(call) failed\n"); + return 1; + } + + ACL_CHECK(aclInit(nullptr)); + inited = true; + ACL_CHECK(aclrtSetDevice(device_id)); + device_set = true; + ACL_CHECK(aclrtCreateStream(&stream)); + for (ShapeBuffers* b : {&b4096, &b5120, &b7168}) { + ACL_CHECK(aclrtMalloc(&b->xd, b->elems * sizeof(float), ACL_MEM_MALLOC_HUGE_FIRST)); + ACL_CHECK(aclrtMalloc(&b->yd, b->elems * sizeof(float), ACL_MEM_MALLOC_HUGE_FIRST)); + ACL_CHECK(aclrtMalloc(&b->wd, b->d * sizeof(float), ACL_MEM_MALLOC_HUGE_FIRST)); + ACL_CHECK(aclrtMalloc(&b->rd, 4096 * sizeof(float), ACL_MEM_MALLOC_HUGE_FIRST)); + ACL_CHECK(aclrtMemcpy(b->xd, b->elems * sizeof(float), b->x.data(), + b->elems * sizeof(float), ACL_MEMCPY_HOST_TO_DEVICE)); + ACL_CHECK(aclrtMemcpy(b->yd, b->elems * sizeof(float), b->y.data(), + b->elems * sizeof(float), ACL_MEMCPY_HOST_TO_DEVICE)); + ACL_CHECK(aclrtMemcpy(b->wd, b->d * sizeof(float), b->w.data(), b->d * sizeof(float), + ACL_MEMCPY_HOST_TO_DEVICE)); + ACL_CHECK(aclrtMemcpy(b->rd, 4096 * sizeof(float), b->rstd.data(), + 4096 * sizeof(float), ACL_MEMCPY_HOST_TO_DEVICE)); + } + std::printf("allocated x pointers: 0x%llx 0x%llx 0x%llx\n", + (unsigned long long)b4096.xd, (unsigned long long)b5120.xd, + (unsigned long long)b7168.xd); + + if (only == nullptr || std::strcmp(only, "4096") == 0) { + for (int repeat = 0; repeat < repeats; ++repeat) { + std::printf("launch 4096 repeat=%d\n", repeat); + if (k0(b4096.xd, b4096.yd, b4096.wd, b4096.rd, 1.0e-6f, stream) != 0) { rc = 1; goto cleanup; } + ACL_CHECK(aclrtSynchronizeStream(stream)); + } + } + if (only == nullptr || std::strcmp(only, "5120") == 0) { + for (int repeat = 0; repeat < repeats; ++repeat) { + std::printf("launch 5120 repeat=%d\n", repeat); + if (k1(b5120.xd, b5120.yd, b5120.wd, b5120.rd, 1.0e-6f, stream) != 0) { rc = 1; goto cleanup; } + ACL_CHECK(aclrtSynchronizeStream(stream)); + } + } + if (only == nullptr || std::strcmp(only, "7168") == 0) { + std::printf("launch 7168\n"); + if (k2(b7168.xd, b7168.yd, b7168.wd, b7168.rd, 1.0e-6f, stream) != 0) { rc = 1; goto cleanup; } + ACL_CHECK(aclrtSynchronizeStream(stream)); + ACL_CHECK(aclrtMemcpy(b7168.rstd.data(), 4096 * sizeof(float), b7168.rd, + 4096 * sizeof(float), ACL_MEMCPY_DEVICE_TO_HOST)); + for (size_t row : {size_t(0), size_t(1), size_t(63), size_t(4095)}) { + const float expected = Rstd(b7168, row); + std::printf("7168 row=%zu rstd=%.9g expected=%.9g\n", row, b7168.rstd[row], expected); + if (std::fabs(b7168.rstd[row] - expected) > 2.0e-5f) rc = 1; + } + } + +cleanup: + for (ShapeBuffers* b : {&b4096, &b5120, &b7168}) { + if (b->xd) aclrtFree(b->xd); + if (b->yd) aclrtFree(b->yd); + if (b->wd) aclrtFree(b->wd); + if (b->rd) aclrtFree(b->rd); + } + if (stream) aclrtDestroyStream(stream); + if (device_set) aclrtResetDevice(device_id); + if (inited) aclFinalize(); + if (h0) dlclose(h0); + if (h1) dlclose(h1); + if (h2) dlclose(h2); + return rc; +} diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/kernel.pto b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/kernel.pto new file mode 100644 index 0000000000..0fb9ab9cdb --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/kernel.pto @@ -0,0 +1,2441 @@ +module attributes {pto.target_arch = "a5"} { + module attributes {pto.backend = "vpto", pto.kernel_kind = #pto.kernel_kind, pto.target_arch = "a5"} { + func.func @rmsnorm_d4096_kernel(%arg0: !pto.ptr, %arg1: !pto.ptr, %arg2: !pto.ptr, %arg3: !pto.ptr, %arg4: f32) attributes {pto.entry, pto.kernel_kind = #pto.kernel_kind} { + %c0_i64 = arith.constant 0 : i64 + %0 = pto.castptr %c0_i64 : i64 -> !pto.ptr + %c32_i32 = arith.constant 32 : i32 + %1 = llvm.alloca %c32_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.set_flag[, , ] + pto.set_flag[, , ] + pto.set_flag[, , ] + pto.set_flag[, , ] + %2 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_i64 = arith.constant 1 : i64 + %c16384_i64 = arith.constant 16384 : i64 + %c16384_i64_0 = arith.constant 16384 : i64 + %c0_i64_1 = arith.constant 0 : i64 + %c16384_i64_2 = arith.constant 16384 : i64 + pto.mte_gm_ub %arg1, %2, %c0_i64_1, %c16384_i64_2 nburst(%c1_i64, %c16384_i64, %c16384_i64_0) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + pto.set_flag[, , ] + pto.wait_flag[, , ] + pto.section.simt<<<128, 1, 1>>> { + %3 = pto.get_tid_x : i32 + %4 = pto.get_tid_y : i32 + %5 = pto.get_tid_z : i32 + %6 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32 = arith.constant 2 : i32 + %7 = arith.muli %3, %c2_i32 : i32 + %c0_i32 = arith.constant 0 : i32 + %8 = arith.addi %c0_i32, %7 : i32 + %9 = arith.index_cast %8 : i32 to index + %c4 = arith.constant 4 : index + %10 = arith.muli %9, %c4 : index + %11 = arith.index_cast %10 : index to i64 + %12 = pto.castptr %6 : !pto.ptr -> i64 + %13 = llvm.inttoptr %12 : i64 to !llvm.ptr<6> + %14 = llvm.getelementptr %13[%11] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %15 = llvm.load %14 : !llvm.ptr<6> -> vector<2xf32> + %c0_3 = arith.constant 0 : index + %c4_4 = arith.constant 4 : index + %16 = arith.muli %c0_3, %c4_4 : index + %17 = arith.index_cast %16 : index to i64 + %18 = llvm.getelementptr %1[%17] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %15, %18 : vector<2xf32>, !llvm.ptr + %19 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_5 = arith.constant 2 : i32 + %20 = arith.muli %3, %c2_i32_5 : i32 + %c256_i32 = arith.constant 256 : i32 + %21 = arith.addi %c256_i32, %20 : i32 + %22 = arith.index_cast %21 : i32 to index + %c4_6 = arith.constant 4 : index + %23 = arith.muli %22, %c4_6 : index + %24 = arith.index_cast %23 : index to i64 + %25 = pto.castptr %19 : !pto.ptr -> i64 + %26 = llvm.inttoptr %25 : i64 to !llvm.ptr<6> + %27 = llvm.getelementptr %26[%24] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %28 = llvm.load %27 : !llvm.ptr<6> -> vector<2xf32> + %c2 = arith.constant 2 : index + %c4_7 = arith.constant 4 : index + %29 = arith.muli %c2, %c4_7 : index + %30 = arith.index_cast %29 : index to i64 + %31 = llvm.getelementptr %1[%30] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %28, %31 : vector<2xf32>, !llvm.ptr + %32 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_8 = arith.constant 2 : i32 + %33 = arith.muli %3, %c2_i32_8 : i32 + %c512_i32 = arith.constant 512 : i32 + %34 = arith.addi %c512_i32, %33 : i32 + %35 = arith.index_cast %34 : i32 to index + %c4_9 = arith.constant 4 : index + %36 = arith.muli %35, %c4_9 : index + %37 = arith.index_cast %36 : index to i64 + %38 = pto.castptr %32 : !pto.ptr -> i64 + %39 = llvm.inttoptr %38 : i64 to !llvm.ptr<6> + %40 = llvm.getelementptr %39[%37] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %41 = llvm.load %40 : !llvm.ptr<6> -> vector<2xf32> + %c4_10 = arith.constant 4 : index + %c4_11 = arith.constant 4 : index + %42 = arith.muli %c4_10, %c4_11 : index + %43 = arith.index_cast %42 : index to i64 + %44 = llvm.getelementptr %1[%43] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %41, %44 : vector<2xf32>, !llvm.ptr + %45 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_12 = arith.constant 2 : i32 + %46 = arith.muli %3, %c2_i32_12 : i32 + %c768_i32 = arith.constant 768 : i32 + %47 = arith.addi %c768_i32, %46 : i32 + %48 = arith.index_cast %47 : i32 to index + %c4_13 = arith.constant 4 : index + %49 = arith.muli %48, %c4_13 : index + %50 = arith.index_cast %49 : index to i64 + %51 = pto.castptr %45 : !pto.ptr -> i64 + %52 = llvm.inttoptr %51 : i64 to !llvm.ptr<6> + %53 = llvm.getelementptr %52[%50] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %54 = llvm.load %53 : !llvm.ptr<6> -> vector<2xf32> + %c6 = arith.constant 6 : index + %c4_14 = arith.constant 4 : index + %55 = arith.muli %c6, %c4_14 : index + %56 = arith.index_cast %55 : index to i64 + %57 = llvm.getelementptr %1[%56] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %54, %57 : vector<2xf32>, !llvm.ptr + %58 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_15 = arith.constant 2 : i32 + %59 = arith.muli %3, %c2_i32_15 : i32 + %c1024_i32 = arith.constant 1024 : i32 + %60 = arith.addi %c1024_i32, %59 : i32 + %61 = arith.index_cast %60 : i32 to index + %c4_16 = arith.constant 4 : index + %62 = arith.muli %61, %c4_16 : index + %63 = arith.index_cast %62 : index to i64 + %64 = pto.castptr %58 : !pto.ptr -> i64 + %65 = llvm.inttoptr %64 : i64 to !llvm.ptr<6> + %66 = llvm.getelementptr %65[%63] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %67 = llvm.load %66 : !llvm.ptr<6> -> vector<2xf32> + %c8 = arith.constant 8 : index + %c4_17 = arith.constant 4 : index + %68 = arith.muli %c8, %c4_17 : index + %69 = arith.index_cast %68 : index to i64 + %70 = llvm.getelementptr %1[%69] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %67, %70 : vector<2xf32>, !llvm.ptr + %71 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_18 = arith.constant 2 : i32 + %72 = arith.muli %3, %c2_i32_18 : i32 + %c1280_i32 = arith.constant 1280 : i32 + %73 = arith.addi %c1280_i32, %72 : i32 + %74 = arith.index_cast %73 : i32 to index + %c4_19 = arith.constant 4 : index + %75 = arith.muli %74, %c4_19 : index + %76 = arith.index_cast %75 : index to i64 + %77 = pto.castptr %71 : !pto.ptr -> i64 + %78 = llvm.inttoptr %77 : i64 to !llvm.ptr<6> + %79 = llvm.getelementptr %78[%76] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %80 = llvm.load %79 : !llvm.ptr<6> -> vector<2xf32> + %c10 = arith.constant 10 : index + %c4_20 = arith.constant 4 : index + %81 = arith.muli %c10, %c4_20 : index + %82 = arith.index_cast %81 : index to i64 + %83 = llvm.getelementptr %1[%82] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %80, %83 : vector<2xf32>, !llvm.ptr + %84 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_21 = arith.constant 2 : i32 + %85 = arith.muli %3, %c2_i32_21 : i32 + %c1536_i32 = arith.constant 1536 : i32 + %86 = arith.addi %c1536_i32, %85 : i32 + %87 = arith.index_cast %86 : i32 to index + %c4_22 = arith.constant 4 : index + %88 = arith.muli %87, %c4_22 : index + %89 = arith.index_cast %88 : index to i64 + %90 = pto.castptr %84 : !pto.ptr -> i64 + %91 = llvm.inttoptr %90 : i64 to !llvm.ptr<6> + %92 = llvm.getelementptr %91[%89] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %93 = llvm.load %92 : !llvm.ptr<6> -> vector<2xf32> + %c12 = arith.constant 12 : index + %c4_23 = arith.constant 4 : index + %94 = arith.muli %c12, %c4_23 : index + %95 = arith.index_cast %94 : index to i64 + %96 = llvm.getelementptr %1[%95] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %93, %96 : vector<2xf32>, !llvm.ptr + %97 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_24 = arith.constant 2 : i32 + %98 = arith.muli %3, %c2_i32_24 : i32 + %c1792_i32 = arith.constant 1792 : i32 + %99 = arith.addi %c1792_i32, %98 : i32 + %100 = arith.index_cast %99 : i32 to index + %c4_25 = arith.constant 4 : index + %101 = arith.muli %100, %c4_25 : index + %102 = arith.index_cast %101 : index to i64 + %103 = pto.castptr %97 : !pto.ptr -> i64 + %104 = llvm.inttoptr %103 : i64 to !llvm.ptr<6> + %105 = llvm.getelementptr %104[%102] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %106 = llvm.load %105 : !llvm.ptr<6> -> vector<2xf32> + %c14 = arith.constant 14 : index + %c4_26 = arith.constant 4 : index + %107 = arith.muli %c14, %c4_26 : index + %108 = arith.index_cast %107 : index to i64 + %109 = llvm.getelementptr %1[%108] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %106, %109 : vector<2xf32>, !llvm.ptr + %110 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_27 = arith.constant 2 : i32 + %111 = arith.muli %3, %c2_i32_27 : i32 + %c2048_i32 = arith.constant 2048 : i32 + %112 = arith.addi %c2048_i32, %111 : i32 + %113 = arith.index_cast %112 : i32 to index + %c4_28 = arith.constant 4 : index + %114 = arith.muli %113, %c4_28 : index + %115 = arith.index_cast %114 : index to i64 + %116 = pto.castptr %110 : !pto.ptr -> i64 + %117 = llvm.inttoptr %116 : i64 to !llvm.ptr<6> + %118 = llvm.getelementptr %117[%115] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %119 = llvm.load %118 : !llvm.ptr<6> -> vector<2xf32> + %c16 = arith.constant 16 : index + %c4_29 = arith.constant 4 : index + %120 = arith.muli %c16, %c4_29 : index + %121 = arith.index_cast %120 : index to i64 + %122 = llvm.getelementptr %1[%121] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %119, %122 : vector<2xf32>, !llvm.ptr + %123 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_30 = arith.constant 2 : i32 + %124 = arith.muli %3, %c2_i32_30 : i32 + %c2304_i32 = arith.constant 2304 : i32 + %125 = arith.addi %c2304_i32, %124 : i32 + %126 = arith.index_cast %125 : i32 to index + %c4_31 = arith.constant 4 : index + %127 = arith.muli %126, %c4_31 : index + %128 = arith.index_cast %127 : index to i64 + %129 = pto.castptr %123 : !pto.ptr -> i64 + %130 = llvm.inttoptr %129 : i64 to !llvm.ptr<6> + %131 = llvm.getelementptr %130[%128] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %132 = llvm.load %131 : !llvm.ptr<6> -> vector<2xf32> + %c18 = arith.constant 18 : index + %c4_32 = arith.constant 4 : index + %133 = arith.muli %c18, %c4_32 : index + %134 = arith.index_cast %133 : index to i64 + %135 = llvm.getelementptr %1[%134] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %132, %135 : vector<2xf32>, !llvm.ptr + %136 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_33 = arith.constant 2 : i32 + %137 = arith.muli %3, %c2_i32_33 : i32 + %c2560_i32 = arith.constant 2560 : i32 + %138 = arith.addi %c2560_i32, %137 : i32 + %139 = arith.index_cast %138 : i32 to index + %c4_34 = arith.constant 4 : index + %140 = arith.muli %139, %c4_34 : index + %141 = arith.index_cast %140 : index to i64 + %142 = pto.castptr %136 : !pto.ptr -> i64 + %143 = llvm.inttoptr %142 : i64 to !llvm.ptr<6> + %144 = llvm.getelementptr %143[%141] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %145 = llvm.load %144 : !llvm.ptr<6> -> vector<2xf32> + %c20 = arith.constant 20 : index + %c4_35 = arith.constant 4 : index + %146 = arith.muli %c20, %c4_35 : index + %147 = arith.index_cast %146 : index to i64 + %148 = llvm.getelementptr %1[%147] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %145, %148 : vector<2xf32>, !llvm.ptr + %149 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_36 = arith.constant 2 : i32 + %150 = arith.muli %3, %c2_i32_36 : i32 + %c2816_i32 = arith.constant 2816 : i32 + %151 = arith.addi %c2816_i32, %150 : i32 + %152 = arith.index_cast %151 : i32 to index + %c4_37 = arith.constant 4 : index + %153 = arith.muli %152, %c4_37 : index + %154 = arith.index_cast %153 : index to i64 + %155 = pto.castptr %149 : !pto.ptr -> i64 + %156 = llvm.inttoptr %155 : i64 to !llvm.ptr<6> + %157 = llvm.getelementptr %156[%154] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %158 = llvm.load %157 : !llvm.ptr<6> -> vector<2xf32> + %c22 = arith.constant 22 : index + %c4_38 = arith.constant 4 : index + %159 = arith.muli %c22, %c4_38 : index + %160 = arith.index_cast %159 : index to i64 + %161 = llvm.getelementptr %1[%160] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %158, %161 : vector<2xf32>, !llvm.ptr + %162 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_39 = arith.constant 2 : i32 + %163 = arith.muli %3, %c2_i32_39 : i32 + %c3072_i32 = arith.constant 3072 : i32 + %164 = arith.addi %c3072_i32, %163 : i32 + %165 = arith.index_cast %164 : i32 to index + %c4_40 = arith.constant 4 : index + %166 = arith.muli %165, %c4_40 : index + %167 = arith.index_cast %166 : index to i64 + %168 = pto.castptr %162 : !pto.ptr -> i64 + %169 = llvm.inttoptr %168 : i64 to !llvm.ptr<6> + %170 = llvm.getelementptr %169[%167] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %171 = llvm.load %170 : !llvm.ptr<6> -> vector<2xf32> + %c24 = arith.constant 24 : index + %c4_41 = arith.constant 4 : index + %172 = arith.muli %c24, %c4_41 : index + %173 = arith.index_cast %172 : index to i64 + %174 = llvm.getelementptr %1[%173] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %171, %174 : vector<2xf32>, !llvm.ptr + %175 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_42 = arith.constant 2 : i32 + %176 = arith.muli %3, %c2_i32_42 : i32 + %c3328_i32 = arith.constant 3328 : i32 + %177 = arith.addi %c3328_i32, %176 : i32 + %178 = arith.index_cast %177 : i32 to index + %c4_43 = arith.constant 4 : index + %179 = arith.muli %178, %c4_43 : index + %180 = arith.index_cast %179 : index to i64 + %181 = pto.castptr %175 : !pto.ptr -> i64 + %182 = llvm.inttoptr %181 : i64 to !llvm.ptr<6> + %183 = llvm.getelementptr %182[%180] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %184 = llvm.load %183 : !llvm.ptr<6> -> vector<2xf32> + %c26 = arith.constant 26 : index + %c4_44 = arith.constant 4 : index + %185 = arith.muli %c26, %c4_44 : index + %186 = arith.index_cast %185 : index to i64 + %187 = llvm.getelementptr %1[%186] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %184, %187 : vector<2xf32>, !llvm.ptr + %188 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_45 = arith.constant 2 : i32 + %189 = arith.muli %3, %c2_i32_45 : i32 + %c3584_i32 = arith.constant 3584 : i32 + %190 = arith.addi %c3584_i32, %189 : i32 + %191 = arith.index_cast %190 : i32 to index + %c4_46 = arith.constant 4 : index + %192 = arith.muli %191, %c4_46 : index + %193 = arith.index_cast %192 : index to i64 + %194 = pto.castptr %188 : !pto.ptr -> i64 + %195 = llvm.inttoptr %194 : i64 to !llvm.ptr<6> + %196 = llvm.getelementptr %195[%193] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %197 = llvm.load %196 : !llvm.ptr<6> -> vector<2xf32> + %c28 = arith.constant 28 : index + %c4_47 = arith.constant 4 : index + %198 = arith.muli %c28, %c4_47 : index + %199 = arith.index_cast %198 : index to i64 + %200 = llvm.getelementptr %1[%199] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %197, %200 : vector<2xf32>, !llvm.ptr + %201 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_48 = arith.constant 2 : i32 + %202 = arith.muli %3, %c2_i32_48 : i32 + %c3840_i32 = arith.constant 3840 : i32 + %203 = arith.addi %c3840_i32, %202 : i32 + %204 = arith.index_cast %203 : i32 to index + %c4_49 = arith.constant 4 : index + %205 = arith.muli %204, %c4_49 : index + %206 = arith.index_cast %205 : index to i64 + %207 = pto.castptr %201 : !pto.ptr -> i64 + %208 = llvm.inttoptr %207 : i64 to !llvm.ptr<6> + %209 = llvm.getelementptr %208[%206] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %210 = llvm.load %209 : !llvm.ptr<6> -> vector<2xf32> + %c30 = arith.constant 30 : index + %c4_50 = arith.constant 4 : index + %211 = arith.muli %c30, %c4_50 : index + %212 = arith.index_cast %211 : index to i64 + %213 = llvm.getelementptr %1[%212] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %210, %213 : vector<2xf32>, !llvm.ptr + } + %c0 = arith.constant 0 : index + %c64 = arith.constant 64 : index + %c1 = arith.constant 1 : index + scf.for %arg5 = %c0 to %c64 step %c1 { + %c1_3 = arith.constant 1 : index + %3 = arith.andi %arg5, %c1_3 : index + pto.wait_flag_dyn[, , %3] + %c262144 = arith.constant 262144 : index + %4 = arith.muli %arg5, %c262144 : index + %5 = pto.get_block_idx + %c4096_i64 = arith.constant 4096 : i64 + %6 = arith.muli %5, %c4096_i64 : i64 + %7 = arith.index_cast %6 : i64 to index + %8 = arith.addi %4, %7 : index + %9 = pto.addptr %arg2, %8 : -> + %10 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_4 = arith.constant 1 : index + %11 = arith.andi %arg5, %c1_4 : index + %c4096 = arith.constant 4096 : index + %12 = arith.muli %11, %c4096 : index + %c4096_5 = arith.constant 4096 : index + %13 = arith.addi %12, %c4096_5 : index + %14 = pto.addptr %10, %13 : -> + %c1_i64_6 = arith.constant 1 : i64 + %c16384_i64_7 = arith.constant 16384 : i64 + %c16384_i64_8 = arith.constant 16384 : i64 + %c0_i64_9 = arith.constant 0 : i64 + %c16384_i64_10 = arith.constant 16384 : i64 + pto.mte_gm_ub %9, %14, %c0_i64_9, %c16384_i64_10 nburst(%c1_i64_6, %c16384_i64_7, %c16384_i64_8) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + %c1_11 = arith.constant 1 : index + %15 = arith.andi %arg5, %c1_11 : index + pto.set_flag_dyn[, , %15] + %c1_12 = arith.constant 1 : index + %16 = arith.andi %arg5, %c1_12 : index + pto.wait_flag_dyn[, , %16] + %c1_13 = arith.constant 1 : index + %17 = arith.andi %arg5, %c1_13 : index + pto.wait_flag_dyn[, , %17] + pto.section.simt<<<128, 1, 1>>> { + %c32_i32_33 = arith.constant 32 : i32 + %42 = llvm.alloca %c32_i32_33 x f32 : (i32) -> !llvm.ptr + %c1_i32 = arith.constant 1 : i32 + %43 = llvm.alloca %c1_i32 x f32 : (i32) -> !llvm.ptr + %44 = pto.get_tid_x : i32 + %45 = pto.get_tid_y : i32 + %46 = pto.get_tid_z : i32 + %47 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_34 = arith.constant 1 : index + %48 = arith.andi %arg5, %c1_34 : index + %c4096_35 = arith.constant 4096 : index + %49 = arith.muli %48, %c4096_35 : index + %c0_36 = arith.constant 0 : index + %50 = arith.addi %49, %c0_36 : index + %c2_i32 = arith.constant 2 : i32 + %51 = arith.muli %44, %c2_i32 : i32 + %52 = arith.index_cast %51 : i32 to index + %53 = arith.addi %50, %52 : index + %c4096_37 = arith.constant 4096 : index + %54 = arith.addi %53, %c4096_37 : index + %c4 = arith.constant 4 : index + %55 = arith.muli %54, %c4 : index + %56 = arith.index_cast %55 : index to i64 + %57 = pto.castptr %47 : !pto.ptr -> i64 + %58 = llvm.inttoptr %57 : i64 to !llvm.ptr<6> + %59 = llvm.getelementptr %58[%56] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %60 = llvm.load %59 : !llvm.ptr<6> -> vector<2xf32> + %c0_38 = arith.constant 0 : index + %c4_39 = arith.constant 4 : index + %61 = arith.muli %c0_38, %c4_39 : index + %62 = arith.index_cast %61 : index to i64 + %63 = llvm.getelementptr %42[%62] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %60, %63 : vector<2xf32>, !llvm.ptr + %64 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_40 = arith.constant 1 : index + %65 = arith.andi %arg5, %c1_40 : index + %c4096_41 = arith.constant 4096 : index + %66 = arith.muli %65, %c4096_41 : index + %c256 = arith.constant 256 : index + %67 = arith.addi %66, %c256 : index + %c2_i32_42 = arith.constant 2 : i32 + %68 = arith.muli %44, %c2_i32_42 : i32 + %69 = arith.index_cast %68 : i32 to index + %70 = arith.addi %67, %69 : index + %c4096_43 = arith.constant 4096 : index + %71 = arith.addi %70, %c4096_43 : index + %c4_44 = arith.constant 4 : index + %72 = arith.muli %71, %c4_44 : index + %73 = arith.index_cast %72 : index to i64 + %74 = pto.castptr %64 : !pto.ptr -> i64 + %75 = llvm.inttoptr %74 : i64 to !llvm.ptr<6> + %76 = llvm.getelementptr %75[%73] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %77 = llvm.load %76 : !llvm.ptr<6> -> vector<2xf32> + %c2 = arith.constant 2 : index + %c4_45 = arith.constant 4 : index + %78 = arith.muli %c2, %c4_45 : index + %79 = arith.index_cast %78 : index to i64 + %80 = llvm.getelementptr %42[%79] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %77, %80 : vector<2xf32>, !llvm.ptr + %81 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_46 = arith.constant 1 : index + %82 = arith.andi %arg5, %c1_46 : index + %c4096_47 = arith.constant 4096 : index + %83 = arith.muli %82, %c4096_47 : index + %c512 = arith.constant 512 : index + %84 = arith.addi %83, %c512 : index + %c2_i32_48 = arith.constant 2 : i32 + %85 = arith.muli %44, %c2_i32_48 : i32 + %86 = arith.index_cast %85 : i32 to index + %87 = arith.addi %84, %86 : index + %c4096_49 = arith.constant 4096 : index + %88 = arith.addi %87, %c4096_49 : index + %c4_50 = arith.constant 4 : index + %89 = arith.muli %88, %c4_50 : index + %90 = arith.index_cast %89 : index to i64 + %91 = pto.castptr %81 : !pto.ptr -> i64 + %92 = llvm.inttoptr %91 : i64 to !llvm.ptr<6> + %93 = llvm.getelementptr %92[%90] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %94 = llvm.load %93 : !llvm.ptr<6> -> vector<2xf32> + %c4_51 = arith.constant 4 : index + %c4_52 = arith.constant 4 : index + %95 = arith.muli %c4_51, %c4_52 : index + %96 = arith.index_cast %95 : index to i64 + %97 = llvm.getelementptr %42[%96] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %94, %97 : vector<2xf32>, !llvm.ptr + %98 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_53 = arith.constant 1 : index + %99 = arith.andi %arg5, %c1_53 : index + %c4096_54 = arith.constant 4096 : index + %100 = arith.muli %99, %c4096_54 : index + %c768 = arith.constant 768 : index + %101 = arith.addi %100, %c768 : index + %c2_i32_55 = arith.constant 2 : i32 + %102 = arith.muli %44, %c2_i32_55 : i32 + %103 = arith.index_cast %102 : i32 to index + %104 = arith.addi %101, %103 : index + %c4096_56 = arith.constant 4096 : index + %105 = arith.addi %104, %c4096_56 : index + %c4_57 = arith.constant 4 : index + %106 = arith.muli %105, %c4_57 : index + %107 = arith.index_cast %106 : index to i64 + %108 = pto.castptr %98 : !pto.ptr -> i64 + %109 = llvm.inttoptr %108 : i64 to !llvm.ptr<6> + %110 = llvm.getelementptr %109[%107] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %111 = llvm.load %110 : !llvm.ptr<6> -> vector<2xf32> + %c6 = arith.constant 6 : index + %c4_58 = arith.constant 4 : index + %112 = arith.muli %c6, %c4_58 : index + %113 = arith.index_cast %112 : index to i64 + %114 = llvm.getelementptr %42[%113] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %111, %114 : vector<2xf32>, !llvm.ptr + %115 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_59 = arith.constant 1 : index + %116 = arith.andi %arg5, %c1_59 : index + %c4096_60 = arith.constant 4096 : index + %117 = arith.muli %116, %c4096_60 : index + %c1024 = arith.constant 1024 : index + %118 = arith.addi %117, %c1024 : index + %c2_i32_61 = arith.constant 2 : i32 + %119 = arith.muli %44, %c2_i32_61 : i32 + %120 = arith.index_cast %119 : i32 to index + %121 = arith.addi %118, %120 : index + %c4096_62 = arith.constant 4096 : index + %122 = arith.addi %121, %c4096_62 : index + %c4_63 = arith.constant 4 : index + %123 = arith.muli %122, %c4_63 : index + %124 = arith.index_cast %123 : index to i64 + %125 = pto.castptr %115 : !pto.ptr -> i64 + %126 = llvm.inttoptr %125 : i64 to !llvm.ptr<6> + %127 = llvm.getelementptr %126[%124] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %128 = llvm.load %127 : !llvm.ptr<6> -> vector<2xf32> + %c8_64 = arith.constant 8 : index + %c4_65 = arith.constant 4 : index + %129 = arith.muli %c8_64, %c4_65 : index + %130 = arith.index_cast %129 : index to i64 + %131 = llvm.getelementptr %42[%130] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %128, %131 : vector<2xf32>, !llvm.ptr + %132 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_66 = arith.constant 1 : index + %133 = arith.andi %arg5, %c1_66 : index + %c4096_67 = arith.constant 4096 : index + %134 = arith.muli %133, %c4096_67 : index + %c1280 = arith.constant 1280 : index + %135 = arith.addi %134, %c1280 : index + %c2_i32_68 = arith.constant 2 : i32 + %136 = arith.muli %44, %c2_i32_68 : i32 + %137 = arith.index_cast %136 : i32 to index + %138 = arith.addi %135, %137 : index + %c4096_69 = arith.constant 4096 : index + %139 = arith.addi %138, %c4096_69 : index + %c4_70 = arith.constant 4 : index + %140 = arith.muli %139, %c4_70 : index + %141 = arith.index_cast %140 : index to i64 + %142 = pto.castptr %132 : !pto.ptr -> i64 + %143 = llvm.inttoptr %142 : i64 to !llvm.ptr<6> + %144 = llvm.getelementptr %143[%141] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %145 = llvm.load %144 : !llvm.ptr<6> -> vector<2xf32> + %c10 = arith.constant 10 : index + %c4_71 = arith.constant 4 : index + %146 = arith.muli %c10, %c4_71 : index + %147 = arith.index_cast %146 : index to i64 + %148 = llvm.getelementptr %42[%147] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %145, %148 : vector<2xf32>, !llvm.ptr + %149 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_72 = arith.constant 1 : index + %150 = arith.andi %arg5, %c1_72 : index + %c4096_73 = arith.constant 4096 : index + %151 = arith.muli %150, %c4096_73 : index + %c1536 = arith.constant 1536 : index + %152 = arith.addi %151, %c1536 : index + %c2_i32_74 = arith.constant 2 : i32 + %153 = arith.muli %44, %c2_i32_74 : i32 + %154 = arith.index_cast %153 : i32 to index + %155 = arith.addi %152, %154 : index + %c4096_75 = arith.constant 4096 : index + %156 = arith.addi %155, %c4096_75 : index + %c4_76 = arith.constant 4 : index + %157 = arith.muli %156, %c4_76 : index + %158 = arith.index_cast %157 : index to i64 + %159 = pto.castptr %149 : !pto.ptr -> i64 + %160 = llvm.inttoptr %159 : i64 to !llvm.ptr<6> + %161 = llvm.getelementptr %160[%158] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %162 = llvm.load %161 : !llvm.ptr<6> -> vector<2xf32> + %c12 = arith.constant 12 : index + %c4_77 = arith.constant 4 : index + %163 = arith.muli %c12, %c4_77 : index + %164 = arith.index_cast %163 : index to i64 + %165 = llvm.getelementptr %42[%164] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %162, %165 : vector<2xf32>, !llvm.ptr + %166 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_78 = arith.constant 1 : index + %167 = arith.andi %arg5, %c1_78 : index + %c4096_79 = arith.constant 4096 : index + %168 = arith.muli %167, %c4096_79 : index + %c1792 = arith.constant 1792 : index + %169 = arith.addi %168, %c1792 : index + %c2_i32_80 = arith.constant 2 : i32 + %170 = arith.muli %44, %c2_i32_80 : i32 + %171 = arith.index_cast %170 : i32 to index + %172 = arith.addi %169, %171 : index + %c4096_81 = arith.constant 4096 : index + %173 = arith.addi %172, %c4096_81 : index + %c4_82 = arith.constant 4 : index + %174 = arith.muli %173, %c4_82 : index + %175 = arith.index_cast %174 : index to i64 + %176 = pto.castptr %166 : !pto.ptr -> i64 + %177 = llvm.inttoptr %176 : i64 to !llvm.ptr<6> + %178 = llvm.getelementptr %177[%175] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %179 = llvm.load %178 : !llvm.ptr<6> -> vector<2xf32> + %c14 = arith.constant 14 : index + %c4_83 = arith.constant 4 : index + %180 = arith.muli %c14, %c4_83 : index + %181 = arith.index_cast %180 : index to i64 + %182 = llvm.getelementptr %42[%181] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %179, %182 : vector<2xf32>, !llvm.ptr + %183 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_84 = arith.constant 1 : index + %184 = arith.andi %arg5, %c1_84 : index + %c4096_85 = arith.constant 4096 : index + %185 = arith.muli %184, %c4096_85 : index + %c2048 = arith.constant 2048 : index + %186 = arith.addi %185, %c2048 : index + %c2_i32_86 = arith.constant 2 : i32 + %187 = arith.muli %44, %c2_i32_86 : i32 + %188 = arith.index_cast %187 : i32 to index + %189 = arith.addi %186, %188 : index + %c4096_87 = arith.constant 4096 : index + %190 = arith.addi %189, %c4096_87 : index + %c4_88 = arith.constant 4 : index + %191 = arith.muli %190, %c4_88 : index + %192 = arith.index_cast %191 : index to i64 + %193 = pto.castptr %183 : !pto.ptr -> i64 + %194 = llvm.inttoptr %193 : i64 to !llvm.ptr<6> + %195 = llvm.getelementptr %194[%192] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %196 = llvm.load %195 : !llvm.ptr<6> -> vector<2xf32> + %c16 = arith.constant 16 : index + %c4_89 = arith.constant 4 : index + %197 = arith.muli %c16, %c4_89 : index + %198 = arith.index_cast %197 : index to i64 + %199 = llvm.getelementptr %42[%198] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %196, %199 : vector<2xf32>, !llvm.ptr + %200 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_90 = arith.constant 1 : index + %201 = arith.andi %arg5, %c1_90 : index + %c4096_91 = arith.constant 4096 : index + %202 = arith.muli %201, %c4096_91 : index + %c2304 = arith.constant 2304 : index + %203 = arith.addi %202, %c2304 : index + %c2_i32_92 = arith.constant 2 : i32 + %204 = arith.muli %44, %c2_i32_92 : i32 + %205 = arith.index_cast %204 : i32 to index + %206 = arith.addi %203, %205 : index + %c4096_93 = arith.constant 4096 : index + %207 = arith.addi %206, %c4096_93 : index + %c4_94 = arith.constant 4 : index + %208 = arith.muli %207, %c4_94 : index + %209 = arith.index_cast %208 : index to i64 + %210 = pto.castptr %200 : !pto.ptr -> i64 + %211 = llvm.inttoptr %210 : i64 to !llvm.ptr<6> + %212 = llvm.getelementptr %211[%209] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %213 = llvm.load %212 : !llvm.ptr<6> -> vector<2xf32> + %c18 = arith.constant 18 : index + %c4_95 = arith.constant 4 : index + %214 = arith.muli %c18, %c4_95 : index + %215 = arith.index_cast %214 : index to i64 + %216 = llvm.getelementptr %42[%215] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %213, %216 : vector<2xf32>, !llvm.ptr + %217 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_96 = arith.constant 1 : index + %218 = arith.andi %arg5, %c1_96 : index + %c4096_97 = arith.constant 4096 : index + %219 = arith.muli %218, %c4096_97 : index + %c2560 = arith.constant 2560 : index + %220 = arith.addi %219, %c2560 : index + %c2_i32_98 = arith.constant 2 : i32 + %221 = arith.muli %44, %c2_i32_98 : i32 + %222 = arith.index_cast %221 : i32 to index + %223 = arith.addi %220, %222 : index + %c4096_99 = arith.constant 4096 : index + %224 = arith.addi %223, %c4096_99 : index + %c4_100 = arith.constant 4 : index + %225 = arith.muli %224, %c4_100 : index + %226 = arith.index_cast %225 : index to i64 + %227 = pto.castptr %217 : !pto.ptr -> i64 + %228 = llvm.inttoptr %227 : i64 to !llvm.ptr<6> + %229 = llvm.getelementptr %228[%226] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %230 = llvm.load %229 : !llvm.ptr<6> -> vector<2xf32> + %c20 = arith.constant 20 : index + %c4_101 = arith.constant 4 : index + %231 = arith.muli %c20, %c4_101 : index + %232 = arith.index_cast %231 : index to i64 + %233 = llvm.getelementptr %42[%232] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %230, %233 : vector<2xf32>, !llvm.ptr + %234 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_102 = arith.constant 1 : index + %235 = arith.andi %arg5, %c1_102 : index + %c4096_103 = arith.constant 4096 : index + %236 = arith.muli %235, %c4096_103 : index + %c2816 = arith.constant 2816 : index + %237 = arith.addi %236, %c2816 : index + %c2_i32_104 = arith.constant 2 : i32 + %238 = arith.muli %44, %c2_i32_104 : i32 + %239 = arith.index_cast %238 : i32 to index + %240 = arith.addi %237, %239 : index + %c4096_105 = arith.constant 4096 : index + %241 = arith.addi %240, %c4096_105 : index + %c4_106 = arith.constant 4 : index + %242 = arith.muli %241, %c4_106 : index + %243 = arith.index_cast %242 : index to i64 + %244 = pto.castptr %234 : !pto.ptr -> i64 + %245 = llvm.inttoptr %244 : i64 to !llvm.ptr<6> + %246 = llvm.getelementptr %245[%243] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %247 = llvm.load %246 : !llvm.ptr<6> -> vector<2xf32> + %c22 = arith.constant 22 : index + %c4_107 = arith.constant 4 : index + %248 = arith.muli %c22, %c4_107 : index + %249 = arith.index_cast %248 : index to i64 + %250 = llvm.getelementptr %42[%249] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %247, %250 : vector<2xf32>, !llvm.ptr + %251 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_108 = arith.constant 1 : index + %252 = arith.andi %arg5, %c1_108 : index + %c4096_109 = arith.constant 4096 : index + %253 = arith.muli %252, %c4096_109 : index + %c3072 = arith.constant 3072 : index + %254 = arith.addi %253, %c3072 : index + %c2_i32_110 = arith.constant 2 : i32 + %255 = arith.muli %44, %c2_i32_110 : i32 + %256 = arith.index_cast %255 : i32 to index + %257 = arith.addi %254, %256 : index + %c4096_111 = arith.constant 4096 : index + %258 = arith.addi %257, %c4096_111 : index + %c4_112 = arith.constant 4 : index + %259 = arith.muli %258, %c4_112 : index + %260 = arith.index_cast %259 : index to i64 + %261 = pto.castptr %251 : !pto.ptr -> i64 + %262 = llvm.inttoptr %261 : i64 to !llvm.ptr<6> + %263 = llvm.getelementptr %262[%260] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %264 = llvm.load %263 : !llvm.ptr<6> -> vector<2xf32> + %c24 = arith.constant 24 : index + %c4_113 = arith.constant 4 : index + %265 = arith.muli %c24, %c4_113 : index + %266 = arith.index_cast %265 : index to i64 + %267 = llvm.getelementptr %42[%266] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %264, %267 : vector<2xf32>, !llvm.ptr + %268 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_114 = arith.constant 1 : index + %269 = arith.andi %arg5, %c1_114 : index + %c4096_115 = arith.constant 4096 : index + %270 = arith.muli %269, %c4096_115 : index + %c3328 = arith.constant 3328 : index + %271 = arith.addi %270, %c3328 : index + %c2_i32_116 = arith.constant 2 : i32 + %272 = arith.muli %44, %c2_i32_116 : i32 + %273 = arith.index_cast %272 : i32 to index + %274 = arith.addi %271, %273 : index + %c4096_117 = arith.constant 4096 : index + %275 = arith.addi %274, %c4096_117 : index + %c4_118 = arith.constant 4 : index + %276 = arith.muli %275, %c4_118 : index + %277 = arith.index_cast %276 : index to i64 + %278 = pto.castptr %268 : !pto.ptr -> i64 + %279 = llvm.inttoptr %278 : i64 to !llvm.ptr<6> + %280 = llvm.getelementptr %279[%277] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %281 = llvm.load %280 : !llvm.ptr<6> -> vector<2xf32> + %c26 = arith.constant 26 : index + %c4_119 = arith.constant 4 : index + %282 = arith.muli %c26, %c4_119 : index + %283 = arith.index_cast %282 : index to i64 + %284 = llvm.getelementptr %42[%283] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %281, %284 : vector<2xf32>, !llvm.ptr + %285 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_120 = arith.constant 1 : index + %286 = arith.andi %arg5, %c1_120 : index + %c4096_121 = arith.constant 4096 : index + %287 = arith.muli %286, %c4096_121 : index + %c3584 = arith.constant 3584 : index + %288 = arith.addi %287, %c3584 : index + %c2_i32_122 = arith.constant 2 : i32 + %289 = arith.muli %44, %c2_i32_122 : i32 + %290 = arith.index_cast %289 : i32 to index + %291 = arith.addi %288, %290 : index + %c4096_123 = arith.constant 4096 : index + %292 = arith.addi %291, %c4096_123 : index + %c4_124 = arith.constant 4 : index + %293 = arith.muli %292, %c4_124 : index + %294 = arith.index_cast %293 : index to i64 + %295 = pto.castptr %285 : !pto.ptr -> i64 + %296 = llvm.inttoptr %295 : i64 to !llvm.ptr<6> + %297 = llvm.getelementptr %296[%294] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %298 = llvm.load %297 : !llvm.ptr<6> -> vector<2xf32> + %c28 = arith.constant 28 : index + %c4_125 = arith.constant 4 : index + %299 = arith.muli %c28, %c4_125 : index + %300 = arith.index_cast %299 : index to i64 + %301 = llvm.getelementptr %42[%300] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %298, %301 : vector<2xf32>, !llvm.ptr + %302 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_126 = arith.constant 1 : index + %303 = arith.andi %arg5, %c1_126 : index + %c4096_127 = arith.constant 4096 : index + %304 = arith.muli %303, %c4096_127 : index + %c3840 = arith.constant 3840 : index + %305 = arith.addi %304, %c3840 : index + %c2_i32_128 = arith.constant 2 : i32 + %306 = arith.muli %44, %c2_i32_128 : i32 + %307 = arith.index_cast %306 : i32 to index + %308 = arith.addi %305, %307 : index + %c4096_129 = arith.constant 4096 : index + %309 = arith.addi %308, %c4096_129 : index + %c4_130 = arith.constant 4 : index + %310 = arith.muli %309, %c4_130 : index + %311 = arith.index_cast %310 : index to i64 + %312 = pto.castptr %302 : !pto.ptr -> i64 + %313 = llvm.inttoptr %312 : i64 to !llvm.ptr<6> + %314 = llvm.getelementptr %313[%311] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %315 = llvm.load %314 : !llvm.ptr<6> -> vector<2xf32> + %c30 = arith.constant 30 : index + %c4_131 = arith.constant 4 : index + %316 = arith.muli %c30, %c4_131 : index + %317 = arith.index_cast %316 : index to i64 + %318 = llvm.getelementptr %42[%317] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %315, %318 : vector<2xf32>, !llvm.ptr + %c0_132 = arith.constant 0 : index + %cst = arith.constant 0.000000e+00 : f32 + %c4_133 = arith.constant 4 : index + %319 = arith.muli %c0_132, %c4_133 : index + %320 = arith.index_cast %319 : index to i64 + %321 = llvm.getelementptr %43[%320] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %cst, %321 : f32, !llvm.ptr + %c0_134 = arith.constant 0 : index + %c4_135 = arith.constant 4 : index + %322 = arith.muli %c0_134, %c4_135 : index + %323 = arith.index_cast %322 : index to i64 + %324 = llvm.getelementptr %43[%323] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %325 = llvm.load %324 : !llvm.ptr -> f32 + %c0_136 = arith.constant 0 : index + %c4_137 = arith.constant 4 : index + %326 = arith.muli %c0_136, %c4_137 : index + %327 = arith.index_cast %326 : index to i64 + %328 = llvm.getelementptr %42[%327] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %329 = llvm.load %328 : !llvm.ptr -> f32 + %c0_138 = arith.constant 0 : index + %c4_139 = arith.constant 4 : index + %330 = arith.muli %c0_138, %c4_139 : index + %331 = arith.index_cast %330 : index to i64 + %332 = llvm.getelementptr %42[%331] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %333 = llvm.load %332 : !llvm.ptr -> f32 + %334 = arith.mulf %329, %333 : f32 + %335 = arith.addf %325, %334 : f32 + %c0_140 = arith.constant 0 : index + %c4_141 = arith.constant 4 : index + %336 = arith.muli %c0_140, %c4_141 : index + %337 = arith.index_cast %336 : index to i64 + %338 = llvm.getelementptr %43[%337] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %335, %338 : f32, !llvm.ptr + %c0_142 = arith.constant 0 : index + %c4_143 = arith.constant 4 : index + %339 = arith.muli %c0_142, %c4_143 : index + %340 = arith.index_cast %339 : index to i64 + %341 = llvm.getelementptr %43[%340] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %342 = llvm.load %341 : !llvm.ptr -> f32 + %c1_144 = arith.constant 1 : index + %c4_145 = arith.constant 4 : index + %343 = arith.muli %c1_144, %c4_145 : index + %344 = arith.index_cast %343 : index to i64 + %345 = llvm.getelementptr %42[%344] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %346 = llvm.load %345 : !llvm.ptr -> f32 + %c1_146 = arith.constant 1 : index + %c4_147 = arith.constant 4 : index + %347 = arith.muli %c1_146, %c4_147 : index + %348 = arith.index_cast %347 : index to i64 + %349 = llvm.getelementptr %42[%348] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %350 = llvm.load %349 : !llvm.ptr -> f32 + %351 = arith.mulf %346, %350 : f32 + %352 = arith.addf %342, %351 : f32 + %c0_148 = arith.constant 0 : index + %c4_149 = arith.constant 4 : index + %353 = arith.muli %c0_148, %c4_149 : index + %354 = arith.index_cast %353 : index to i64 + %355 = llvm.getelementptr %43[%354] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %352, %355 : f32, !llvm.ptr + %c0_150 = arith.constant 0 : index + %c4_151 = arith.constant 4 : index + %356 = arith.muli %c0_150, %c4_151 : index + %357 = arith.index_cast %356 : index to i64 + %358 = llvm.getelementptr %43[%357] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %359 = llvm.load %358 : !llvm.ptr -> f32 + %c2_152 = arith.constant 2 : index + %c4_153 = arith.constant 4 : index + %360 = arith.muli %c2_152, %c4_153 : index + %361 = arith.index_cast %360 : index to i64 + %362 = llvm.getelementptr %42[%361] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %363 = llvm.load %362 : !llvm.ptr -> f32 + %c2_154 = arith.constant 2 : index + %c4_155 = arith.constant 4 : index + %364 = arith.muli %c2_154, %c4_155 : index + %365 = arith.index_cast %364 : index to i64 + %366 = llvm.getelementptr %42[%365] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %367 = llvm.load %366 : !llvm.ptr -> f32 + %368 = arith.mulf %363, %367 : f32 + %369 = arith.addf %359, %368 : f32 + %c0_156 = arith.constant 0 : index + %c4_157 = arith.constant 4 : index + %370 = arith.muli %c0_156, %c4_157 : index + %371 = arith.index_cast %370 : index to i64 + %372 = llvm.getelementptr %43[%371] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %369, %372 : f32, !llvm.ptr + %c0_158 = arith.constant 0 : index + %c4_159 = arith.constant 4 : index + %373 = arith.muli %c0_158, %c4_159 : index + %374 = arith.index_cast %373 : index to i64 + %375 = llvm.getelementptr %43[%374] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %376 = llvm.load %375 : !llvm.ptr -> f32 + %c3 = arith.constant 3 : index + %c4_160 = arith.constant 4 : index + %377 = arith.muli %c3, %c4_160 : index + %378 = arith.index_cast %377 : index to i64 + %379 = llvm.getelementptr %42[%378] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %380 = llvm.load %379 : !llvm.ptr -> f32 + %c3_161 = arith.constant 3 : index + %c4_162 = arith.constant 4 : index + %381 = arith.muli %c3_161, %c4_162 : index + %382 = arith.index_cast %381 : index to i64 + %383 = llvm.getelementptr %42[%382] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %384 = llvm.load %383 : !llvm.ptr -> f32 + %385 = arith.mulf %380, %384 : f32 + %386 = arith.addf %376, %385 : f32 + %c0_163 = arith.constant 0 : index + %c4_164 = arith.constant 4 : index + %387 = arith.muli %c0_163, %c4_164 : index + %388 = arith.index_cast %387 : index to i64 + %389 = llvm.getelementptr %43[%388] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %386, %389 : f32, !llvm.ptr + %c0_165 = arith.constant 0 : index + %c4_166 = arith.constant 4 : index + %390 = arith.muli %c0_165, %c4_166 : index + %391 = arith.index_cast %390 : index to i64 + %392 = llvm.getelementptr %43[%391] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %393 = llvm.load %392 : !llvm.ptr -> f32 + %c4_167 = arith.constant 4 : index + %c4_168 = arith.constant 4 : index + %394 = arith.muli %c4_167, %c4_168 : index + %395 = arith.index_cast %394 : index to i64 + %396 = llvm.getelementptr %42[%395] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %397 = llvm.load %396 : !llvm.ptr -> f32 + %c4_169 = arith.constant 4 : index + %c4_170 = arith.constant 4 : index + %398 = arith.muli %c4_169, %c4_170 : index + %399 = arith.index_cast %398 : index to i64 + %400 = llvm.getelementptr %42[%399] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %401 = llvm.load %400 : !llvm.ptr -> f32 + %402 = arith.mulf %397, %401 : f32 + %403 = arith.addf %393, %402 : f32 + %c0_171 = arith.constant 0 : index + %c4_172 = arith.constant 4 : index + %404 = arith.muli %c0_171, %c4_172 : index + %405 = arith.index_cast %404 : index to i64 + %406 = llvm.getelementptr %43[%405] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %403, %406 : f32, !llvm.ptr + %c0_173 = arith.constant 0 : index + %c4_174 = arith.constant 4 : index + %407 = arith.muli %c0_173, %c4_174 : index + %408 = arith.index_cast %407 : index to i64 + %409 = llvm.getelementptr %43[%408] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %410 = llvm.load %409 : !llvm.ptr -> f32 + %c5 = arith.constant 5 : index + %c4_175 = arith.constant 4 : index + %411 = arith.muli %c5, %c4_175 : index + %412 = arith.index_cast %411 : index to i64 + %413 = llvm.getelementptr %42[%412] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %414 = llvm.load %413 : !llvm.ptr -> f32 + %c5_176 = arith.constant 5 : index + %c4_177 = arith.constant 4 : index + %415 = arith.muli %c5_176, %c4_177 : index + %416 = arith.index_cast %415 : index to i64 + %417 = llvm.getelementptr %42[%416] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %418 = llvm.load %417 : !llvm.ptr -> f32 + %419 = arith.mulf %414, %418 : f32 + %420 = arith.addf %410, %419 : f32 + %c0_178 = arith.constant 0 : index + %c4_179 = arith.constant 4 : index + %421 = arith.muli %c0_178, %c4_179 : index + %422 = arith.index_cast %421 : index to i64 + %423 = llvm.getelementptr %43[%422] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %420, %423 : f32, !llvm.ptr + %c0_180 = arith.constant 0 : index + %c4_181 = arith.constant 4 : index + %424 = arith.muli %c0_180, %c4_181 : index + %425 = arith.index_cast %424 : index to i64 + %426 = llvm.getelementptr %43[%425] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %427 = llvm.load %426 : !llvm.ptr -> f32 + %c6_182 = arith.constant 6 : index + %c4_183 = arith.constant 4 : index + %428 = arith.muli %c6_182, %c4_183 : index + %429 = arith.index_cast %428 : index to i64 + %430 = llvm.getelementptr %42[%429] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %431 = llvm.load %430 : !llvm.ptr -> f32 + %c6_184 = arith.constant 6 : index + %c4_185 = arith.constant 4 : index + %432 = arith.muli %c6_184, %c4_185 : index + %433 = arith.index_cast %432 : index to i64 + %434 = llvm.getelementptr %42[%433] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %435 = llvm.load %434 : !llvm.ptr -> f32 + %436 = arith.mulf %431, %435 : f32 + %437 = arith.addf %427, %436 : f32 + %c0_186 = arith.constant 0 : index + %c4_187 = arith.constant 4 : index + %438 = arith.muli %c0_186, %c4_187 : index + %439 = arith.index_cast %438 : index to i64 + %440 = llvm.getelementptr %43[%439] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %437, %440 : f32, !llvm.ptr + %c0_188 = arith.constant 0 : index + %c4_189 = arith.constant 4 : index + %441 = arith.muli %c0_188, %c4_189 : index + %442 = arith.index_cast %441 : index to i64 + %443 = llvm.getelementptr %43[%442] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %444 = llvm.load %443 : !llvm.ptr -> f32 + %c7 = arith.constant 7 : index + %c4_190 = arith.constant 4 : index + %445 = arith.muli %c7, %c4_190 : index + %446 = arith.index_cast %445 : index to i64 + %447 = llvm.getelementptr %42[%446] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %448 = llvm.load %447 : !llvm.ptr -> f32 + %c7_191 = arith.constant 7 : index + %c4_192 = arith.constant 4 : index + %449 = arith.muli %c7_191, %c4_192 : index + %450 = arith.index_cast %449 : index to i64 + %451 = llvm.getelementptr %42[%450] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %452 = llvm.load %451 : !llvm.ptr -> f32 + %453 = arith.mulf %448, %452 : f32 + %454 = arith.addf %444, %453 : f32 + %c0_193 = arith.constant 0 : index + %c4_194 = arith.constant 4 : index + %455 = arith.muli %c0_193, %c4_194 : index + %456 = arith.index_cast %455 : index to i64 + %457 = llvm.getelementptr %43[%456] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %454, %457 : f32, !llvm.ptr + %c0_195 = arith.constant 0 : index + %c4_196 = arith.constant 4 : index + %458 = arith.muli %c0_195, %c4_196 : index + %459 = arith.index_cast %458 : index to i64 + %460 = llvm.getelementptr %43[%459] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %461 = llvm.load %460 : !llvm.ptr -> f32 + %c8_197 = arith.constant 8 : index + %c4_198 = arith.constant 4 : index + %462 = arith.muli %c8_197, %c4_198 : index + %463 = arith.index_cast %462 : index to i64 + %464 = llvm.getelementptr %42[%463] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %465 = llvm.load %464 : !llvm.ptr -> f32 + %c8_199 = arith.constant 8 : index + %c4_200 = arith.constant 4 : index + %466 = arith.muli %c8_199, %c4_200 : index + %467 = arith.index_cast %466 : index to i64 + %468 = llvm.getelementptr %42[%467] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %469 = llvm.load %468 : !llvm.ptr -> f32 + %470 = arith.mulf %465, %469 : f32 + %471 = arith.addf %461, %470 : f32 + %c0_201 = arith.constant 0 : index + %c4_202 = arith.constant 4 : index + %472 = arith.muli %c0_201, %c4_202 : index + %473 = arith.index_cast %472 : index to i64 + %474 = llvm.getelementptr %43[%473] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %471, %474 : f32, !llvm.ptr + %c0_203 = arith.constant 0 : index + %c4_204 = arith.constant 4 : index + %475 = arith.muli %c0_203, %c4_204 : index + %476 = arith.index_cast %475 : index to i64 + %477 = llvm.getelementptr %43[%476] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %478 = llvm.load %477 : !llvm.ptr -> f32 + %c9 = arith.constant 9 : index + %c4_205 = arith.constant 4 : index + %479 = arith.muli %c9, %c4_205 : index + %480 = arith.index_cast %479 : index to i64 + %481 = llvm.getelementptr %42[%480] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %482 = llvm.load %481 : !llvm.ptr -> f32 + %c9_206 = arith.constant 9 : index + %c4_207 = arith.constant 4 : index + %483 = arith.muli %c9_206, %c4_207 : index + %484 = arith.index_cast %483 : index to i64 + %485 = llvm.getelementptr %42[%484] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %486 = llvm.load %485 : !llvm.ptr -> f32 + %487 = arith.mulf %482, %486 : f32 + %488 = arith.addf %478, %487 : f32 + %c0_208 = arith.constant 0 : index + %c4_209 = arith.constant 4 : index + %489 = arith.muli %c0_208, %c4_209 : index + %490 = arith.index_cast %489 : index to i64 + %491 = llvm.getelementptr %43[%490] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %488, %491 : f32, !llvm.ptr + %c0_210 = arith.constant 0 : index + %c4_211 = arith.constant 4 : index + %492 = arith.muli %c0_210, %c4_211 : index + %493 = arith.index_cast %492 : index to i64 + %494 = llvm.getelementptr %43[%493] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %495 = llvm.load %494 : !llvm.ptr -> f32 + %c10_212 = arith.constant 10 : index + %c4_213 = arith.constant 4 : index + %496 = arith.muli %c10_212, %c4_213 : index + %497 = arith.index_cast %496 : index to i64 + %498 = llvm.getelementptr %42[%497] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %499 = llvm.load %498 : !llvm.ptr -> f32 + %c10_214 = arith.constant 10 : index + %c4_215 = arith.constant 4 : index + %500 = arith.muli %c10_214, %c4_215 : index + %501 = arith.index_cast %500 : index to i64 + %502 = llvm.getelementptr %42[%501] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %503 = llvm.load %502 : !llvm.ptr -> f32 + %504 = arith.mulf %499, %503 : f32 + %505 = arith.addf %495, %504 : f32 + %c0_216 = arith.constant 0 : index + %c4_217 = arith.constant 4 : index + %506 = arith.muli %c0_216, %c4_217 : index + %507 = arith.index_cast %506 : index to i64 + %508 = llvm.getelementptr %43[%507] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %505, %508 : f32, !llvm.ptr + %c0_218 = arith.constant 0 : index + %c4_219 = arith.constant 4 : index + %509 = arith.muli %c0_218, %c4_219 : index + %510 = arith.index_cast %509 : index to i64 + %511 = llvm.getelementptr %43[%510] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %512 = llvm.load %511 : !llvm.ptr -> f32 + %c11 = arith.constant 11 : index + %c4_220 = arith.constant 4 : index + %513 = arith.muli %c11, %c4_220 : index + %514 = arith.index_cast %513 : index to i64 + %515 = llvm.getelementptr %42[%514] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %516 = llvm.load %515 : !llvm.ptr -> f32 + %c11_221 = arith.constant 11 : index + %c4_222 = arith.constant 4 : index + %517 = arith.muli %c11_221, %c4_222 : index + %518 = arith.index_cast %517 : index to i64 + %519 = llvm.getelementptr %42[%518] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %520 = llvm.load %519 : !llvm.ptr -> f32 + %521 = arith.mulf %516, %520 : f32 + %522 = arith.addf %512, %521 : f32 + %c0_223 = arith.constant 0 : index + %c4_224 = arith.constant 4 : index + %523 = arith.muli %c0_223, %c4_224 : index + %524 = arith.index_cast %523 : index to i64 + %525 = llvm.getelementptr %43[%524] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %522, %525 : f32, !llvm.ptr + %c0_225 = arith.constant 0 : index + %c4_226 = arith.constant 4 : index + %526 = arith.muli %c0_225, %c4_226 : index + %527 = arith.index_cast %526 : index to i64 + %528 = llvm.getelementptr %43[%527] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %529 = llvm.load %528 : !llvm.ptr -> f32 + %c12_227 = arith.constant 12 : index + %c4_228 = arith.constant 4 : index + %530 = arith.muli %c12_227, %c4_228 : index + %531 = arith.index_cast %530 : index to i64 + %532 = llvm.getelementptr %42[%531] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %533 = llvm.load %532 : !llvm.ptr -> f32 + %c12_229 = arith.constant 12 : index + %c4_230 = arith.constant 4 : index + %534 = arith.muli %c12_229, %c4_230 : index + %535 = arith.index_cast %534 : index to i64 + %536 = llvm.getelementptr %42[%535] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %537 = llvm.load %536 : !llvm.ptr -> f32 + %538 = arith.mulf %533, %537 : f32 + %539 = arith.addf %529, %538 : f32 + %c0_231 = arith.constant 0 : index + %c4_232 = arith.constant 4 : index + %540 = arith.muli %c0_231, %c4_232 : index + %541 = arith.index_cast %540 : index to i64 + %542 = llvm.getelementptr %43[%541] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %539, %542 : f32, !llvm.ptr + %c0_233 = arith.constant 0 : index + %c4_234 = arith.constant 4 : index + %543 = arith.muli %c0_233, %c4_234 : index + %544 = arith.index_cast %543 : index to i64 + %545 = llvm.getelementptr %43[%544] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %546 = llvm.load %545 : !llvm.ptr -> f32 + %c13 = arith.constant 13 : index + %c4_235 = arith.constant 4 : index + %547 = arith.muli %c13, %c4_235 : index + %548 = arith.index_cast %547 : index to i64 + %549 = llvm.getelementptr %42[%548] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %550 = llvm.load %549 : !llvm.ptr -> f32 + %c13_236 = arith.constant 13 : index + %c4_237 = arith.constant 4 : index + %551 = arith.muli %c13_236, %c4_237 : index + %552 = arith.index_cast %551 : index to i64 + %553 = llvm.getelementptr %42[%552] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %554 = llvm.load %553 : !llvm.ptr -> f32 + %555 = arith.mulf %550, %554 : f32 + %556 = arith.addf %546, %555 : f32 + %c0_238 = arith.constant 0 : index + %c4_239 = arith.constant 4 : index + %557 = arith.muli %c0_238, %c4_239 : index + %558 = arith.index_cast %557 : index to i64 + %559 = llvm.getelementptr %43[%558] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %556, %559 : f32, !llvm.ptr + %c0_240 = arith.constant 0 : index + %c4_241 = arith.constant 4 : index + %560 = arith.muli %c0_240, %c4_241 : index + %561 = arith.index_cast %560 : index to i64 + %562 = llvm.getelementptr %43[%561] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %563 = llvm.load %562 : !llvm.ptr -> f32 + %c14_242 = arith.constant 14 : index + %c4_243 = arith.constant 4 : index + %564 = arith.muli %c14_242, %c4_243 : index + %565 = arith.index_cast %564 : index to i64 + %566 = llvm.getelementptr %42[%565] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %567 = llvm.load %566 : !llvm.ptr -> f32 + %c14_244 = arith.constant 14 : index + %c4_245 = arith.constant 4 : index + %568 = arith.muli %c14_244, %c4_245 : index + %569 = arith.index_cast %568 : index to i64 + %570 = llvm.getelementptr %42[%569] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %571 = llvm.load %570 : !llvm.ptr -> f32 + %572 = arith.mulf %567, %571 : f32 + %573 = arith.addf %563, %572 : f32 + %c0_246 = arith.constant 0 : index + %c4_247 = arith.constant 4 : index + %574 = arith.muli %c0_246, %c4_247 : index + %575 = arith.index_cast %574 : index to i64 + %576 = llvm.getelementptr %43[%575] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %573, %576 : f32, !llvm.ptr + %c0_248 = arith.constant 0 : index + %c4_249 = arith.constant 4 : index + %577 = arith.muli %c0_248, %c4_249 : index + %578 = arith.index_cast %577 : index to i64 + %579 = llvm.getelementptr %43[%578] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %580 = llvm.load %579 : !llvm.ptr -> f32 + %c15 = arith.constant 15 : index + %c4_250 = arith.constant 4 : index + %581 = arith.muli %c15, %c4_250 : index + %582 = arith.index_cast %581 : index to i64 + %583 = llvm.getelementptr %42[%582] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %584 = llvm.load %583 : !llvm.ptr -> f32 + %c15_251 = arith.constant 15 : index + %c4_252 = arith.constant 4 : index + %585 = arith.muli %c15_251, %c4_252 : index + %586 = arith.index_cast %585 : index to i64 + %587 = llvm.getelementptr %42[%586] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %588 = llvm.load %587 : !llvm.ptr -> f32 + %589 = arith.mulf %584, %588 : f32 + %590 = arith.addf %580, %589 : f32 + %c0_253 = arith.constant 0 : index + %c4_254 = arith.constant 4 : index + %591 = arith.muli %c0_253, %c4_254 : index + %592 = arith.index_cast %591 : index to i64 + %593 = llvm.getelementptr %43[%592] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %590, %593 : f32, !llvm.ptr + %c0_255 = arith.constant 0 : index + %c4_256 = arith.constant 4 : index + %594 = arith.muli %c0_255, %c4_256 : index + %595 = arith.index_cast %594 : index to i64 + %596 = llvm.getelementptr %43[%595] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %597 = llvm.load %596 : !llvm.ptr -> f32 + %c16_257 = arith.constant 16 : index + %c4_258 = arith.constant 4 : index + %598 = arith.muli %c16_257, %c4_258 : index + %599 = arith.index_cast %598 : index to i64 + %600 = llvm.getelementptr %42[%599] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %601 = llvm.load %600 : !llvm.ptr -> f32 + %c16_259 = arith.constant 16 : index + %c4_260 = arith.constant 4 : index + %602 = arith.muli %c16_259, %c4_260 : index + %603 = arith.index_cast %602 : index to i64 + %604 = llvm.getelementptr %42[%603] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %605 = llvm.load %604 : !llvm.ptr -> f32 + %606 = arith.mulf %601, %605 : f32 + %607 = arith.addf %597, %606 : f32 + %c0_261 = arith.constant 0 : index + %c4_262 = arith.constant 4 : index + %608 = arith.muli %c0_261, %c4_262 : index + %609 = arith.index_cast %608 : index to i64 + %610 = llvm.getelementptr %43[%609] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %607, %610 : f32, !llvm.ptr + %c0_263 = arith.constant 0 : index + %c4_264 = arith.constant 4 : index + %611 = arith.muli %c0_263, %c4_264 : index + %612 = arith.index_cast %611 : index to i64 + %613 = llvm.getelementptr %43[%612] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %614 = llvm.load %613 : !llvm.ptr -> f32 + %c17 = arith.constant 17 : index + %c4_265 = arith.constant 4 : index + %615 = arith.muli %c17, %c4_265 : index + %616 = arith.index_cast %615 : index to i64 + %617 = llvm.getelementptr %42[%616] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %618 = llvm.load %617 : !llvm.ptr -> f32 + %c17_266 = arith.constant 17 : index + %c4_267 = arith.constant 4 : index + %619 = arith.muli %c17_266, %c4_267 : index + %620 = arith.index_cast %619 : index to i64 + %621 = llvm.getelementptr %42[%620] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %622 = llvm.load %621 : !llvm.ptr -> f32 + %623 = arith.mulf %618, %622 : f32 + %624 = arith.addf %614, %623 : f32 + %c0_268 = arith.constant 0 : index + %c4_269 = arith.constant 4 : index + %625 = arith.muli %c0_268, %c4_269 : index + %626 = arith.index_cast %625 : index to i64 + %627 = llvm.getelementptr %43[%626] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %624, %627 : f32, !llvm.ptr + %c0_270 = arith.constant 0 : index + %c4_271 = arith.constant 4 : index + %628 = arith.muli %c0_270, %c4_271 : index + %629 = arith.index_cast %628 : index to i64 + %630 = llvm.getelementptr %43[%629] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %631 = llvm.load %630 : !llvm.ptr -> f32 + %c18_272 = arith.constant 18 : index + %c4_273 = arith.constant 4 : index + %632 = arith.muli %c18_272, %c4_273 : index + %633 = arith.index_cast %632 : index to i64 + %634 = llvm.getelementptr %42[%633] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %635 = llvm.load %634 : !llvm.ptr -> f32 + %c18_274 = arith.constant 18 : index + %c4_275 = arith.constant 4 : index + %636 = arith.muli %c18_274, %c4_275 : index + %637 = arith.index_cast %636 : index to i64 + %638 = llvm.getelementptr %42[%637] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %639 = llvm.load %638 : !llvm.ptr -> f32 + %640 = arith.mulf %635, %639 : f32 + %641 = arith.addf %631, %640 : f32 + %c0_276 = arith.constant 0 : index + %c4_277 = arith.constant 4 : index + %642 = arith.muli %c0_276, %c4_277 : index + %643 = arith.index_cast %642 : index to i64 + %644 = llvm.getelementptr %43[%643] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %641, %644 : f32, !llvm.ptr + %c0_278 = arith.constant 0 : index + %c4_279 = arith.constant 4 : index + %645 = arith.muli %c0_278, %c4_279 : index + %646 = arith.index_cast %645 : index to i64 + %647 = llvm.getelementptr %43[%646] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %648 = llvm.load %647 : !llvm.ptr -> f32 + %c19 = arith.constant 19 : index + %c4_280 = arith.constant 4 : index + %649 = arith.muli %c19, %c4_280 : index + %650 = arith.index_cast %649 : index to i64 + %651 = llvm.getelementptr %42[%650] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %652 = llvm.load %651 : !llvm.ptr -> f32 + %c19_281 = arith.constant 19 : index + %c4_282 = arith.constant 4 : index + %653 = arith.muli %c19_281, %c4_282 : index + %654 = arith.index_cast %653 : index to i64 + %655 = llvm.getelementptr %42[%654] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %656 = llvm.load %655 : !llvm.ptr -> f32 + %657 = arith.mulf %652, %656 : f32 + %658 = arith.addf %648, %657 : f32 + %c0_283 = arith.constant 0 : index + %c4_284 = arith.constant 4 : index + %659 = arith.muli %c0_283, %c4_284 : index + %660 = arith.index_cast %659 : index to i64 + %661 = llvm.getelementptr %43[%660] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %658, %661 : f32, !llvm.ptr + %c0_285 = arith.constant 0 : index + %c4_286 = arith.constant 4 : index + %662 = arith.muli %c0_285, %c4_286 : index + %663 = arith.index_cast %662 : index to i64 + %664 = llvm.getelementptr %43[%663] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %665 = llvm.load %664 : !llvm.ptr -> f32 + %c20_287 = arith.constant 20 : index + %c4_288 = arith.constant 4 : index + %666 = arith.muli %c20_287, %c4_288 : index + %667 = arith.index_cast %666 : index to i64 + %668 = llvm.getelementptr %42[%667] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %669 = llvm.load %668 : !llvm.ptr -> f32 + %c20_289 = arith.constant 20 : index + %c4_290 = arith.constant 4 : index + %670 = arith.muli %c20_289, %c4_290 : index + %671 = arith.index_cast %670 : index to i64 + %672 = llvm.getelementptr %42[%671] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %673 = llvm.load %672 : !llvm.ptr -> f32 + %674 = arith.mulf %669, %673 : f32 + %675 = arith.addf %665, %674 : f32 + %c0_291 = arith.constant 0 : index + %c4_292 = arith.constant 4 : index + %676 = arith.muli %c0_291, %c4_292 : index + %677 = arith.index_cast %676 : index to i64 + %678 = llvm.getelementptr %43[%677] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %675, %678 : f32, !llvm.ptr + %c0_293 = arith.constant 0 : index + %c4_294 = arith.constant 4 : index + %679 = arith.muli %c0_293, %c4_294 : index + %680 = arith.index_cast %679 : index to i64 + %681 = llvm.getelementptr %43[%680] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %682 = llvm.load %681 : !llvm.ptr -> f32 + %c21 = arith.constant 21 : index + %c4_295 = arith.constant 4 : index + %683 = arith.muli %c21, %c4_295 : index + %684 = arith.index_cast %683 : index to i64 + %685 = llvm.getelementptr %42[%684] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %686 = llvm.load %685 : !llvm.ptr -> f32 + %c21_296 = arith.constant 21 : index + %c4_297 = arith.constant 4 : index + %687 = arith.muli %c21_296, %c4_297 : index + %688 = arith.index_cast %687 : index to i64 + %689 = llvm.getelementptr %42[%688] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %690 = llvm.load %689 : !llvm.ptr -> f32 + %691 = arith.mulf %686, %690 : f32 + %692 = arith.addf %682, %691 : f32 + %c0_298 = arith.constant 0 : index + %c4_299 = arith.constant 4 : index + %693 = arith.muli %c0_298, %c4_299 : index + %694 = arith.index_cast %693 : index to i64 + %695 = llvm.getelementptr %43[%694] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %692, %695 : f32, !llvm.ptr + %c0_300 = arith.constant 0 : index + %c4_301 = arith.constant 4 : index + %696 = arith.muli %c0_300, %c4_301 : index + %697 = arith.index_cast %696 : index to i64 + %698 = llvm.getelementptr %43[%697] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %699 = llvm.load %698 : !llvm.ptr -> f32 + %c22_302 = arith.constant 22 : index + %c4_303 = arith.constant 4 : index + %700 = arith.muli %c22_302, %c4_303 : index + %701 = arith.index_cast %700 : index to i64 + %702 = llvm.getelementptr %42[%701] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %703 = llvm.load %702 : !llvm.ptr -> f32 + %c22_304 = arith.constant 22 : index + %c4_305 = arith.constant 4 : index + %704 = arith.muli %c22_304, %c4_305 : index + %705 = arith.index_cast %704 : index to i64 + %706 = llvm.getelementptr %42[%705] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %707 = llvm.load %706 : !llvm.ptr -> f32 + %708 = arith.mulf %703, %707 : f32 + %709 = arith.addf %699, %708 : f32 + %c0_306 = arith.constant 0 : index + %c4_307 = arith.constant 4 : index + %710 = arith.muli %c0_306, %c4_307 : index + %711 = arith.index_cast %710 : index to i64 + %712 = llvm.getelementptr %43[%711] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %709, %712 : f32, !llvm.ptr + %c0_308 = arith.constant 0 : index + %c4_309 = arith.constant 4 : index + %713 = arith.muli %c0_308, %c4_309 : index + %714 = arith.index_cast %713 : index to i64 + %715 = llvm.getelementptr %43[%714] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %716 = llvm.load %715 : !llvm.ptr -> f32 + %c23 = arith.constant 23 : index + %c4_310 = arith.constant 4 : index + %717 = arith.muli %c23, %c4_310 : index + %718 = arith.index_cast %717 : index to i64 + %719 = llvm.getelementptr %42[%718] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %720 = llvm.load %719 : !llvm.ptr -> f32 + %c23_311 = arith.constant 23 : index + %c4_312 = arith.constant 4 : index + %721 = arith.muli %c23_311, %c4_312 : index + %722 = arith.index_cast %721 : index to i64 + %723 = llvm.getelementptr %42[%722] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %724 = llvm.load %723 : !llvm.ptr -> f32 + %725 = arith.mulf %720, %724 : f32 + %726 = arith.addf %716, %725 : f32 + %c0_313 = arith.constant 0 : index + %c4_314 = arith.constant 4 : index + %727 = arith.muli %c0_313, %c4_314 : index + %728 = arith.index_cast %727 : index to i64 + %729 = llvm.getelementptr %43[%728] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %726, %729 : f32, !llvm.ptr + %c0_315 = arith.constant 0 : index + %c4_316 = arith.constant 4 : index + %730 = arith.muli %c0_315, %c4_316 : index + %731 = arith.index_cast %730 : index to i64 + %732 = llvm.getelementptr %43[%731] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %733 = llvm.load %732 : !llvm.ptr -> f32 + %c24_317 = arith.constant 24 : index + %c4_318 = arith.constant 4 : index + %734 = arith.muli %c24_317, %c4_318 : index + %735 = arith.index_cast %734 : index to i64 + %736 = llvm.getelementptr %42[%735] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %737 = llvm.load %736 : !llvm.ptr -> f32 + %c24_319 = arith.constant 24 : index + %c4_320 = arith.constant 4 : index + %738 = arith.muli %c24_319, %c4_320 : index + %739 = arith.index_cast %738 : index to i64 + %740 = llvm.getelementptr %42[%739] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %741 = llvm.load %740 : !llvm.ptr -> f32 + %742 = arith.mulf %737, %741 : f32 + %743 = arith.addf %733, %742 : f32 + %c0_321 = arith.constant 0 : index + %c4_322 = arith.constant 4 : index + %744 = arith.muli %c0_321, %c4_322 : index + %745 = arith.index_cast %744 : index to i64 + %746 = llvm.getelementptr %43[%745] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %743, %746 : f32, !llvm.ptr + %c0_323 = arith.constant 0 : index + %c4_324 = arith.constant 4 : index + %747 = arith.muli %c0_323, %c4_324 : index + %748 = arith.index_cast %747 : index to i64 + %749 = llvm.getelementptr %43[%748] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %750 = llvm.load %749 : !llvm.ptr -> f32 + %c25 = arith.constant 25 : index + %c4_325 = arith.constant 4 : index + %751 = arith.muli %c25, %c4_325 : index + %752 = arith.index_cast %751 : index to i64 + %753 = llvm.getelementptr %42[%752] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %754 = llvm.load %753 : !llvm.ptr -> f32 + %c25_326 = arith.constant 25 : index + %c4_327 = arith.constant 4 : index + %755 = arith.muli %c25_326, %c4_327 : index + %756 = arith.index_cast %755 : index to i64 + %757 = llvm.getelementptr %42[%756] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %758 = llvm.load %757 : !llvm.ptr -> f32 + %759 = arith.mulf %754, %758 : f32 + %760 = arith.addf %750, %759 : f32 + %c0_328 = arith.constant 0 : index + %c4_329 = arith.constant 4 : index + %761 = arith.muli %c0_328, %c4_329 : index + %762 = arith.index_cast %761 : index to i64 + %763 = llvm.getelementptr %43[%762] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %760, %763 : f32, !llvm.ptr + %c0_330 = arith.constant 0 : index + %c4_331 = arith.constant 4 : index + %764 = arith.muli %c0_330, %c4_331 : index + %765 = arith.index_cast %764 : index to i64 + %766 = llvm.getelementptr %43[%765] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %767 = llvm.load %766 : !llvm.ptr -> f32 + %c26_332 = arith.constant 26 : index + %c4_333 = arith.constant 4 : index + %768 = arith.muli %c26_332, %c4_333 : index + %769 = arith.index_cast %768 : index to i64 + %770 = llvm.getelementptr %42[%769] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %771 = llvm.load %770 : !llvm.ptr -> f32 + %c26_334 = arith.constant 26 : index + %c4_335 = arith.constant 4 : index + %772 = arith.muli %c26_334, %c4_335 : index + %773 = arith.index_cast %772 : index to i64 + %774 = llvm.getelementptr %42[%773] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %775 = llvm.load %774 : !llvm.ptr -> f32 + %776 = arith.mulf %771, %775 : f32 + %777 = arith.addf %767, %776 : f32 + %c0_336 = arith.constant 0 : index + %c4_337 = arith.constant 4 : index + %778 = arith.muli %c0_336, %c4_337 : index + %779 = arith.index_cast %778 : index to i64 + %780 = llvm.getelementptr %43[%779] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %777, %780 : f32, !llvm.ptr + %c0_338 = arith.constant 0 : index + %c4_339 = arith.constant 4 : index + %781 = arith.muli %c0_338, %c4_339 : index + %782 = arith.index_cast %781 : index to i64 + %783 = llvm.getelementptr %43[%782] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %784 = llvm.load %783 : !llvm.ptr -> f32 + %c27 = arith.constant 27 : index + %c4_340 = arith.constant 4 : index + %785 = arith.muli %c27, %c4_340 : index + %786 = arith.index_cast %785 : index to i64 + %787 = llvm.getelementptr %42[%786] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %788 = llvm.load %787 : !llvm.ptr -> f32 + %c27_341 = arith.constant 27 : index + %c4_342 = arith.constant 4 : index + %789 = arith.muli %c27_341, %c4_342 : index + %790 = arith.index_cast %789 : index to i64 + %791 = llvm.getelementptr %42[%790] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %792 = llvm.load %791 : !llvm.ptr -> f32 + %793 = arith.mulf %788, %792 : f32 + %794 = arith.addf %784, %793 : f32 + %c0_343 = arith.constant 0 : index + %c4_344 = arith.constant 4 : index + %795 = arith.muli %c0_343, %c4_344 : index + %796 = arith.index_cast %795 : index to i64 + %797 = llvm.getelementptr %43[%796] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %794, %797 : f32, !llvm.ptr + %c0_345 = arith.constant 0 : index + %c4_346 = arith.constant 4 : index + %798 = arith.muli %c0_345, %c4_346 : index + %799 = arith.index_cast %798 : index to i64 + %800 = llvm.getelementptr %43[%799] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %801 = llvm.load %800 : !llvm.ptr -> f32 + %c28_347 = arith.constant 28 : index + %c4_348 = arith.constant 4 : index + %802 = arith.muli %c28_347, %c4_348 : index + %803 = arith.index_cast %802 : index to i64 + %804 = llvm.getelementptr %42[%803] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %805 = llvm.load %804 : !llvm.ptr -> f32 + %c28_349 = arith.constant 28 : index + %c4_350 = arith.constant 4 : index + %806 = arith.muli %c28_349, %c4_350 : index + %807 = arith.index_cast %806 : index to i64 + %808 = llvm.getelementptr %42[%807] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %809 = llvm.load %808 : !llvm.ptr -> f32 + %810 = arith.mulf %805, %809 : f32 + %811 = arith.addf %801, %810 : f32 + %c0_351 = arith.constant 0 : index + %c4_352 = arith.constant 4 : index + %812 = arith.muli %c0_351, %c4_352 : index + %813 = arith.index_cast %812 : index to i64 + %814 = llvm.getelementptr %43[%813] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %811, %814 : f32, !llvm.ptr + %c0_353 = arith.constant 0 : index + %c4_354 = arith.constant 4 : index + %815 = arith.muli %c0_353, %c4_354 : index + %816 = arith.index_cast %815 : index to i64 + %817 = llvm.getelementptr %43[%816] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %818 = llvm.load %817 : !llvm.ptr -> f32 + %c29 = arith.constant 29 : index + %c4_355 = arith.constant 4 : index + %819 = arith.muli %c29, %c4_355 : index + %820 = arith.index_cast %819 : index to i64 + %821 = llvm.getelementptr %42[%820] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %822 = llvm.load %821 : !llvm.ptr -> f32 + %c29_356 = arith.constant 29 : index + %c4_357 = arith.constant 4 : index + %823 = arith.muli %c29_356, %c4_357 : index + %824 = arith.index_cast %823 : index to i64 + %825 = llvm.getelementptr %42[%824] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %826 = llvm.load %825 : !llvm.ptr -> f32 + %827 = arith.mulf %822, %826 : f32 + %828 = arith.addf %818, %827 : f32 + %c0_358 = arith.constant 0 : index + %c4_359 = arith.constant 4 : index + %829 = arith.muli %c0_358, %c4_359 : index + %830 = arith.index_cast %829 : index to i64 + %831 = llvm.getelementptr %43[%830] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %828, %831 : f32, !llvm.ptr + %c0_360 = arith.constant 0 : index + %c4_361 = arith.constant 4 : index + %832 = arith.muli %c0_360, %c4_361 : index + %833 = arith.index_cast %832 : index to i64 + %834 = llvm.getelementptr %43[%833] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %835 = llvm.load %834 : !llvm.ptr -> f32 + %c30_362 = arith.constant 30 : index + %c4_363 = arith.constant 4 : index + %836 = arith.muli %c30_362, %c4_363 : index + %837 = arith.index_cast %836 : index to i64 + %838 = llvm.getelementptr %42[%837] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %839 = llvm.load %838 : !llvm.ptr -> f32 + %c30_364 = arith.constant 30 : index + %c4_365 = arith.constant 4 : index + %840 = arith.muli %c30_364, %c4_365 : index + %841 = arith.index_cast %840 : index to i64 + %842 = llvm.getelementptr %42[%841] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %843 = llvm.load %842 : !llvm.ptr -> f32 + %844 = arith.mulf %839, %843 : f32 + %845 = arith.addf %835, %844 : f32 + %c0_366 = arith.constant 0 : index + %c4_367 = arith.constant 4 : index + %846 = arith.muli %c0_366, %c4_367 : index + %847 = arith.index_cast %846 : index to i64 + %848 = llvm.getelementptr %43[%847] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %845, %848 : f32, !llvm.ptr + %c0_368 = arith.constant 0 : index + %c4_369 = arith.constant 4 : index + %849 = arith.muli %c0_368, %c4_369 : index + %850 = arith.index_cast %849 : index to i64 + %851 = llvm.getelementptr %43[%850] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %852 = llvm.load %851 : !llvm.ptr -> f32 + %c31 = arith.constant 31 : index + %c4_370 = arith.constant 4 : index + %853 = arith.muli %c31, %c4_370 : index + %854 = arith.index_cast %853 : index to i64 + %855 = llvm.getelementptr %42[%854] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %856 = llvm.load %855 : !llvm.ptr -> f32 + %c31_371 = arith.constant 31 : index + %c4_372 = arith.constant 4 : index + %857 = arith.muli %c31_371, %c4_372 : index + %858 = arith.index_cast %857 : index to i64 + %859 = llvm.getelementptr %42[%858] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %860 = llvm.load %859 : !llvm.ptr -> f32 + %861 = arith.mulf %856, %860 : f32 + %862 = arith.addf %852, %861 : f32 + %c0_373 = arith.constant 0 : index + %c4_374 = arith.constant 4 : index + %863 = arith.muli %c0_373, %c4_374 : index + %864 = arith.index_cast %863 : index to i64 + %865 = llvm.getelementptr %43[%864] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %862, %865 : f32, !llvm.ptr + %c0_375 = arith.constant 0 : index + %c4_376 = arith.constant 4 : index + %866 = arith.muli %c0_375, %c4_376 : index + %867 = arith.index_cast %866 : index to i64 + %868 = llvm.getelementptr %43[%867] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %869 = llvm.load %868 : !llvm.ptr -> f32 + %870 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c20480 = arith.constant 20480 : index + %871 = pto.addptr %870, %c20480 : -> + %cst_377 = arith.constant 0.000000e+00 : f32 + %872 = pto.get_tid_x : i32 + %c32_i32_378 = arith.constant 32 : i32 + %873 = arith.floordivsi %872, %c32_i32_378 : i32 + %874 = pto.get_laneid : i32 + %875 = pto.redux_add %869 : f32 -> f32 + %c1_i32_379 = arith.constant 1 : i32 + %876 = arith.cmpi slt, %874, %c1_i32_379 : i32 + scf.if %876 { + %c1_i32_599 = arith.constant 1 : i32 + %1347 = arith.muli %873, %c1_i32_599 : i32 + %1348 = arith.addi %1347, %874 : i32 + %1349 = arith.index_cast %1348 : i32 to index + pto.store %875, %871[%1349] : !pto.ptr, f32 + } + pto.syncthreads + %c32_i32_380 = arith.constant 32 : i32 + %877 = arith.cmpi slt, %872, %c32_i32_380 : i32 + %878 = scf.if %877 -> (f32) { + %c4_i32 = arith.constant 4 : i32 + %1347 = arith.cmpi slt, %874, %c4_i32 : i32 + %1348 = arith.index_cast %874 : i32 to index + %1349 = pto.load %871[%1348] : !pto.ptr -> f32 + %1350 = arith.select %1347, %1349, %cst_377 : f32 + %1351 = pto.redux_add %1350 : f32 -> f32 + scf.yield %1351 : f32 + } else { + scf.yield %cst_377 : f32 + } + %c1_i32_381 = arith.constant 1 : i32 + %879 = arith.cmpi slt, %872, %c1_i32_381 : i32 + scf.if %879 { + %1347 = arith.index_cast %872 : i32 to index + pto.store %878, %871[%1347] : !pto.ptr, f32 + } + pto.syncthreads + %c1_i32_382 = arith.constant 1 : i32 + %880 = arith.remsi %872, %c1_i32_382 : i32 + %881 = arith.index_cast %880 : i32 to index + %882 = pto.load %871[%881] : !pto.ptr -> f32 + pto.syncthreads + %c0_383 = arith.constant 0 : index + %c4_384 = arith.constant 4 : index + %883 = arith.muli %c0_383, %c4_384 : index + %884 = arith.index_cast %883 : index to i64 + %885 = llvm.getelementptr %43[%884] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %882, %885 : f32, !llvm.ptr + %c0_385 = arith.constant 0 : index + %c4_386 = arith.constant 4 : index + %886 = arith.muli %c0_385, %c4_386 : index + %887 = arith.index_cast %886 : index to i64 + %888 = llvm.getelementptr %43[%887] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %889 = llvm.load %888 : !llvm.ptr -> f32 + %cst_387 = arith.constant 4.096000e+03 : f32 + %890 = arith.divf %889, %cst_387 : f32 + %891 = arith.addf %890, %arg4 : f32 + %892 = pto.sqrt %891 : f32 -> f32 + %cst_388 = arith.constant 1.000000e+00 : f32 + %893 = arith.divf %cst_388, %892 : f32 + %894 = pto.sqrt %891 : f32 -> f32 + %cst_389 = arith.constant 1.000000e+00 : f32 + %895 = arith.divf %cst_389, %894 : f32 + %896 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_390 = arith.constant 1 : index + %897 = arith.andi %arg5, %c1_390 : index + %c8_391 = arith.constant 8 : index + %898 = arith.muli %897, %c8_391 : index + pto.store %895, %896[%898] : !pto.ptr, f32 + %c0_392 = arith.constant 0 : index + %c4_393 = arith.constant 4 : index + %899 = arith.muli %c0_392, %c4_393 : index + %900 = arith.index_cast %899 : index to i64 + %901 = llvm.getelementptr %42[%900] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %902 = llvm.load %901 : !llvm.ptr -> vector<2xf32> + %903 = pto.sqrt %891 : f32 -> f32 + %cst_394 = arith.constant 1.000000e+00 : f32 + %904 = arith.divf %cst_394, %903 : f32 + %905 = llvm.mlir.undef : vector<2xf32> + %c0_i32 = arith.constant 0 : i32 + %906 = llvm.insertelement %904, %905[%c0_i32 : i32] : vector<2xf32> + %c1_i32_395 = arith.constant 1 : i32 + %907 = llvm.insertelement %904, %906[%c1_i32_395 : i32] : vector<2xf32> + %908 = arith.mulf %902, %907 : vector<2xf32> + %c0_396 = arith.constant 0 : index + %c4_397 = arith.constant 4 : index + %909 = arith.muli %c0_396, %c4_397 : index + %910 = arith.index_cast %909 : index to i64 + %911 = llvm.getelementptr %1[%910] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %912 = llvm.load %911 : !llvm.ptr -> vector<2xf32> + %913 = arith.mulf %908, %912 : vector<2xf32> + %914 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_398 = arith.constant 1 : index + %915 = arith.andi %arg5, %c1_398 : index + %c4096_399 = arith.constant 4096 : index + %916 = arith.muli %915, %c4096_399 : index + %c0_400 = arith.constant 0 : index + %917 = arith.addi %916, %c0_400 : index + %c2_i32_401 = arith.constant 2 : i32 + %918 = arith.muli %44, %c2_i32_401 : i32 + %919 = arith.index_cast %918 : i32 to index + %920 = arith.addi %917, %919 : index + %c12288_402 = arith.constant 12288 : index + %921 = arith.addi %920, %c12288_402 : index + %c4_403 = arith.constant 4 : index + %922 = arith.muli %921, %c4_403 : index + %923 = arith.index_cast %922 : index to i64 + %924 = pto.castptr %914 : !pto.ptr -> i64 + %925 = llvm.inttoptr %924 : i64 to !llvm.ptr<6> + %926 = llvm.getelementptr %925[%923] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %913, %926 : vector<2xf32>, !llvm.ptr<6> + %c2_404 = arith.constant 2 : index + %c4_405 = arith.constant 4 : index + %927 = arith.muli %c2_404, %c4_405 : index + %928 = arith.index_cast %927 : index to i64 + %929 = llvm.getelementptr %42[%928] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %930 = llvm.load %929 : !llvm.ptr -> vector<2xf32> + %931 = pto.sqrt %891 : f32 -> f32 + %cst_406 = arith.constant 1.000000e+00 : f32 + %932 = arith.divf %cst_406, %931 : f32 + %933 = llvm.mlir.undef : vector<2xf32> + %c0_i32_407 = arith.constant 0 : i32 + %934 = llvm.insertelement %932, %933[%c0_i32_407 : i32] : vector<2xf32> + %c1_i32_408 = arith.constant 1 : i32 + %935 = llvm.insertelement %932, %934[%c1_i32_408 : i32] : vector<2xf32> + %936 = arith.mulf %930, %935 : vector<2xf32> + %c2_409 = arith.constant 2 : index + %c4_410 = arith.constant 4 : index + %937 = arith.muli %c2_409, %c4_410 : index + %938 = arith.index_cast %937 : index to i64 + %939 = llvm.getelementptr %1[%938] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %940 = llvm.load %939 : !llvm.ptr -> vector<2xf32> + %941 = arith.mulf %936, %940 : vector<2xf32> + %942 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_411 = arith.constant 1 : index + %943 = arith.andi %arg5, %c1_411 : index + %c4096_412 = arith.constant 4096 : index + %944 = arith.muli %943, %c4096_412 : index + %c256_413 = arith.constant 256 : index + %945 = arith.addi %944, %c256_413 : index + %c2_i32_414 = arith.constant 2 : i32 + %946 = arith.muli %44, %c2_i32_414 : i32 + %947 = arith.index_cast %946 : i32 to index + %948 = arith.addi %945, %947 : index + %c12288_415 = arith.constant 12288 : index + %949 = arith.addi %948, %c12288_415 : index + %c4_416 = arith.constant 4 : index + %950 = arith.muli %949, %c4_416 : index + %951 = arith.index_cast %950 : index to i64 + %952 = pto.castptr %942 : !pto.ptr -> i64 + %953 = llvm.inttoptr %952 : i64 to !llvm.ptr<6> + %954 = llvm.getelementptr %953[%951] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %941, %954 : vector<2xf32>, !llvm.ptr<6> + %c4_417 = arith.constant 4 : index + %c4_418 = arith.constant 4 : index + %955 = arith.muli %c4_417, %c4_418 : index + %956 = arith.index_cast %955 : index to i64 + %957 = llvm.getelementptr %42[%956] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %958 = llvm.load %957 : !llvm.ptr -> vector<2xf32> + %959 = pto.sqrt %891 : f32 -> f32 + %cst_419 = arith.constant 1.000000e+00 : f32 + %960 = arith.divf %cst_419, %959 : f32 + %961 = llvm.mlir.undef : vector<2xf32> + %c0_i32_420 = arith.constant 0 : i32 + %962 = llvm.insertelement %960, %961[%c0_i32_420 : i32] : vector<2xf32> + %c1_i32_421 = arith.constant 1 : i32 + %963 = llvm.insertelement %960, %962[%c1_i32_421 : i32] : vector<2xf32> + %964 = arith.mulf %958, %963 : vector<2xf32> + %c4_422 = arith.constant 4 : index + %c4_423 = arith.constant 4 : index + %965 = arith.muli %c4_422, %c4_423 : index + %966 = arith.index_cast %965 : index to i64 + %967 = llvm.getelementptr %1[%966] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %968 = llvm.load %967 : !llvm.ptr -> vector<2xf32> + %969 = arith.mulf %964, %968 : vector<2xf32> + %970 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_424 = arith.constant 1 : index + %971 = arith.andi %arg5, %c1_424 : index + %c4096_425 = arith.constant 4096 : index + %972 = arith.muli %971, %c4096_425 : index + %c512_426 = arith.constant 512 : index + %973 = arith.addi %972, %c512_426 : index + %c2_i32_427 = arith.constant 2 : i32 + %974 = arith.muli %44, %c2_i32_427 : i32 + %975 = arith.index_cast %974 : i32 to index + %976 = arith.addi %973, %975 : index + %c12288_428 = arith.constant 12288 : index + %977 = arith.addi %976, %c12288_428 : index + %c4_429 = arith.constant 4 : index + %978 = arith.muli %977, %c4_429 : index + %979 = arith.index_cast %978 : index to i64 + %980 = pto.castptr %970 : !pto.ptr -> i64 + %981 = llvm.inttoptr %980 : i64 to !llvm.ptr<6> + %982 = llvm.getelementptr %981[%979] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %969, %982 : vector<2xf32>, !llvm.ptr<6> + %c6_430 = arith.constant 6 : index + %c4_431 = arith.constant 4 : index + %983 = arith.muli %c6_430, %c4_431 : index + %984 = arith.index_cast %983 : index to i64 + %985 = llvm.getelementptr %42[%984] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %986 = llvm.load %985 : !llvm.ptr -> vector<2xf32> + %987 = pto.sqrt %891 : f32 -> f32 + %cst_432 = arith.constant 1.000000e+00 : f32 + %988 = arith.divf %cst_432, %987 : f32 + %989 = llvm.mlir.undef : vector<2xf32> + %c0_i32_433 = arith.constant 0 : i32 + %990 = llvm.insertelement %988, %989[%c0_i32_433 : i32] : vector<2xf32> + %c1_i32_434 = arith.constant 1 : i32 + %991 = llvm.insertelement %988, %990[%c1_i32_434 : i32] : vector<2xf32> + %992 = arith.mulf %986, %991 : vector<2xf32> + %c6_435 = arith.constant 6 : index + %c4_436 = arith.constant 4 : index + %993 = arith.muli %c6_435, %c4_436 : index + %994 = arith.index_cast %993 : index to i64 + %995 = llvm.getelementptr %1[%994] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %996 = llvm.load %995 : !llvm.ptr -> vector<2xf32> + %997 = arith.mulf %992, %996 : vector<2xf32> + %998 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_437 = arith.constant 1 : index + %999 = arith.andi %arg5, %c1_437 : index + %c4096_438 = arith.constant 4096 : index + %1000 = arith.muli %999, %c4096_438 : index + %c768_439 = arith.constant 768 : index + %1001 = arith.addi %1000, %c768_439 : index + %c2_i32_440 = arith.constant 2 : i32 + %1002 = arith.muli %44, %c2_i32_440 : i32 + %1003 = arith.index_cast %1002 : i32 to index + %1004 = arith.addi %1001, %1003 : index + %c12288_441 = arith.constant 12288 : index + %1005 = arith.addi %1004, %c12288_441 : index + %c4_442 = arith.constant 4 : index + %1006 = arith.muli %1005, %c4_442 : index + %1007 = arith.index_cast %1006 : index to i64 + %1008 = pto.castptr %998 : !pto.ptr -> i64 + %1009 = llvm.inttoptr %1008 : i64 to !llvm.ptr<6> + %1010 = llvm.getelementptr %1009[%1007] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %997, %1010 : vector<2xf32>, !llvm.ptr<6> + %c8_443 = arith.constant 8 : index + %c4_444 = arith.constant 4 : index + %1011 = arith.muli %c8_443, %c4_444 : index + %1012 = arith.index_cast %1011 : index to i64 + %1013 = llvm.getelementptr %42[%1012] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1014 = llvm.load %1013 : !llvm.ptr -> vector<2xf32> + %1015 = pto.sqrt %891 : f32 -> f32 + %cst_445 = arith.constant 1.000000e+00 : f32 + %1016 = arith.divf %cst_445, %1015 : f32 + %1017 = llvm.mlir.undef : vector<2xf32> + %c0_i32_446 = arith.constant 0 : i32 + %1018 = llvm.insertelement %1016, %1017[%c0_i32_446 : i32] : vector<2xf32> + %c1_i32_447 = arith.constant 1 : i32 + %1019 = llvm.insertelement %1016, %1018[%c1_i32_447 : i32] : vector<2xf32> + %1020 = arith.mulf %1014, %1019 : vector<2xf32> + %c8_448 = arith.constant 8 : index + %c4_449 = arith.constant 4 : index + %1021 = arith.muli %c8_448, %c4_449 : index + %1022 = arith.index_cast %1021 : index to i64 + %1023 = llvm.getelementptr %1[%1022] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1024 = llvm.load %1023 : !llvm.ptr -> vector<2xf32> + %1025 = arith.mulf %1020, %1024 : vector<2xf32> + %1026 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_450 = arith.constant 1 : index + %1027 = arith.andi %arg5, %c1_450 : index + %c4096_451 = arith.constant 4096 : index + %1028 = arith.muli %1027, %c4096_451 : index + %c1024_452 = arith.constant 1024 : index + %1029 = arith.addi %1028, %c1024_452 : index + %c2_i32_453 = arith.constant 2 : i32 + %1030 = arith.muli %44, %c2_i32_453 : i32 + %1031 = arith.index_cast %1030 : i32 to index + %1032 = arith.addi %1029, %1031 : index + %c12288_454 = arith.constant 12288 : index + %1033 = arith.addi %1032, %c12288_454 : index + %c4_455 = arith.constant 4 : index + %1034 = arith.muli %1033, %c4_455 : index + %1035 = arith.index_cast %1034 : index to i64 + %1036 = pto.castptr %1026 : !pto.ptr -> i64 + %1037 = llvm.inttoptr %1036 : i64 to !llvm.ptr<6> + %1038 = llvm.getelementptr %1037[%1035] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1025, %1038 : vector<2xf32>, !llvm.ptr<6> + %c10_456 = arith.constant 10 : index + %c4_457 = arith.constant 4 : index + %1039 = arith.muli %c10_456, %c4_457 : index + %1040 = arith.index_cast %1039 : index to i64 + %1041 = llvm.getelementptr %42[%1040] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1042 = llvm.load %1041 : !llvm.ptr -> vector<2xf32> + %1043 = pto.sqrt %891 : f32 -> f32 + %cst_458 = arith.constant 1.000000e+00 : f32 + %1044 = arith.divf %cst_458, %1043 : f32 + %1045 = llvm.mlir.undef : vector<2xf32> + %c0_i32_459 = arith.constant 0 : i32 + %1046 = llvm.insertelement %1044, %1045[%c0_i32_459 : i32] : vector<2xf32> + %c1_i32_460 = arith.constant 1 : i32 + %1047 = llvm.insertelement %1044, %1046[%c1_i32_460 : i32] : vector<2xf32> + %1048 = arith.mulf %1042, %1047 : vector<2xf32> + %c10_461 = arith.constant 10 : index + %c4_462 = arith.constant 4 : index + %1049 = arith.muli %c10_461, %c4_462 : index + %1050 = arith.index_cast %1049 : index to i64 + %1051 = llvm.getelementptr %1[%1050] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1052 = llvm.load %1051 : !llvm.ptr -> vector<2xf32> + %1053 = arith.mulf %1048, %1052 : vector<2xf32> + %1054 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_463 = arith.constant 1 : index + %1055 = arith.andi %arg5, %c1_463 : index + %c4096_464 = arith.constant 4096 : index + %1056 = arith.muli %1055, %c4096_464 : index + %c1280_465 = arith.constant 1280 : index + %1057 = arith.addi %1056, %c1280_465 : index + %c2_i32_466 = arith.constant 2 : i32 + %1058 = arith.muli %44, %c2_i32_466 : i32 + %1059 = arith.index_cast %1058 : i32 to index + %1060 = arith.addi %1057, %1059 : index + %c12288_467 = arith.constant 12288 : index + %1061 = arith.addi %1060, %c12288_467 : index + %c4_468 = arith.constant 4 : index + %1062 = arith.muli %1061, %c4_468 : index + %1063 = arith.index_cast %1062 : index to i64 + %1064 = pto.castptr %1054 : !pto.ptr -> i64 + %1065 = llvm.inttoptr %1064 : i64 to !llvm.ptr<6> + %1066 = llvm.getelementptr %1065[%1063] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1053, %1066 : vector<2xf32>, !llvm.ptr<6> + %c12_469 = arith.constant 12 : index + %c4_470 = arith.constant 4 : index + %1067 = arith.muli %c12_469, %c4_470 : index + %1068 = arith.index_cast %1067 : index to i64 + %1069 = llvm.getelementptr %42[%1068] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1070 = llvm.load %1069 : !llvm.ptr -> vector<2xf32> + %1071 = pto.sqrt %891 : f32 -> f32 + %cst_471 = arith.constant 1.000000e+00 : f32 + %1072 = arith.divf %cst_471, %1071 : f32 + %1073 = llvm.mlir.undef : vector<2xf32> + %c0_i32_472 = arith.constant 0 : i32 + %1074 = llvm.insertelement %1072, %1073[%c0_i32_472 : i32] : vector<2xf32> + %c1_i32_473 = arith.constant 1 : i32 + %1075 = llvm.insertelement %1072, %1074[%c1_i32_473 : i32] : vector<2xf32> + %1076 = arith.mulf %1070, %1075 : vector<2xf32> + %c12_474 = arith.constant 12 : index + %c4_475 = arith.constant 4 : index + %1077 = arith.muli %c12_474, %c4_475 : index + %1078 = arith.index_cast %1077 : index to i64 + %1079 = llvm.getelementptr %1[%1078] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1080 = llvm.load %1079 : !llvm.ptr -> vector<2xf32> + %1081 = arith.mulf %1076, %1080 : vector<2xf32> + %1082 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_476 = arith.constant 1 : index + %1083 = arith.andi %arg5, %c1_476 : index + %c4096_477 = arith.constant 4096 : index + %1084 = arith.muli %1083, %c4096_477 : index + %c1536_478 = arith.constant 1536 : index + %1085 = arith.addi %1084, %c1536_478 : index + %c2_i32_479 = arith.constant 2 : i32 + %1086 = arith.muli %44, %c2_i32_479 : i32 + %1087 = arith.index_cast %1086 : i32 to index + %1088 = arith.addi %1085, %1087 : index + %c12288_480 = arith.constant 12288 : index + %1089 = arith.addi %1088, %c12288_480 : index + %c4_481 = arith.constant 4 : index + %1090 = arith.muli %1089, %c4_481 : index + %1091 = arith.index_cast %1090 : index to i64 + %1092 = pto.castptr %1082 : !pto.ptr -> i64 + %1093 = llvm.inttoptr %1092 : i64 to !llvm.ptr<6> + %1094 = llvm.getelementptr %1093[%1091] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1081, %1094 : vector<2xf32>, !llvm.ptr<6> + %c14_482 = arith.constant 14 : index + %c4_483 = arith.constant 4 : index + %1095 = arith.muli %c14_482, %c4_483 : index + %1096 = arith.index_cast %1095 : index to i64 + %1097 = llvm.getelementptr %42[%1096] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1098 = llvm.load %1097 : !llvm.ptr -> vector<2xf32> + %1099 = pto.sqrt %891 : f32 -> f32 + %cst_484 = arith.constant 1.000000e+00 : f32 + %1100 = arith.divf %cst_484, %1099 : f32 + %1101 = llvm.mlir.undef : vector<2xf32> + %c0_i32_485 = arith.constant 0 : i32 + %1102 = llvm.insertelement %1100, %1101[%c0_i32_485 : i32] : vector<2xf32> + %c1_i32_486 = arith.constant 1 : i32 + %1103 = llvm.insertelement %1100, %1102[%c1_i32_486 : i32] : vector<2xf32> + %1104 = arith.mulf %1098, %1103 : vector<2xf32> + %c14_487 = arith.constant 14 : index + %c4_488 = arith.constant 4 : index + %1105 = arith.muli %c14_487, %c4_488 : index + %1106 = arith.index_cast %1105 : index to i64 + %1107 = llvm.getelementptr %1[%1106] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1108 = llvm.load %1107 : !llvm.ptr -> vector<2xf32> + %1109 = arith.mulf %1104, %1108 : vector<2xf32> + %1110 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_489 = arith.constant 1 : index + %1111 = arith.andi %arg5, %c1_489 : index + %c4096_490 = arith.constant 4096 : index + %1112 = arith.muli %1111, %c4096_490 : index + %c1792_491 = arith.constant 1792 : index + %1113 = arith.addi %1112, %c1792_491 : index + %c2_i32_492 = arith.constant 2 : i32 + %1114 = arith.muli %44, %c2_i32_492 : i32 + %1115 = arith.index_cast %1114 : i32 to index + %1116 = arith.addi %1113, %1115 : index + %c12288_493 = arith.constant 12288 : index + %1117 = arith.addi %1116, %c12288_493 : index + %c4_494 = arith.constant 4 : index + %1118 = arith.muli %1117, %c4_494 : index + %1119 = arith.index_cast %1118 : index to i64 + %1120 = pto.castptr %1110 : !pto.ptr -> i64 + %1121 = llvm.inttoptr %1120 : i64 to !llvm.ptr<6> + %1122 = llvm.getelementptr %1121[%1119] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1109, %1122 : vector<2xf32>, !llvm.ptr<6> + %c16_495 = arith.constant 16 : index + %c4_496 = arith.constant 4 : index + %1123 = arith.muli %c16_495, %c4_496 : index + %1124 = arith.index_cast %1123 : index to i64 + %1125 = llvm.getelementptr %42[%1124] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1126 = llvm.load %1125 : !llvm.ptr -> vector<2xf32> + %1127 = pto.sqrt %891 : f32 -> f32 + %cst_497 = arith.constant 1.000000e+00 : f32 + %1128 = arith.divf %cst_497, %1127 : f32 + %1129 = llvm.mlir.undef : vector<2xf32> + %c0_i32_498 = arith.constant 0 : i32 + %1130 = llvm.insertelement %1128, %1129[%c0_i32_498 : i32] : vector<2xf32> + %c1_i32_499 = arith.constant 1 : i32 + %1131 = llvm.insertelement %1128, %1130[%c1_i32_499 : i32] : vector<2xf32> + %1132 = arith.mulf %1126, %1131 : vector<2xf32> + %c16_500 = arith.constant 16 : index + %c4_501 = arith.constant 4 : index + %1133 = arith.muli %c16_500, %c4_501 : index + %1134 = arith.index_cast %1133 : index to i64 + %1135 = llvm.getelementptr %1[%1134] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1136 = llvm.load %1135 : !llvm.ptr -> vector<2xf32> + %1137 = arith.mulf %1132, %1136 : vector<2xf32> + %1138 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_502 = arith.constant 1 : index + %1139 = arith.andi %arg5, %c1_502 : index + %c4096_503 = arith.constant 4096 : index + %1140 = arith.muli %1139, %c4096_503 : index + %c2048_504 = arith.constant 2048 : index + %1141 = arith.addi %1140, %c2048_504 : index + %c2_i32_505 = arith.constant 2 : i32 + %1142 = arith.muli %44, %c2_i32_505 : i32 + %1143 = arith.index_cast %1142 : i32 to index + %1144 = arith.addi %1141, %1143 : index + %c12288_506 = arith.constant 12288 : index + %1145 = arith.addi %1144, %c12288_506 : index + %c4_507 = arith.constant 4 : index + %1146 = arith.muli %1145, %c4_507 : index + %1147 = arith.index_cast %1146 : index to i64 + %1148 = pto.castptr %1138 : !pto.ptr -> i64 + %1149 = llvm.inttoptr %1148 : i64 to !llvm.ptr<6> + %1150 = llvm.getelementptr %1149[%1147] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1137, %1150 : vector<2xf32>, !llvm.ptr<6> + %c18_508 = arith.constant 18 : index + %c4_509 = arith.constant 4 : index + %1151 = arith.muli %c18_508, %c4_509 : index + %1152 = arith.index_cast %1151 : index to i64 + %1153 = llvm.getelementptr %42[%1152] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1154 = llvm.load %1153 : !llvm.ptr -> vector<2xf32> + %1155 = pto.sqrt %891 : f32 -> f32 + %cst_510 = arith.constant 1.000000e+00 : f32 + %1156 = arith.divf %cst_510, %1155 : f32 + %1157 = llvm.mlir.undef : vector<2xf32> + %c0_i32_511 = arith.constant 0 : i32 + %1158 = llvm.insertelement %1156, %1157[%c0_i32_511 : i32] : vector<2xf32> + %c1_i32_512 = arith.constant 1 : i32 + %1159 = llvm.insertelement %1156, %1158[%c1_i32_512 : i32] : vector<2xf32> + %1160 = arith.mulf %1154, %1159 : vector<2xf32> + %c18_513 = arith.constant 18 : index + %c4_514 = arith.constant 4 : index + %1161 = arith.muli %c18_513, %c4_514 : index + %1162 = arith.index_cast %1161 : index to i64 + %1163 = llvm.getelementptr %1[%1162] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1164 = llvm.load %1163 : !llvm.ptr -> vector<2xf32> + %1165 = arith.mulf %1160, %1164 : vector<2xf32> + %1166 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_515 = arith.constant 1 : index + %1167 = arith.andi %arg5, %c1_515 : index + %c4096_516 = arith.constant 4096 : index + %1168 = arith.muli %1167, %c4096_516 : index + %c2304_517 = arith.constant 2304 : index + %1169 = arith.addi %1168, %c2304_517 : index + %c2_i32_518 = arith.constant 2 : i32 + %1170 = arith.muli %44, %c2_i32_518 : i32 + %1171 = arith.index_cast %1170 : i32 to index + %1172 = arith.addi %1169, %1171 : index + %c12288_519 = arith.constant 12288 : index + %1173 = arith.addi %1172, %c12288_519 : index + %c4_520 = arith.constant 4 : index + %1174 = arith.muli %1173, %c4_520 : index + %1175 = arith.index_cast %1174 : index to i64 + %1176 = pto.castptr %1166 : !pto.ptr -> i64 + %1177 = llvm.inttoptr %1176 : i64 to !llvm.ptr<6> + %1178 = llvm.getelementptr %1177[%1175] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1165, %1178 : vector<2xf32>, !llvm.ptr<6> + %c20_521 = arith.constant 20 : index + %c4_522 = arith.constant 4 : index + %1179 = arith.muli %c20_521, %c4_522 : index + %1180 = arith.index_cast %1179 : index to i64 + %1181 = llvm.getelementptr %42[%1180] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1182 = llvm.load %1181 : !llvm.ptr -> vector<2xf32> + %1183 = pto.sqrt %891 : f32 -> f32 + %cst_523 = arith.constant 1.000000e+00 : f32 + %1184 = arith.divf %cst_523, %1183 : f32 + %1185 = llvm.mlir.undef : vector<2xf32> + %c0_i32_524 = arith.constant 0 : i32 + %1186 = llvm.insertelement %1184, %1185[%c0_i32_524 : i32] : vector<2xf32> + %c1_i32_525 = arith.constant 1 : i32 + %1187 = llvm.insertelement %1184, %1186[%c1_i32_525 : i32] : vector<2xf32> + %1188 = arith.mulf %1182, %1187 : vector<2xf32> + %c20_526 = arith.constant 20 : index + %c4_527 = arith.constant 4 : index + %1189 = arith.muli %c20_526, %c4_527 : index + %1190 = arith.index_cast %1189 : index to i64 + %1191 = llvm.getelementptr %1[%1190] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1192 = llvm.load %1191 : !llvm.ptr -> vector<2xf32> + %1193 = arith.mulf %1188, %1192 : vector<2xf32> + %1194 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_528 = arith.constant 1 : index + %1195 = arith.andi %arg5, %c1_528 : index + %c4096_529 = arith.constant 4096 : index + %1196 = arith.muli %1195, %c4096_529 : index + %c2560_530 = arith.constant 2560 : index + %1197 = arith.addi %1196, %c2560_530 : index + %c2_i32_531 = arith.constant 2 : i32 + %1198 = arith.muli %44, %c2_i32_531 : i32 + %1199 = arith.index_cast %1198 : i32 to index + %1200 = arith.addi %1197, %1199 : index + %c12288_532 = arith.constant 12288 : index + %1201 = arith.addi %1200, %c12288_532 : index + %c4_533 = arith.constant 4 : index + %1202 = arith.muli %1201, %c4_533 : index + %1203 = arith.index_cast %1202 : index to i64 + %1204 = pto.castptr %1194 : !pto.ptr -> i64 + %1205 = llvm.inttoptr %1204 : i64 to !llvm.ptr<6> + %1206 = llvm.getelementptr %1205[%1203] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1193, %1206 : vector<2xf32>, !llvm.ptr<6> + %c22_534 = arith.constant 22 : index + %c4_535 = arith.constant 4 : index + %1207 = arith.muli %c22_534, %c4_535 : index + %1208 = arith.index_cast %1207 : index to i64 + %1209 = llvm.getelementptr %42[%1208] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1210 = llvm.load %1209 : !llvm.ptr -> vector<2xf32> + %1211 = pto.sqrt %891 : f32 -> f32 + %cst_536 = arith.constant 1.000000e+00 : f32 + %1212 = arith.divf %cst_536, %1211 : f32 + %1213 = llvm.mlir.undef : vector<2xf32> + %c0_i32_537 = arith.constant 0 : i32 + %1214 = llvm.insertelement %1212, %1213[%c0_i32_537 : i32] : vector<2xf32> + %c1_i32_538 = arith.constant 1 : i32 + %1215 = llvm.insertelement %1212, %1214[%c1_i32_538 : i32] : vector<2xf32> + %1216 = arith.mulf %1210, %1215 : vector<2xf32> + %c22_539 = arith.constant 22 : index + %c4_540 = arith.constant 4 : index + %1217 = arith.muli %c22_539, %c4_540 : index + %1218 = arith.index_cast %1217 : index to i64 + %1219 = llvm.getelementptr %1[%1218] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1220 = llvm.load %1219 : !llvm.ptr -> vector<2xf32> + %1221 = arith.mulf %1216, %1220 : vector<2xf32> + %1222 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_541 = arith.constant 1 : index + %1223 = arith.andi %arg5, %c1_541 : index + %c4096_542 = arith.constant 4096 : index + %1224 = arith.muli %1223, %c4096_542 : index + %c2816_543 = arith.constant 2816 : index + %1225 = arith.addi %1224, %c2816_543 : index + %c2_i32_544 = arith.constant 2 : i32 + %1226 = arith.muli %44, %c2_i32_544 : i32 + %1227 = arith.index_cast %1226 : i32 to index + %1228 = arith.addi %1225, %1227 : index + %c12288_545 = arith.constant 12288 : index + %1229 = arith.addi %1228, %c12288_545 : index + %c4_546 = arith.constant 4 : index + %1230 = arith.muli %1229, %c4_546 : index + %1231 = arith.index_cast %1230 : index to i64 + %1232 = pto.castptr %1222 : !pto.ptr -> i64 + %1233 = llvm.inttoptr %1232 : i64 to !llvm.ptr<6> + %1234 = llvm.getelementptr %1233[%1231] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1221, %1234 : vector<2xf32>, !llvm.ptr<6> + %c24_547 = arith.constant 24 : index + %c4_548 = arith.constant 4 : index + %1235 = arith.muli %c24_547, %c4_548 : index + %1236 = arith.index_cast %1235 : index to i64 + %1237 = llvm.getelementptr %42[%1236] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1238 = llvm.load %1237 : !llvm.ptr -> vector<2xf32> + %1239 = pto.sqrt %891 : f32 -> f32 + %cst_549 = arith.constant 1.000000e+00 : f32 + %1240 = arith.divf %cst_549, %1239 : f32 + %1241 = llvm.mlir.undef : vector<2xf32> + %c0_i32_550 = arith.constant 0 : i32 + %1242 = llvm.insertelement %1240, %1241[%c0_i32_550 : i32] : vector<2xf32> + %c1_i32_551 = arith.constant 1 : i32 + %1243 = llvm.insertelement %1240, %1242[%c1_i32_551 : i32] : vector<2xf32> + %1244 = arith.mulf %1238, %1243 : vector<2xf32> + %c24_552 = arith.constant 24 : index + %c4_553 = arith.constant 4 : index + %1245 = arith.muli %c24_552, %c4_553 : index + %1246 = arith.index_cast %1245 : index to i64 + %1247 = llvm.getelementptr %1[%1246] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1248 = llvm.load %1247 : !llvm.ptr -> vector<2xf32> + %1249 = arith.mulf %1244, %1248 : vector<2xf32> + %1250 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_554 = arith.constant 1 : index + %1251 = arith.andi %arg5, %c1_554 : index + %c4096_555 = arith.constant 4096 : index + %1252 = arith.muli %1251, %c4096_555 : index + %c3072_556 = arith.constant 3072 : index + %1253 = arith.addi %1252, %c3072_556 : index + %c2_i32_557 = arith.constant 2 : i32 + %1254 = arith.muli %44, %c2_i32_557 : i32 + %1255 = arith.index_cast %1254 : i32 to index + %1256 = arith.addi %1253, %1255 : index + %c12288_558 = arith.constant 12288 : index + %1257 = arith.addi %1256, %c12288_558 : index + %c4_559 = arith.constant 4 : index + %1258 = arith.muli %1257, %c4_559 : index + %1259 = arith.index_cast %1258 : index to i64 + %1260 = pto.castptr %1250 : !pto.ptr -> i64 + %1261 = llvm.inttoptr %1260 : i64 to !llvm.ptr<6> + %1262 = llvm.getelementptr %1261[%1259] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1249, %1262 : vector<2xf32>, !llvm.ptr<6> + %c26_560 = arith.constant 26 : index + %c4_561 = arith.constant 4 : index + %1263 = arith.muli %c26_560, %c4_561 : index + %1264 = arith.index_cast %1263 : index to i64 + %1265 = llvm.getelementptr %42[%1264] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1266 = llvm.load %1265 : !llvm.ptr -> vector<2xf32> + %1267 = pto.sqrt %891 : f32 -> f32 + %cst_562 = arith.constant 1.000000e+00 : f32 + %1268 = arith.divf %cst_562, %1267 : f32 + %1269 = llvm.mlir.undef : vector<2xf32> + %c0_i32_563 = arith.constant 0 : i32 + %1270 = llvm.insertelement %1268, %1269[%c0_i32_563 : i32] : vector<2xf32> + %c1_i32_564 = arith.constant 1 : i32 + %1271 = llvm.insertelement %1268, %1270[%c1_i32_564 : i32] : vector<2xf32> + %1272 = arith.mulf %1266, %1271 : vector<2xf32> + %c26_565 = arith.constant 26 : index + %c4_566 = arith.constant 4 : index + %1273 = arith.muli %c26_565, %c4_566 : index + %1274 = arith.index_cast %1273 : index to i64 + %1275 = llvm.getelementptr %1[%1274] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1276 = llvm.load %1275 : !llvm.ptr -> vector<2xf32> + %1277 = arith.mulf %1272, %1276 : vector<2xf32> + %1278 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_567 = arith.constant 1 : index + %1279 = arith.andi %arg5, %c1_567 : index + %c4096_568 = arith.constant 4096 : index + %1280 = arith.muli %1279, %c4096_568 : index + %c3328_569 = arith.constant 3328 : index + %1281 = arith.addi %1280, %c3328_569 : index + %c2_i32_570 = arith.constant 2 : i32 + %1282 = arith.muli %44, %c2_i32_570 : i32 + %1283 = arith.index_cast %1282 : i32 to index + %1284 = arith.addi %1281, %1283 : index + %c12288_571 = arith.constant 12288 : index + %1285 = arith.addi %1284, %c12288_571 : index + %c4_572 = arith.constant 4 : index + %1286 = arith.muli %1285, %c4_572 : index + %1287 = arith.index_cast %1286 : index to i64 + %1288 = pto.castptr %1278 : !pto.ptr -> i64 + %1289 = llvm.inttoptr %1288 : i64 to !llvm.ptr<6> + %1290 = llvm.getelementptr %1289[%1287] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1277, %1290 : vector<2xf32>, !llvm.ptr<6> + %c28_573 = arith.constant 28 : index + %c4_574 = arith.constant 4 : index + %1291 = arith.muli %c28_573, %c4_574 : index + %1292 = arith.index_cast %1291 : index to i64 + %1293 = llvm.getelementptr %42[%1292] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1294 = llvm.load %1293 : !llvm.ptr -> vector<2xf32> + %1295 = pto.sqrt %891 : f32 -> f32 + %cst_575 = arith.constant 1.000000e+00 : f32 + %1296 = arith.divf %cst_575, %1295 : f32 + %1297 = llvm.mlir.undef : vector<2xf32> + %c0_i32_576 = arith.constant 0 : i32 + %1298 = llvm.insertelement %1296, %1297[%c0_i32_576 : i32] : vector<2xf32> + %c1_i32_577 = arith.constant 1 : i32 + %1299 = llvm.insertelement %1296, %1298[%c1_i32_577 : i32] : vector<2xf32> + %1300 = arith.mulf %1294, %1299 : vector<2xf32> + %c28_578 = arith.constant 28 : index + %c4_579 = arith.constant 4 : index + %1301 = arith.muli %c28_578, %c4_579 : index + %1302 = arith.index_cast %1301 : index to i64 + %1303 = llvm.getelementptr %1[%1302] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1304 = llvm.load %1303 : !llvm.ptr -> vector<2xf32> + %1305 = arith.mulf %1300, %1304 : vector<2xf32> + %1306 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_580 = arith.constant 1 : index + %1307 = arith.andi %arg5, %c1_580 : index + %c4096_581 = arith.constant 4096 : index + %1308 = arith.muli %1307, %c4096_581 : index + %c3584_582 = arith.constant 3584 : index + %1309 = arith.addi %1308, %c3584_582 : index + %c2_i32_583 = arith.constant 2 : i32 + %1310 = arith.muli %44, %c2_i32_583 : i32 + %1311 = arith.index_cast %1310 : i32 to index + %1312 = arith.addi %1309, %1311 : index + %c12288_584 = arith.constant 12288 : index + %1313 = arith.addi %1312, %c12288_584 : index + %c4_585 = arith.constant 4 : index + %1314 = arith.muli %1313, %c4_585 : index + %1315 = arith.index_cast %1314 : index to i64 + %1316 = pto.castptr %1306 : !pto.ptr -> i64 + %1317 = llvm.inttoptr %1316 : i64 to !llvm.ptr<6> + %1318 = llvm.getelementptr %1317[%1315] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1305, %1318 : vector<2xf32>, !llvm.ptr<6> + %c30_586 = arith.constant 30 : index + %c4_587 = arith.constant 4 : index + %1319 = arith.muli %c30_586, %c4_587 : index + %1320 = arith.index_cast %1319 : index to i64 + %1321 = llvm.getelementptr %42[%1320] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1322 = llvm.load %1321 : !llvm.ptr -> vector<2xf32> + %1323 = pto.sqrt %891 : f32 -> f32 + %cst_588 = arith.constant 1.000000e+00 : f32 + %1324 = arith.divf %cst_588, %1323 : f32 + %1325 = llvm.mlir.undef : vector<2xf32> + %c0_i32_589 = arith.constant 0 : i32 + %1326 = llvm.insertelement %1324, %1325[%c0_i32_589 : i32] : vector<2xf32> + %c1_i32_590 = arith.constant 1 : i32 + %1327 = llvm.insertelement %1324, %1326[%c1_i32_590 : i32] : vector<2xf32> + %1328 = arith.mulf %1322, %1327 : vector<2xf32> + %c30_591 = arith.constant 30 : index + %c4_592 = arith.constant 4 : index + %1329 = arith.muli %c30_591, %c4_592 : index + %1330 = arith.index_cast %1329 : index to i64 + %1331 = llvm.getelementptr %1[%1330] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1332 = llvm.load %1331 : !llvm.ptr -> vector<2xf32> + %1333 = arith.mulf %1328, %1332 : vector<2xf32> + %1334 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_593 = arith.constant 1 : index + %1335 = arith.andi %arg5, %c1_593 : index + %c4096_594 = arith.constant 4096 : index + %1336 = arith.muli %1335, %c4096_594 : index + %c3840_595 = arith.constant 3840 : index + %1337 = arith.addi %1336, %c3840_595 : index + %c2_i32_596 = arith.constant 2 : i32 + %1338 = arith.muli %44, %c2_i32_596 : i32 + %1339 = arith.index_cast %1338 : i32 to index + %1340 = arith.addi %1337, %1339 : index + %c12288_597 = arith.constant 12288 : index + %1341 = arith.addi %1340, %c12288_597 : index + %c4_598 = arith.constant 4 : index + %1342 = arith.muli %1341, %c4_598 : index + %1343 = arith.index_cast %1342 : index to i64 + %1344 = pto.castptr %1334 : !pto.ptr -> i64 + %1345 = llvm.inttoptr %1344 : i64 to !llvm.ptr<6> + %1346 = llvm.getelementptr %1345[%1343] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1333, %1346 : vector<2xf32>, !llvm.ptr<6> + } + %c1_14 = arith.constant 1 : index + %18 = arith.andi %arg5, %c1_14 : index + pto.set_flag_dyn[, , %18] + %c1_15 = arith.constant 1 : index + %19 = arith.andi %arg5, %c1_15 : index + pto.set_flag_dyn[, , %19] + %c1_16 = arith.constant 1 : index + %20 = arith.andi %arg5, %c1_16 : index + pto.wait_flag_dyn[, , %20] + %21 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_17 = arith.constant 1 : index + %22 = arith.andi %arg5, %c1_17 : index + %c8 = arith.constant 8 : index + %23 = arith.muli %22, %c8 : index + %24 = pto.addptr %21, %23 : -> + %c64_18 = arith.constant 64 : index + %25 = arith.muli %arg5, %c64_18 : index + %26 = pto.get_block_idx + %27 = arith.index_cast %26 : i64 to index + %28 = arith.addi %25, %27 : index + %29 = pto.addptr %arg0, %28 : -> + %c1_i64_19 = arith.constant 1 : i64 + %c4_i64 = arith.constant 4 : i64 + %c4_i64_20 = arith.constant 4 : i64 + %c4_i64_21 = arith.constant 4 : i64 + %c0_i64_22 = arith.constant 0 : i64 + pto.mte_ub_gm %24, %29, %c4_i64_21 nburst(%c1_i64_19, %c4_i64, %c4_i64_20) l2_cache_ctl(%c0_i64_22) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + %30 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_23 = arith.constant 1 : index + %31 = arith.andi %arg5, %c1_23 : index + %c4096_24 = arith.constant 4096 : index + %32 = arith.muli %31, %c4096_24 : index + %c12288 = arith.constant 12288 : index + %33 = arith.addi %32, %c12288 : index + %34 = pto.addptr %30, %33 : -> + %c262144_25 = arith.constant 262144 : index + %35 = arith.muli %arg5, %c262144_25 : index + %36 = pto.get_block_idx + %c4096_i64_26 = arith.constant 4096 : i64 + %37 = arith.muli %36, %c4096_i64_26 : i64 + %38 = arith.index_cast %37 : i64 to index + %39 = arith.addi %35, %38 : index + %40 = pto.addptr %arg3, %39 : -> + %c1_i64_27 = arith.constant 1 : i64 + %c16384_i64_28 = arith.constant 16384 : i64 + %c16384_i64_29 = arith.constant 16384 : i64 + %c16384_i64_30 = arith.constant 16384 : i64 + %c0_i64_31 = arith.constant 0 : i64 + pto.mte_ub_gm %34, %40, %c16384_i64_30 nburst(%c1_i64_27, %c16384_i64_28, %c16384_i64_29) l2_cache_ctl(%c0_i64_31) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + %c1_32 = arith.constant 1 : index + %41 = arith.andi %arg5, %c1_32 : index + pto.set_flag_dyn[, , %41] + } + pto.wait_flag[, , ] + pto.wait_flag[, , ] + pto.wait_flag[, , ] + pto.wait_flag[, , ] + return + } + } +} diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/kernel.ptodsl.py b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/kernel.ptodsl.py new file mode 100644 index 0000000000..fa7b84a354 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/kernel.ptodsl.py @@ -0,0 +1,55 @@ +from ptodsl import pto, scalar +from ptodsl._ops import _coerce_i64 as _tl_coerce_i64 +from ptodsl._surface_values import wrap_surface_value as _tl_wrap_surface_value + +@pto.jit(name="rmsnorm_d4096_kernel", kernel_kind="vector", target="a5", mode="explicit") +def rmsnorm_d4096_kernel(RSTD: pto.ptr(pto.f32, "gm"), W: pto.ptr(pto.f32, "gm"), X: pto.ptr(pto.f32, "gm"), Y: pto.ptr(pto.f32, "gm"), eps: pto.f32): + buf_dyn_shmem = pto.castptr(pto.const(0, dtype=pto.i64), pto.ptr(pto.i8, "ub")) + w_frag = [None] * 32 + w_frag_1 = pto.alloc_buffer((32,), pto.f32) + pto.set_flag("MTE3", "V", event_id=0) + pto.set_flag("MTE3", "V", event_id=1) + pto.set_flag("V", "MTE2", event_id=0) + pto.set_flag("V", "MTE2", event_id=1) + pto.mte_gm_ub(W, pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), 0, 16384, nburst=(1, 16384, 16384)) + pto.set_flag("MTE2", "V", event_id=2) + pto.wait_flag("MTE2", "V", event_id=2) + with pto.simt(128, 1, 1): + simtvf_tx = pto.get_tid_x() + simtvf_ty = pto.get_tid_y() + simtvf_tz = pto.get_tid_z() + for i in pto.static_range(0, 16): + scalar.store(scalar.load(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), (i * 256) + (simtvf_tx * 2), contiguous=2), w_frag_1, i * 2) + with pto.for_(0, 64, step=1) as t: + pto.wait_flag("V", "MTE2", event_id=t & 1) + pto.mte_gm_ub(pto.addptr(X, (t * 262144) + (pto.get_block_idx() * 4096)), pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((t & 1) * 4096) + 4096), 0, 16384, nburst=(1, 16384, 16384)) + pto.set_flag("MTE2", "V", event_id=t & 1) + pto.wait_flag("MTE3", "V", event_id=t & 1) + pto.wait_flag("MTE2", "V", event_id=t & 1) + with pto.simt(128, 1, 1): + x_frag = pto.alloc_buffer((32,), pto.f32) + sum_sq = pto.alloc_buffer((1,), pto.f32) + simtvf_tx_1 = pto.get_tid_x() + simtvf_ty_1 = pto.get_tid_y() + simtvf_tz_1 = pto.get_tid_z() + for i_1 in pto.static_range(0, 16): + scalar.store(scalar.load(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((((t & 1) * 4096) + (i_1 * 256)) + (simtvf_tx_1 * 2)) + 4096, contiguous=2), x_frag, i_1 * 2) + scalar.store(float.fromhex('0x0p+0'), sum_sq, 0) + for i_2 in pto.static_range(0, 32): + scalar.store(scalar.load(sum_sq, 0) + (scalar.load(x_frag, i_2) * scalar.load(x_frag, i_2)), sum_sq, 0) + scalar.store(pto.simt_allreduce_sum(scalar.load(sum_sq, 0), threads=128, scale=1, thread_offset=0, scratch=pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), 20480)), sum_sq, 0) + var = (scalar.load(sum_sq, 0) / float.fromhex('0x1p+12')) + eps + rstd_val = float.fromhex('0x1p+0') / pto.sqrt(var) + scalar.store(float.fromhex('0x1p+0') / pto.sqrt(var), pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), (t & 1) * 8) + for i_3 in pto.static_range(0, 16): + scalar.store((scalar.load(x_frag, i_3 * 2, contiguous=2) * pto.Vec(pto.f32, 2, init=(float.fromhex('0x1p+0') / pto.sqrt(var)))) * scalar.load(w_frag_1, i_3 * 2, contiguous=2), pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((((t & 1) * 4096) + (i_3 * 256)) + (simtvf_tx_1 * 2)) + 12288) + pto.set_flag("V", "MTE3", event_id=t & 1) + pto.set_flag("V", "MTE2", event_id=t & 1) + pto.wait_flag("V", "MTE3", event_id=t & 1) + pto.mte_ub_gm(pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), (t & 1) * 8), pto.addptr(RSTD, (t * 64) + pto.get_block_idx()), 4, nburst=(1, 4, 4)) + pto.mte_ub_gm(pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((t & 1) * 4096) + 12288), pto.addptr(Y, (t * 262144) + (pto.get_block_idx() * 4096)), 16384, nburst=(1, 16384, 16384)) + pto.set_flag("MTE3", "V", event_id=t & 1) + pto.wait_flag("MTE3", "V", event_id=0) + pto.wait_flag("MTE3", "V", event_id=1) + pto.wait_flag("V", "MTE2", event_id=0) + pto.wait_flag("V", "MTE2", event_id=1) diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/kernel_vpto.ll b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/kernel_vpto.ll new file mode 100644 index 0000000000..4538c7500d --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/kernel_vpto.ll @@ -0,0 +1,866 @@ +; ModuleID = 'ptoas.hivm.official.vector' +source_filename = "ptoas.hivm.official.vector" + +; Unknown intrinsic +declare void @llvm.hivm.SET.FLAG.IMM(i64, i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.MOV.OUT.TO.UB.ALIGN.V2.f32.DV(ptr addrspace(6), ptr addrspace(1), i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.WAIT.FLAG.IMM(i64, i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.store.vfsimt.info(i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.WAIT.FLAG.REG(i64, i64, i64) #0 + +; Unknown intrinsic +declare i64 @llvm.hivm.GET.BLOCK.IDX() #0 + +; Unknown intrinsic +declare void @llvm.hivm.SET.FLAG.REG(i64, i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.MOV.UB.TO.OUT.ALIGN.V2.DV(ptr addrspace(1), ptr addrspace(6), i64, i64) #0 + +; Unknown intrinsic +declare i32 @llvm.hivm.get.TID.X() #0 + +; Unknown intrinsic +declare i32 @llvm.hivm.get.laneID() #0 + +; Unknown intrinsic +declare float @llvm.hivm.redux.add.f32(float) #0 + +; Unknown intrinsic +declare void @llvm.hivm.sync.workitems() #0 + +; Function Attrs: nocallback nofree nosync nounwind speculatable willreturn memory(none) +declare float @llvm.sqrt.f32(float) #1 + +define void @rmsnorm_d4096_kernel_mix_aiv(ptr addrspace(1) %0, ptr addrspace(1) %1, ptr addrspace(1) %2, ptr addrspace(1) %3, float %4) #2 { + call void @llvm.hivm.SET.FLAG.IMM(i64 5, i64 1, i64 0) + call void @llvm.hivm.SET.FLAG.IMM(i64 5, i64 1, i64 1) + call void @llvm.hivm.SET.FLAG.IMM(i64 1, i64 4, i64 0) + call void @llvm.hivm.SET.FLAG.IMM(i64 1, i64 4, i64 1) + call void @llvm.hivm.MOV.OUT.TO.UB.ALIGN.V2.f32.DV(ptr addrspace(6) null, ptr addrspace(1) %1, i64 549755813904, i64 18014398509498368) + call void @llvm.hivm.SET.FLAG.IMM(i64 4, i64 1, i64 2) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 4, i64 1, i64 2) + call void @llvm.hivm.store.vfsimt.info(i64 4295032960) + call simt_entry void @rmsnorm_d4096_kernel_simt_0(ptr addrspace(6) null) + br label %6 + +6: ; preds = %9, %5 + %7 = phi i64 [ %27, %9 ], [ 0, %5 ] + %8 = icmp slt i64 %7, 64 + br i1 %8, label %9, label %28 + +9: ; preds = %6 + %10 = and i64 %7, 1 + call void @llvm.hivm.WAIT.FLAG.REG(i64 1, i64 4, i64 %10) + %11 = mul i64 %7, 262144 + %12 = call i64 @llvm.hivm.GET.BLOCK.IDX() + %13 = mul i64 %12, 4096 + %14 = add i64 %11, %13 + %15 = getelementptr float, ptr addrspace(1) %2, i64 %14 + %16 = mul i64 %10, 4096 + %17 = add i64 %16, 4096 + %18 = getelementptr float, ptr addrspace(6) null, i64 %17 + call void @llvm.hivm.MOV.OUT.TO.UB.ALIGN.V2.f32.DV(ptr addrspace(6) %18, ptr addrspace(1) %15, i64 549755813904, i64 18014398509498368) + call void @llvm.hivm.SET.FLAG.REG(i64 4, i64 1, i64 %10) + call void @llvm.hivm.WAIT.FLAG.REG(i64 5, i64 1, i64 %10) + call void @llvm.hivm.WAIT.FLAG.REG(i64 4, i64 1, i64 %10) + call void @llvm.hivm.store.vfsimt.info(i64 4295032960) + call simt_entry void @rmsnorm_d4096_kernel_simt_1(i64 %16, ptr addrspace(6) null, float %4, i64 %10) + call void @llvm.hivm.SET.FLAG.REG(i64 1, i64 5, i64 %10) + call void @llvm.hivm.SET.FLAG.REG(i64 1, i64 4, i64 %10) + call void @llvm.hivm.WAIT.FLAG.REG(i64 1, i64 5, i64 %10) + %19 = mul i64 %10, 8 + %20 = getelementptr float, ptr addrspace(6) null, i64 %19 + %21 = mul i64 %7, 64 + %22 = add i64 %21, %12 + %23 = getelementptr float, ptr addrspace(1) %0, i64 %22 + call void @llvm.hivm.MOV.UB.TO.OUT.ALIGN.V2.DV(ptr addrspace(1) %23, ptr addrspace(6) %20, i64 134217744, i64 4398046511108) + %24 = add i64 %16, 12288 + %25 = getelementptr float, ptr addrspace(6) null, i64 %24 + %26 = getelementptr float, ptr addrspace(1) %3, i64 %14 + call void @llvm.hivm.MOV.UB.TO.OUT.ALIGN.V2.DV(ptr addrspace(1) %26, ptr addrspace(6) %25, i64 549755813904, i64 18014398509498368) + call void @llvm.hivm.SET.FLAG.REG(i64 5, i64 1, i64 %10) + %27 = add i64 %7, 1 + br label %6 + +28: ; preds = %6 + call void @llvm.hivm.WAIT.FLAG.IMM(i64 5, i64 1, i64 0) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 5, i64 1, i64 1) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 1, i64 4, i64 0) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 1, i64 4, i64 1) + ret void +} + +; Function Attrs: noinline +define linkonce_odr simt_entry void @rmsnorm_d4096_kernel_simt_0(ptr addrspace(6) %0) #3 !annotation !8 !annotation !9 { + %2 = call i32 @llvm.hivm.get.TID.X() + %3 = mul i32 %2, 2 + %4 = sext i32 %3 to i64 + %5 = mul i64 %4, 4 + %6 = ptrtoint ptr addrspace(6) %0 to i64 + %7 = inttoptr i64 %6 to ptr addrspace(6) + %8 = getelementptr i8, ptr addrspace(6) %7, i64 %5 + %9 = load <2 x float>, ptr addrspace(6) %8, align 8 + %10 = extractelement <2 x float> %9, i32 0 + %11 = extractelement <2 x float> %9, i32 1 + %12 = add i32 %3, 256 + %13 = sext i32 %12 to i64 + %14 = mul i64 %13, 4 + %15 = getelementptr i8, ptr addrspace(6) %7, i64 %14 + %16 = load <2 x float>, ptr addrspace(6) %15, align 8 + %17 = extractelement <2 x float> %16, i32 0 + %18 = extractelement <2 x float> %16, i32 1 + %19 = add i32 %3, 512 + %20 = sext i32 %19 to i64 + %21 = mul i64 %20, 4 + %22 = getelementptr i8, ptr addrspace(6) %7, i64 %21 + %23 = load <2 x float>, ptr addrspace(6) %22, align 8 + %24 = extractelement <2 x float> %23, i32 0 + %25 = extractelement <2 x float> %23, i32 1 + %26 = add i32 %3, 768 + %27 = sext i32 %26 to i64 + %28 = mul i64 %27, 4 + %29 = getelementptr i8, ptr addrspace(6) %7, i64 %28 + %30 = load <2 x float>, ptr addrspace(6) %29, align 8 + %31 = extractelement <2 x float> %30, i32 0 + %32 = extractelement <2 x float> %30, i32 1 + %33 = add i32 %3, 1024 + %34 = sext i32 %33 to i64 + %35 = mul i64 %34, 4 + %36 = getelementptr i8, ptr addrspace(6) %7, i64 %35 + %37 = load <2 x float>, ptr addrspace(6) %36, align 8 + %38 = extractelement <2 x float> %37, i32 0 + %39 = extractelement <2 x float> %37, i32 1 + %40 = add i32 %3, 1280 + %41 = sext i32 %40 to i64 + %42 = mul i64 %41, 4 + %43 = getelementptr i8, ptr addrspace(6) %7, i64 %42 + %44 = load <2 x float>, ptr addrspace(6) %43, align 8 + %45 = extractelement <2 x float> %44, i32 0 + %46 = extractelement <2 x float> %44, i32 1 + %47 = add i32 %3, 1536 + %48 = sext i32 %47 to i64 + %49 = mul i64 %48, 4 + %50 = getelementptr i8, ptr addrspace(6) %7, i64 %49 + %51 = load <2 x float>, ptr addrspace(6) %50, align 8 + %52 = extractelement <2 x float> %51, i32 0 + %53 = extractelement <2 x float> %51, i32 1 + %54 = add i32 %3, 1792 + %55 = sext i32 %54 to i64 + %56 = mul i64 %55, 4 + %57 = getelementptr i8, ptr addrspace(6) %7, i64 %56 + %58 = load <2 x float>, ptr addrspace(6) %57, align 8 + %59 = extractelement <2 x float> %58, i32 0 + %60 = extractelement <2 x float> %58, i32 1 + %61 = add i32 %3, 2048 + %62 = sext i32 %61 to i64 + %63 = mul i64 %62, 4 + %64 = getelementptr i8, ptr addrspace(6) %7, i64 %63 + %65 = load <2 x float>, ptr addrspace(6) %64, align 8 + %66 = extractelement <2 x float> %65, i32 0 + %67 = extractelement <2 x float> %65, i32 1 + %68 = add i32 %3, 2304 + %69 = sext i32 %68 to i64 + %70 = mul i64 %69, 4 + %71 = getelementptr i8, ptr addrspace(6) %7, i64 %70 + %72 = load <2 x float>, ptr addrspace(6) %71, align 8 + %73 = extractelement <2 x float> %72, i32 0 + %74 = extractelement <2 x float> %72, i32 1 + %75 = add i32 %3, 2560 + %76 = sext i32 %75 to i64 + %77 = mul i64 %76, 4 + %78 = getelementptr i8, ptr addrspace(6) %7, i64 %77 + %79 = load <2 x float>, ptr addrspace(6) %78, align 8 + %80 = extractelement <2 x float> %79, i32 0 + %81 = extractelement <2 x float> %79, i32 1 + %82 = add i32 %3, 2816 + %83 = sext i32 %82 to i64 + %84 = mul i64 %83, 4 + %85 = getelementptr i8, ptr addrspace(6) %7, i64 %84 + %86 = load <2 x float>, ptr addrspace(6) %85, align 8 + %87 = extractelement <2 x float> %86, i32 0 + %88 = extractelement <2 x float> %86, i32 1 + %89 = add i32 %3, 3072 + %90 = sext i32 %89 to i64 + %91 = mul i64 %90, 4 + %92 = getelementptr i8, ptr addrspace(6) %7, i64 %91 + %93 = load <2 x float>, ptr addrspace(6) %92, align 8 + %94 = extractelement <2 x float> %93, i32 0 + %95 = extractelement <2 x float> %93, i32 1 + %96 = add i32 %3, 3328 + %97 = sext i32 %96 to i64 + %98 = mul i64 %97, 4 + %99 = getelementptr i8, ptr addrspace(6) %7, i64 %98 + %100 = load <2 x float>, ptr addrspace(6) %99, align 8 + %101 = extractelement <2 x float> %100, i32 0 + %102 = extractelement <2 x float> %100, i32 1 + %103 = add i32 %3, 3584 + %104 = sext i32 %103 to i64 + %105 = mul i64 %104, 4 + %106 = getelementptr i8, ptr addrspace(6) %7, i64 %105 + %107 = load <2 x float>, ptr addrspace(6) %106, align 8 + %108 = extractelement <2 x float> %107, i32 0 + %109 = extractelement <2 x float> %107, i32 1 + %110 = add i32 %3, 3840 + %111 = sext i32 %110 to i64 + %112 = mul i64 %111, 4 + %113 = getelementptr i8, ptr addrspace(6) %7, i64 %112 + %114 = load <2 x float>, ptr addrspace(6) %113, align 8 + %115 = extractelement <2 x float> %114, i32 0 + %116 = extractelement <2 x float> %114, i32 1 + %117 = bitcast float %10 to i32 + %118 = bitcast float %11 to i32 + %119 = bitcast float %17 to i32 + %120 = bitcast float %18 to i32 + %121 = bitcast float %24 to i32 + %122 = bitcast float %25 to i32 + %123 = bitcast float %31 to i32 + %124 = bitcast float %32 to i32 + %125 = bitcast float %38 to i32 + %126 = bitcast float %39 to i32 + %127 = bitcast float %45 to i32 + %128 = bitcast float %46 to i32 + %129 = bitcast float %52 to i32 + %130 = bitcast float %53 to i32 + %131 = bitcast float %59 to i32 + %132 = bitcast float %60 to i32 + %133 = bitcast float %66 to i32 + %134 = bitcast float %67 to i32 + %135 = bitcast float %73 to i32 + %136 = bitcast float %74 to i32 + %137 = bitcast float %80 to i32 + %138 = bitcast float %81 to i32 + %139 = bitcast float %87 to i32 + %140 = bitcast float %88 to i32 + %141 = bitcast float %94 to i32 + %142 = bitcast float %95 to i32 + %143 = bitcast float %101 to i32 + %144 = bitcast float %102 to i32 + %145 = bitcast float %108 to i32 + %146 = bitcast float %109 to i32 + %147 = bitcast float %115 to i32 + %148 = bitcast float %116 to i32 + %149 = call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "", "={TPER4},={TPER5},={TPER6},={TPER7},={TPER8},={TPER9},={TPER10},={TPER11},={TPER12},={TPER13},={TPER14},={TPER15},={TPER16},={TPER17},={TPER18},={TPER19},={TPER20},={TPER21},={TPER22},={TPER23},={TPER24},={TPER25},={TPER26},={TPER27},={TPER28},={TPER29},={TPER30},={TPER31},={TPER32},={TPER33},={TPER34},={TPER35},0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31"(i32 %117, i32 %118, i32 %119, i32 %120, i32 %121, i32 %122, i32 %123, i32 %124, i32 %125, i32 %126, i32 %127, i32 %128, i32 %129, i32 %130, i32 %131, i32 %132, i32 %133, i32 %134, i32 %135, i32 %136, i32 %137, i32 %138, i32 %139, i32 %140, i32 %141, i32 %142, i32 %143, i32 %144, i32 %145, i32 %146, i32 %147, i32 %148) + ret void +} + +; Function Attrs: noinline +define linkonce_odr simt_entry void @rmsnorm_d4096_kernel_simt_1(i64 %0, ptr addrspace(6) %1, float %2, i64 %3) #3 !annotation !8 !annotation !9 { + %5 = call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "", "={TPER4},={TPER5},={TPER6},={TPER7},={TPER8},={TPER9},={TPER10},={TPER11},={TPER12},={TPER13},={TPER14},={TPER15},={TPER16},={TPER17},={TPER18},={TPER19},={TPER20},={TPER21},={TPER22},={TPER23},={TPER24},={TPER25},={TPER26},={TPER27},={TPER28},={TPER29},={TPER30},={TPER31},={TPER32},={TPER33},={TPER34},={TPER35}"() + %6 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 0 + %7 = bitcast i32 %6 to float + %8 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 1 + %9 = bitcast i32 %8 to float + %10 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 2 + %11 = bitcast i32 %10 to float + %12 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 3 + %13 = bitcast i32 %12 to float + %14 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 4 + %15 = bitcast i32 %14 to float + %16 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 5 + %17 = bitcast i32 %16 to float + %18 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 6 + %19 = bitcast i32 %18 to float + %20 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 7 + %21 = bitcast i32 %20 to float + %22 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 8 + %23 = bitcast i32 %22 to float + %24 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 9 + %25 = bitcast i32 %24 to float + %26 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 10 + %27 = bitcast i32 %26 to float + %28 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 11 + %29 = bitcast i32 %28 to float + %30 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 12 + %31 = bitcast i32 %30 to float + %32 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 13 + %33 = bitcast i32 %32 to float + %34 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 14 + %35 = bitcast i32 %34 to float + %36 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 15 + %37 = bitcast i32 %36 to float + %38 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 16 + %39 = bitcast i32 %38 to float + %40 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 17 + %41 = bitcast i32 %40 to float + %42 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 18 + %43 = bitcast i32 %42 to float + %44 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 19 + %45 = bitcast i32 %44 to float + %46 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 20 + %47 = bitcast i32 %46 to float + %48 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 21 + %49 = bitcast i32 %48 to float + %50 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 22 + %51 = bitcast i32 %50 to float + %52 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 23 + %53 = bitcast i32 %52 to float + %54 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 24 + %55 = bitcast i32 %54 to float + %56 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 25 + %57 = bitcast i32 %56 to float + %58 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 26 + %59 = bitcast i32 %58 to float + %60 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 27 + %61 = bitcast i32 %60 to float + %62 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 28 + %63 = bitcast i32 %62 to float + %64 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 29 + %65 = bitcast i32 %64 to float + %66 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 30 + %67 = bitcast i32 %66 to float + %68 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %5, 31 + %69 = bitcast i32 %68 to float + %70 = alloca float, i32 32, align 4 + %71 = alloca float, align 4 + %72 = call i32 @llvm.hivm.get.TID.X() + %73 = mul i32 %72, 2 + %74 = sext i32 %73 to i64 + %75 = add i64 %0, %74 + %76 = add i64 %75, 4096 + %77 = mul i64 %76, 4 + %78 = ptrtoint ptr addrspace(6) %1 to i64 + %79 = inttoptr i64 %78 to ptr addrspace(6) + %80 = getelementptr i8, ptr addrspace(6) %79, i64 %77 + %81 = load <2 x float>, ptr addrspace(6) %80, align 8 + store <2 x float> %81, ptr %70, align 8 + %82 = add i64 %0, 256 + %83 = add i64 %82, %74 + %84 = add i64 %83, 4096 + %85 = mul i64 %84, 4 + %86 = getelementptr i8, ptr addrspace(6) %79, i64 %85 + %87 = load <2 x float>, ptr addrspace(6) %86, align 8 + %88 = getelementptr i8, ptr %70, i32 8 + store <2 x float> %87, ptr %88, align 8 + %89 = add i64 %0, 512 + %90 = add i64 %89, %74 + %91 = add i64 %90, 4096 + %92 = mul i64 %91, 4 + %93 = getelementptr i8, ptr addrspace(6) %79, i64 %92 + %94 = load <2 x float>, ptr addrspace(6) %93, align 8 + %95 = getelementptr i8, ptr %70, i32 16 + store <2 x float> %94, ptr %95, align 8 + %96 = add i64 %0, 768 + %97 = add i64 %96, %74 + %98 = add i64 %97, 4096 + %99 = mul i64 %98, 4 + %100 = getelementptr i8, ptr addrspace(6) %79, i64 %99 + %101 = load <2 x float>, ptr addrspace(6) %100, align 8 + %102 = getelementptr i8, ptr %70, i32 24 + store <2 x float> %101, ptr %102, align 8 + %103 = add i64 %0, 1024 + %104 = add i64 %103, %74 + %105 = add i64 %104, 4096 + %106 = mul i64 %105, 4 + %107 = getelementptr i8, ptr addrspace(6) %79, i64 %106 + %108 = load <2 x float>, ptr addrspace(6) %107, align 8 + %109 = getelementptr i8, ptr %70, i32 32 + store <2 x float> %108, ptr %109, align 8 + %110 = add i64 %0, 1280 + %111 = add i64 %110, %74 + %112 = add i64 %111, 4096 + %113 = mul i64 %112, 4 + %114 = getelementptr i8, ptr addrspace(6) %79, i64 %113 + %115 = load <2 x float>, ptr addrspace(6) %114, align 8 + %116 = getelementptr i8, ptr %70, i32 40 + store <2 x float> %115, ptr %116, align 8 + %117 = add i64 %0, 1536 + %118 = add i64 %117, %74 + %119 = add i64 %118, 4096 + %120 = mul i64 %119, 4 + %121 = getelementptr i8, ptr addrspace(6) %79, i64 %120 + %122 = load <2 x float>, ptr addrspace(6) %121, align 8 + %123 = getelementptr i8, ptr %70, i32 48 + store <2 x float> %122, ptr %123, align 8 + %124 = add i64 %0, 1792 + %125 = add i64 %124, %74 + %126 = add i64 %125, 4096 + %127 = mul i64 %126, 4 + %128 = getelementptr i8, ptr addrspace(6) %79, i64 %127 + %129 = load <2 x float>, ptr addrspace(6) %128, align 8 + %130 = getelementptr i8, ptr %70, i32 56 + store <2 x float> %129, ptr %130, align 8 + %131 = add i64 %0, 2048 + %132 = add i64 %131, %74 + %133 = add i64 %132, 4096 + %134 = mul i64 %133, 4 + %135 = getelementptr i8, ptr addrspace(6) %79, i64 %134 + %136 = load <2 x float>, ptr addrspace(6) %135, align 8 + %137 = getelementptr i8, ptr %70, i32 64 + store <2 x float> %136, ptr %137, align 8 + %138 = add i64 %0, 2304 + %139 = add i64 %138, %74 + %140 = add i64 %139, 4096 + %141 = mul i64 %140, 4 + %142 = getelementptr i8, ptr addrspace(6) %79, i64 %141 + %143 = load <2 x float>, ptr addrspace(6) %142, align 8 + %144 = getelementptr i8, ptr %70, i32 72 + store <2 x float> %143, ptr %144, align 8 + %145 = add i64 %0, 2560 + %146 = add i64 %145, %74 + %147 = add i64 %146, 4096 + %148 = mul i64 %147, 4 + %149 = getelementptr i8, ptr addrspace(6) %79, i64 %148 + %150 = load <2 x float>, ptr addrspace(6) %149, align 8 + %151 = getelementptr i8, ptr %70, i32 80 + store <2 x float> %150, ptr %151, align 8 + %152 = add i64 %0, 2816 + %153 = add i64 %152, %74 + %154 = add i64 %153, 4096 + %155 = mul i64 %154, 4 + %156 = getelementptr i8, ptr addrspace(6) %79, i64 %155 + %157 = load <2 x float>, ptr addrspace(6) %156, align 8 + %158 = getelementptr i8, ptr %70, i32 88 + store <2 x float> %157, ptr %158, align 8 + %159 = add i64 %0, 3072 + %160 = add i64 %159, %74 + %161 = add i64 %160, 4096 + %162 = mul i64 %161, 4 + %163 = getelementptr i8, ptr addrspace(6) %79, i64 %162 + %164 = load <2 x float>, ptr addrspace(6) %163, align 8 + %165 = getelementptr i8, ptr %70, i32 96 + store <2 x float> %164, ptr %165, align 8 + %166 = add i64 %0, 3328 + %167 = add i64 %166, %74 + %168 = add i64 %167, 4096 + %169 = mul i64 %168, 4 + %170 = getelementptr i8, ptr addrspace(6) %79, i64 %169 + %171 = load <2 x float>, ptr addrspace(6) %170, align 8 + %172 = getelementptr i8, ptr %70, i32 104 + store <2 x float> %171, ptr %172, align 8 + %173 = add i64 %0, 3584 + %174 = add i64 %173, %74 + %175 = add i64 %174, 4096 + %176 = mul i64 %175, 4 + %177 = getelementptr i8, ptr addrspace(6) %79, i64 %176 + %178 = load <2 x float>, ptr addrspace(6) %177, align 8 + %179 = getelementptr i8, ptr %70, i32 112 + store <2 x float> %178, ptr %179, align 8 + %180 = add i64 %0, 3840 + %181 = add i64 %180, %74 + %182 = add i64 %181, 4096 + %183 = mul i64 %182, 4 + %184 = getelementptr i8, ptr addrspace(6) %79, i64 %183 + %185 = load <2 x float>, ptr addrspace(6) %184, align 8 + %186 = getelementptr i8, ptr %70, i32 120 + store <2 x float> %185, ptr %186, align 8 + store float 0.000000e+00, ptr %71, align 4 + %187 = load float, ptr %71, align 4 + %188 = load float, ptr %70, align 4 + %189 = fmul float %188, %188 + %190 = fadd float %187, %189 + store float %190, ptr %71, align 4 + %191 = load float, ptr %71, align 4 + %192 = getelementptr i8, ptr %70, i32 4 + %193 = load float, ptr %192, align 4 + %194 = fmul float %193, %193 + %195 = fadd float %191, %194 + store float %195, ptr %71, align 4 + %196 = load float, ptr %71, align 4 + %197 = load float, ptr %88, align 4 + %198 = fmul float %197, %197 + %199 = fadd float %196, %198 + store float %199, ptr %71, align 4 + %200 = load float, ptr %71, align 4 + %201 = getelementptr i8, ptr %70, i32 12 + %202 = load float, ptr %201, align 4 + %203 = fmul float %202, %202 + %204 = fadd float %200, %203 + store float %204, ptr %71, align 4 + %205 = load float, ptr %71, align 4 + %206 = load float, ptr %95, align 4 + %207 = fmul float %206, %206 + %208 = fadd float %205, %207 + store float %208, ptr %71, align 4 + %209 = load float, ptr %71, align 4 + %210 = getelementptr i8, ptr %70, i32 20 + %211 = load float, ptr %210, align 4 + %212 = fmul float %211, %211 + %213 = fadd float %209, %212 + store float %213, ptr %71, align 4 + %214 = load float, ptr %71, align 4 + %215 = load float, ptr %102, align 4 + %216 = fmul float %215, %215 + %217 = fadd float %214, %216 + store float %217, ptr %71, align 4 + %218 = load float, ptr %71, align 4 + %219 = getelementptr i8, ptr %70, i32 28 + %220 = load float, ptr %219, align 4 + %221 = fmul float %220, %220 + %222 = fadd float %218, %221 + store float %222, ptr %71, align 4 + %223 = load float, ptr %71, align 4 + %224 = load float, ptr %109, align 4 + %225 = fmul float %224, %224 + %226 = fadd float %223, %225 + store float %226, ptr %71, align 4 + %227 = load float, ptr %71, align 4 + %228 = getelementptr i8, ptr %70, i32 36 + %229 = load float, ptr %228, align 4 + %230 = fmul float %229, %229 + %231 = fadd float %227, %230 + store float %231, ptr %71, align 4 + %232 = load float, ptr %71, align 4 + %233 = load float, ptr %116, align 4 + %234 = fmul float %233, %233 + %235 = fadd float %232, %234 + store float %235, ptr %71, align 4 + %236 = load float, ptr %71, align 4 + %237 = getelementptr i8, ptr %70, i32 44 + %238 = load float, ptr %237, align 4 + %239 = fmul float %238, %238 + %240 = fadd float %236, %239 + store float %240, ptr %71, align 4 + %241 = load float, ptr %71, align 4 + %242 = load float, ptr %123, align 4 + %243 = fmul float %242, %242 + %244 = fadd float %241, %243 + store float %244, ptr %71, align 4 + %245 = load float, ptr %71, align 4 + %246 = getelementptr i8, ptr %70, i32 52 + %247 = load float, ptr %246, align 4 + %248 = fmul float %247, %247 + %249 = fadd float %245, %248 + store float %249, ptr %71, align 4 + %250 = load float, ptr %71, align 4 + %251 = load float, ptr %130, align 4 + %252 = fmul float %251, %251 + %253 = fadd float %250, %252 + store float %253, ptr %71, align 4 + %254 = load float, ptr %71, align 4 + %255 = getelementptr i8, ptr %70, i32 60 + %256 = load float, ptr %255, align 4 + %257 = fmul float %256, %256 + %258 = fadd float %254, %257 + store float %258, ptr %71, align 4 + %259 = load float, ptr %71, align 4 + %260 = load float, ptr %137, align 4 + %261 = fmul float %260, %260 + %262 = fadd float %259, %261 + store float %262, ptr %71, align 4 + %263 = load float, ptr %71, align 4 + %264 = getelementptr i8, ptr %70, i32 68 + %265 = load float, ptr %264, align 4 + %266 = fmul float %265, %265 + %267 = fadd float %263, %266 + store float %267, ptr %71, align 4 + %268 = load float, ptr %71, align 4 + %269 = load float, ptr %144, align 4 + %270 = fmul float %269, %269 + %271 = fadd float %268, %270 + store float %271, ptr %71, align 4 + %272 = load float, ptr %71, align 4 + %273 = getelementptr i8, ptr %70, i32 76 + %274 = load float, ptr %273, align 4 + %275 = fmul float %274, %274 + %276 = fadd float %272, %275 + store float %276, ptr %71, align 4 + %277 = load float, ptr %71, align 4 + %278 = load float, ptr %151, align 4 + %279 = fmul float %278, %278 + %280 = fadd float %277, %279 + store float %280, ptr %71, align 4 + %281 = load float, ptr %71, align 4 + %282 = getelementptr i8, ptr %70, i32 84 + %283 = load float, ptr %282, align 4 + %284 = fmul float %283, %283 + %285 = fadd float %281, %284 + store float %285, ptr %71, align 4 + %286 = load float, ptr %71, align 4 + %287 = load float, ptr %158, align 4 + %288 = fmul float %287, %287 + %289 = fadd float %286, %288 + store float %289, ptr %71, align 4 + %290 = load float, ptr %71, align 4 + %291 = getelementptr i8, ptr %70, i32 92 + %292 = load float, ptr %291, align 4 + %293 = fmul float %292, %292 + %294 = fadd float %290, %293 + store float %294, ptr %71, align 4 + %295 = load float, ptr %71, align 4 + %296 = load float, ptr %165, align 4 + %297 = fmul float %296, %296 + %298 = fadd float %295, %297 + store float %298, ptr %71, align 4 + %299 = load float, ptr %71, align 4 + %300 = getelementptr i8, ptr %70, i32 100 + %301 = load float, ptr %300, align 4 + %302 = fmul float %301, %301 + %303 = fadd float %299, %302 + store float %303, ptr %71, align 4 + %304 = load float, ptr %71, align 4 + %305 = load float, ptr %172, align 4 + %306 = fmul float %305, %305 + %307 = fadd float %304, %306 + store float %307, ptr %71, align 4 + %308 = load float, ptr %71, align 4 + %309 = getelementptr i8, ptr %70, i32 108 + %310 = load float, ptr %309, align 4 + %311 = fmul float %310, %310 + %312 = fadd float %308, %311 + store float %312, ptr %71, align 4 + %313 = load float, ptr %71, align 4 + %314 = load float, ptr %179, align 4 + %315 = fmul float %314, %314 + %316 = fadd float %313, %315 + store float %316, ptr %71, align 4 + %317 = load float, ptr %71, align 4 + %318 = getelementptr i8, ptr %70, i32 116 + %319 = load float, ptr %318, align 4 + %320 = fmul float %319, %319 + %321 = fadd float %317, %320 + store float %321, ptr %71, align 4 + %322 = load float, ptr %71, align 4 + %323 = load float, ptr %186, align 4 + %324 = fmul float %323, %323 + %325 = fadd float %322, %324 + store float %325, ptr %71, align 4 + %326 = load float, ptr %71, align 4 + %327 = getelementptr i8, ptr %70, i32 124 + %328 = load float, ptr %327, align 4 + %329 = fmul float %328, %328 + %330 = fadd float %326, %329 + store float %330, ptr %71, align 4 + %331 = load float, ptr %71, align 4 + %332 = getelementptr float, ptr addrspace(6) %1, i64 20480 + %333 = sdiv i32 %72, 32 + %334 = mul i32 %333, 32 + %335 = icmp ne i32 %72, %334 + %336 = icmp slt i32 %72, 0 + %337 = icmp ne i1 %336, false + %338 = and i1 %335, %337 + %339 = add i32 %333, -1 + %340 = select i1 %338, i32 %339, i32 %333 + %341 = call i32 @llvm.hivm.get.laneID() + %342 = call float @llvm.hivm.redux.add.f32(float %331) + %343 = icmp slt i32 %341, 1 + br i1 %343, label %344, label %348 + +344: ; preds = %4 + %345 = add i32 %340, %341 + %346 = sext i32 %345 to i64 + %347 = getelementptr float, ptr addrspace(6) %332, i64 %346 + store float %342, ptr addrspace(6) %347, align 4 + br label %348 + +348: ; preds = %344, %4 + call void @llvm.hivm.sync.workitems() + %349 = icmp slt i32 %72, 32 + br i1 %349, label %350, label %357 + +350: ; preds = %348 + %351 = icmp slt i32 %341, 4 + %352 = sext i32 %341 to i64 + %353 = getelementptr float, ptr addrspace(6) %332, i64 %352 + %354 = load float, ptr addrspace(6) %353, align 4 + %355 = select i1 %351, float %354, float 0.000000e+00 + %356 = call float @llvm.hivm.redux.add.f32(float %355) + br label %358 + +357: ; preds = %348 + br label %358 + +358: ; preds = %350, %357 + %359 = phi float [ 0.000000e+00, %357 ], [ %356, %350 ] + br label %360 + +360: ; preds = %358 + %361 = icmp slt i32 %72, 1 + br i1 %361, label %362, label %365 + +362: ; preds = %360 + %363 = sext i32 %72 to i64 + %364 = getelementptr float, ptr addrspace(6) %332, i64 %363 + store float %359, ptr addrspace(6) %364, align 4 + br label %365 + +365: ; preds = %362, %360 + call void @llvm.hivm.sync.workitems() + %366 = getelementptr float, ptr addrspace(6) %332, i64 0 + %367 = load float, ptr addrspace(6) %366, align 4 + call void @llvm.hivm.sync.workitems() + store float %367, ptr %71, align 4 + %368 = load float, ptr %71, align 4 + %369 = fdiv float %368, 4.096000e+03 + %370 = fadd float %369, %2 + %371 = call float @llvm.sqrt.f32(float %370) + %372 = fdiv float 1.000000e+00, %371 + %373 = mul i64 %3, 8 + %374 = getelementptr float, ptr addrspace(6) %1, i64 %373 + store float %372, ptr addrspace(6) %374, align 4 + %375 = load <2 x float>, ptr %70, align 8 + %376 = insertelement <2 x float> undef, float %372, i32 0 + %377 = insertelement <2 x float> %376, float %372, i32 1 + %378 = fmul <2 x float> %375, %377 + %379 = insertelement <2 x float> poison, float %7, i32 0 + %380 = insertelement <2 x float> %379, float %9, i32 1 + %381 = fmul <2 x float> %378, %380 + %382 = add i64 %75, 12288 + %383 = mul i64 %382, 4 + %384 = getelementptr i8, ptr addrspace(6) %79, i64 %383 + store <2 x float> %381, ptr addrspace(6) %384, align 8 + %385 = load <2 x float>, ptr %88, align 8 + %386 = fmul <2 x float> %385, %377 + %387 = insertelement <2 x float> poison, float %11, i32 0 + %388 = insertelement <2 x float> %387, float %13, i32 1 + %389 = fmul <2 x float> %386, %388 + %390 = add i64 %83, 12288 + %391 = mul i64 %390, 4 + %392 = getelementptr i8, ptr addrspace(6) %79, i64 %391 + store <2 x float> %389, ptr addrspace(6) %392, align 8 + %393 = load <2 x float>, ptr %95, align 8 + %394 = fmul <2 x float> %393, %377 + %395 = insertelement <2 x float> poison, float %15, i32 0 + %396 = insertelement <2 x float> %395, float %17, i32 1 + %397 = fmul <2 x float> %394, %396 + %398 = add i64 %90, 12288 + %399 = mul i64 %398, 4 + %400 = getelementptr i8, ptr addrspace(6) %79, i64 %399 + store <2 x float> %397, ptr addrspace(6) %400, align 8 + %401 = load <2 x float>, ptr %102, align 8 + %402 = fmul <2 x float> %401, %377 + %403 = insertelement <2 x float> poison, float %19, i32 0 + %404 = insertelement <2 x float> %403, float %21, i32 1 + %405 = fmul <2 x float> %402, %404 + %406 = add i64 %97, 12288 + %407 = mul i64 %406, 4 + %408 = getelementptr i8, ptr addrspace(6) %79, i64 %407 + store <2 x float> %405, ptr addrspace(6) %408, align 8 + %409 = load <2 x float>, ptr %109, align 8 + %410 = fmul <2 x float> %409, %377 + %411 = insertelement <2 x float> poison, float %23, i32 0 + %412 = insertelement <2 x float> %411, float %25, i32 1 + %413 = fmul <2 x float> %410, %412 + %414 = add i64 %104, 12288 + %415 = mul i64 %414, 4 + %416 = getelementptr i8, ptr addrspace(6) %79, i64 %415 + store <2 x float> %413, ptr addrspace(6) %416, align 8 + %417 = load <2 x float>, ptr %116, align 8 + %418 = fmul <2 x float> %417, %377 + %419 = insertelement <2 x float> poison, float %27, i32 0 + %420 = insertelement <2 x float> %419, float %29, i32 1 + %421 = fmul <2 x float> %418, %420 + %422 = add i64 %111, 12288 + %423 = mul i64 %422, 4 + %424 = getelementptr i8, ptr addrspace(6) %79, i64 %423 + store <2 x float> %421, ptr addrspace(6) %424, align 8 + %425 = load <2 x float>, ptr %123, align 8 + %426 = fmul <2 x float> %425, %377 + %427 = insertelement <2 x float> poison, float %31, i32 0 + %428 = insertelement <2 x float> %427, float %33, i32 1 + %429 = fmul <2 x float> %426, %428 + %430 = add i64 %118, 12288 + %431 = mul i64 %430, 4 + %432 = getelementptr i8, ptr addrspace(6) %79, i64 %431 + store <2 x float> %429, ptr addrspace(6) %432, align 8 + %433 = load <2 x float>, ptr %130, align 8 + %434 = fmul <2 x float> %433, %377 + %435 = insertelement <2 x float> poison, float %35, i32 0 + %436 = insertelement <2 x float> %435, float %37, i32 1 + %437 = fmul <2 x float> %434, %436 + %438 = add i64 %125, 12288 + %439 = mul i64 %438, 4 + %440 = getelementptr i8, ptr addrspace(6) %79, i64 %439 + store <2 x float> %437, ptr addrspace(6) %440, align 8 + %441 = load <2 x float>, ptr %137, align 8 + %442 = fmul <2 x float> %441, %377 + %443 = insertelement <2 x float> poison, float %39, i32 0 + %444 = insertelement <2 x float> %443, float %41, i32 1 + %445 = fmul <2 x float> %442, %444 + %446 = add i64 %132, 12288 + %447 = mul i64 %446, 4 + %448 = getelementptr i8, ptr addrspace(6) %79, i64 %447 + store <2 x float> %445, ptr addrspace(6) %448, align 8 + %449 = load <2 x float>, ptr %144, align 8 + %450 = fmul <2 x float> %449, %377 + %451 = insertelement <2 x float> poison, float %43, i32 0 + %452 = insertelement <2 x float> %451, float %45, i32 1 + %453 = fmul <2 x float> %450, %452 + %454 = add i64 %139, 12288 + %455 = mul i64 %454, 4 + %456 = getelementptr i8, ptr addrspace(6) %79, i64 %455 + store <2 x float> %453, ptr addrspace(6) %456, align 8 + %457 = load <2 x float>, ptr %151, align 8 + %458 = fmul <2 x float> %457, %377 + %459 = insertelement <2 x float> poison, float %47, i32 0 + %460 = insertelement <2 x float> %459, float %49, i32 1 + %461 = fmul <2 x float> %458, %460 + %462 = add i64 %146, 12288 + %463 = mul i64 %462, 4 + %464 = getelementptr i8, ptr addrspace(6) %79, i64 %463 + store <2 x float> %461, ptr addrspace(6) %464, align 8 + %465 = load <2 x float>, ptr %158, align 8 + %466 = fmul <2 x float> %465, %377 + %467 = insertelement <2 x float> poison, float %51, i32 0 + %468 = insertelement <2 x float> %467, float %53, i32 1 + %469 = fmul <2 x float> %466, %468 + %470 = add i64 %153, 12288 + %471 = mul i64 %470, 4 + %472 = getelementptr i8, ptr addrspace(6) %79, i64 %471 + store <2 x float> %469, ptr addrspace(6) %472, align 8 + %473 = load <2 x float>, ptr %165, align 8 + %474 = fmul <2 x float> %473, %377 + %475 = insertelement <2 x float> poison, float %55, i32 0 + %476 = insertelement <2 x float> %475, float %57, i32 1 + %477 = fmul <2 x float> %474, %476 + %478 = add i64 %160, 12288 + %479 = mul i64 %478, 4 + %480 = getelementptr i8, ptr addrspace(6) %79, i64 %479 + store <2 x float> %477, ptr addrspace(6) %480, align 8 + %481 = load <2 x float>, ptr %172, align 8 + %482 = fmul <2 x float> %481, %377 + %483 = insertelement <2 x float> poison, float %59, i32 0 + %484 = insertelement <2 x float> %483, float %61, i32 1 + %485 = fmul <2 x float> %482, %484 + %486 = add i64 %167, 12288 + %487 = mul i64 %486, 4 + %488 = getelementptr i8, ptr addrspace(6) %79, i64 %487 + store <2 x float> %485, ptr addrspace(6) %488, align 8 + %489 = load <2 x float>, ptr %179, align 8 + %490 = fmul <2 x float> %489, %377 + %491 = insertelement <2 x float> poison, float %63, i32 0 + %492 = insertelement <2 x float> %491, float %65, i32 1 + %493 = fmul <2 x float> %490, %492 + %494 = add i64 %174, 12288 + %495 = mul i64 %494, 4 + %496 = getelementptr i8, ptr addrspace(6) %79, i64 %495 + store <2 x float> %493, ptr addrspace(6) %496, align 8 + %497 = load <2 x float>, ptr %186, align 8 + %498 = fmul <2 x float> %497, %377 + %499 = insertelement <2 x float> poison, float %67, i32 0 + %500 = insertelement <2 x float> %499, float %69, i32 1 + %501 = fmul <2 x float> %498, %500 + %502 = add i64 %181, 12288 + %503 = mul i64 %502, 4 + %504 = getelementptr i8, ptr addrspace(6) %79, i64 %503 + store <2 x float> %501, ptr addrspace(6) %504, align 8 + %505 = call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "", "={TPER4},={TPER5},={TPER6},={TPER7},={TPER8},={TPER9},={TPER10},={TPER11},={TPER12},={TPER13},={TPER14},={TPER15},={TPER16},={TPER17},={TPER18},={TPER19},={TPER20},={TPER21},={TPER22},={TPER23},={TPER24},={TPER25},={TPER26},={TPER27},={TPER28},={TPER29},={TPER30},={TPER31},={TPER32},={TPER33},={TPER34},={TPER35},0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31"(i32 %6, i32 %8, i32 %10, i32 %12, i32 %14, i32 %16, i32 %18, i32 %20, i32 %22, i32 %24, i32 %26, i32 %28, i32 %30, i32 %32, i32 %34, i32 %36, i32 %38, i32 %40, i32 %42, i32 %44, i32 %46, i32 %48, i32 %50, i32 %52, i32 %54, i32 %56, i32 %58, i32 %60, i32 %62, i32 %64, i32 %66, i32 %68) + ret void +} + +attributes #0 = { "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } +attributes #1 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } +attributes #2 = { "target-cpu"="dav-c310-vec" "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } +attributes #3 = { noinline "target-cpu"="dav-c310-vec" "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } + +!llvm.module.flags = !{!0} +!hivm.annotations = !{!1, !2, !3, !4, !5, !6, !7} + +!0 = !{i32 2, !"Debug Info Version", i32 3} +!1 = !{ptr @rmsnorm_d4096_kernel_mix_aiv, !"kernel", i32 1} +!2 = !{ptr @rmsnorm_d4096_kernel_mix_aiv, !"kernel_with_simd", i32 1} +!3 = !{ptr @rmsnorm_d4096_kernel_mix_aiv, !"kernel_with_simt", i32 1} +!4 = distinct !{null, !"simt-max-threads", i32 128} +!5 = distinct !{null, !"simt-max-registers", i32 128} +!6 = distinct !{null, !"simt-max-threads", i32 128} +!7 = distinct !{null, !"simt-max-registers", i32 128} +!8 = !{!"simt-max-threads", i32 128} +!9 = !{!"simt-max-registers", i32 128} diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/launch.cpp b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/launch.cpp new file mode 100644 index 0000000000..ef9357fe09 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d4096/launch.cpp @@ -0,0 +1,12 @@ +#ifndef AICORE +#define AICORE [aicore] +#endif +extern "C" __global__ AICORE void rmsnorm_d4096_kernel( + __gm__ void*, __gm__ void*, __gm__ void*, __gm__ void*, float); +extern "C" int call( + float* X, float* Y, float* W, float* RSTD, float eps, void* stream) { + rmsnorm_d4096_kernel<<<64, 82432, stream>>>( + (__gm__ float*)RSTD, (__gm__ float*)W, (__gm__ float*)X, + (__gm__ float*)Y, eps); + return 0; +} diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/kernel.pto b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/kernel.pto new file mode 100644 index 0000000000..04c0967670 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/kernel.pto @@ -0,0 +1,1763 @@ +module attributes {pto.target_arch = "a5"} { + module attributes {pto.backend = "vpto", pto.kernel_kind = #pto.kernel_kind, pto.target_arch = "a5"} { + func.func @rmsnorm_d5120_kernel(%arg0: !pto.ptr, %arg1: !pto.ptr, %arg2: !pto.ptr, %arg3: !pto.ptr, %arg4: f32) attributes {pto.entry, pto.kernel_kind = #pto.kernel_kind} { + %c0_i64 = arith.constant 0 : i64 + %0 = pto.castptr %c0_i64 : i64 -> !pto.ptr + %c32_i32 = arith.constant 32 : i32 + %1 = llvm.alloca %c32_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.set_flag[, , ] + pto.set_flag[, , ] + pto.set_flag[, , ] + pto.set_flag[, , ] + %2 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_i64 = arith.constant 1 : i64 + %c20480_i64 = arith.constant 20480 : i64 + %c20480_i64_0 = arith.constant 20480 : i64 + %c0_i64_1 = arith.constant 0 : i64 + %c20480_i64_2 = arith.constant 20480 : i64 + pto.mte_gm_ub %arg1, %2, %c0_i64_1, %c20480_i64_2 nburst(%c1_i64, %c20480_i64, %c20480_i64_0) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + pto.set_flag[, , ] + pto.wait_flag[, , ] + pto.section.simt<<<256, 1, 1>>> { + %3 = pto.get_tid_x : i32 + %4 = pto.get_tid_y : i32 + %5 = pto.get_tid_z : i32 + %6 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32 = arith.constant 2 : i32 + %7 = arith.muli %3, %c2_i32 : i32 + %c0_i32 = arith.constant 0 : i32 + %8 = arith.addi %c0_i32, %7 : i32 + %9 = arith.index_cast %8 : i32 to index + %c4 = arith.constant 4 : index + %10 = arith.muli %9, %c4 : index + %11 = arith.index_cast %10 : index to i64 + %12 = pto.castptr %6 : !pto.ptr -> i64 + %13 = llvm.inttoptr %12 : i64 to !llvm.ptr<6> + %14 = llvm.getelementptr %13[%11] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %15 = llvm.load %14 : !llvm.ptr<6> -> vector<2xf32> + %c0_3 = arith.constant 0 : index + %c4_4 = arith.constant 4 : index + %16 = arith.muli %c0_3, %c4_4 : index + %17 = arith.index_cast %16 : index to i64 + %18 = llvm.getelementptr %1[%17] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %15, %18 : vector<2xf32>, !llvm.ptr + %19 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_5 = arith.constant 2 : i32 + %20 = arith.muli %3, %c2_i32_5 : i32 + %c512_i32 = arith.constant 512 : i32 + %21 = arith.addi %c512_i32, %20 : i32 + %22 = arith.index_cast %21 : i32 to index + %c4_6 = arith.constant 4 : index + %23 = arith.muli %22, %c4_6 : index + %24 = arith.index_cast %23 : index to i64 + %25 = pto.castptr %19 : !pto.ptr -> i64 + %26 = llvm.inttoptr %25 : i64 to !llvm.ptr<6> + %27 = llvm.getelementptr %26[%24] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %28 = llvm.load %27 : !llvm.ptr<6> -> vector<2xf32> + %c2 = arith.constant 2 : index + %c4_7 = arith.constant 4 : index + %29 = arith.muli %c2, %c4_7 : index + %30 = arith.index_cast %29 : index to i64 + %31 = llvm.getelementptr %1[%30] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %28, %31 : vector<2xf32>, !llvm.ptr + %32 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_8 = arith.constant 2 : i32 + %33 = arith.muli %3, %c2_i32_8 : i32 + %c1024_i32 = arith.constant 1024 : i32 + %34 = arith.addi %c1024_i32, %33 : i32 + %35 = arith.index_cast %34 : i32 to index + %c4_9 = arith.constant 4 : index + %36 = arith.muli %35, %c4_9 : index + %37 = arith.index_cast %36 : index to i64 + %38 = pto.castptr %32 : !pto.ptr -> i64 + %39 = llvm.inttoptr %38 : i64 to !llvm.ptr<6> + %40 = llvm.getelementptr %39[%37] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %41 = llvm.load %40 : !llvm.ptr<6> -> vector<2xf32> + %c4_10 = arith.constant 4 : index + %c4_11 = arith.constant 4 : index + %42 = arith.muli %c4_10, %c4_11 : index + %43 = arith.index_cast %42 : index to i64 + %44 = llvm.getelementptr %1[%43] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %41, %44 : vector<2xf32>, !llvm.ptr + %45 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_12 = arith.constant 2 : i32 + %46 = arith.muli %3, %c2_i32_12 : i32 + %c1536_i32 = arith.constant 1536 : i32 + %47 = arith.addi %c1536_i32, %46 : i32 + %48 = arith.index_cast %47 : i32 to index + %c4_13 = arith.constant 4 : index + %49 = arith.muli %48, %c4_13 : index + %50 = arith.index_cast %49 : index to i64 + %51 = pto.castptr %45 : !pto.ptr -> i64 + %52 = llvm.inttoptr %51 : i64 to !llvm.ptr<6> + %53 = llvm.getelementptr %52[%50] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %54 = llvm.load %53 : !llvm.ptr<6> -> vector<2xf32> + %c6 = arith.constant 6 : index + %c4_14 = arith.constant 4 : index + %55 = arith.muli %c6, %c4_14 : index + %56 = arith.index_cast %55 : index to i64 + %57 = llvm.getelementptr %1[%56] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %54, %57 : vector<2xf32>, !llvm.ptr + %58 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_15 = arith.constant 2 : i32 + %59 = arith.muli %3, %c2_i32_15 : i32 + %c2048_i32 = arith.constant 2048 : i32 + %60 = arith.addi %c2048_i32, %59 : i32 + %61 = arith.index_cast %60 : i32 to index + %c4_16 = arith.constant 4 : index + %62 = arith.muli %61, %c4_16 : index + %63 = arith.index_cast %62 : index to i64 + %64 = pto.castptr %58 : !pto.ptr -> i64 + %65 = llvm.inttoptr %64 : i64 to !llvm.ptr<6> + %66 = llvm.getelementptr %65[%63] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %67 = llvm.load %66 : !llvm.ptr<6> -> vector<2xf32> + %c8 = arith.constant 8 : index + %c4_17 = arith.constant 4 : index + %68 = arith.muli %c8, %c4_17 : index + %69 = arith.index_cast %68 : index to i64 + %70 = llvm.getelementptr %1[%69] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %67, %70 : vector<2xf32>, !llvm.ptr + %71 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_18 = arith.constant 2 : i32 + %72 = arith.muli %3, %c2_i32_18 : i32 + %c2560_i32 = arith.constant 2560 : i32 + %73 = arith.addi %c2560_i32, %72 : i32 + %74 = arith.index_cast %73 : i32 to index + %c4_19 = arith.constant 4 : index + %75 = arith.muli %74, %c4_19 : index + %76 = arith.index_cast %75 : index to i64 + %77 = pto.castptr %71 : !pto.ptr -> i64 + %78 = llvm.inttoptr %77 : i64 to !llvm.ptr<6> + %79 = llvm.getelementptr %78[%76] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %80 = llvm.load %79 : !llvm.ptr<6> -> vector<2xf32> + %c10 = arith.constant 10 : index + %c4_20 = arith.constant 4 : index + %81 = arith.muli %c10, %c4_20 : index + %82 = arith.index_cast %81 : index to i64 + %83 = llvm.getelementptr %1[%82] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %80, %83 : vector<2xf32>, !llvm.ptr + %84 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_21 = arith.constant 2 : i32 + %85 = arith.muli %3, %c2_i32_21 : i32 + %c3072_i32 = arith.constant 3072 : i32 + %86 = arith.addi %c3072_i32, %85 : i32 + %87 = arith.index_cast %86 : i32 to index + %c4_22 = arith.constant 4 : index + %88 = arith.muli %87, %c4_22 : index + %89 = arith.index_cast %88 : index to i64 + %90 = pto.castptr %84 : !pto.ptr -> i64 + %91 = llvm.inttoptr %90 : i64 to !llvm.ptr<6> + %92 = llvm.getelementptr %91[%89] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %93 = llvm.load %92 : !llvm.ptr<6> -> vector<2xf32> + %c12 = arith.constant 12 : index + %c4_23 = arith.constant 4 : index + %94 = arith.muli %c12, %c4_23 : index + %95 = arith.index_cast %94 : index to i64 + %96 = llvm.getelementptr %1[%95] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %93, %96 : vector<2xf32>, !llvm.ptr + %97 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_24 = arith.constant 2 : i32 + %98 = arith.muli %3, %c2_i32_24 : i32 + %c3584_i32 = arith.constant 3584 : i32 + %99 = arith.addi %c3584_i32, %98 : i32 + %100 = arith.index_cast %99 : i32 to index + %c4_25 = arith.constant 4 : index + %101 = arith.muli %100, %c4_25 : index + %102 = arith.index_cast %101 : index to i64 + %103 = pto.castptr %97 : !pto.ptr -> i64 + %104 = llvm.inttoptr %103 : i64 to !llvm.ptr<6> + %105 = llvm.getelementptr %104[%102] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %106 = llvm.load %105 : !llvm.ptr<6> -> vector<2xf32> + %c14 = arith.constant 14 : index + %c4_26 = arith.constant 4 : index + %107 = arith.muli %c14, %c4_26 : index + %108 = arith.index_cast %107 : index to i64 + %109 = llvm.getelementptr %1[%108] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %106, %109 : vector<2xf32>, !llvm.ptr + %110 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_27 = arith.constant 2 : i32 + %111 = arith.muli %3, %c2_i32_27 : i32 + %c4096_i32 = arith.constant 4096 : i32 + %112 = arith.addi %c4096_i32, %111 : i32 + %113 = arith.index_cast %112 : i32 to index + %c4_28 = arith.constant 4 : index + %114 = arith.muli %113, %c4_28 : index + %115 = arith.index_cast %114 : index to i64 + %116 = pto.castptr %110 : !pto.ptr -> i64 + %117 = llvm.inttoptr %116 : i64 to !llvm.ptr<6> + %118 = llvm.getelementptr %117[%115] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %119 = llvm.load %118 : !llvm.ptr<6> -> vector<2xf32> + %c16 = arith.constant 16 : index + %c4_29 = arith.constant 4 : index + %120 = arith.muli %c16, %c4_29 : index + %121 = arith.index_cast %120 : index to i64 + %122 = llvm.getelementptr %1[%121] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %119, %122 : vector<2xf32>, !llvm.ptr + %123 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_30 = arith.constant 2 : i32 + %124 = arith.muli %3, %c2_i32_30 : i32 + %c4608_i32 = arith.constant 4608 : i32 + %125 = arith.addi %c4608_i32, %124 : i32 + %126 = arith.index_cast %125 : i32 to index + %c4_31 = arith.constant 4 : index + %127 = arith.muli %126, %c4_31 : index + %128 = arith.index_cast %127 : index to i64 + %129 = pto.castptr %123 : !pto.ptr -> i64 + %130 = llvm.inttoptr %129 : i64 to !llvm.ptr<6> + %131 = llvm.getelementptr %130[%128] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %132 = llvm.load %131 : !llvm.ptr<6> -> vector<2xf32> + %c18 = arith.constant 18 : index + %c4_32 = arith.constant 4 : index + %133 = arith.muli %c18, %c4_32 : index + %134 = arith.index_cast %133 : index to i64 + %135 = llvm.getelementptr %1[%134] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %132, %135 : vector<2xf32>, !llvm.ptr + } + %c0 = arith.constant 0 : index + %c64 = arith.constant 64 : index + %c1 = arith.constant 1 : index + scf.for %arg5 = %c0 to %c64 step %c1 { + %c1_3 = arith.constant 1 : index + %3 = arith.andi %arg5, %c1_3 : index + pto.wait_flag_dyn[, , %3] + %c327680 = arith.constant 327680 : index + %4 = arith.muli %arg5, %c327680 : index + %5 = pto.get_block_idx + %c5120_i64 = arith.constant 5120 : i64 + %6 = arith.muli %5, %c5120_i64 : i64 + %7 = arith.index_cast %6 : i64 to index + %8 = arith.addi %4, %7 : index + %9 = pto.addptr %arg2, %8 : -> + %10 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_4 = arith.constant 1 : index + %11 = arith.andi %arg5, %c1_4 : index + %c8192 = arith.constant 8192 : index + %12 = arith.muli %11, %c8192 : index + %c5120 = arith.constant 5120 : index + %13 = arith.addi %12, %c5120 : index + %14 = pto.addptr %10, %13 : -> + %c1_i64_5 = arith.constant 1 : i64 + %c20480_i64_6 = arith.constant 20480 : i64 + %c20480_i64_7 = arith.constant 20480 : i64 + %c0_i64_8 = arith.constant 0 : i64 + %c20480_i64_9 = arith.constant 20480 : i64 + pto.mte_gm_ub %9, %14, %c0_i64_8, %c20480_i64_9 nburst(%c1_i64_5, %c20480_i64_6, %c20480_i64_7) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + %c1_10 = arith.constant 1 : index + %15 = arith.andi %arg5, %c1_10 : index + pto.set_flag_dyn[, , %15] + %c1_11 = arith.constant 1 : index + %16 = arith.andi %arg5, %c1_11 : index + pto.wait_flag_dyn[, , %16] + %c1_12 = arith.constant 1 : index + %17 = arith.andi %arg5, %c1_12 : index + pto.wait_flag_dyn[, , %17] + pto.section.simt<<<256, 1, 1>>> { + %c32_i32_32 = arith.constant 32 : i32 + %42 = llvm.alloca %c32_i32_32 x f32 : (i32) -> !llvm.ptr + %c1_i32 = arith.constant 1 : i32 + %43 = llvm.alloca %c1_i32 x f32 : (i32) -> !llvm.ptr + %44 = pto.get_tid_x : i32 + %45 = pto.get_tid_y : i32 + %46 = pto.get_tid_z : i32 + %47 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_33 = arith.constant 1 : index + %48 = arith.andi %arg5, %c1_33 : index + %c8192_34 = arith.constant 8192 : index + %49 = arith.muli %48, %c8192_34 : index + %c0_35 = arith.constant 0 : index + %50 = arith.addi %49, %c0_35 : index + %c2_i32 = arith.constant 2 : i32 + %51 = arith.muli %44, %c2_i32 : i32 + %52 = arith.index_cast %51 : i32 to index + %53 = arith.addi %50, %52 : index + %c5120_36 = arith.constant 5120 : index + %54 = arith.addi %53, %c5120_36 : index + %c4 = arith.constant 4 : index + %55 = arith.muli %54, %c4 : index + %56 = arith.index_cast %55 : index to i64 + %57 = pto.castptr %47 : !pto.ptr -> i64 + %58 = llvm.inttoptr %57 : i64 to !llvm.ptr<6> + %59 = llvm.getelementptr %58[%56] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %60 = llvm.load %59 : !llvm.ptr<6> -> vector<2xf32> + %c0_37 = arith.constant 0 : index + %c4_38 = arith.constant 4 : index + %61 = arith.muli %c0_37, %c4_38 : index + %62 = arith.index_cast %61 : index to i64 + %63 = llvm.getelementptr %42[%62] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %60, %63 : vector<2xf32>, !llvm.ptr + %64 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_39 = arith.constant 1 : index + %65 = arith.andi %arg5, %c1_39 : index + %c8192_40 = arith.constant 8192 : index + %66 = arith.muli %65, %c8192_40 : index + %c512 = arith.constant 512 : index + %67 = arith.addi %66, %c512 : index + %c2_i32_41 = arith.constant 2 : i32 + %68 = arith.muli %44, %c2_i32_41 : i32 + %69 = arith.index_cast %68 : i32 to index + %70 = arith.addi %67, %69 : index + %c5120_42 = arith.constant 5120 : index + %71 = arith.addi %70, %c5120_42 : index + %c4_43 = arith.constant 4 : index + %72 = arith.muli %71, %c4_43 : index + %73 = arith.index_cast %72 : index to i64 + %74 = pto.castptr %64 : !pto.ptr -> i64 + %75 = llvm.inttoptr %74 : i64 to !llvm.ptr<6> + %76 = llvm.getelementptr %75[%73] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %77 = llvm.load %76 : !llvm.ptr<6> -> vector<2xf32> + %c2 = arith.constant 2 : index + %c4_44 = arith.constant 4 : index + %78 = arith.muli %c2, %c4_44 : index + %79 = arith.index_cast %78 : index to i64 + %80 = llvm.getelementptr %42[%79] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %77, %80 : vector<2xf32>, !llvm.ptr + %81 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_45 = arith.constant 1 : index + %82 = arith.andi %arg5, %c1_45 : index + %c8192_46 = arith.constant 8192 : index + %83 = arith.muli %82, %c8192_46 : index + %c1024 = arith.constant 1024 : index + %84 = arith.addi %83, %c1024 : index + %c2_i32_47 = arith.constant 2 : i32 + %85 = arith.muli %44, %c2_i32_47 : i32 + %86 = arith.index_cast %85 : i32 to index + %87 = arith.addi %84, %86 : index + %c5120_48 = arith.constant 5120 : index + %88 = arith.addi %87, %c5120_48 : index + %c4_49 = arith.constant 4 : index + %89 = arith.muli %88, %c4_49 : index + %90 = arith.index_cast %89 : index to i64 + %91 = pto.castptr %81 : !pto.ptr -> i64 + %92 = llvm.inttoptr %91 : i64 to !llvm.ptr<6> + %93 = llvm.getelementptr %92[%90] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %94 = llvm.load %93 : !llvm.ptr<6> -> vector<2xf32> + %c4_50 = arith.constant 4 : index + %c4_51 = arith.constant 4 : index + %95 = arith.muli %c4_50, %c4_51 : index + %96 = arith.index_cast %95 : index to i64 + %97 = llvm.getelementptr %42[%96] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %94, %97 : vector<2xf32>, !llvm.ptr + %98 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_52 = arith.constant 1 : index + %99 = arith.andi %arg5, %c1_52 : index + %c8192_53 = arith.constant 8192 : index + %100 = arith.muli %99, %c8192_53 : index + %c1536 = arith.constant 1536 : index + %101 = arith.addi %100, %c1536 : index + %c2_i32_54 = arith.constant 2 : i32 + %102 = arith.muli %44, %c2_i32_54 : i32 + %103 = arith.index_cast %102 : i32 to index + %104 = arith.addi %101, %103 : index + %c5120_55 = arith.constant 5120 : index + %105 = arith.addi %104, %c5120_55 : index + %c4_56 = arith.constant 4 : index + %106 = arith.muli %105, %c4_56 : index + %107 = arith.index_cast %106 : index to i64 + %108 = pto.castptr %98 : !pto.ptr -> i64 + %109 = llvm.inttoptr %108 : i64 to !llvm.ptr<6> + %110 = llvm.getelementptr %109[%107] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %111 = llvm.load %110 : !llvm.ptr<6> -> vector<2xf32> + %c6 = arith.constant 6 : index + %c4_57 = arith.constant 4 : index + %112 = arith.muli %c6, %c4_57 : index + %113 = arith.index_cast %112 : index to i64 + %114 = llvm.getelementptr %42[%113] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %111, %114 : vector<2xf32>, !llvm.ptr + %115 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_58 = arith.constant 1 : index + %116 = arith.andi %arg5, %c1_58 : index + %c8192_59 = arith.constant 8192 : index + %117 = arith.muli %116, %c8192_59 : index + %c2048 = arith.constant 2048 : index + %118 = arith.addi %117, %c2048 : index + %c2_i32_60 = arith.constant 2 : i32 + %119 = arith.muli %44, %c2_i32_60 : i32 + %120 = arith.index_cast %119 : i32 to index + %121 = arith.addi %118, %120 : index + %c5120_61 = arith.constant 5120 : index + %122 = arith.addi %121, %c5120_61 : index + %c4_62 = arith.constant 4 : index + %123 = arith.muli %122, %c4_62 : index + %124 = arith.index_cast %123 : index to i64 + %125 = pto.castptr %115 : !pto.ptr -> i64 + %126 = llvm.inttoptr %125 : i64 to !llvm.ptr<6> + %127 = llvm.getelementptr %126[%124] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %128 = llvm.load %127 : !llvm.ptr<6> -> vector<2xf32> + %c8_63 = arith.constant 8 : index + %c4_64 = arith.constant 4 : index + %129 = arith.muli %c8_63, %c4_64 : index + %130 = arith.index_cast %129 : index to i64 + %131 = llvm.getelementptr %42[%130] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %128, %131 : vector<2xf32>, !llvm.ptr + %132 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_65 = arith.constant 1 : index + %133 = arith.andi %arg5, %c1_65 : index + %c8192_66 = arith.constant 8192 : index + %134 = arith.muli %133, %c8192_66 : index + %c2560 = arith.constant 2560 : index + %135 = arith.addi %134, %c2560 : index + %c2_i32_67 = arith.constant 2 : i32 + %136 = arith.muli %44, %c2_i32_67 : i32 + %137 = arith.index_cast %136 : i32 to index + %138 = arith.addi %135, %137 : index + %c5120_68 = arith.constant 5120 : index + %139 = arith.addi %138, %c5120_68 : index + %c4_69 = arith.constant 4 : index + %140 = arith.muli %139, %c4_69 : index + %141 = arith.index_cast %140 : index to i64 + %142 = pto.castptr %132 : !pto.ptr -> i64 + %143 = llvm.inttoptr %142 : i64 to !llvm.ptr<6> + %144 = llvm.getelementptr %143[%141] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %145 = llvm.load %144 : !llvm.ptr<6> -> vector<2xf32> + %c10 = arith.constant 10 : index + %c4_70 = arith.constant 4 : index + %146 = arith.muli %c10, %c4_70 : index + %147 = arith.index_cast %146 : index to i64 + %148 = llvm.getelementptr %42[%147] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %145, %148 : vector<2xf32>, !llvm.ptr + %149 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_71 = arith.constant 1 : index + %150 = arith.andi %arg5, %c1_71 : index + %c8192_72 = arith.constant 8192 : index + %151 = arith.muli %150, %c8192_72 : index + %c3072 = arith.constant 3072 : index + %152 = arith.addi %151, %c3072 : index + %c2_i32_73 = arith.constant 2 : i32 + %153 = arith.muli %44, %c2_i32_73 : i32 + %154 = arith.index_cast %153 : i32 to index + %155 = arith.addi %152, %154 : index + %c5120_74 = arith.constant 5120 : index + %156 = arith.addi %155, %c5120_74 : index + %c4_75 = arith.constant 4 : index + %157 = arith.muli %156, %c4_75 : index + %158 = arith.index_cast %157 : index to i64 + %159 = pto.castptr %149 : !pto.ptr -> i64 + %160 = llvm.inttoptr %159 : i64 to !llvm.ptr<6> + %161 = llvm.getelementptr %160[%158] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %162 = llvm.load %161 : !llvm.ptr<6> -> vector<2xf32> + %c12 = arith.constant 12 : index + %c4_76 = arith.constant 4 : index + %163 = arith.muli %c12, %c4_76 : index + %164 = arith.index_cast %163 : index to i64 + %165 = llvm.getelementptr %42[%164] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %162, %165 : vector<2xf32>, !llvm.ptr + %166 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_77 = arith.constant 1 : index + %167 = arith.andi %arg5, %c1_77 : index + %c8192_78 = arith.constant 8192 : index + %168 = arith.muli %167, %c8192_78 : index + %c3584 = arith.constant 3584 : index + %169 = arith.addi %168, %c3584 : index + %c2_i32_79 = arith.constant 2 : i32 + %170 = arith.muli %44, %c2_i32_79 : i32 + %171 = arith.index_cast %170 : i32 to index + %172 = arith.addi %169, %171 : index + %c5120_80 = arith.constant 5120 : index + %173 = arith.addi %172, %c5120_80 : index + %c4_81 = arith.constant 4 : index + %174 = arith.muli %173, %c4_81 : index + %175 = arith.index_cast %174 : index to i64 + %176 = pto.castptr %166 : !pto.ptr -> i64 + %177 = llvm.inttoptr %176 : i64 to !llvm.ptr<6> + %178 = llvm.getelementptr %177[%175] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %179 = llvm.load %178 : !llvm.ptr<6> -> vector<2xf32> + %c14 = arith.constant 14 : index + %c4_82 = arith.constant 4 : index + %180 = arith.muli %c14, %c4_82 : index + %181 = arith.index_cast %180 : index to i64 + %182 = llvm.getelementptr %42[%181] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %179, %182 : vector<2xf32>, !llvm.ptr + %183 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_83 = arith.constant 1 : index + %184 = arith.andi %arg5, %c1_83 : index + %c8192_84 = arith.constant 8192 : index + %185 = arith.muli %184, %c8192_84 : index + %c4096 = arith.constant 4096 : index + %186 = arith.addi %185, %c4096 : index + %c2_i32_85 = arith.constant 2 : i32 + %187 = arith.muli %44, %c2_i32_85 : i32 + %188 = arith.index_cast %187 : i32 to index + %189 = arith.addi %186, %188 : index + %c5120_86 = arith.constant 5120 : index + %190 = arith.addi %189, %c5120_86 : index + %c4_87 = arith.constant 4 : index + %191 = arith.muli %190, %c4_87 : index + %192 = arith.index_cast %191 : index to i64 + %193 = pto.castptr %183 : !pto.ptr -> i64 + %194 = llvm.inttoptr %193 : i64 to !llvm.ptr<6> + %195 = llvm.getelementptr %194[%192] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %196 = llvm.load %195 : !llvm.ptr<6> -> vector<2xf32> + %c16 = arith.constant 16 : index + %c4_88 = arith.constant 4 : index + %197 = arith.muli %c16, %c4_88 : index + %198 = arith.index_cast %197 : index to i64 + %199 = llvm.getelementptr %42[%198] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %196, %199 : vector<2xf32>, !llvm.ptr + %200 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_89 = arith.constant 1 : index + %201 = arith.andi %arg5, %c1_89 : index + %c8192_90 = arith.constant 8192 : index + %202 = arith.muli %201, %c8192_90 : index + %c4608 = arith.constant 4608 : index + %203 = arith.addi %202, %c4608 : index + %c2_i32_91 = arith.constant 2 : i32 + %204 = arith.muli %44, %c2_i32_91 : i32 + %205 = arith.index_cast %204 : i32 to index + %206 = arith.addi %203, %205 : index + %c5120_92 = arith.constant 5120 : index + %207 = arith.addi %206, %c5120_92 : index + %c4_93 = arith.constant 4 : index + %208 = arith.muli %207, %c4_93 : index + %209 = arith.index_cast %208 : index to i64 + %210 = pto.castptr %200 : !pto.ptr -> i64 + %211 = llvm.inttoptr %210 : i64 to !llvm.ptr<6> + %212 = llvm.getelementptr %211[%209] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %213 = llvm.load %212 : !llvm.ptr<6> -> vector<2xf32> + %c18 = arith.constant 18 : index + %c4_94 = arith.constant 4 : index + %214 = arith.muli %c18, %c4_94 : index + %215 = arith.index_cast %214 : index to i64 + %216 = llvm.getelementptr %42[%215] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %213, %216 : vector<2xf32>, !llvm.ptr + %217 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_95 = arith.constant 1 : index + %218 = arith.andi %arg5, %c1_95 : index + %c8192_96 = arith.constant 8192 : index + %219 = arith.muli %218, %c8192_96 : index + %c5120_97 = arith.constant 5120 : index + %220 = arith.addi %219, %c5120_97 : index + %c2_i32_98 = arith.constant 2 : i32 + %221 = arith.muli %44, %c2_i32_98 : i32 + %222 = arith.index_cast %221 : i32 to index + %223 = arith.addi %220, %222 : index + %c5120_99 = arith.constant 5120 : index + %224 = arith.addi %223, %c5120_99 : index + %c4_100 = arith.constant 4 : index + %225 = arith.muli %224, %c4_100 : index + %226 = arith.index_cast %225 : index to i64 + %227 = pto.castptr %217 : !pto.ptr -> i64 + %228 = llvm.inttoptr %227 : i64 to !llvm.ptr<6> + %229 = llvm.getelementptr %228[%226] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %230 = llvm.load %229 : !llvm.ptr<6> -> vector<2xf32> + %c20 = arith.constant 20 : index + %c4_101 = arith.constant 4 : index + %231 = arith.muli %c20, %c4_101 : index + %232 = arith.index_cast %231 : index to i64 + %233 = llvm.getelementptr %42[%232] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %230, %233 : vector<2xf32>, !llvm.ptr + %234 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_102 = arith.constant 1 : index + %235 = arith.andi %arg5, %c1_102 : index + %c8192_103 = arith.constant 8192 : index + %236 = arith.muli %235, %c8192_103 : index + %c5632 = arith.constant 5632 : index + %237 = arith.addi %236, %c5632 : index + %c2_i32_104 = arith.constant 2 : i32 + %238 = arith.muli %44, %c2_i32_104 : i32 + %239 = arith.index_cast %238 : i32 to index + %240 = arith.addi %237, %239 : index + %c5120_105 = arith.constant 5120 : index + %241 = arith.addi %240, %c5120_105 : index + %c4_106 = arith.constant 4 : index + %242 = arith.muli %241, %c4_106 : index + %243 = arith.index_cast %242 : index to i64 + %244 = pto.castptr %234 : !pto.ptr -> i64 + %245 = llvm.inttoptr %244 : i64 to !llvm.ptr<6> + %246 = llvm.getelementptr %245[%243] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %247 = llvm.load %246 : !llvm.ptr<6> -> vector<2xf32> + %c22 = arith.constant 22 : index + %c4_107 = arith.constant 4 : index + %248 = arith.muli %c22, %c4_107 : index + %249 = arith.index_cast %248 : index to i64 + %250 = llvm.getelementptr %42[%249] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %247, %250 : vector<2xf32>, !llvm.ptr + %251 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_108 = arith.constant 1 : index + %252 = arith.andi %arg5, %c1_108 : index + %c8192_109 = arith.constant 8192 : index + %253 = arith.muli %252, %c8192_109 : index + %c6144 = arith.constant 6144 : index + %254 = arith.addi %253, %c6144 : index + %c2_i32_110 = arith.constant 2 : i32 + %255 = arith.muli %44, %c2_i32_110 : i32 + %256 = arith.index_cast %255 : i32 to index + %257 = arith.addi %254, %256 : index + %c5120_111 = arith.constant 5120 : index + %258 = arith.addi %257, %c5120_111 : index + %c4_112 = arith.constant 4 : index + %259 = arith.muli %258, %c4_112 : index + %260 = arith.index_cast %259 : index to i64 + %261 = pto.castptr %251 : !pto.ptr -> i64 + %262 = llvm.inttoptr %261 : i64 to !llvm.ptr<6> + %263 = llvm.getelementptr %262[%260] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %264 = llvm.load %263 : !llvm.ptr<6> -> vector<2xf32> + %c24 = arith.constant 24 : index + %c4_113 = arith.constant 4 : index + %265 = arith.muli %c24, %c4_113 : index + %266 = arith.index_cast %265 : index to i64 + %267 = llvm.getelementptr %42[%266] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %264, %267 : vector<2xf32>, !llvm.ptr + %268 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_114 = arith.constant 1 : index + %269 = arith.andi %arg5, %c1_114 : index + %c8192_115 = arith.constant 8192 : index + %270 = arith.muli %269, %c8192_115 : index + %c6656 = arith.constant 6656 : index + %271 = arith.addi %270, %c6656 : index + %c2_i32_116 = arith.constant 2 : i32 + %272 = arith.muli %44, %c2_i32_116 : i32 + %273 = arith.index_cast %272 : i32 to index + %274 = arith.addi %271, %273 : index + %c5120_117 = arith.constant 5120 : index + %275 = arith.addi %274, %c5120_117 : index + %c4_118 = arith.constant 4 : index + %276 = arith.muli %275, %c4_118 : index + %277 = arith.index_cast %276 : index to i64 + %278 = pto.castptr %268 : !pto.ptr -> i64 + %279 = llvm.inttoptr %278 : i64 to !llvm.ptr<6> + %280 = llvm.getelementptr %279[%277] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %281 = llvm.load %280 : !llvm.ptr<6> -> vector<2xf32> + %c26 = arith.constant 26 : index + %c4_119 = arith.constant 4 : index + %282 = arith.muli %c26, %c4_119 : index + %283 = arith.index_cast %282 : index to i64 + %284 = llvm.getelementptr %42[%283] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %281, %284 : vector<2xf32>, !llvm.ptr + %285 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_120 = arith.constant 1 : index + %286 = arith.andi %arg5, %c1_120 : index + %c8192_121 = arith.constant 8192 : index + %287 = arith.muli %286, %c8192_121 : index + %c7168 = arith.constant 7168 : index + %288 = arith.addi %287, %c7168 : index + %c2_i32_122 = arith.constant 2 : i32 + %289 = arith.muli %44, %c2_i32_122 : i32 + %290 = arith.index_cast %289 : i32 to index + %291 = arith.addi %288, %290 : index + %c5120_123 = arith.constant 5120 : index + %292 = arith.addi %291, %c5120_123 : index + %c4_124 = arith.constant 4 : index + %293 = arith.muli %292, %c4_124 : index + %294 = arith.index_cast %293 : index to i64 + %295 = pto.castptr %285 : !pto.ptr -> i64 + %296 = llvm.inttoptr %295 : i64 to !llvm.ptr<6> + %297 = llvm.getelementptr %296[%294] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %298 = llvm.load %297 : !llvm.ptr<6> -> vector<2xf32> + %c28 = arith.constant 28 : index + %c4_125 = arith.constant 4 : index + %299 = arith.muli %c28, %c4_125 : index + %300 = arith.index_cast %299 : index to i64 + %301 = llvm.getelementptr %42[%300] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %298, %301 : vector<2xf32>, !llvm.ptr + %302 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_126 = arith.constant 1 : index + %303 = arith.andi %arg5, %c1_126 : index + %c8192_127 = arith.constant 8192 : index + %304 = arith.muli %303, %c8192_127 : index + %c7680 = arith.constant 7680 : index + %305 = arith.addi %304, %c7680 : index + %c2_i32_128 = arith.constant 2 : i32 + %306 = arith.muli %44, %c2_i32_128 : i32 + %307 = arith.index_cast %306 : i32 to index + %308 = arith.addi %305, %307 : index + %c5120_129 = arith.constant 5120 : index + %309 = arith.addi %308, %c5120_129 : index + %c4_130 = arith.constant 4 : index + %310 = arith.muli %309, %c4_130 : index + %311 = arith.index_cast %310 : index to i64 + %312 = pto.castptr %302 : !pto.ptr -> i64 + %313 = llvm.inttoptr %312 : i64 to !llvm.ptr<6> + %314 = llvm.getelementptr %313[%311] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %315 = llvm.load %314 : !llvm.ptr<6> -> vector<2xf32> + %c30 = arith.constant 30 : index + %c4_131 = arith.constant 4 : index + %316 = arith.muli %c30, %c4_131 : index + %317 = arith.index_cast %316 : index to i64 + %318 = llvm.getelementptr %42[%317] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %315, %318 : vector<2xf32>, !llvm.ptr + %c0_132 = arith.constant 0 : index + %cst = arith.constant 0.000000e+00 : f32 + %c4_133 = arith.constant 4 : index + %319 = arith.muli %c0_132, %c4_133 : index + %320 = arith.index_cast %319 : index to i64 + %321 = llvm.getelementptr %43[%320] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %cst, %321 : f32, !llvm.ptr + %c0_134 = arith.constant 0 : index + %c4_135 = arith.constant 4 : index + %322 = arith.muli %c0_134, %c4_135 : index + %323 = arith.index_cast %322 : index to i64 + %324 = llvm.getelementptr %43[%323] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %325 = llvm.load %324 : !llvm.ptr -> f32 + %c0_136 = arith.constant 0 : index + %c4_137 = arith.constant 4 : index + %326 = arith.muli %c0_136, %c4_137 : index + %327 = arith.index_cast %326 : index to i64 + %328 = llvm.getelementptr %42[%327] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %329 = llvm.load %328 : !llvm.ptr -> f32 + %c0_138 = arith.constant 0 : index + %c4_139 = arith.constant 4 : index + %330 = arith.muli %c0_138, %c4_139 : index + %331 = arith.index_cast %330 : index to i64 + %332 = llvm.getelementptr %42[%331] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %333 = llvm.load %332 : !llvm.ptr -> f32 + %334 = arith.mulf %329, %333 : f32 + %335 = arith.addf %325, %334 : f32 + %c0_140 = arith.constant 0 : index + %c4_141 = arith.constant 4 : index + %336 = arith.muli %c0_140, %c4_141 : index + %337 = arith.index_cast %336 : index to i64 + %338 = llvm.getelementptr %43[%337] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %335, %338 : f32, !llvm.ptr + %c0_142 = arith.constant 0 : index + %c4_143 = arith.constant 4 : index + %339 = arith.muli %c0_142, %c4_143 : index + %340 = arith.index_cast %339 : index to i64 + %341 = llvm.getelementptr %43[%340] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %342 = llvm.load %341 : !llvm.ptr -> f32 + %c1_144 = arith.constant 1 : index + %c4_145 = arith.constant 4 : index + %343 = arith.muli %c1_144, %c4_145 : index + %344 = arith.index_cast %343 : index to i64 + %345 = llvm.getelementptr %42[%344] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %346 = llvm.load %345 : !llvm.ptr -> f32 + %c1_146 = arith.constant 1 : index + %c4_147 = arith.constant 4 : index + %347 = arith.muli %c1_146, %c4_147 : index + %348 = arith.index_cast %347 : index to i64 + %349 = llvm.getelementptr %42[%348] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %350 = llvm.load %349 : !llvm.ptr -> f32 + %351 = arith.mulf %346, %350 : f32 + %352 = arith.addf %342, %351 : f32 + %c0_148 = arith.constant 0 : index + %c4_149 = arith.constant 4 : index + %353 = arith.muli %c0_148, %c4_149 : index + %354 = arith.index_cast %353 : index to i64 + %355 = llvm.getelementptr %43[%354] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %352, %355 : f32, !llvm.ptr + %c0_150 = arith.constant 0 : index + %c4_151 = arith.constant 4 : index + %356 = arith.muli %c0_150, %c4_151 : index + %357 = arith.index_cast %356 : index to i64 + %358 = llvm.getelementptr %43[%357] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %359 = llvm.load %358 : !llvm.ptr -> f32 + %c2_152 = arith.constant 2 : index + %c4_153 = arith.constant 4 : index + %360 = arith.muli %c2_152, %c4_153 : index + %361 = arith.index_cast %360 : index to i64 + %362 = llvm.getelementptr %42[%361] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %363 = llvm.load %362 : !llvm.ptr -> f32 + %c2_154 = arith.constant 2 : index + %c4_155 = arith.constant 4 : index + %364 = arith.muli %c2_154, %c4_155 : index + %365 = arith.index_cast %364 : index to i64 + %366 = llvm.getelementptr %42[%365] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %367 = llvm.load %366 : !llvm.ptr -> f32 + %368 = arith.mulf %363, %367 : f32 + %369 = arith.addf %359, %368 : f32 + %c0_156 = arith.constant 0 : index + %c4_157 = arith.constant 4 : index + %370 = arith.muli %c0_156, %c4_157 : index + %371 = arith.index_cast %370 : index to i64 + %372 = llvm.getelementptr %43[%371] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %369, %372 : f32, !llvm.ptr + %c0_158 = arith.constant 0 : index + %c4_159 = arith.constant 4 : index + %373 = arith.muli %c0_158, %c4_159 : index + %374 = arith.index_cast %373 : index to i64 + %375 = llvm.getelementptr %43[%374] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %376 = llvm.load %375 : !llvm.ptr -> f32 + %c3 = arith.constant 3 : index + %c4_160 = arith.constant 4 : index + %377 = arith.muli %c3, %c4_160 : index + %378 = arith.index_cast %377 : index to i64 + %379 = llvm.getelementptr %42[%378] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %380 = llvm.load %379 : !llvm.ptr -> f32 + %c3_161 = arith.constant 3 : index + %c4_162 = arith.constant 4 : index + %381 = arith.muli %c3_161, %c4_162 : index + %382 = arith.index_cast %381 : index to i64 + %383 = llvm.getelementptr %42[%382] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %384 = llvm.load %383 : !llvm.ptr -> f32 + %385 = arith.mulf %380, %384 : f32 + %386 = arith.addf %376, %385 : f32 + %c0_163 = arith.constant 0 : index + %c4_164 = arith.constant 4 : index + %387 = arith.muli %c0_163, %c4_164 : index + %388 = arith.index_cast %387 : index to i64 + %389 = llvm.getelementptr %43[%388] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %386, %389 : f32, !llvm.ptr + %c0_165 = arith.constant 0 : index + %c4_166 = arith.constant 4 : index + %390 = arith.muli %c0_165, %c4_166 : index + %391 = arith.index_cast %390 : index to i64 + %392 = llvm.getelementptr %43[%391] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %393 = llvm.load %392 : !llvm.ptr -> f32 + %c4_167 = arith.constant 4 : index + %c4_168 = arith.constant 4 : index + %394 = arith.muli %c4_167, %c4_168 : index + %395 = arith.index_cast %394 : index to i64 + %396 = llvm.getelementptr %42[%395] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %397 = llvm.load %396 : !llvm.ptr -> f32 + %c4_169 = arith.constant 4 : index + %c4_170 = arith.constant 4 : index + %398 = arith.muli %c4_169, %c4_170 : index + %399 = arith.index_cast %398 : index to i64 + %400 = llvm.getelementptr %42[%399] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %401 = llvm.load %400 : !llvm.ptr -> f32 + %402 = arith.mulf %397, %401 : f32 + %403 = arith.addf %393, %402 : f32 + %c0_171 = arith.constant 0 : index + %c4_172 = arith.constant 4 : index + %404 = arith.muli %c0_171, %c4_172 : index + %405 = arith.index_cast %404 : index to i64 + %406 = llvm.getelementptr %43[%405] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %403, %406 : f32, !llvm.ptr + %c0_173 = arith.constant 0 : index + %c4_174 = arith.constant 4 : index + %407 = arith.muli %c0_173, %c4_174 : index + %408 = arith.index_cast %407 : index to i64 + %409 = llvm.getelementptr %43[%408] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %410 = llvm.load %409 : !llvm.ptr -> f32 + %c5 = arith.constant 5 : index + %c4_175 = arith.constant 4 : index + %411 = arith.muli %c5, %c4_175 : index + %412 = arith.index_cast %411 : index to i64 + %413 = llvm.getelementptr %42[%412] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %414 = llvm.load %413 : !llvm.ptr -> f32 + %c5_176 = arith.constant 5 : index + %c4_177 = arith.constant 4 : index + %415 = arith.muli %c5_176, %c4_177 : index + %416 = arith.index_cast %415 : index to i64 + %417 = llvm.getelementptr %42[%416] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %418 = llvm.load %417 : !llvm.ptr -> f32 + %419 = arith.mulf %414, %418 : f32 + %420 = arith.addf %410, %419 : f32 + %c0_178 = arith.constant 0 : index + %c4_179 = arith.constant 4 : index + %421 = arith.muli %c0_178, %c4_179 : index + %422 = arith.index_cast %421 : index to i64 + %423 = llvm.getelementptr %43[%422] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %420, %423 : f32, !llvm.ptr + %c0_180 = arith.constant 0 : index + %c4_181 = arith.constant 4 : index + %424 = arith.muli %c0_180, %c4_181 : index + %425 = arith.index_cast %424 : index to i64 + %426 = llvm.getelementptr %43[%425] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %427 = llvm.load %426 : !llvm.ptr -> f32 + %c6_182 = arith.constant 6 : index + %c4_183 = arith.constant 4 : index + %428 = arith.muli %c6_182, %c4_183 : index + %429 = arith.index_cast %428 : index to i64 + %430 = llvm.getelementptr %42[%429] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %431 = llvm.load %430 : !llvm.ptr -> f32 + %c6_184 = arith.constant 6 : index + %c4_185 = arith.constant 4 : index + %432 = arith.muli %c6_184, %c4_185 : index + %433 = arith.index_cast %432 : index to i64 + %434 = llvm.getelementptr %42[%433] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %435 = llvm.load %434 : !llvm.ptr -> f32 + %436 = arith.mulf %431, %435 : f32 + %437 = arith.addf %427, %436 : f32 + %c0_186 = arith.constant 0 : index + %c4_187 = arith.constant 4 : index + %438 = arith.muli %c0_186, %c4_187 : index + %439 = arith.index_cast %438 : index to i64 + %440 = llvm.getelementptr %43[%439] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %437, %440 : f32, !llvm.ptr + %c0_188 = arith.constant 0 : index + %c4_189 = arith.constant 4 : index + %441 = arith.muli %c0_188, %c4_189 : index + %442 = arith.index_cast %441 : index to i64 + %443 = llvm.getelementptr %43[%442] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %444 = llvm.load %443 : !llvm.ptr -> f32 + %c7 = arith.constant 7 : index + %c4_190 = arith.constant 4 : index + %445 = arith.muli %c7, %c4_190 : index + %446 = arith.index_cast %445 : index to i64 + %447 = llvm.getelementptr %42[%446] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %448 = llvm.load %447 : !llvm.ptr -> f32 + %c7_191 = arith.constant 7 : index + %c4_192 = arith.constant 4 : index + %449 = arith.muli %c7_191, %c4_192 : index + %450 = arith.index_cast %449 : index to i64 + %451 = llvm.getelementptr %42[%450] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %452 = llvm.load %451 : !llvm.ptr -> f32 + %453 = arith.mulf %448, %452 : f32 + %454 = arith.addf %444, %453 : f32 + %c0_193 = arith.constant 0 : index + %c4_194 = arith.constant 4 : index + %455 = arith.muli %c0_193, %c4_194 : index + %456 = arith.index_cast %455 : index to i64 + %457 = llvm.getelementptr %43[%456] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %454, %457 : f32, !llvm.ptr + %c0_195 = arith.constant 0 : index + %c4_196 = arith.constant 4 : index + %458 = arith.muli %c0_195, %c4_196 : index + %459 = arith.index_cast %458 : index to i64 + %460 = llvm.getelementptr %43[%459] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %461 = llvm.load %460 : !llvm.ptr -> f32 + %c8_197 = arith.constant 8 : index + %c4_198 = arith.constant 4 : index + %462 = arith.muli %c8_197, %c4_198 : index + %463 = arith.index_cast %462 : index to i64 + %464 = llvm.getelementptr %42[%463] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %465 = llvm.load %464 : !llvm.ptr -> f32 + %c8_199 = arith.constant 8 : index + %c4_200 = arith.constant 4 : index + %466 = arith.muli %c8_199, %c4_200 : index + %467 = arith.index_cast %466 : index to i64 + %468 = llvm.getelementptr %42[%467] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %469 = llvm.load %468 : !llvm.ptr -> f32 + %470 = arith.mulf %465, %469 : f32 + %471 = arith.addf %461, %470 : f32 + %c0_201 = arith.constant 0 : index + %c4_202 = arith.constant 4 : index + %472 = arith.muli %c0_201, %c4_202 : index + %473 = arith.index_cast %472 : index to i64 + %474 = llvm.getelementptr %43[%473] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %471, %474 : f32, !llvm.ptr + %c0_203 = arith.constant 0 : index + %c4_204 = arith.constant 4 : index + %475 = arith.muli %c0_203, %c4_204 : index + %476 = arith.index_cast %475 : index to i64 + %477 = llvm.getelementptr %43[%476] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %478 = llvm.load %477 : !llvm.ptr -> f32 + %c9 = arith.constant 9 : index + %c4_205 = arith.constant 4 : index + %479 = arith.muli %c9, %c4_205 : index + %480 = arith.index_cast %479 : index to i64 + %481 = llvm.getelementptr %42[%480] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %482 = llvm.load %481 : !llvm.ptr -> f32 + %c9_206 = arith.constant 9 : index + %c4_207 = arith.constant 4 : index + %483 = arith.muli %c9_206, %c4_207 : index + %484 = arith.index_cast %483 : index to i64 + %485 = llvm.getelementptr %42[%484] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %486 = llvm.load %485 : !llvm.ptr -> f32 + %487 = arith.mulf %482, %486 : f32 + %488 = arith.addf %478, %487 : f32 + %c0_208 = arith.constant 0 : index + %c4_209 = arith.constant 4 : index + %489 = arith.muli %c0_208, %c4_209 : index + %490 = arith.index_cast %489 : index to i64 + %491 = llvm.getelementptr %43[%490] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %488, %491 : f32, !llvm.ptr + %c0_210 = arith.constant 0 : index + %c4_211 = arith.constant 4 : index + %492 = arith.muli %c0_210, %c4_211 : index + %493 = arith.index_cast %492 : index to i64 + %494 = llvm.getelementptr %43[%493] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %495 = llvm.load %494 : !llvm.ptr -> f32 + %c10_212 = arith.constant 10 : index + %c4_213 = arith.constant 4 : index + %496 = arith.muli %c10_212, %c4_213 : index + %497 = arith.index_cast %496 : index to i64 + %498 = llvm.getelementptr %42[%497] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %499 = llvm.load %498 : !llvm.ptr -> f32 + %c10_214 = arith.constant 10 : index + %c4_215 = arith.constant 4 : index + %500 = arith.muli %c10_214, %c4_215 : index + %501 = arith.index_cast %500 : index to i64 + %502 = llvm.getelementptr %42[%501] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %503 = llvm.load %502 : !llvm.ptr -> f32 + %504 = arith.mulf %499, %503 : f32 + %505 = arith.addf %495, %504 : f32 + %c0_216 = arith.constant 0 : index + %c4_217 = arith.constant 4 : index + %506 = arith.muli %c0_216, %c4_217 : index + %507 = arith.index_cast %506 : index to i64 + %508 = llvm.getelementptr %43[%507] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %505, %508 : f32, !llvm.ptr + %c0_218 = arith.constant 0 : index + %c4_219 = arith.constant 4 : index + %509 = arith.muli %c0_218, %c4_219 : index + %510 = arith.index_cast %509 : index to i64 + %511 = llvm.getelementptr %43[%510] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %512 = llvm.load %511 : !llvm.ptr -> f32 + %c11 = arith.constant 11 : index + %c4_220 = arith.constant 4 : index + %513 = arith.muli %c11, %c4_220 : index + %514 = arith.index_cast %513 : index to i64 + %515 = llvm.getelementptr %42[%514] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %516 = llvm.load %515 : !llvm.ptr -> f32 + %c11_221 = arith.constant 11 : index + %c4_222 = arith.constant 4 : index + %517 = arith.muli %c11_221, %c4_222 : index + %518 = arith.index_cast %517 : index to i64 + %519 = llvm.getelementptr %42[%518] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %520 = llvm.load %519 : !llvm.ptr -> f32 + %521 = arith.mulf %516, %520 : f32 + %522 = arith.addf %512, %521 : f32 + %c0_223 = arith.constant 0 : index + %c4_224 = arith.constant 4 : index + %523 = arith.muli %c0_223, %c4_224 : index + %524 = arith.index_cast %523 : index to i64 + %525 = llvm.getelementptr %43[%524] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %522, %525 : f32, !llvm.ptr + %c0_225 = arith.constant 0 : index + %c4_226 = arith.constant 4 : index + %526 = arith.muli %c0_225, %c4_226 : index + %527 = arith.index_cast %526 : index to i64 + %528 = llvm.getelementptr %43[%527] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %529 = llvm.load %528 : !llvm.ptr -> f32 + %c12_227 = arith.constant 12 : index + %c4_228 = arith.constant 4 : index + %530 = arith.muli %c12_227, %c4_228 : index + %531 = arith.index_cast %530 : index to i64 + %532 = llvm.getelementptr %42[%531] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %533 = llvm.load %532 : !llvm.ptr -> f32 + %c12_229 = arith.constant 12 : index + %c4_230 = arith.constant 4 : index + %534 = arith.muli %c12_229, %c4_230 : index + %535 = arith.index_cast %534 : index to i64 + %536 = llvm.getelementptr %42[%535] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %537 = llvm.load %536 : !llvm.ptr -> f32 + %538 = arith.mulf %533, %537 : f32 + %539 = arith.addf %529, %538 : f32 + %c0_231 = arith.constant 0 : index + %c4_232 = arith.constant 4 : index + %540 = arith.muli %c0_231, %c4_232 : index + %541 = arith.index_cast %540 : index to i64 + %542 = llvm.getelementptr %43[%541] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %539, %542 : f32, !llvm.ptr + %c0_233 = arith.constant 0 : index + %c4_234 = arith.constant 4 : index + %543 = arith.muli %c0_233, %c4_234 : index + %544 = arith.index_cast %543 : index to i64 + %545 = llvm.getelementptr %43[%544] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %546 = llvm.load %545 : !llvm.ptr -> f32 + %c13 = arith.constant 13 : index + %c4_235 = arith.constant 4 : index + %547 = arith.muli %c13, %c4_235 : index + %548 = arith.index_cast %547 : index to i64 + %549 = llvm.getelementptr %42[%548] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %550 = llvm.load %549 : !llvm.ptr -> f32 + %c13_236 = arith.constant 13 : index + %c4_237 = arith.constant 4 : index + %551 = arith.muli %c13_236, %c4_237 : index + %552 = arith.index_cast %551 : index to i64 + %553 = llvm.getelementptr %42[%552] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %554 = llvm.load %553 : !llvm.ptr -> f32 + %555 = arith.mulf %550, %554 : f32 + %556 = arith.addf %546, %555 : f32 + %c0_238 = arith.constant 0 : index + %c4_239 = arith.constant 4 : index + %557 = arith.muli %c0_238, %c4_239 : index + %558 = arith.index_cast %557 : index to i64 + %559 = llvm.getelementptr %43[%558] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %556, %559 : f32, !llvm.ptr + %c0_240 = arith.constant 0 : index + %c4_241 = arith.constant 4 : index + %560 = arith.muli %c0_240, %c4_241 : index + %561 = arith.index_cast %560 : index to i64 + %562 = llvm.getelementptr %43[%561] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %563 = llvm.load %562 : !llvm.ptr -> f32 + %c14_242 = arith.constant 14 : index + %c4_243 = arith.constant 4 : index + %564 = arith.muli %c14_242, %c4_243 : index + %565 = arith.index_cast %564 : index to i64 + %566 = llvm.getelementptr %42[%565] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %567 = llvm.load %566 : !llvm.ptr -> f32 + %c14_244 = arith.constant 14 : index + %c4_245 = arith.constant 4 : index + %568 = arith.muli %c14_244, %c4_245 : index + %569 = arith.index_cast %568 : index to i64 + %570 = llvm.getelementptr %42[%569] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %571 = llvm.load %570 : !llvm.ptr -> f32 + %572 = arith.mulf %567, %571 : f32 + %573 = arith.addf %563, %572 : f32 + %c0_246 = arith.constant 0 : index + %c4_247 = arith.constant 4 : index + %574 = arith.muli %c0_246, %c4_247 : index + %575 = arith.index_cast %574 : index to i64 + %576 = llvm.getelementptr %43[%575] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %573, %576 : f32, !llvm.ptr + %c0_248 = arith.constant 0 : index + %c4_249 = arith.constant 4 : index + %577 = arith.muli %c0_248, %c4_249 : index + %578 = arith.index_cast %577 : index to i64 + %579 = llvm.getelementptr %43[%578] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %580 = llvm.load %579 : !llvm.ptr -> f32 + %c15 = arith.constant 15 : index + %c4_250 = arith.constant 4 : index + %581 = arith.muli %c15, %c4_250 : index + %582 = arith.index_cast %581 : index to i64 + %583 = llvm.getelementptr %42[%582] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %584 = llvm.load %583 : !llvm.ptr -> f32 + %c15_251 = arith.constant 15 : index + %c4_252 = arith.constant 4 : index + %585 = arith.muli %c15_251, %c4_252 : index + %586 = arith.index_cast %585 : index to i64 + %587 = llvm.getelementptr %42[%586] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %588 = llvm.load %587 : !llvm.ptr -> f32 + %589 = arith.mulf %584, %588 : f32 + %590 = arith.addf %580, %589 : f32 + %c0_253 = arith.constant 0 : index + %c4_254 = arith.constant 4 : index + %591 = arith.muli %c0_253, %c4_254 : index + %592 = arith.index_cast %591 : index to i64 + %593 = llvm.getelementptr %43[%592] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %590, %593 : f32, !llvm.ptr + %c0_255 = arith.constant 0 : index + %c4_256 = arith.constant 4 : index + %594 = arith.muli %c0_255, %c4_256 : index + %595 = arith.index_cast %594 : index to i64 + %596 = llvm.getelementptr %43[%595] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %597 = llvm.load %596 : !llvm.ptr -> f32 + %c16_257 = arith.constant 16 : index + %c4_258 = arith.constant 4 : index + %598 = arith.muli %c16_257, %c4_258 : index + %599 = arith.index_cast %598 : index to i64 + %600 = llvm.getelementptr %42[%599] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %601 = llvm.load %600 : !llvm.ptr -> f32 + %c16_259 = arith.constant 16 : index + %c4_260 = arith.constant 4 : index + %602 = arith.muli %c16_259, %c4_260 : index + %603 = arith.index_cast %602 : index to i64 + %604 = llvm.getelementptr %42[%603] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %605 = llvm.load %604 : !llvm.ptr -> f32 + %606 = arith.mulf %601, %605 : f32 + %607 = arith.addf %597, %606 : f32 + %c0_261 = arith.constant 0 : index + %c4_262 = arith.constant 4 : index + %608 = arith.muli %c0_261, %c4_262 : index + %609 = arith.index_cast %608 : index to i64 + %610 = llvm.getelementptr %43[%609] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %607, %610 : f32, !llvm.ptr + %c0_263 = arith.constant 0 : index + %c4_264 = arith.constant 4 : index + %611 = arith.muli %c0_263, %c4_264 : index + %612 = arith.index_cast %611 : index to i64 + %613 = llvm.getelementptr %43[%612] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %614 = llvm.load %613 : !llvm.ptr -> f32 + %c17 = arith.constant 17 : index + %c4_265 = arith.constant 4 : index + %615 = arith.muli %c17, %c4_265 : index + %616 = arith.index_cast %615 : index to i64 + %617 = llvm.getelementptr %42[%616] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %618 = llvm.load %617 : !llvm.ptr -> f32 + %c17_266 = arith.constant 17 : index + %c4_267 = arith.constant 4 : index + %619 = arith.muli %c17_266, %c4_267 : index + %620 = arith.index_cast %619 : index to i64 + %621 = llvm.getelementptr %42[%620] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %622 = llvm.load %621 : !llvm.ptr -> f32 + %623 = arith.mulf %618, %622 : f32 + %624 = arith.addf %614, %623 : f32 + %c0_268 = arith.constant 0 : index + %c4_269 = arith.constant 4 : index + %625 = arith.muli %c0_268, %c4_269 : index + %626 = arith.index_cast %625 : index to i64 + %627 = llvm.getelementptr %43[%626] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %624, %627 : f32, !llvm.ptr + %c0_270 = arith.constant 0 : index + %c4_271 = arith.constant 4 : index + %628 = arith.muli %c0_270, %c4_271 : index + %629 = arith.index_cast %628 : index to i64 + %630 = llvm.getelementptr %43[%629] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %631 = llvm.load %630 : !llvm.ptr -> f32 + %c18_272 = arith.constant 18 : index + %c4_273 = arith.constant 4 : index + %632 = arith.muli %c18_272, %c4_273 : index + %633 = arith.index_cast %632 : index to i64 + %634 = llvm.getelementptr %42[%633] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %635 = llvm.load %634 : !llvm.ptr -> f32 + %c18_274 = arith.constant 18 : index + %c4_275 = arith.constant 4 : index + %636 = arith.muli %c18_274, %c4_275 : index + %637 = arith.index_cast %636 : index to i64 + %638 = llvm.getelementptr %42[%637] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %639 = llvm.load %638 : !llvm.ptr -> f32 + %640 = arith.mulf %635, %639 : f32 + %641 = arith.addf %631, %640 : f32 + %c0_276 = arith.constant 0 : index + %c4_277 = arith.constant 4 : index + %642 = arith.muli %c0_276, %c4_277 : index + %643 = arith.index_cast %642 : index to i64 + %644 = llvm.getelementptr %43[%643] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %641, %644 : f32, !llvm.ptr + %c0_278 = arith.constant 0 : index + %c4_279 = arith.constant 4 : index + %645 = arith.muli %c0_278, %c4_279 : index + %646 = arith.index_cast %645 : index to i64 + %647 = llvm.getelementptr %43[%646] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %648 = llvm.load %647 : !llvm.ptr -> f32 + %c19 = arith.constant 19 : index + %c4_280 = arith.constant 4 : index + %649 = arith.muli %c19, %c4_280 : index + %650 = arith.index_cast %649 : index to i64 + %651 = llvm.getelementptr %42[%650] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %652 = llvm.load %651 : !llvm.ptr -> f32 + %c19_281 = arith.constant 19 : index + %c4_282 = arith.constant 4 : index + %653 = arith.muli %c19_281, %c4_282 : index + %654 = arith.index_cast %653 : index to i64 + %655 = llvm.getelementptr %42[%654] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %656 = llvm.load %655 : !llvm.ptr -> f32 + %657 = arith.mulf %652, %656 : f32 + %658 = arith.addf %648, %657 : f32 + %c0_283 = arith.constant 0 : index + %c4_284 = arith.constant 4 : index + %659 = arith.muli %c0_283, %c4_284 : index + %660 = arith.index_cast %659 : index to i64 + %661 = llvm.getelementptr %43[%660] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %658, %661 : f32, !llvm.ptr + %c0_285 = arith.constant 0 : index + %c4_286 = arith.constant 4 : index + %662 = arith.muli %c0_285, %c4_286 : index + %663 = arith.index_cast %662 : index to i64 + %664 = llvm.getelementptr %43[%663] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %665 = llvm.load %664 : !llvm.ptr -> f32 + %666 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c37888 = arith.constant 37888 : index + %667 = pto.addptr %666, %c37888 : -> + %cst_287 = arith.constant 0.000000e+00 : f32 + %668 = pto.get_tid_x : i32 + %c32_i32_288 = arith.constant 32 : i32 + %669 = arith.floordivsi %668, %c32_i32_288 : i32 + %670 = pto.get_laneid : i32 + %671 = pto.redux_add %665 : f32 -> f32 + %c1_i32_289 = arith.constant 1 : i32 + %672 = arith.cmpi slt, %670, %c1_i32_289 : i32 + scf.if %672 { + %c1_i32_431 = arith.constant 1 : i32 + %975 = arith.muli %669, %c1_i32_431 : i32 + %976 = arith.addi %975, %670 : i32 + %977 = arith.index_cast %976 : i32 to index + pto.store %671, %667[%977] : !pto.ptr, f32 + } + pto.syncthreads + %c32_i32_290 = arith.constant 32 : i32 + %673 = arith.cmpi slt, %668, %c32_i32_290 : i32 + %674 = scf.if %673 -> (f32) { + %c8_i32 = arith.constant 8 : i32 + %975 = arith.cmpi slt, %670, %c8_i32 : i32 + %976 = arith.index_cast %670 : i32 to index + %977 = pto.load %667[%976] : !pto.ptr -> f32 + %978 = arith.select %975, %977, %cst_287 : f32 + %979 = pto.redux_add %978 : f32 -> f32 + scf.yield %979 : f32 + } else { + scf.yield %cst_287 : f32 + } + %c1_i32_291 = arith.constant 1 : i32 + %675 = arith.cmpi slt, %668, %c1_i32_291 : i32 + scf.if %675 { + %975 = arith.index_cast %668 : i32 to index + pto.store %674, %667[%975] : !pto.ptr, f32 + } + pto.syncthreads + %c1_i32_292 = arith.constant 1 : i32 + %676 = arith.remsi %668, %c1_i32_292 : i32 + %677 = arith.index_cast %676 : i32 to index + %678 = pto.load %667[%677] : !pto.ptr -> f32 + pto.syncthreads + %c0_293 = arith.constant 0 : index + %c4_294 = arith.constant 4 : index + %679 = arith.muli %c0_293, %c4_294 : index + %680 = arith.index_cast %679 : index to i64 + %681 = llvm.getelementptr %43[%680] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %678, %681 : f32, !llvm.ptr + %c0_295 = arith.constant 0 : index + %c4_296 = arith.constant 4 : index + %682 = arith.muli %c0_295, %c4_296 : index + %683 = arith.index_cast %682 : index to i64 + %684 = llvm.getelementptr %43[%683] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %685 = llvm.load %684 : !llvm.ptr -> f32 + %cst_297 = arith.constant 5.120000e+03 : f32 + %686 = arith.divf %685, %cst_297 : f32 + %687 = arith.addf %686, %arg4 : f32 + %688 = pto.sqrt %687 : f32 -> f32 + %cst_298 = arith.constant 1.000000e+00 : f32 + %689 = arith.divf %cst_298, %688 : f32 + %690 = pto.sqrt %687 : f32 -> f32 + %cst_299 = arith.constant 1.000000e+00 : f32 + %691 = arith.divf %cst_299, %690 : f32 + %692 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_300 = arith.constant 1 : index + %693 = arith.andi %arg5, %c1_300 : index + %c8_301 = arith.constant 8 : index + %694 = arith.muli %693, %c8_301 : index + pto.store %691, %692[%694] : !pto.ptr, f32 + %c0_302 = arith.constant 0 : index + %c4_303 = arith.constant 4 : index + %695 = arith.muli %c0_302, %c4_303 : index + %696 = arith.index_cast %695 : index to i64 + %697 = llvm.getelementptr %42[%696] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %698 = llvm.load %697 : !llvm.ptr -> vector<2xf32> + %699 = pto.sqrt %687 : f32 -> f32 + %cst_304 = arith.constant 1.000000e+00 : f32 + %700 = arith.divf %cst_304, %699 : f32 + %701 = llvm.mlir.undef : vector<2xf32> + %c0_i32 = arith.constant 0 : i32 + %702 = llvm.insertelement %700, %701[%c0_i32 : i32] : vector<2xf32> + %c1_i32_305 = arith.constant 1 : i32 + %703 = llvm.insertelement %700, %702[%c1_i32_305 : i32] : vector<2xf32> + %704 = arith.mulf %698, %703 : vector<2xf32> + %c0_306 = arith.constant 0 : index + %c4_307 = arith.constant 4 : index + %705 = arith.muli %c0_306, %c4_307 : index + %706 = arith.index_cast %705 : index to i64 + %707 = llvm.getelementptr %1[%706] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %708 = llvm.load %707 : !llvm.ptr -> vector<2xf32> + %709 = arith.mulf %704, %708 : vector<2xf32> + %710 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_308 = arith.constant 1 : index + %711 = arith.andi %arg5, %c1_308 : index + %c8192_309 = arith.constant 8192 : index + %712 = arith.muli %711, %c8192_309 : index + %c0_310 = arith.constant 0 : index + %713 = arith.addi %712, %c0_310 : index + %c2_i32_311 = arith.constant 2 : i32 + %714 = arith.muli %44, %c2_i32_311 : i32 + %715 = arith.index_cast %714 : i32 to index + %716 = arith.addi %713, %715 : index + %c21504_312 = arith.constant 21504 : index + %717 = arith.addi %716, %c21504_312 : index + %c4_313 = arith.constant 4 : index + %718 = arith.muli %717, %c4_313 : index + %719 = arith.index_cast %718 : index to i64 + %720 = pto.castptr %710 : !pto.ptr -> i64 + %721 = llvm.inttoptr %720 : i64 to !llvm.ptr<6> + %722 = llvm.getelementptr %721[%719] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %709, %722 : vector<2xf32>, !llvm.ptr<6> + %c2_314 = arith.constant 2 : index + %c4_315 = arith.constant 4 : index + %723 = arith.muli %c2_314, %c4_315 : index + %724 = arith.index_cast %723 : index to i64 + %725 = llvm.getelementptr %42[%724] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %726 = llvm.load %725 : !llvm.ptr -> vector<2xf32> + %727 = pto.sqrt %687 : f32 -> f32 + %cst_316 = arith.constant 1.000000e+00 : f32 + %728 = arith.divf %cst_316, %727 : f32 + %729 = llvm.mlir.undef : vector<2xf32> + %c0_i32_317 = arith.constant 0 : i32 + %730 = llvm.insertelement %728, %729[%c0_i32_317 : i32] : vector<2xf32> + %c1_i32_318 = arith.constant 1 : i32 + %731 = llvm.insertelement %728, %730[%c1_i32_318 : i32] : vector<2xf32> + %732 = arith.mulf %726, %731 : vector<2xf32> + %c2_319 = arith.constant 2 : index + %c4_320 = arith.constant 4 : index + %733 = arith.muli %c2_319, %c4_320 : index + %734 = arith.index_cast %733 : index to i64 + %735 = llvm.getelementptr %1[%734] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %736 = llvm.load %735 : !llvm.ptr -> vector<2xf32> + %737 = arith.mulf %732, %736 : vector<2xf32> + %738 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_321 = arith.constant 1 : index + %739 = arith.andi %arg5, %c1_321 : index + %c8192_322 = arith.constant 8192 : index + %740 = arith.muli %739, %c8192_322 : index + %c512_323 = arith.constant 512 : index + %741 = arith.addi %740, %c512_323 : index + %c2_i32_324 = arith.constant 2 : i32 + %742 = arith.muli %44, %c2_i32_324 : i32 + %743 = arith.index_cast %742 : i32 to index + %744 = arith.addi %741, %743 : index + %c21504_325 = arith.constant 21504 : index + %745 = arith.addi %744, %c21504_325 : index + %c4_326 = arith.constant 4 : index + %746 = arith.muli %745, %c4_326 : index + %747 = arith.index_cast %746 : index to i64 + %748 = pto.castptr %738 : !pto.ptr -> i64 + %749 = llvm.inttoptr %748 : i64 to !llvm.ptr<6> + %750 = llvm.getelementptr %749[%747] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %737, %750 : vector<2xf32>, !llvm.ptr<6> + %c4_327 = arith.constant 4 : index + %c4_328 = arith.constant 4 : index + %751 = arith.muli %c4_327, %c4_328 : index + %752 = arith.index_cast %751 : index to i64 + %753 = llvm.getelementptr %42[%752] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %754 = llvm.load %753 : !llvm.ptr -> vector<2xf32> + %755 = pto.sqrt %687 : f32 -> f32 + %cst_329 = arith.constant 1.000000e+00 : f32 + %756 = arith.divf %cst_329, %755 : f32 + %757 = llvm.mlir.undef : vector<2xf32> + %c0_i32_330 = arith.constant 0 : i32 + %758 = llvm.insertelement %756, %757[%c0_i32_330 : i32] : vector<2xf32> + %c1_i32_331 = arith.constant 1 : i32 + %759 = llvm.insertelement %756, %758[%c1_i32_331 : i32] : vector<2xf32> + %760 = arith.mulf %754, %759 : vector<2xf32> + %c4_332 = arith.constant 4 : index + %c4_333 = arith.constant 4 : index + %761 = arith.muli %c4_332, %c4_333 : index + %762 = arith.index_cast %761 : index to i64 + %763 = llvm.getelementptr %1[%762] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %764 = llvm.load %763 : !llvm.ptr -> vector<2xf32> + %765 = arith.mulf %760, %764 : vector<2xf32> + %766 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_334 = arith.constant 1 : index + %767 = arith.andi %arg5, %c1_334 : index + %c8192_335 = arith.constant 8192 : index + %768 = arith.muli %767, %c8192_335 : index + %c1024_336 = arith.constant 1024 : index + %769 = arith.addi %768, %c1024_336 : index + %c2_i32_337 = arith.constant 2 : i32 + %770 = arith.muli %44, %c2_i32_337 : i32 + %771 = arith.index_cast %770 : i32 to index + %772 = arith.addi %769, %771 : index + %c21504_338 = arith.constant 21504 : index + %773 = arith.addi %772, %c21504_338 : index + %c4_339 = arith.constant 4 : index + %774 = arith.muli %773, %c4_339 : index + %775 = arith.index_cast %774 : index to i64 + %776 = pto.castptr %766 : !pto.ptr -> i64 + %777 = llvm.inttoptr %776 : i64 to !llvm.ptr<6> + %778 = llvm.getelementptr %777[%775] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %765, %778 : vector<2xf32>, !llvm.ptr<6> + %c6_340 = arith.constant 6 : index + %c4_341 = arith.constant 4 : index + %779 = arith.muli %c6_340, %c4_341 : index + %780 = arith.index_cast %779 : index to i64 + %781 = llvm.getelementptr %42[%780] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %782 = llvm.load %781 : !llvm.ptr -> vector<2xf32> + %783 = pto.sqrt %687 : f32 -> f32 + %cst_342 = arith.constant 1.000000e+00 : f32 + %784 = arith.divf %cst_342, %783 : f32 + %785 = llvm.mlir.undef : vector<2xf32> + %c0_i32_343 = arith.constant 0 : i32 + %786 = llvm.insertelement %784, %785[%c0_i32_343 : i32] : vector<2xf32> + %c1_i32_344 = arith.constant 1 : i32 + %787 = llvm.insertelement %784, %786[%c1_i32_344 : i32] : vector<2xf32> + %788 = arith.mulf %782, %787 : vector<2xf32> + %c6_345 = arith.constant 6 : index + %c4_346 = arith.constant 4 : index + %789 = arith.muli %c6_345, %c4_346 : index + %790 = arith.index_cast %789 : index to i64 + %791 = llvm.getelementptr %1[%790] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %792 = llvm.load %791 : !llvm.ptr -> vector<2xf32> + %793 = arith.mulf %788, %792 : vector<2xf32> + %794 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_347 = arith.constant 1 : index + %795 = arith.andi %arg5, %c1_347 : index + %c8192_348 = arith.constant 8192 : index + %796 = arith.muli %795, %c8192_348 : index + %c1536_349 = arith.constant 1536 : index + %797 = arith.addi %796, %c1536_349 : index + %c2_i32_350 = arith.constant 2 : i32 + %798 = arith.muli %44, %c2_i32_350 : i32 + %799 = arith.index_cast %798 : i32 to index + %800 = arith.addi %797, %799 : index + %c21504_351 = arith.constant 21504 : index + %801 = arith.addi %800, %c21504_351 : index + %c4_352 = arith.constant 4 : index + %802 = arith.muli %801, %c4_352 : index + %803 = arith.index_cast %802 : index to i64 + %804 = pto.castptr %794 : !pto.ptr -> i64 + %805 = llvm.inttoptr %804 : i64 to !llvm.ptr<6> + %806 = llvm.getelementptr %805[%803] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %793, %806 : vector<2xf32>, !llvm.ptr<6> + %c8_353 = arith.constant 8 : index + %c4_354 = arith.constant 4 : index + %807 = arith.muli %c8_353, %c4_354 : index + %808 = arith.index_cast %807 : index to i64 + %809 = llvm.getelementptr %42[%808] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %810 = llvm.load %809 : !llvm.ptr -> vector<2xf32> + %811 = pto.sqrt %687 : f32 -> f32 + %cst_355 = arith.constant 1.000000e+00 : f32 + %812 = arith.divf %cst_355, %811 : f32 + %813 = llvm.mlir.undef : vector<2xf32> + %c0_i32_356 = arith.constant 0 : i32 + %814 = llvm.insertelement %812, %813[%c0_i32_356 : i32] : vector<2xf32> + %c1_i32_357 = arith.constant 1 : i32 + %815 = llvm.insertelement %812, %814[%c1_i32_357 : i32] : vector<2xf32> + %816 = arith.mulf %810, %815 : vector<2xf32> + %c8_358 = arith.constant 8 : index + %c4_359 = arith.constant 4 : index + %817 = arith.muli %c8_358, %c4_359 : index + %818 = arith.index_cast %817 : index to i64 + %819 = llvm.getelementptr %1[%818] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %820 = llvm.load %819 : !llvm.ptr -> vector<2xf32> + %821 = arith.mulf %816, %820 : vector<2xf32> + %822 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_360 = arith.constant 1 : index + %823 = arith.andi %arg5, %c1_360 : index + %c8192_361 = arith.constant 8192 : index + %824 = arith.muli %823, %c8192_361 : index + %c2048_362 = arith.constant 2048 : index + %825 = arith.addi %824, %c2048_362 : index + %c2_i32_363 = arith.constant 2 : i32 + %826 = arith.muli %44, %c2_i32_363 : i32 + %827 = arith.index_cast %826 : i32 to index + %828 = arith.addi %825, %827 : index + %c21504_364 = arith.constant 21504 : index + %829 = arith.addi %828, %c21504_364 : index + %c4_365 = arith.constant 4 : index + %830 = arith.muli %829, %c4_365 : index + %831 = arith.index_cast %830 : index to i64 + %832 = pto.castptr %822 : !pto.ptr -> i64 + %833 = llvm.inttoptr %832 : i64 to !llvm.ptr<6> + %834 = llvm.getelementptr %833[%831] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %821, %834 : vector<2xf32>, !llvm.ptr<6> + %c10_366 = arith.constant 10 : index + %c4_367 = arith.constant 4 : index + %835 = arith.muli %c10_366, %c4_367 : index + %836 = arith.index_cast %835 : index to i64 + %837 = llvm.getelementptr %42[%836] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %838 = llvm.load %837 : !llvm.ptr -> vector<2xf32> + %839 = pto.sqrt %687 : f32 -> f32 + %cst_368 = arith.constant 1.000000e+00 : f32 + %840 = arith.divf %cst_368, %839 : f32 + %841 = llvm.mlir.undef : vector<2xf32> + %c0_i32_369 = arith.constant 0 : i32 + %842 = llvm.insertelement %840, %841[%c0_i32_369 : i32] : vector<2xf32> + %c1_i32_370 = arith.constant 1 : i32 + %843 = llvm.insertelement %840, %842[%c1_i32_370 : i32] : vector<2xf32> + %844 = arith.mulf %838, %843 : vector<2xf32> + %c10_371 = arith.constant 10 : index + %c4_372 = arith.constant 4 : index + %845 = arith.muli %c10_371, %c4_372 : index + %846 = arith.index_cast %845 : index to i64 + %847 = llvm.getelementptr %1[%846] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %848 = llvm.load %847 : !llvm.ptr -> vector<2xf32> + %849 = arith.mulf %844, %848 : vector<2xf32> + %850 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_373 = arith.constant 1 : index + %851 = arith.andi %arg5, %c1_373 : index + %c8192_374 = arith.constant 8192 : index + %852 = arith.muli %851, %c8192_374 : index + %c2560_375 = arith.constant 2560 : index + %853 = arith.addi %852, %c2560_375 : index + %c2_i32_376 = arith.constant 2 : i32 + %854 = arith.muli %44, %c2_i32_376 : i32 + %855 = arith.index_cast %854 : i32 to index + %856 = arith.addi %853, %855 : index + %c21504_377 = arith.constant 21504 : index + %857 = arith.addi %856, %c21504_377 : index + %c4_378 = arith.constant 4 : index + %858 = arith.muli %857, %c4_378 : index + %859 = arith.index_cast %858 : index to i64 + %860 = pto.castptr %850 : !pto.ptr -> i64 + %861 = llvm.inttoptr %860 : i64 to !llvm.ptr<6> + %862 = llvm.getelementptr %861[%859] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %849, %862 : vector<2xf32>, !llvm.ptr<6> + %c12_379 = arith.constant 12 : index + %c4_380 = arith.constant 4 : index + %863 = arith.muli %c12_379, %c4_380 : index + %864 = arith.index_cast %863 : index to i64 + %865 = llvm.getelementptr %42[%864] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %866 = llvm.load %865 : !llvm.ptr -> vector<2xf32> + %867 = pto.sqrt %687 : f32 -> f32 + %cst_381 = arith.constant 1.000000e+00 : f32 + %868 = arith.divf %cst_381, %867 : f32 + %869 = llvm.mlir.undef : vector<2xf32> + %c0_i32_382 = arith.constant 0 : i32 + %870 = llvm.insertelement %868, %869[%c0_i32_382 : i32] : vector<2xf32> + %c1_i32_383 = arith.constant 1 : i32 + %871 = llvm.insertelement %868, %870[%c1_i32_383 : i32] : vector<2xf32> + %872 = arith.mulf %866, %871 : vector<2xf32> + %c12_384 = arith.constant 12 : index + %c4_385 = arith.constant 4 : index + %873 = arith.muli %c12_384, %c4_385 : index + %874 = arith.index_cast %873 : index to i64 + %875 = llvm.getelementptr %1[%874] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %876 = llvm.load %875 : !llvm.ptr -> vector<2xf32> + %877 = arith.mulf %872, %876 : vector<2xf32> + %878 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_386 = arith.constant 1 : index + %879 = arith.andi %arg5, %c1_386 : index + %c8192_387 = arith.constant 8192 : index + %880 = arith.muli %879, %c8192_387 : index + %c3072_388 = arith.constant 3072 : index + %881 = arith.addi %880, %c3072_388 : index + %c2_i32_389 = arith.constant 2 : i32 + %882 = arith.muli %44, %c2_i32_389 : i32 + %883 = arith.index_cast %882 : i32 to index + %884 = arith.addi %881, %883 : index + %c21504_390 = arith.constant 21504 : index + %885 = arith.addi %884, %c21504_390 : index + %c4_391 = arith.constant 4 : index + %886 = arith.muli %885, %c4_391 : index + %887 = arith.index_cast %886 : index to i64 + %888 = pto.castptr %878 : !pto.ptr -> i64 + %889 = llvm.inttoptr %888 : i64 to !llvm.ptr<6> + %890 = llvm.getelementptr %889[%887] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %877, %890 : vector<2xf32>, !llvm.ptr<6> + %c14_392 = arith.constant 14 : index + %c4_393 = arith.constant 4 : index + %891 = arith.muli %c14_392, %c4_393 : index + %892 = arith.index_cast %891 : index to i64 + %893 = llvm.getelementptr %42[%892] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %894 = llvm.load %893 : !llvm.ptr -> vector<2xf32> + %895 = pto.sqrt %687 : f32 -> f32 + %cst_394 = arith.constant 1.000000e+00 : f32 + %896 = arith.divf %cst_394, %895 : f32 + %897 = llvm.mlir.undef : vector<2xf32> + %c0_i32_395 = arith.constant 0 : i32 + %898 = llvm.insertelement %896, %897[%c0_i32_395 : i32] : vector<2xf32> + %c1_i32_396 = arith.constant 1 : i32 + %899 = llvm.insertelement %896, %898[%c1_i32_396 : i32] : vector<2xf32> + %900 = arith.mulf %894, %899 : vector<2xf32> + %c14_397 = arith.constant 14 : index + %c4_398 = arith.constant 4 : index + %901 = arith.muli %c14_397, %c4_398 : index + %902 = arith.index_cast %901 : index to i64 + %903 = llvm.getelementptr %1[%902] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %904 = llvm.load %903 : !llvm.ptr -> vector<2xf32> + %905 = arith.mulf %900, %904 : vector<2xf32> + %906 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_399 = arith.constant 1 : index + %907 = arith.andi %arg5, %c1_399 : index + %c8192_400 = arith.constant 8192 : index + %908 = arith.muli %907, %c8192_400 : index + %c3584_401 = arith.constant 3584 : index + %909 = arith.addi %908, %c3584_401 : index + %c2_i32_402 = arith.constant 2 : i32 + %910 = arith.muli %44, %c2_i32_402 : i32 + %911 = arith.index_cast %910 : i32 to index + %912 = arith.addi %909, %911 : index + %c21504_403 = arith.constant 21504 : index + %913 = arith.addi %912, %c21504_403 : index + %c4_404 = arith.constant 4 : index + %914 = arith.muli %913, %c4_404 : index + %915 = arith.index_cast %914 : index to i64 + %916 = pto.castptr %906 : !pto.ptr -> i64 + %917 = llvm.inttoptr %916 : i64 to !llvm.ptr<6> + %918 = llvm.getelementptr %917[%915] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %905, %918 : vector<2xf32>, !llvm.ptr<6> + %c16_405 = arith.constant 16 : index + %c4_406 = arith.constant 4 : index + %919 = arith.muli %c16_405, %c4_406 : index + %920 = arith.index_cast %919 : index to i64 + %921 = llvm.getelementptr %42[%920] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %922 = llvm.load %921 : !llvm.ptr -> vector<2xf32> + %923 = pto.sqrt %687 : f32 -> f32 + %cst_407 = arith.constant 1.000000e+00 : f32 + %924 = arith.divf %cst_407, %923 : f32 + %925 = llvm.mlir.undef : vector<2xf32> + %c0_i32_408 = arith.constant 0 : i32 + %926 = llvm.insertelement %924, %925[%c0_i32_408 : i32] : vector<2xf32> + %c1_i32_409 = arith.constant 1 : i32 + %927 = llvm.insertelement %924, %926[%c1_i32_409 : i32] : vector<2xf32> + %928 = arith.mulf %922, %927 : vector<2xf32> + %c16_410 = arith.constant 16 : index + %c4_411 = arith.constant 4 : index + %929 = arith.muli %c16_410, %c4_411 : index + %930 = arith.index_cast %929 : index to i64 + %931 = llvm.getelementptr %1[%930] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %932 = llvm.load %931 : !llvm.ptr -> vector<2xf32> + %933 = arith.mulf %928, %932 : vector<2xf32> + %934 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_412 = arith.constant 1 : index + %935 = arith.andi %arg5, %c1_412 : index + %c8192_413 = arith.constant 8192 : index + %936 = arith.muli %935, %c8192_413 : index + %c4096_414 = arith.constant 4096 : index + %937 = arith.addi %936, %c4096_414 : index + %c2_i32_415 = arith.constant 2 : i32 + %938 = arith.muli %44, %c2_i32_415 : i32 + %939 = arith.index_cast %938 : i32 to index + %940 = arith.addi %937, %939 : index + %c21504_416 = arith.constant 21504 : index + %941 = arith.addi %940, %c21504_416 : index + %c4_417 = arith.constant 4 : index + %942 = arith.muli %941, %c4_417 : index + %943 = arith.index_cast %942 : index to i64 + %944 = pto.castptr %934 : !pto.ptr -> i64 + %945 = llvm.inttoptr %944 : i64 to !llvm.ptr<6> + %946 = llvm.getelementptr %945[%943] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %933, %946 : vector<2xf32>, !llvm.ptr<6> + %c18_418 = arith.constant 18 : index + %c4_419 = arith.constant 4 : index + %947 = arith.muli %c18_418, %c4_419 : index + %948 = arith.index_cast %947 : index to i64 + %949 = llvm.getelementptr %42[%948] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %950 = llvm.load %949 : !llvm.ptr -> vector<2xf32> + %951 = pto.sqrt %687 : f32 -> f32 + %cst_420 = arith.constant 1.000000e+00 : f32 + %952 = arith.divf %cst_420, %951 : f32 + %953 = llvm.mlir.undef : vector<2xf32> + %c0_i32_421 = arith.constant 0 : i32 + %954 = llvm.insertelement %952, %953[%c0_i32_421 : i32] : vector<2xf32> + %c1_i32_422 = arith.constant 1 : i32 + %955 = llvm.insertelement %952, %954[%c1_i32_422 : i32] : vector<2xf32> + %956 = arith.mulf %950, %955 : vector<2xf32> + %c18_423 = arith.constant 18 : index + %c4_424 = arith.constant 4 : index + %957 = arith.muli %c18_423, %c4_424 : index + %958 = arith.index_cast %957 : index to i64 + %959 = llvm.getelementptr %1[%958] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %960 = llvm.load %959 : !llvm.ptr -> vector<2xf32> + %961 = arith.mulf %956, %960 : vector<2xf32> + %962 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_425 = arith.constant 1 : index + %963 = arith.andi %arg5, %c1_425 : index + %c8192_426 = arith.constant 8192 : index + %964 = arith.muli %963, %c8192_426 : index + %c4608_427 = arith.constant 4608 : index + %965 = arith.addi %964, %c4608_427 : index + %c2_i32_428 = arith.constant 2 : i32 + %966 = arith.muli %44, %c2_i32_428 : i32 + %967 = arith.index_cast %966 : i32 to index + %968 = arith.addi %965, %967 : index + %c21504_429 = arith.constant 21504 : index + %969 = arith.addi %968, %c21504_429 : index + %c4_430 = arith.constant 4 : index + %970 = arith.muli %969, %c4_430 : index + %971 = arith.index_cast %970 : index to i64 + %972 = pto.castptr %962 : !pto.ptr -> i64 + %973 = llvm.inttoptr %972 : i64 to !llvm.ptr<6> + %974 = llvm.getelementptr %973[%971] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %961, %974 : vector<2xf32>, !llvm.ptr<6> + } + %c1_13 = arith.constant 1 : index + %18 = arith.andi %arg5, %c1_13 : index + pto.set_flag_dyn[, , %18] + %c1_14 = arith.constant 1 : index + %19 = arith.andi %arg5, %c1_14 : index + pto.set_flag_dyn[, , %19] + %c1_15 = arith.constant 1 : index + %20 = arith.andi %arg5, %c1_15 : index + pto.wait_flag_dyn[, , %20] + %21 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_16 = arith.constant 1 : index + %22 = arith.andi %arg5, %c1_16 : index + %c8 = arith.constant 8 : index + %23 = arith.muli %22, %c8 : index + %24 = pto.addptr %21, %23 : -> + %c64_17 = arith.constant 64 : index + %25 = arith.muli %arg5, %c64_17 : index + %26 = pto.get_block_idx + %27 = arith.index_cast %26 : i64 to index + %28 = arith.addi %25, %27 : index + %29 = pto.addptr %arg0, %28 : -> + %c1_i64_18 = arith.constant 1 : i64 + %c4_i64 = arith.constant 4 : i64 + %c4_i64_19 = arith.constant 4 : i64 + %c4_i64_20 = arith.constant 4 : i64 + %c0_i64_21 = arith.constant 0 : i64 + pto.mte_ub_gm %24, %29, %c4_i64_20 nburst(%c1_i64_18, %c4_i64, %c4_i64_19) l2_cache_ctl(%c0_i64_21) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + %30 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_22 = arith.constant 1 : index + %31 = arith.andi %arg5, %c1_22 : index + %c8192_23 = arith.constant 8192 : index + %32 = arith.muli %31, %c8192_23 : index + %c21504 = arith.constant 21504 : index + %33 = arith.addi %32, %c21504 : index + %34 = pto.addptr %30, %33 : -> + %c327680_24 = arith.constant 327680 : index + %35 = arith.muli %arg5, %c327680_24 : index + %36 = pto.get_block_idx + %c5120_i64_25 = arith.constant 5120 : i64 + %37 = arith.muli %36, %c5120_i64_25 : i64 + %38 = arith.index_cast %37 : i64 to index + %39 = arith.addi %35, %38 : index + %40 = pto.addptr %arg3, %39 : -> + %c1_i64_26 = arith.constant 1 : i64 + %c20480_i64_27 = arith.constant 20480 : i64 + %c20480_i64_28 = arith.constant 20480 : i64 + %c20480_i64_29 = arith.constant 20480 : i64 + %c0_i64_30 = arith.constant 0 : i64 + pto.mte_ub_gm %34, %40, %c20480_i64_29 nburst(%c1_i64_26, %c20480_i64_27, %c20480_i64_28) l2_cache_ctl(%c0_i64_30) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + %c1_31 = arith.constant 1 : index + %41 = arith.andi %arg5, %c1_31 : index + pto.set_flag_dyn[, , %41] + } + pto.wait_flag[, , ] + pto.wait_flag[, , ] + pto.wait_flag[, , ] + pto.wait_flag[, , ] + return + } + } +} diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/kernel.ptodsl.py b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/kernel.ptodsl.py new file mode 100644 index 0000000000..e080c053f0 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/kernel.ptodsl.py @@ -0,0 +1,58 @@ +from ptodsl import pto, scalar +from ptodsl._ops import _coerce_i64 as _tl_coerce_i64 +from ptodsl._surface_values import wrap_surface_value as _tl_wrap_surface_value + +@pto.jit(name="rmsnorm_d5120_kernel", kernel_kind="vector", target="a5", mode="explicit") +def rmsnorm_d5120_kernel(RSTD: pto.ptr(pto.f32, "gm"), W: pto.ptr(pto.f32, "gm"), X: pto.ptr(pto.f32, "gm"), Y: pto.ptr(pto.f32, "gm"), eps: pto.f32): + buf_dyn_shmem = pto.castptr(pto.const(0, dtype=pto.i64), pto.ptr(pto.i8, "ub")) + w_frag = [None] * 32 + w_frag_1 = pto.alloc_buffer((32,), pto.f32) + pto.set_flag("MTE3", "V", event_id=0) + pto.set_flag("MTE3", "V", event_id=1) + pto.set_flag("V", "MTE2", event_id=0) + pto.set_flag("V", "MTE2", event_id=1) + pto.mte_gm_ub(W, pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), 0, 20480, nburst=(1, 20480, 20480)) + pto.set_flag("MTE2", "V", event_id=2) + pto.wait_flag("MTE2", "V", event_id=2) + with pto.simt(256, 1, 1): + simtvf_tx = pto.get_tid_x() + simtvf_ty = pto.get_tid_y() + simtvf_tz = pto.get_tid_z() + for i in pto.static_range(0, 16): + if i < 10: + scalar.store(scalar.load(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), (i * 512) + (simtvf_tx * 2), contiguous=2), w_frag_1, i * 2) + with pto.for_(0, 64, step=1) as t: + pto.wait_flag("V", "MTE2", event_id=t & 1) + pto.mte_gm_ub(pto.addptr(X, (t * 327680) + (pto.get_block_idx() * 5120)), pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((t & 1) * 8192) + 5120), 0, 20480, nburst=(1, 20480, 20480)) + pto.set_flag("MTE2", "V", event_id=t & 1) + pto.wait_flag("MTE3", "V", event_id=t & 1) + pto.wait_flag("MTE2", "V", event_id=t & 1) + with pto.simt(256, 1, 1): + x_frag = pto.alloc_buffer((32,), pto.f32) + sum_sq = pto.alloc_buffer((1,), pto.f32) + simtvf_tx_1 = pto.get_tid_x() + simtvf_ty_1 = pto.get_tid_y() + simtvf_tz_1 = pto.get_tid_z() + for i_1 in pto.static_range(0, 16): + scalar.store(scalar.load(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((((t & 1) * 8192) + (i_1 * 512)) + (simtvf_tx_1 * 2)) + 5120, contiguous=2), x_frag, i_1 * 2) + scalar.store(float.fromhex('0x0p+0'), sum_sq, 0) + for i_2 in pto.static_range(0, 32): + if i_2 < 20: + scalar.store(scalar.load(sum_sq, 0) + (scalar.load(x_frag, i_2) * scalar.load(x_frag, i_2)), sum_sq, 0) + scalar.store(pto.simt_allreduce_sum(scalar.load(sum_sq, 0), threads=256, scale=1, thread_offset=0, scratch=pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), 37888)), sum_sq, 0) + var = (scalar.load(sum_sq, 0) / float.fromhex('0x1.4p+12')) + eps + rstd_val = float.fromhex('0x1p+0') / pto.sqrt(var) + scalar.store(float.fromhex('0x1p+0') / pto.sqrt(var), pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), (t & 1) * 8) + for i_3 in pto.static_range(0, 16): + if i_3 < 10: + scalar.store((scalar.load(x_frag, i_3 * 2, contiguous=2) * pto.Vec(pto.f32, 2, init=(float.fromhex('0x1p+0') / pto.sqrt(var)))) * scalar.load(w_frag_1, i_3 * 2, contiguous=2), pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((((t & 1) * 8192) + (i_3 * 512)) + (simtvf_tx_1 * 2)) + 21504) + pto.set_flag("V", "MTE3", event_id=t & 1) + pto.set_flag("V", "MTE2", event_id=t & 1) + pto.wait_flag("V", "MTE3", event_id=t & 1) + pto.mte_ub_gm(pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), (t & 1) * 8), pto.addptr(RSTD, (t * 64) + pto.get_block_idx()), 4, nburst=(1, 4, 4)) + pto.mte_ub_gm(pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((t & 1) * 8192) + 21504), pto.addptr(Y, (t * 327680) + (pto.get_block_idx() * 5120)), 20480, nburst=(1, 20480, 20480)) + pto.set_flag("MTE3", "V", event_id=t & 1) + pto.wait_flag("MTE3", "V", event_id=0) + pto.wait_flag("MTE3", "V", event_id=1) + pto.wait_flag("V", "MTE2", event_id=0) + pto.wait_flag("V", "MTE2", event_id=1) diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/kernel_vpto.ll b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/kernel_vpto.ll new file mode 100644 index 0000000000..9d58b7ffde --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/kernel_vpto.ll @@ -0,0 +1,667 @@ +; ModuleID = 'ptoas.hivm.official.vector' +source_filename = "ptoas.hivm.official.vector" + +; Unknown intrinsic +declare void @llvm.hivm.SET.FLAG.IMM(i64, i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.MOV.OUT.TO.UB.ALIGN.V2.f32.DV(ptr addrspace(6), ptr addrspace(1), i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.WAIT.FLAG.IMM(i64, i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.store.vfsimt.info(i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.WAIT.FLAG.REG(i64, i64, i64) #0 + +; Unknown intrinsic +declare i64 @llvm.hivm.GET.BLOCK.IDX() #0 + +; Unknown intrinsic +declare void @llvm.hivm.SET.FLAG.REG(i64, i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.MOV.UB.TO.OUT.ALIGN.V2.DV(ptr addrspace(1), ptr addrspace(6), i64, i64) #0 + +; Unknown intrinsic +declare i32 @llvm.hivm.get.TID.X() #0 + +; Unknown intrinsic +declare i32 @llvm.hivm.get.laneID() #0 + +; Unknown intrinsic +declare float @llvm.hivm.redux.add.f32(float) #0 + +; Unknown intrinsic +declare void @llvm.hivm.sync.workitems() #0 + +; Function Attrs: nocallback nofree nosync nounwind speculatable willreturn memory(none) +declare float @llvm.sqrt.f32(float) #1 + +define void @rmsnorm_d5120_kernel_mix_aiv(ptr addrspace(1) %0, ptr addrspace(1) %1, ptr addrspace(1) %2, ptr addrspace(1) %3, float %4) #2 { + call void @llvm.hivm.SET.FLAG.IMM(i64 5, i64 1, i64 0) + call void @llvm.hivm.SET.FLAG.IMM(i64 5, i64 1, i64 1) + call void @llvm.hivm.SET.FLAG.IMM(i64 1, i64 4, i64 0) + call void @llvm.hivm.SET.FLAG.IMM(i64 1, i64 4, i64 1) + call void @llvm.hivm.MOV.OUT.TO.UB.ALIGN.V2.f32.DV(ptr addrspace(6) null, ptr addrspace(1) %1, i64 687194767376, i64 22517998136872960) + call void @llvm.hivm.SET.FLAG.IMM(i64 4, i64 1, i64 2) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 4, i64 1, i64 2) + call void @llvm.hivm.store.vfsimt.info(i64 4295033088) + call simt_entry void @rmsnorm_d5120_kernel_simt_0(ptr addrspace(6) null) + br label %6 + +6: ; preds = %9, %5 + %7 = phi i64 [ %27, %9 ], [ 0, %5 ] + %8 = icmp slt i64 %7, 64 + br i1 %8, label %9, label %28 + +9: ; preds = %6 + %10 = and i64 %7, 1 + call void @llvm.hivm.WAIT.FLAG.REG(i64 1, i64 4, i64 %10) + %11 = mul i64 %7, 327680 + %12 = call i64 @llvm.hivm.GET.BLOCK.IDX() + %13 = mul i64 %12, 5120 + %14 = add i64 %11, %13 + %15 = getelementptr float, ptr addrspace(1) %2, i64 %14 + %16 = mul i64 %10, 8192 + %17 = add i64 %16, 5120 + %18 = getelementptr float, ptr addrspace(6) null, i64 %17 + call void @llvm.hivm.MOV.OUT.TO.UB.ALIGN.V2.f32.DV(ptr addrspace(6) %18, ptr addrspace(1) %15, i64 687194767376, i64 22517998136872960) + call void @llvm.hivm.SET.FLAG.REG(i64 4, i64 1, i64 %10) + call void @llvm.hivm.WAIT.FLAG.REG(i64 5, i64 1, i64 %10) + call void @llvm.hivm.WAIT.FLAG.REG(i64 4, i64 1, i64 %10) + call void @llvm.hivm.store.vfsimt.info(i64 4295033088) + call simt_entry void @rmsnorm_d5120_kernel_simt_1(i64 %16, ptr addrspace(6) null, i64 %17, float %4, i64 %10) + call void @llvm.hivm.SET.FLAG.REG(i64 1, i64 5, i64 %10) + call void @llvm.hivm.SET.FLAG.REG(i64 1, i64 4, i64 %10) + call void @llvm.hivm.WAIT.FLAG.REG(i64 1, i64 5, i64 %10) + %19 = mul i64 %10, 8 + %20 = getelementptr float, ptr addrspace(6) null, i64 %19 + %21 = mul i64 %7, 64 + %22 = add i64 %21, %12 + %23 = getelementptr float, ptr addrspace(1) %0, i64 %22 + call void @llvm.hivm.MOV.UB.TO.OUT.ALIGN.V2.DV(ptr addrspace(1) %23, ptr addrspace(6) %20, i64 134217744, i64 4398046511108) + %24 = add i64 %16, 21504 + %25 = getelementptr float, ptr addrspace(6) null, i64 %24 + %26 = getelementptr float, ptr addrspace(1) %3, i64 %14 + call void @llvm.hivm.MOV.UB.TO.OUT.ALIGN.V2.DV(ptr addrspace(1) %26, ptr addrspace(6) %25, i64 687194767376, i64 22517998136872960) + call void @llvm.hivm.SET.FLAG.REG(i64 5, i64 1, i64 %10) + %27 = add i64 %7, 1 + br label %6 + +28: ; preds = %6 + call void @llvm.hivm.WAIT.FLAG.IMM(i64 5, i64 1, i64 0) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 5, i64 1, i64 1) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 1, i64 4, i64 0) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 1, i64 4, i64 1) + ret void +} + +; Function Attrs: noinline +define linkonce_odr simt_entry void @rmsnorm_d5120_kernel_simt_0(ptr addrspace(6) %0) #3 !annotation !8 !annotation !9 { + %2 = call i32 @llvm.hivm.get.TID.X() + %3 = mul i32 %2, 2 + %4 = sext i32 %3 to i64 + %5 = mul i64 %4, 4 + %6 = ptrtoint ptr addrspace(6) %0 to i64 + %7 = inttoptr i64 %6 to ptr addrspace(6) + %8 = getelementptr i8, ptr addrspace(6) %7, i64 %5 + %9 = load <2 x float>, ptr addrspace(6) %8, align 8 + %10 = extractelement <2 x float> %9, i32 0 + %11 = extractelement <2 x float> %9, i32 1 + %12 = add i32 %3, 512 + %13 = sext i32 %12 to i64 + %14 = mul i64 %13, 4 + %15 = getelementptr i8, ptr addrspace(6) %7, i64 %14 + %16 = load <2 x float>, ptr addrspace(6) %15, align 8 + %17 = extractelement <2 x float> %16, i32 0 + %18 = extractelement <2 x float> %16, i32 1 + %19 = add i32 %3, 1024 + %20 = sext i32 %19 to i64 + %21 = mul i64 %20, 4 + %22 = getelementptr i8, ptr addrspace(6) %7, i64 %21 + %23 = load <2 x float>, ptr addrspace(6) %22, align 8 + %24 = extractelement <2 x float> %23, i32 0 + %25 = extractelement <2 x float> %23, i32 1 + %26 = add i32 %3, 1536 + %27 = sext i32 %26 to i64 + %28 = mul i64 %27, 4 + %29 = getelementptr i8, ptr addrspace(6) %7, i64 %28 + %30 = load <2 x float>, ptr addrspace(6) %29, align 8 + %31 = extractelement <2 x float> %30, i32 0 + %32 = extractelement <2 x float> %30, i32 1 + %33 = add i32 %3, 2048 + %34 = sext i32 %33 to i64 + %35 = mul i64 %34, 4 + %36 = getelementptr i8, ptr addrspace(6) %7, i64 %35 + %37 = load <2 x float>, ptr addrspace(6) %36, align 8 + %38 = extractelement <2 x float> %37, i32 0 + %39 = extractelement <2 x float> %37, i32 1 + %40 = add i32 %3, 2560 + %41 = sext i32 %40 to i64 + %42 = mul i64 %41, 4 + %43 = getelementptr i8, ptr addrspace(6) %7, i64 %42 + %44 = load <2 x float>, ptr addrspace(6) %43, align 8 + %45 = extractelement <2 x float> %44, i32 0 + %46 = extractelement <2 x float> %44, i32 1 + %47 = add i32 %3, 3072 + %48 = sext i32 %47 to i64 + %49 = mul i64 %48, 4 + %50 = getelementptr i8, ptr addrspace(6) %7, i64 %49 + %51 = load <2 x float>, ptr addrspace(6) %50, align 8 + %52 = extractelement <2 x float> %51, i32 0 + %53 = extractelement <2 x float> %51, i32 1 + %54 = add i32 %3, 3584 + %55 = sext i32 %54 to i64 + %56 = mul i64 %55, 4 + %57 = getelementptr i8, ptr addrspace(6) %7, i64 %56 + %58 = load <2 x float>, ptr addrspace(6) %57, align 8 + %59 = extractelement <2 x float> %58, i32 0 + %60 = extractelement <2 x float> %58, i32 1 + %61 = add i32 %3, 4096 + %62 = sext i32 %61 to i64 + %63 = mul i64 %62, 4 + %64 = getelementptr i8, ptr addrspace(6) %7, i64 %63 + %65 = load <2 x float>, ptr addrspace(6) %64, align 8 + %66 = extractelement <2 x float> %65, i32 0 + %67 = extractelement <2 x float> %65, i32 1 + %68 = add i32 %3, 4608 + %69 = sext i32 %68 to i64 + %70 = mul i64 %69, 4 + %71 = getelementptr i8, ptr addrspace(6) %7, i64 %70 + %72 = load <2 x float>, ptr addrspace(6) %71, align 8 + %73 = extractelement <2 x float> %72, i32 0 + %74 = extractelement <2 x float> %72, i32 1 + %75 = bitcast float %10 to i32 + %76 = bitcast float %11 to i32 + %77 = bitcast float %17 to i32 + %78 = bitcast float %18 to i32 + %79 = bitcast float %24 to i32 + %80 = bitcast float %25 to i32 + %81 = bitcast float %31 to i32 + %82 = bitcast float %32 to i32 + %83 = bitcast float %38 to i32 + %84 = bitcast float %39 to i32 + %85 = bitcast float %45 to i32 + %86 = bitcast float %46 to i32 + %87 = bitcast float %52 to i32 + %88 = bitcast float %53 to i32 + %89 = bitcast float %59 to i32 + %90 = bitcast float %60 to i32 + %91 = bitcast float %66 to i32 + %92 = bitcast float %67 to i32 + %93 = bitcast float %73 to i32 + %94 = bitcast float %74 to i32 + %95 = call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "", "={TPER4},={TPER5},={TPER6},={TPER7},={TPER8},={TPER9},={TPER10},={TPER11},={TPER12},={TPER13},={TPER14},={TPER15},={TPER16},={TPER17},={TPER18},={TPER19},={TPER20},={TPER21},={TPER22},={TPER23},0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19"(i32 %75, i32 %76, i32 %77, i32 %78, i32 %79, i32 %80, i32 %81, i32 %82, i32 %83, i32 %84, i32 %85, i32 %86, i32 %87, i32 %88, i32 %89, i32 %90, i32 %91, i32 %92, i32 %93, i32 %94) + ret void +} + +; Function Attrs: noinline +define linkonce_odr simt_entry void @rmsnorm_d5120_kernel_simt_1(i64 %0, ptr addrspace(6) %1, i64 %2, float %3, i64 %4) #3 !annotation !8 !annotation !9 { + %6 = call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "", "={TPER4},={TPER5},={TPER6},={TPER7},={TPER8},={TPER9},={TPER10},={TPER11},={TPER12},={TPER13},={TPER14},={TPER15},={TPER16},={TPER17},={TPER18},={TPER19},={TPER20},={TPER21},={TPER22},={TPER23}"() + %7 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 0 + %8 = bitcast i32 %7 to float + %9 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 1 + %10 = bitcast i32 %9 to float + %11 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 2 + %12 = bitcast i32 %11 to float + %13 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 3 + %14 = bitcast i32 %13 to float + %15 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 4 + %16 = bitcast i32 %15 to float + %17 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 5 + %18 = bitcast i32 %17 to float + %19 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 6 + %20 = bitcast i32 %19 to float + %21 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 7 + %22 = bitcast i32 %21 to float + %23 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 8 + %24 = bitcast i32 %23 to float + %25 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 9 + %26 = bitcast i32 %25 to float + %27 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 10 + %28 = bitcast i32 %27 to float + %29 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 11 + %30 = bitcast i32 %29 to float + %31 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 12 + %32 = bitcast i32 %31 to float + %33 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 13 + %34 = bitcast i32 %33 to float + %35 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 14 + %36 = bitcast i32 %35 to float + %37 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 15 + %38 = bitcast i32 %37 to float + %39 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 16 + %40 = bitcast i32 %39 to float + %41 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 17 + %42 = bitcast i32 %41 to float + %43 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 18 + %44 = bitcast i32 %43 to float + %45 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 19 + %46 = bitcast i32 %45 to float + %47 = alloca float, i32 32, align 4 + %48 = alloca float, align 4 + %49 = call i32 @llvm.hivm.get.TID.X() + %50 = mul i32 %49, 2 + %51 = sext i32 %50 to i64 + %52 = add i64 %0, %51 + %53 = add i64 %52, 5120 + %54 = mul i64 %53, 4 + %55 = ptrtoint ptr addrspace(6) %1 to i64 + %56 = inttoptr i64 %55 to ptr addrspace(6) + %57 = getelementptr i8, ptr addrspace(6) %56, i64 %54 + %58 = load <2 x float>, ptr addrspace(6) %57, align 8 + store <2 x float> %58, ptr %47, align 8 + %59 = add i64 %0, 512 + %60 = add i64 %59, %51 + %61 = add i64 %60, 5120 + %62 = mul i64 %61, 4 + %63 = getelementptr i8, ptr addrspace(6) %56, i64 %62 + %64 = load <2 x float>, ptr addrspace(6) %63, align 8 + %65 = getelementptr i8, ptr %47, i32 8 + store <2 x float> %64, ptr %65, align 8 + %66 = add i64 %0, 1024 + %67 = add i64 %66, %51 + %68 = add i64 %67, 5120 + %69 = mul i64 %68, 4 + %70 = getelementptr i8, ptr addrspace(6) %56, i64 %69 + %71 = load <2 x float>, ptr addrspace(6) %70, align 8 + %72 = getelementptr i8, ptr %47, i32 16 + store <2 x float> %71, ptr %72, align 8 + %73 = add i64 %0, 1536 + %74 = add i64 %73, %51 + %75 = add i64 %74, 5120 + %76 = mul i64 %75, 4 + %77 = getelementptr i8, ptr addrspace(6) %56, i64 %76 + %78 = load <2 x float>, ptr addrspace(6) %77, align 8 + %79 = getelementptr i8, ptr %47, i32 24 + store <2 x float> %78, ptr %79, align 8 + %80 = add i64 %0, 2048 + %81 = add i64 %80, %51 + %82 = add i64 %81, 5120 + %83 = mul i64 %82, 4 + %84 = getelementptr i8, ptr addrspace(6) %56, i64 %83 + %85 = load <2 x float>, ptr addrspace(6) %84, align 8 + %86 = getelementptr i8, ptr %47, i32 32 + store <2 x float> %85, ptr %86, align 8 + %87 = add i64 %0, 2560 + %88 = add i64 %87, %51 + %89 = add i64 %88, 5120 + %90 = mul i64 %89, 4 + %91 = getelementptr i8, ptr addrspace(6) %56, i64 %90 + %92 = load <2 x float>, ptr addrspace(6) %91, align 8 + %93 = getelementptr i8, ptr %47, i32 40 + store <2 x float> %92, ptr %93, align 8 + %94 = add i64 %0, 3072 + %95 = add i64 %94, %51 + %96 = add i64 %95, 5120 + %97 = mul i64 %96, 4 + %98 = getelementptr i8, ptr addrspace(6) %56, i64 %97 + %99 = load <2 x float>, ptr addrspace(6) %98, align 8 + %100 = getelementptr i8, ptr %47, i32 48 + store <2 x float> %99, ptr %100, align 8 + %101 = add i64 %0, 3584 + %102 = add i64 %101, %51 + %103 = add i64 %102, 5120 + %104 = mul i64 %103, 4 + %105 = getelementptr i8, ptr addrspace(6) %56, i64 %104 + %106 = load <2 x float>, ptr addrspace(6) %105, align 8 + %107 = getelementptr i8, ptr %47, i32 56 + store <2 x float> %106, ptr %107, align 8 + %108 = add i64 %0, 4096 + %109 = add i64 %108, %51 + %110 = add i64 %109, 5120 + %111 = mul i64 %110, 4 + %112 = getelementptr i8, ptr addrspace(6) %56, i64 %111 + %113 = load <2 x float>, ptr addrspace(6) %112, align 8 + %114 = getelementptr i8, ptr %47, i32 64 + store <2 x float> %113, ptr %114, align 8 + %115 = add i64 %0, 4608 + %116 = add i64 %115, %51 + %117 = add i64 %116, 5120 + %118 = mul i64 %117, 4 + %119 = getelementptr i8, ptr addrspace(6) %56, i64 %118 + %120 = load <2 x float>, ptr addrspace(6) %119, align 8 + %121 = getelementptr i8, ptr %47, i32 72 + store <2 x float> %120, ptr %121, align 8 + %122 = add i64 %2, %51 + %123 = add i64 %122, 5120 + %124 = mul i64 %123, 4 + %125 = getelementptr i8, ptr addrspace(6) %56, i64 %124 + %126 = load <2 x float>, ptr addrspace(6) %125, align 8 + %127 = getelementptr i8, ptr %47, i32 80 + store <2 x float> %126, ptr %127, align 8 + %128 = add i64 %0, 5632 + %129 = add i64 %128, %51 + %130 = add i64 %129, 5120 + %131 = mul i64 %130, 4 + %132 = getelementptr i8, ptr addrspace(6) %56, i64 %131 + %133 = load <2 x float>, ptr addrspace(6) %132, align 8 + %134 = getelementptr i8, ptr %47, i32 88 + store <2 x float> %133, ptr %134, align 8 + %135 = add i64 %0, 6144 + %136 = add i64 %135, %51 + %137 = add i64 %136, 5120 + %138 = mul i64 %137, 4 + %139 = getelementptr i8, ptr addrspace(6) %56, i64 %138 + %140 = load <2 x float>, ptr addrspace(6) %139, align 8 + %141 = getelementptr i8, ptr %47, i32 96 + store <2 x float> %140, ptr %141, align 8 + %142 = add i64 %0, 6656 + %143 = add i64 %142, %51 + %144 = add i64 %143, 5120 + %145 = mul i64 %144, 4 + %146 = getelementptr i8, ptr addrspace(6) %56, i64 %145 + %147 = load <2 x float>, ptr addrspace(6) %146, align 8 + %148 = getelementptr i8, ptr %47, i32 104 + store <2 x float> %147, ptr %148, align 8 + %149 = add i64 %0, 7168 + %150 = add i64 %149, %51 + %151 = add i64 %150, 5120 + %152 = mul i64 %151, 4 + %153 = getelementptr i8, ptr addrspace(6) %56, i64 %152 + %154 = load <2 x float>, ptr addrspace(6) %153, align 8 + %155 = getelementptr i8, ptr %47, i32 112 + store <2 x float> %154, ptr %155, align 8 + %156 = add i64 %0, 7680 + %157 = add i64 %156, %51 + %158 = add i64 %157, 5120 + %159 = mul i64 %158, 4 + %160 = getelementptr i8, ptr addrspace(6) %56, i64 %159 + %161 = load <2 x float>, ptr addrspace(6) %160, align 8 + %162 = getelementptr i8, ptr %47, i32 120 + store <2 x float> %161, ptr %162, align 8 + store float 0.000000e+00, ptr %48, align 4 + %163 = load float, ptr %48, align 4 + %164 = load float, ptr %47, align 4 + %165 = fmul float %164, %164 + %166 = fadd float %163, %165 + store float %166, ptr %48, align 4 + %167 = load float, ptr %48, align 4 + %168 = getelementptr i8, ptr %47, i32 4 + %169 = load float, ptr %168, align 4 + %170 = fmul float %169, %169 + %171 = fadd float %167, %170 + store float %171, ptr %48, align 4 + %172 = load float, ptr %48, align 4 + %173 = load float, ptr %65, align 4 + %174 = fmul float %173, %173 + %175 = fadd float %172, %174 + store float %175, ptr %48, align 4 + %176 = load float, ptr %48, align 4 + %177 = getelementptr i8, ptr %47, i32 12 + %178 = load float, ptr %177, align 4 + %179 = fmul float %178, %178 + %180 = fadd float %176, %179 + store float %180, ptr %48, align 4 + %181 = load float, ptr %48, align 4 + %182 = load float, ptr %72, align 4 + %183 = fmul float %182, %182 + %184 = fadd float %181, %183 + store float %184, ptr %48, align 4 + %185 = load float, ptr %48, align 4 + %186 = getelementptr i8, ptr %47, i32 20 + %187 = load float, ptr %186, align 4 + %188 = fmul float %187, %187 + %189 = fadd float %185, %188 + store float %189, ptr %48, align 4 + %190 = load float, ptr %48, align 4 + %191 = load float, ptr %79, align 4 + %192 = fmul float %191, %191 + %193 = fadd float %190, %192 + store float %193, ptr %48, align 4 + %194 = load float, ptr %48, align 4 + %195 = getelementptr i8, ptr %47, i32 28 + %196 = load float, ptr %195, align 4 + %197 = fmul float %196, %196 + %198 = fadd float %194, %197 + store float %198, ptr %48, align 4 + %199 = load float, ptr %48, align 4 + %200 = load float, ptr %86, align 4 + %201 = fmul float %200, %200 + %202 = fadd float %199, %201 + store float %202, ptr %48, align 4 + %203 = load float, ptr %48, align 4 + %204 = getelementptr i8, ptr %47, i32 36 + %205 = load float, ptr %204, align 4 + %206 = fmul float %205, %205 + %207 = fadd float %203, %206 + store float %207, ptr %48, align 4 + %208 = load float, ptr %48, align 4 + %209 = load float, ptr %93, align 4 + %210 = fmul float %209, %209 + %211 = fadd float %208, %210 + store float %211, ptr %48, align 4 + %212 = load float, ptr %48, align 4 + %213 = getelementptr i8, ptr %47, i32 44 + %214 = load float, ptr %213, align 4 + %215 = fmul float %214, %214 + %216 = fadd float %212, %215 + store float %216, ptr %48, align 4 + %217 = load float, ptr %48, align 4 + %218 = load float, ptr %100, align 4 + %219 = fmul float %218, %218 + %220 = fadd float %217, %219 + store float %220, ptr %48, align 4 + %221 = load float, ptr %48, align 4 + %222 = getelementptr i8, ptr %47, i32 52 + %223 = load float, ptr %222, align 4 + %224 = fmul float %223, %223 + %225 = fadd float %221, %224 + store float %225, ptr %48, align 4 + %226 = load float, ptr %48, align 4 + %227 = load float, ptr %107, align 4 + %228 = fmul float %227, %227 + %229 = fadd float %226, %228 + store float %229, ptr %48, align 4 + %230 = load float, ptr %48, align 4 + %231 = getelementptr i8, ptr %47, i32 60 + %232 = load float, ptr %231, align 4 + %233 = fmul float %232, %232 + %234 = fadd float %230, %233 + store float %234, ptr %48, align 4 + %235 = load float, ptr %48, align 4 + %236 = load float, ptr %114, align 4 + %237 = fmul float %236, %236 + %238 = fadd float %235, %237 + store float %238, ptr %48, align 4 + %239 = load float, ptr %48, align 4 + %240 = getelementptr i8, ptr %47, i32 68 + %241 = load float, ptr %240, align 4 + %242 = fmul float %241, %241 + %243 = fadd float %239, %242 + store float %243, ptr %48, align 4 + %244 = load float, ptr %48, align 4 + %245 = load float, ptr %121, align 4 + %246 = fmul float %245, %245 + %247 = fadd float %244, %246 + store float %247, ptr %48, align 4 + %248 = load float, ptr %48, align 4 + %249 = getelementptr i8, ptr %47, i32 76 + %250 = load float, ptr %249, align 4 + %251 = fmul float %250, %250 + %252 = fadd float %248, %251 + store float %252, ptr %48, align 4 + %253 = load float, ptr %48, align 4 + %254 = getelementptr float, ptr addrspace(6) %1, i64 37888 + %255 = sdiv i32 %49, 32 + %256 = mul i32 %255, 32 + %257 = icmp ne i32 %49, %256 + %258 = icmp slt i32 %49, 0 + %259 = icmp ne i1 %258, false + %260 = and i1 %257, %259 + %261 = add i32 %255, -1 + %262 = select i1 %260, i32 %261, i32 %255 + %263 = call i32 @llvm.hivm.get.laneID() + %264 = call float @llvm.hivm.redux.add.f32(float %253) + %265 = icmp slt i32 %263, 1 + br i1 %265, label %266, label %270 + +266: ; preds = %5 + %267 = add i32 %262, %263 + %268 = sext i32 %267 to i64 + %269 = getelementptr float, ptr addrspace(6) %254, i64 %268 + store float %264, ptr addrspace(6) %269, align 4 + br label %270 + +270: ; preds = %266, %5 + call void @llvm.hivm.sync.workitems() + %271 = icmp slt i32 %49, 32 + br i1 %271, label %272, label %279 + +272: ; preds = %270 + %273 = icmp slt i32 %263, 8 + %274 = sext i32 %263 to i64 + %275 = getelementptr float, ptr addrspace(6) %254, i64 %274 + %276 = load float, ptr addrspace(6) %275, align 4 + %277 = select i1 %273, float %276, float 0.000000e+00 + %278 = call float @llvm.hivm.redux.add.f32(float %277) + br label %280 + +279: ; preds = %270 + br label %280 + +280: ; preds = %272, %279 + %281 = phi float [ 0.000000e+00, %279 ], [ %278, %272 ] + br label %282 + +282: ; preds = %280 + %283 = icmp slt i32 %49, 1 + br i1 %283, label %284, label %287 + +284: ; preds = %282 + %285 = sext i32 %49 to i64 + %286 = getelementptr float, ptr addrspace(6) %254, i64 %285 + store float %281, ptr addrspace(6) %286, align 4 + br label %287 + +287: ; preds = %284, %282 + call void @llvm.hivm.sync.workitems() + %288 = getelementptr float, ptr addrspace(6) %254, i64 0 + %289 = load float, ptr addrspace(6) %288, align 4 + call void @llvm.hivm.sync.workitems() + store float %289, ptr %48, align 4 + %290 = load float, ptr %48, align 4 + %291 = fdiv float %290, 5.120000e+03 + %292 = fadd float %291, %3 + %293 = call float @llvm.sqrt.f32(float %292) + %294 = fdiv float 1.000000e+00, %293 + %295 = mul i64 %4, 8 + %296 = getelementptr float, ptr addrspace(6) %1, i64 %295 + store float %294, ptr addrspace(6) %296, align 4 + %297 = load <2 x float>, ptr %47, align 8 + %298 = insertelement <2 x float> undef, float %294, i32 0 + %299 = insertelement <2 x float> %298, float %294, i32 1 + %300 = fmul <2 x float> %297, %299 + %301 = insertelement <2 x float> poison, float %8, i32 0 + %302 = insertelement <2 x float> %301, float %10, i32 1 + %303 = fmul <2 x float> %300, %302 + %304 = add i64 %52, 21504 + %305 = mul i64 %304, 4 + %306 = getelementptr i8, ptr addrspace(6) %56, i64 %305 + store <2 x float> %303, ptr addrspace(6) %306, align 8 + %307 = load <2 x float>, ptr %65, align 8 + %308 = fmul <2 x float> %307, %299 + %309 = insertelement <2 x float> poison, float %12, i32 0 + %310 = insertelement <2 x float> %309, float %14, i32 1 + %311 = fmul <2 x float> %308, %310 + %312 = add i64 %60, 21504 + %313 = mul i64 %312, 4 + %314 = getelementptr i8, ptr addrspace(6) %56, i64 %313 + store <2 x float> %311, ptr addrspace(6) %314, align 8 + %315 = load <2 x float>, ptr %72, align 8 + %316 = fmul <2 x float> %315, %299 + %317 = insertelement <2 x float> poison, float %16, i32 0 + %318 = insertelement <2 x float> %317, float %18, i32 1 + %319 = fmul <2 x float> %316, %318 + %320 = add i64 %67, 21504 + %321 = mul i64 %320, 4 + %322 = getelementptr i8, ptr addrspace(6) %56, i64 %321 + store <2 x float> %319, ptr addrspace(6) %322, align 8 + %323 = load <2 x float>, ptr %79, align 8 + %324 = fmul <2 x float> %323, %299 + %325 = insertelement <2 x float> poison, float %20, i32 0 + %326 = insertelement <2 x float> %325, float %22, i32 1 + %327 = fmul <2 x float> %324, %326 + %328 = add i64 %74, 21504 + %329 = mul i64 %328, 4 + %330 = getelementptr i8, ptr addrspace(6) %56, i64 %329 + store <2 x float> %327, ptr addrspace(6) %330, align 8 + %331 = load <2 x float>, ptr %86, align 8 + %332 = fmul <2 x float> %331, %299 + %333 = insertelement <2 x float> poison, float %24, i32 0 + %334 = insertelement <2 x float> %333, float %26, i32 1 + %335 = fmul <2 x float> %332, %334 + %336 = add i64 %81, 21504 + %337 = mul i64 %336, 4 + %338 = getelementptr i8, ptr addrspace(6) %56, i64 %337 + store <2 x float> %335, ptr addrspace(6) %338, align 8 + %339 = load <2 x float>, ptr %93, align 8 + %340 = fmul <2 x float> %339, %299 + %341 = insertelement <2 x float> poison, float %28, i32 0 + %342 = insertelement <2 x float> %341, float %30, i32 1 + %343 = fmul <2 x float> %340, %342 + %344 = add i64 %88, 21504 + %345 = mul i64 %344, 4 + %346 = getelementptr i8, ptr addrspace(6) %56, i64 %345 + store <2 x float> %343, ptr addrspace(6) %346, align 8 + %347 = load <2 x float>, ptr %100, align 8 + %348 = fmul <2 x float> %347, %299 + %349 = insertelement <2 x float> poison, float %32, i32 0 + %350 = insertelement <2 x float> %349, float %34, i32 1 + %351 = fmul <2 x float> %348, %350 + %352 = add i64 %95, 21504 + %353 = mul i64 %352, 4 + %354 = getelementptr i8, ptr addrspace(6) %56, i64 %353 + store <2 x float> %351, ptr addrspace(6) %354, align 8 + %355 = load <2 x float>, ptr %107, align 8 + %356 = fmul <2 x float> %355, %299 + %357 = insertelement <2 x float> poison, float %36, i32 0 + %358 = insertelement <2 x float> %357, float %38, i32 1 + %359 = fmul <2 x float> %356, %358 + %360 = add i64 %102, 21504 + %361 = mul i64 %360, 4 + %362 = getelementptr i8, ptr addrspace(6) %56, i64 %361 + store <2 x float> %359, ptr addrspace(6) %362, align 8 + %363 = load <2 x float>, ptr %114, align 8 + %364 = fmul <2 x float> %363, %299 + %365 = insertelement <2 x float> poison, float %40, i32 0 + %366 = insertelement <2 x float> %365, float %42, i32 1 + %367 = fmul <2 x float> %364, %366 + %368 = add i64 %109, 21504 + %369 = mul i64 %368, 4 + %370 = getelementptr i8, ptr addrspace(6) %56, i64 %369 + store <2 x float> %367, ptr addrspace(6) %370, align 8 + %371 = load <2 x float>, ptr %121, align 8 + %372 = fmul <2 x float> %371, %299 + %373 = insertelement <2 x float> poison, float %44, i32 0 + %374 = insertelement <2 x float> %373, float %46, i32 1 + %375 = fmul <2 x float> %372, %374 + %376 = add i64 %116, 21504 + %377 = mul i64 %376, 4 + %378 = getelementptr i8, ptr addrspace(6) %56, i64 %377 + store <2 x float> %375, ptr addrspace(6) %378, align 8 + %379 = call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "", "={TPER4},={TPER5},={TPER6},={TPER7},={TPER8},={TPER9},={TPER10},={TPER11},={TPER12},={TPER13},={TPER14},={TPER15},={TPER16},={TPER17},={TPER18},={TPER19},={TPER20},={TPER21},={TPER22},={TPER23},0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19"(i32 %7, i32 %9, i32 %11, i32 %13, i32 %15, i32 %17, i32 %19, i32 %21, i32 %23, i32 %25, i32 %27, i32 %29, i32 %31, i32 %33, i32 %35, i32 %37, i32 %39, i32 %41, i32 %43, i32 %45) + ret void +} + +attributes #0 = { "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } +attributes #1 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } +attributes #2 = { "target-cpu"="dav-c310-vec" "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } +attributes #3 = { noinline "target-cpu"="dav-c310-vec" "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } + +!llvm.module.flags = !{!0} +!hivm.annotations = !{!1, !2, !3, !4, !5, !6, !7} + +!0 = !{i32 2, !"Debug Info Version", i32 3} +!1 = !{ptr @rmsnorm_d5120_kernel_mix_aiv, !"kernel", i32 1} +!2 = !{ptr @rmsnorm_d5120_kernel_mix_aiv, !"kernel_with_simd", i32 1} +!3 = !{ptr @rmsnorm_d5120_kernel_mix_aiv, !"kernel_with_simt", i32 1} +!4 = distinct !{null, !"simt-max-threads", i32 256} +!5 = distinct !{null, !"simt-max-registers", i32 128} +!6 = distinct !{null, !"simt-max-threads", i32 256} +!7 = distinct !{null, !"simt-max-registers", i32 128} +!8 = !{!"simt-max-threads", i32 256} +!9 = !{!"simt-max-registers", i32 128} diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/launch.cpp b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/launch.cpp new file mode 100644 index 0000000000..233cdc1237 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d5120/launch.cpp @@ -0,0 +1,12 @@ +#ifndef AICORE +#define AICORE [aicore] +#endif +extern "C" __global__ AICORE void rmsnorm_d5120_kernel( + __gm__ void*, __gm__ void*, __gm__ void*, __gm__ void*, float); +extern "C" int call( + float* X, float* Y, float* W, float* RSTD, float eps, void* stream) { + rmsnorm_d5120_kernel<<<64, 152576, stream>>>( + (__gm__ float*)RSTD, (__gm__ float*)W, (__gm__ float*)X, + (__gm__ float*)Y, eps); + return 0; +} diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/kernel.pto b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/kernel.pto new file mode 100644 index 0000000000..ac0612c4c4 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/kernel.pto @@ -0,0 +1,2215 @@ +module attributes {pto.target_arch = "a5"} { + module attributes {pto.backend = "vpto", pto.kernel_kind = #pto.kernel_kind, pto.target_arch = "a5"} { + func.func @rmsnorm_d7168_kernel(%arg0: !pto.ptr, %arg1: !pto.ptr, %arg2: !pto.ptr, %arg3: !pto.ptr, %arg4: f32) attributes {pto.entry, pto.kernel_kind = #pto.kernel_kind} { + %c0_i64 = arith.constant 0 : i64 + %0 = pto.castptr %c0_i64 : i64 -> !pto.ptr + %c32_i32 = arith.constant 32 : i32 + %1 = llvm.alloca %c32_i32 x f32 {pto.persistent} : (i32) -> !llvm.ptr + pto.set_flag[, , ] + pto.set_flag[, , ] + pto.set_flag[, , ] + pto.set_flag[, , ] + %2 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_i64 = arith.constant 1 : i64 + %c28672_i64 = arith.constant 28672 : i64 + %c28672_i64_0 = arith.constant 28672 : i64 + %c0_i64_1 = arith.constant 0 : i64 + %c28672_i64_2 = arith.constant 28672 : i64 + pto.mte_gm_ub %arg1, %2, %c0_i64_1, %c28672_i64_2 nburst(%c1_i64, %c28672_i64, %c28672_i64_0) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + pto.set_flag[, , ] + pto.wait_flag[, , ] + pto.section.simt<<<256, 1, 1>>> { + %3 = pto.get_tid_x : i32 + %4 = pto.get_tid_y : i32 + %5 = pto.get_tid_z : i32 + %6 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32 = arith.constant 2 : i32 + %7 = arith.muli %3, %c2_i32 : i32 + %c0_i32 = arith.constant 0 : i32 + %8 = arith.addi %c0_i32, %7 : i32 + %9 = arith.index_cast %8 : i32 to index + %c4 = arith.constant 4 : index + %10 = arith.muli %9, %c4 : index + %11 = arith.index_cast %10 : index to i64 + %12 = pto.castptr %6 : !pto.ptr -> i64 + %13 = llvm.inttoptr %12 : i64 to !llvm.ptr<6> + %14 = llvm.getelementptr %13[%11] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %15 = llvm.load %14 : !llvm.ptr<6> -> vector<2xf32> + %c0_3 = arith.constant 0 : index + %c4_4 = arith.constant 4 : index + %16 = arith.muli %c0_3, %c4_4 : index + %17 = arith.index_cast %16 : index to i64 + %18 = llvm.getelementptr %1[%17] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %15, %18 : vector<2xf32>, !llvm.ptr + %19 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_5 = arith.constant 2 : i32 + %20 = arith.muli %3, %c2_i32_5 : i32 + %c512_i32 = arith.constant 512 : i32 + %21 = arith.addi %c512_i32, %20 : i32 + %22 = arith.index_cast %21 : i32 to index + %c4_6 = arith.constant 4 : index + %23 = arith.muli %22, %c4_6 : index + %24 = arith.index_cast %23 : index to i64 + %25 = pto.castptr %19 : !pto.ptr -> i64 + %26 = llvm.inttoptr %25 : i64 to !llvm.ptr<6> + %27 = llvm.getelementptr %26[%24] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %28 = llvm.load %27 : !llvm.ptr<6> -> vector<2xf32> + %c2 = arith.constant 2 : index + %c4_7 = arith.constant 4 : index + %29 = arith.muli %c2, %c4_7 : index + %30 = arith.index_cast %29 : index to i64 + %31 = llvm.getelementptr %1[%30] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %28, %31 : vector<2xf32>, !llvm.ptr + %32 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_8 = arith.constant 2 : i32 + %33 = arith.muli %3, %c2_i32_8 : i32 + %c1024_i32 = arith.constant 1024 : i32 + %34 = arith.addi %c1024_i32, %33 : i32 + %35 = arith.index_cast %34 : i32 to index + %c4_9 = arith.constant 4 : index + %36 = arith.muli %35, %c4_9 : index + %37 = arith.index_cast %36 : index to i64 + %38 = pto.castptr %32 : !pto.ptr -> i64 + %39 = llvm.inttoptr %38 : i64 to !llvm.ptr<6> + %40 = llvm.getelementptr %39[%37] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %41 = llvm.load %40 : !llvm.ptr<6> -> vector<2xf32> + %c4_10 = arith.constant 4 : index + %c4_11 = arith.constant 4 : index + %42 = arith.muli %c4_10, %c4_11 : index + %43 = arith.index_cast %42 : index to i64 + %44 = llvm.getelementptr %1[%43] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %41, %44 : vector<2xf32>, !llvm.ptr + %45 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_12 = arith.constant 2 : i32 + %46 = arith.muli %3, %c2_i32_12 : i32 + %c1536_i32 = arith.constant 1536 : i32 + %47 = arith.addi %c1536_i32, %46 : i32 + %48 = arith.index_cast %47 : i32 to index + %c4_13 = arith.constant 4 : index + %49 = arith.muli %48, %c4_13 : index + %50 = arith.index_cast %49 : index to i64 + %51 = pto.castptr %45 : !pto.ptr -> i64 + %52 = llvm.inttoptr %51 : i64 to !llvm.ptr<6> + %53 = llvm.getelementptr %52[%50] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %54 = llvm.load %53 : !llvm.ptr<6> -> vector<2xf32> + %c6 = arith.constant 6 : index + %c4_14 = arith.constant 4 : index + %55 = arith.muli %c6, %c4_14 : index + %56 = arith.index_cast %55 : index to i64 + %57 = llvm.getelementptr %1[%56] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %54, %57 : vector<2xf32>, !llvm.ptr + %58 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_15 = arith.constant 2 : i32 + %59 = arith.muli %3, %c2_i32_15 : i32 + %c2048_i32 = arith.constant 2048 : i32 + %60 = arith.addi %c2048_i32, %59 : i32 + %61 = arith.index_cast %60 : i32 to index + %c4_16 = arith.constant 4 : index + %62 = arith.muli %61, %c4_16 : index + %63 = arith.index_cast %62 : index to i64 + %64 = pto.castptr %58 : !pto.ptr -> i64 + %65 = llvm.inttoptr %64 : i64 to !llvm.ptr<6> + %66 = llvm.getelementptr %65[%63] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %67 = llvm.load %66 : !llvm.ptr<6> -> vector<2xf32> + %c8 = arith.constant 8 : index + %c4_17 = arith.constant 4 : index + %68 = arith.muli %c8, %c4_17 : index + %69 = arith.index_cast %68 : index to i64 + %70 = llvm.getelementptr %1[%69] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %67, %70 : vector<2xf32>, !llvm.ptr + %71 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_18 = arith.constant 2 : i32 + %72 = arith.muli %3, %c2_i32_18 : i32 + %c2560_i32 = arith.constant 2560 : i32 + %73 = arith.addi %c2560_i32, %72 : i32 + %74 = arith.index_cast %73 : i32 to index + %c4_19 = arith.constant 4 : index + %75 = arith.muli %74, %c4_19 : index + %76 = arith.index_cast %75 : index to i64 + %77 = pto.castptr %71 : !pto.ptr -> i64 + %78 = llvm.inttoptr %77 : i64 to !llvm.ptr<6> + %79 = llvm.getelementptr %78[%76] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %80 = llvm.load %79 : !llvm.ptr<6> -> vector<2xf32> + %c10 = arith.constant 10 : index + %c4_20 = arith.constant 4 : index + %81 = arith.muli %c10, %c4_20 : index + %82 = arith.index_cast %81 : index to i64 + %83 = llvm.getelementptr %1[%82] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %80, %83 : vector<2xf32>, !llvm.ptr + %84 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_21 = arith.constant 2 : i32 + %85 = arith.muli %3, %c2_i32_21 : i32 + %c3072_i32 = arith.constant 3072 : i32 + %86 = arith.addi %c3072_i32, %85 : i32 + %87 = arith.index_cast %86 : i32 to index + %c4_22 = arith.constant 4 : index + %88 = arith.muli %87, %c4_22 : index + %89 = arith.index_cast %88 : index to i64 + %90 = pto.castptr %84 : !pto.ptr -> i64 + %91 = llvm.inttoptr %90 : i64 to !llvm.ptr<6> + %92 = llvm.getelementptr %91[%89] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %93 = llvm.load %92 : !llvm.ptr<6> -> vector<2xf32> + %c12 = arith.constant 12 : index + %c4_23 = arith.constant 4 : index + %94 = arith.muli %c12, %c4_23 : index + %95 = arith.index_cast %94 : index to i64 + %96 = llvm.getelementptr %1[%95] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %93, %96 : vector<2xf32>, !llvm.ptr + %97 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_24 = arith.constant 2 : i32 + %98 = arith.muli %3, %c2_i32_24 : i32 + %c3584_i32 = arith.constant 3584 : i32 + %99 = arith.addi %c3584_i32, %98 : i32 + %100 = arith.index_cast %99 : i32 to index + %c4_25 = arith.constant 4 : index + %101 = arith.muli %100, %c4_25 : index + %102 = arith.index_cast %101 : index to i64 + %103 = pto.castptr %97 : !pto.ptr -> i64 + %104 = llvm.inttoptr %103 : i64 to !llvm.ptr<6> + %105 = llvm.getelementptr %104[%102] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %106 = llvm.load %105 : !llvm.ptr<6> -> vector<2xf32> + %c14 = arith.constant 14 : index + %c4_26 = arith.constant 4 : index + %107 = arith.muli %c14, %c4_26 : index + %108 = arith.index_cast %107 : index to i64 + %109 = llvm.getelementptr %1[%108] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %106, %109 : vector<2xf32>, !llvm.ptr + %110 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_27 = arith.constant 2 : i32 + %111 = arith.muli %3, %c2_i32_27 : i32 + %c4096_i32 = arith.constant 4096 : i32 + %112 = arith.addi %c4096_i32, %111 : i32 + %113 = arith.index_cast %112 : i32 to index + %c4_28 = arith.constant 4 : index + %114 = arith.muli %113, %c4_28 : index + %115 = arith.index_cast %114 : index to i64 + %116 = pto.castptr %110 : !pto.ptr -> i64 + %117 = llvm.inttoptr %116 : i64 to !llvm.ptr<6> + %118 = llvm.getelementptr %117[%115] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %119 = llvm.load %118 : !llvm.ptr<6> -> vector<2xf32> + %c16 = arith.constant 16 : index + %c4_29 = arith.constant 4 : index + %120 = arith.muli %c16, %c4_29 : index + %121 = arith.index_cast %120 : index to i64 + %122 = llvm.getelementptr %1[%121] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %119, %122 : vector<2xf32>, !llvm.ptr + %123 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_30 = arith.constant 2 : i32 + %124 = arith.muli %3, %c2_i32_30 : i32 + %c4608_i32 = arith.constant 4608 : i32 + %125 = arith.addi %c4608_i32, %124 : i32 + %126 = arith.index_cast %125 : i32 to index + %c4_31 = arith.constant 4 : index + %127 = arith.muli %126, %c4_31 : index + %128 = arith.index_cast %127 : index to i64 + %129 = pto.castptr %123 : !pto.ptr -> i64 + %130 = llvm.inttoptr %129 : i64 to !llvm.ptr<6> + %131 = llvm.getelementptr %130[%128] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %132 = llvm.load %131 : !llvm.ptr<6> -> vector<2xf32> + %c18 = arith.constant 18 : index + %c4_32 = arith.constant 4 : index + %133 = arith.muli %c18, %c4_32 : index + %134 = arith.index_cast %133 : index to i64 + %135 = llvm.getelementptr %1[%134] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %132, %135 : vector<2xf32>, !llvm.ptr + %136 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_33 = arith.constant 2 : i32 + %137 = arith.muli %3, %c2_i32_33 : i32 + %c5120_i32 = arith.constant 5120 : i32 + %138 = arith.addi %c5120_i32, %137 : i32 + %139 = arith.index_cast %138 : i32 to index + %c4_34 = arith.constant 4 : index + %140 = arith.muli %139, %c4_34 : index + %141 = arith.index_cast %140 : index to i64 + %142 = pto.castptr %136 : !pto.ptr -> i64 + %143 = llvm.inttoptr %142 : i64 to !llvm.ptr<6> + %144 = llvm.getelementptr %143[%141] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %145 = llvm.load %144 : !llvm.ptr<6> -> vector<2xf32> + %c20 = arith.constant 20 : index + %c4_35 = arith.constant 4 : index + %146 = arith.muli %c20, %c4_35 : index + %147 = arith.index_cast %146 : index to i64 + %148 = llvm.getelementptr %1[%147] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %145, %148 : vector<2xf32>, !llvm.ptr + %149 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_36 = arith.constant 2 : i32 + %150 = arith.muli %3, %c2_i32_36 : i32 + %c5632_i32 = arith.constant 5632 : i32 + %151 = arith.addi %c5632_i32, %150 : i32 + %152 = arith.index_cast %151 : i32 to index + %c4_37 = arith.constant 4 : index + %153 = arith.muli %152, %c4_37 : index + %154 = arith.index_cast %153 : index to i64 + %155 = pto.castptr %149 : !pto.ptr -> i64 + %156 = llvm.inttoptr %155 : i64 to !llvm.ptr<6> + %157 = llvm.getelementptr %156[%154] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %158 = llvm.load %157 : !llvm.ptr<6> -> vector<2xf32> + %c22 = arith.constant 22 : index + %c4_38 = arith.constant 4 : index + %159 = arith.muli %c22, %c4_38 : index + %160 = arith.index_cast %159 : index to i64 + %161 = llvm.getelementptr %1[%160] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %158, %161 : vector<2xf32>, !llvm.ptr + %162 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_39 = arith.constant 2 : i32 + %163 = arith.muli %3, %c2_i32_39 : i32 + %c6144_i32 = arith.constant 6144 : i32 + %164 = arith.addi %c6144_i32, %163 : i32 + %165 = arith.index_cast %164 : i32 to index + %c4_40 = arith.constant 4 : index + %166 = arith.muli %165, %c4_40 : index + %167 = arith.index_cast %166 : index to i64 + %168 = pto.castptr %162 : !pto.ptr -> i64 + %169 = llvm.inttoptr %168 : i64 to !llvm.ptr<6> + %170 = llvm.getelementptr %169[%167] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %171 = llvm.load %170 : !llvm.ptr<6> -> vector<2xf32> + %c24 = arith.constant 24 : index + %c4_41 = arith.constant 4 : index + %172 = arith.muli %c24, %c4_41 : index + %173 = arith.index_cast %172 : index to i64 + %174 = llvm.getelementptr %1[%173] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %171, %174 : vector<2xf32>, !llvm.ptr + %175 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c2_i32_42 = arith.constant 2 : i32 + %176 = arith.muli %3, %c2_i32_42 : i32 + %c6656_i32 = arith.constant 6656 : i32 + %177 = arith.addi %c6656_i32, %176 : i32 + %178 = arith.index_cast %177 : i32 to index + %c4_43 = arith.constant 4 : index + %179 = arith.muli %178, %c4_43 : index + %180 = arith.index_cast %179 : index to i64 + %181 = pto.castptr %175 : !pto.ptr -> i64 + %182 = llvm.inttoptr %181 : i64 to !llvm.ptr<6> + %183 = llvm.getelementptr %182[%180] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %184 = llvm.load %183 : !llvm.ptr<6> -> vector<2xf32> + %c26 = arith.constant 26 : index + %c4_44 = arith.constant 4 : index + %185 = arith.muli %c26, %c4_44 : index + %186 = arith.index_cast %185 : index to i64 + %187 = llvm.getelementptr %1[%186] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %184, %187 : vector<2xf32>, !llvm.ptr + } + %c0 = arith.constant 0 : index + %c64 = arith.constant 64 : index + %c1 = arith.constant 1 : index + scf.for %arg5 = %c0 to %c64 step %c1 { + %c1_3 = arith.constant 1 : index + %3 = arith.andi %arg5, %c1_3 : index + pto.wait_flag_dyn[, , %3] + %c458752 = arith.constant 458752 : index + %4 = arith.muli %arg5, %c458752 : index + %5 = pto.get_block_idx + %c7168_i64 = arith.constant 7168 : i64 + %6 = arith.muli %5, %c7168_i64 : i64 + %7 = arith.index_cast %6 : i64 to index + %8 = arith.addi %4, %7 : index + %9 = pto.addptr %arg2, %8 : -> + %10 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_4 = arith.constant 1 : index + %11 = arith.andi %arg5, %c1_4 : index + %c8192 = arith.constant 8192 : index + %12 = arith.muli %11, %c8192 : index + %c7168 = arith.constant 7168 : index + %13 = arith.addi %12, %c7168 : index + %14 = pto.addptr %10, %13 : -> + %c1_i64_5 = arith.constant 1 : i64 + %c28672_i64_6 = arith.constant 28672 : i64 + %c28672_i64_7 = arith.constant 28672 : i64 + %c0_i64_8 = arith.constant 0 : i64 + %c28672_i64_9 = arith.constant 28672 : i64 + pto.mte_gm_ub %9, %14, %c0_i64_8, %c28672_i64_9 nburst(%c1_i64_5, %c28672_i64_6, %c28672_i64_7) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + %c1_10 = arith.constant 1 : index + %15 = arith.andi %arg5, %c1_10 : index + pto.set_flag_dyn[, , %15] + %c1_11 = arith.constant 1 : index + %16 = arith.andi %arg5, %c1_11 : index + pto.wait_flag_dyn[, , %16] + %c1_12 = arith.constant 1 : index + %17 = arith.andi %arg5, %c1_12 : index + pto.wait_flag_dyn[, , %17] + pto.section.simt<<<256, 1, 1>>> { + %c32_i32_32 = arith.constant 32 : i32 + %42 = llvm.alloca %c32_i32_32 x f32 : (i32) -> !llvm.ptr + %c1_i32 = arith.constant 1 : i32 + %43 = llvm.alloca %c1_i32 x f32 : (i32) -> !llvm.ptr + %44 = pto.get_tid_x : i32 + %45 = pto.get_tid_y : i32 + %46 = pto.get_tid_z : i32 + %47 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_33 = arith.constant 1 : index + %48 = arith.andi %arg5, %c1_33 : index + %c8192_34 = arith.constant 8192 : index + %49 = arith.muli %48, %c8192_34 : index + %c0_35 = arith.constant 0 : index + %50 = arith.addi %49, %c0_35 : index + %c2_i32 = arith.constant 2 : i32 + %51 = arith.muli %44, %c2_i32 : i32 + %52 = arith.index_cast %51 : i32 to index + %53 = arith.addi %50, %52 : index + %c7168_36 = arith.constant 7168 : index + %54 = arith.addi %53, %c7168_36 : index + %c4 = arith.constant 4 : index + %55 = arith.muli %54, %c4 : index + %56 = arith.index_cast %55 : index to i64 + %57 = pto.castptr %47 : !pto.ptr -> i64 + %58 = llvm.inttoptr %57 : i64 to !llvm.ptr<6> + %59 = llvm.getelementptr %58[%56] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %60 = llvm.load %59 : !llvm.ptr<6> -> vector<2xf32> + %c0_37 = arith.constant 0 : index + %c4_38 = arith.constant 4 : index + %61 = arith.muli %c0_37, %c4_38 : index + %62 = arith.index_cast %61 : index to i64 + %63 = llvm.getelementptr %42[%62] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %60, %63 : vector<2xf32>, !llvm.ptr + %64 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_39 = arith.constant 1 : index + %65 = arith.andi %arg5, %c1_39 : index + %c8192_40 = arith.constant 8192 : index + %66 = arith.muli %65, %c8192_40 : index + %c512 = arith.constant 512 : index + %67 = arith.addi %66, %c512 : index + %c2_i32_41 = arith.constant 2 : i32 + %68 = arith.muli %44, %c2_i32_41 : i32 + %69 = arith.index_cast %68 : i32 to index + %70 = arith.addi %67, %69 : index + %c7168_42 = arith.constant 7168 : index + %71 = arith.addi %70, %c7168_42 : index + %c4_43 = arith.constant 4 : index + %72 = arith.muli %71, %c4_43 : index + %73 = arith.index_cast %72 : index to i64 + %74 = pto.castptr %64 : !pto.ptr -> i64 + %75 = llvm.inttoptr %74 : i64 to !llvm.ptr<6> + %76 = llvm.getelementptr %75[%73] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %77 = llvm.load %76 : !llvm.ptr<6> -> vector<2xf32> + %c2 = arith.constant 2 : index + %c4_44 = arith.constant 4 : index + %78 = arith.muli %c2, %c4_44 : index + %79 = arith.index_cast %78 : index to i64 + %80 = llvm.getelementptr %42[%79] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %77, %80 : vector<2xf32>, !llvm.ptr + %81 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_45 = arith.constant 1 : index + %82 = arith.andi %arg5, %c1_45 : index + %c8192_46 = arith.constant 8192 : index + %83 = arith.muli %82, %c8192_46 : index + %c1024 = arith.constant 1024 : index + %84 = arith.addi %83, %c1024 : index + %c2_i32_47 = arith.constant 2 : i32 + %85 = arith.muli %44, %c2_i32_47 : i32 + %86 = arith.index_cast %85 : i32 to index + %87 = arith.addi %84, %86 : index + %c7168_48 = arith.constant 7168 : index + %88 = arith.addi %87, %c7168_48 : index + %c4_49 = arith.constant 4 : index + %89 = arith.muli %88, %c4_49 : index + %90 = arith.index_cast %89 : index to i64 + %91 = pto.castptr %81 : !pto.ptr -> i64 + %92 = llvm.inttoptr %91 : i64 to !llvm.ptr<6> + %93 = llvm.getelementptr %92[%90] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %94 = llvm.load %93 : !llvm.ptr<6> -> vector<2xf32> + %c4_50 = arith.constant 4 : index + %c4_51 = arith.constant 4 : index + %95 = arith.muli %c4_50, %c4_51 : index + %96 = arith.index_cast %95 : index to i64 + %97 = llvm.getelementptr %42[%96] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %94, %97 : vector<2xf32>, !llvm.ptr + %98 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_52 = arith.constant 1 : index + %99 = arith.andi %arg5, %c1_52 : index + %c8192_53 = arith.constant 8192 : index + %100 = arith.muli %99, %c8192_53 : index + %c1536 = arith.constant 1536 : index + %101 = arith.addi %100, %c1536 : index + %c2_i32_54 = arith.constant 2 : i32 + %102 = arith.muli %44, %c2_i32_54 : i32 + %103 = arith.index_cast %102 : i32 to index + %104 = arith.addi %101, %103 : index + %c7168_55 = arith.constant 7168 : index + %105 = arith.addi %104, %c7168_55 : index + %c4_56 = arith.constant 4 : index + %106 = arith.muli %105, %c4_56 : index + %107 = arith.index_cast %106 : index to i64 + %108 = pto.castptr %98 : !pto.ptr -> i64 + %109 = llvm.inttoptr %108 : i64 to !llvm.ptr<6> + %110 = llvm.getelementptr %109[%107] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %111 = llvm.load %110 : !llvm.ptr<6> -> vector<2xf32> + %c6 = arith.constant 6 : index + %c4_57 = arith.constant 4 : index + %112 = arith.muli %c6, %c4_57 : index + %113 = arith.index_cast %112 : index to i64 + %114 = llvm.getelementptr %42[%113] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %111, %114 : vector<2xf32>, !llvm.ptr + %115 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_58 = arith.constant 1 : index + %116 = arith.andi %arg5, %c1_58 : index + %c8192_59 = arith.constant 8192 : index + %117 = arith.muli %116, %c8192_59 : index + %c2048 = arith.constant 2048 : index + %118 = arith.addi %117, %c2048 : index + %c2_i32_60 = arith.constant 2 : i32 + %119 = arith.muli %44, %c2_i32_60 : i32 + %120 = arith.index_cast %119 : i32 to index + %121 = arith.addi %118, %120 : index + %c7168_61 = arith.constant 7168 : index + %122 = arith.addi %121, %c7168_61 : index + %c4_62 = arith.constant 4 : index + %123 = arith.muli %122, %c4_62 : index + %124 = arith.index_cast %123 : index to i64 + %125 = pto.castptr %115 : !pto.ptr -> i64 + %126 = llvm.inttoptr %125 : i64 to !llvm.ptr<6> + %127 = llvm.getelementptr %126[%124] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %128 = llvm.load %127 : !llvm.ptr<6> -> vector<2xf32> + %c8_63 = arith.constant 8 : index + %c4_64 = arith.constant 4 : index + %129 = arith.muli %c8_63, %c4_64 : index + %130 = arith.index_cast %129 : index to i64 + %131 = llvm.getelementptr %42[%130] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %128, %131 : vector<2xf32>, !llvm.ptr + %132 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_65 = arith.constant 1 : index + %133 = arith.andi %arg5, %c1_65 : index + %c8192_66 = arith.constant 8192 : index + %134 = arith.muli %133, %c8192_66 : index + %c2560 = arith.constant 2560 : index + %135 = arith.addi %134, %c2560 : index + %c2_i32_67 = arith.constant 2 : i32 + %136 = arith.muli %44, %c2_i32_67 : i32 + %137 = arith.index_cast %136 : i32 to index + %138 = arith.addi %135, %137 : index + %c7168_68 = arith.constant 7168 : index + %139 = arith.addi %138, %c7168_68 : index + %c4_69 = arith.constant 4 : index + %140 = arith.muli %139, %c4_69 : index + %141 = arith.index_cast %140 : index to i64 + %142 = pto.castptr %132 : !pto.ptr -> i64 + %143 = llvm.inttoptr %142 : i64 to !llvm.ptr<6> + %144 = llvm.getelementptr %143[%141] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %145 = llvm.load %144 : !llvm.ptr<6> -> vector<2xf32> + %c10 = arith.constant 10 : index + %c4_70 = arith.constant 4 : index + %146 = arith.muli %c10, %c4_70 : index + %147 = arith.index_cast %146 : index to i64 + %148 = llvm.getelementptr %42[%147] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %145, %148 : vector<2xf32>, !llvm.ptr + %149 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_71 = arith.constant 1 : index + %150 = arith.andi %arg5, %c1_71 : index + %c8192_72 = arith.constant 8192 : index + %151 = arith.muli %150, %c8192_72 : index + %c3072 = arith.constant 3072 : index + %152 = arith.addi %151, %c3072 : index + %c2_i32_73 = arith.constant 2 : i32 + %153 = arith.muli %44, %c2_i32_73 : i32 + %154 = arith.index_cast %153 : i32 to index + %155 = arith.addi %152, %154 : index + %c7168_74 = arith.constant 7168 : index + %156 = arith.addi %155, %c7168_74 : index + %c4_75 = arith.constant 4 : index + %157 = arith.muli %156, %c4_75 : index + %158 = arith.index_cast %157 : index to i64 + %159 = pto.castptr %149 : !pto.ptr -> i64 + %160 = llvm.inttoptr %159 : i64 to !llvm.ptr<6> + %161 = llvm.getelementptr %160[%158] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %162 = llvm.load %161 : !llvm.ptr<6> -> vector<2xf32> + %c12 = arith.constant 12 : index + %c4_76 = arith.constant 4 : index + %163 = arith.muli %c12, %c4_76 : index + %164 = arith.index_cast %163 : index to i64 + %165 = llvm.getelementptr %42[%164] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %162, %165 : vector<2xf32>, !llvm.ptr + %166 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_77 = arith.constant 1 : index + %167 = arith.andi %arg5, %c1_77 : index + %c8192_78 = arith.constant 8192 : index + %168 = arith.muli %167, %c8192_78 : index + %c3584 = arith.constant 3584 : index + %169 = arith.addi %168, %c3584 : index + %c2_i32_79 = arith.constant 2 : i32 + %170 = arith.muli %44, %c2_i32_79 : i32 + %171 = arith.index_cast %170 : i32 to index + %172 = arith.addi %169, %171 : index + %c7168_80 = arith.constant 7168 : index + %173 = arith.addi %172, %c7168_80 : index + %c4_81 = arith.constant 4 : index + %174 = arith.muli %173, %c4_81 : index + %175 = arith.index_cast %174 : index to i64 + %176 = pto.castptr %166 : !pto.ptr -> i64 + %177 = llvm.inttoptr %176 : i64 to !llvm.ptr<6> + %178 = llvm.getelementptr %177[%175] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %179 = llvm.load %178 : !llvm.ptr<6> -> vector<2xf32> + %c14 = arith.constant 14 : index + %c4_82 = arith.constant 4 : index + %180 = arith.muli %c14, %c4_82 : index + %181 = arith.index_cast %180 : index to i64 + %182 = llvm.getelementptr %42[%181] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %179, %182 : vector<2xf32>, !llvm.ptr + %183 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_83 = arith.constant 1 : index + %184 = arith.andi %arg5, %c1_83 : index + %c8192_84 = arith.constant 8192 : index + %185 = arith.muli %184, %c8192_84 : index + %c4096 = arith.constant 4096 : index + %186 = arith.addi %185, %c4096 : index + %c2_i32_85 = arith.constant 2 : i32 + %187 = arith.muli %44, %c2_i32_85 : i32 + %188 = arith.index_cast %187 : i32 to index + %189 = arith.addi %186, %188 : index + %c7168_86 = arith.constant 7168 : index + %190 = arith.addi %189, %c7168_86 : index + %c4_87 = arith.constant 4 : index + %191 = arith.muli %190, %c4_87 : index + %192 = arith.index_cast %191 : index to i64 + %193 = pto.castptr %183 : !pto.ptr -> i64 + %194 = llvm.inttoptr %193 : i64 to !llvm.ptr<6> + %195 = llvm.getelementptr %194[%192] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %196 = llvm.load %195 : !llvm.ptr<6> -> vector<2xf32> + %c16 = arith.constant 16 : index + %c4_88 = arith.constant 4 : index + %197 = arith.muli %c16, %c4_88 : index + %198 = arith.index_cast %197 : index to i64 + %199 = llvm.getelementptr %42[%198] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %196, %199 : vector<2xf32>, !llvm.ptr + %200 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_89 = arith.constant 1 : index + %201 = arith.andi %arg5, %c1_89 : index + %c8192_90 = arith.constant 8192 : index + %202 = arith.muli %201, %c8192_90 : index + %c4608 = arith.constant 4608 : index + %203 = arith.addi %202, %c4608 : index + %c2_i32_91 = arith.constant 2 : i32 + %204 = arith.muli %44, %c2_i32_91 : i32 + %205 = arith.index_cast %204 : i32 to index + %206 = arith.addi %203, %205 : index + %c7168_92 = arith.constant 7168 : index + %207 = arith.addi %206, %c7168_92 : index + %c4_93 = arith.constant 4 : index + %208 = arith.muli %207, %c4_93 : index + %209 = arith.index_cast %208 : index to i64 + %210 = pto.castptr %200 : !pto.ptr -> i64 + %211 = llvm.inttoptr %210 : i64 to !llvm.ptr<6> + %212 = llvm.getelementptr %211[%209] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %213 = llvm.load %212 : !llvm.ptr<6> -> vector<2xf32> + %c18 = arith.constant 18 : index + %c4_94 = arith.constant 4 : index + %214 = arith.muli %c18, %c4_94 : index + %215 = arith.index_cast %214 : index to i64 + %216 = llvm.getelementptr %42[%215] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %213, %216 : vector<2xf32>, !llvm.ptr + %217 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_95 = arith.constant 1 : index + %218 = arith.andi %arg5, %c1_95 : index + %c8192_96 = arith.constant 8192 : index + %219 = arith.muli %218, %c8192_96 : index + %c5120 = arith.constant 5120 : index + %220 = arith.addi %219, %c5120 : index + %c2_i32_97 = arith.constant 2 : i32 + %221 = arith.muli %44, %c2_i32_97 : i32 + %222 = arith.index_cast %221 : i32 to index + %223 = arith.addi %220, %222 : index + %c7168_98 = arith.constant 7168 : index + %224 = arith.addi %223, %c7168_98 : index + %c4_99 = arith.constant 4 : index + %225 = arith.muli %224, %c4_99 : index + %226 = arith.index_cast %225 : index to i64 + %227 = pto.castptr %217 : !pto.ptr -> i64 + %228 = llvm.inttoptr %227 : i64 to !llvm.ptr<6> + %229 = llvm.getelementptr %228[%226] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %230 = llvm.load %229 : !llvm.ptr<6> -> vector<2xf32> + %c20 = arith.constant 20 : index + %c4_100 = arith.constant 4 : index + %231 = arith.muli %c20, %c4_100 : index + %232 = arith.index_cast %231 : index to i64 + %233 = llvm.getelementptr %42[%232] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %230, %233 : vector<2xf32>, !llvm.ptr + %234 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_101 = arith.constant 1 : index + %235 = arith.andi %arg5, %c1_101 : index + %c8192_102 = arith.constant 8192 : index + %236 = arith.muli %235, %c8192_102 : index + %c5632 = arith.constant 5632 : index + %237 = arith.addi %236, %c5632 : index + %c2_i32_103 = arith.constant 2 : i32 + %238 = arith.muli %44, %c2_i32_103 : i32 + %239 = arith.index_cast %238 : i32 to index + %240 = arith.addi %237, %239 : index + %c7168_104 = arith.constant 7168 : index + %241 = arith.addi %240, %c7168_104 : index + %c4_105 = arith.constant 4 : index + %242 = arith.muli %241, %c4_105 : index + %243 = arith.index_cast %242 : index to i64 + %244 = pto.castptr %234 : !pto.ptr -> i64 + %245 = llvm.inttoptr %244 : i64 to !llvm.ptr<6> + %246 = llvm.getelementptr %245[%243] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %247 = llvm.load %246 : !llvm.ptr<6> -> vector<2xf32> + %c22 = arith.constant 22 : index + %c4_106 = arith.constant 4 : index + %248 = arith.muli %c22, %c4_106 : index + %249 = arith.index_cast %248 : index to i64 + %250 = llvm.getelementptr %42[%249] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %247, %250 : vector<2xf32>, !llvm.ptr + %251 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_107 = arith.constant 1 : index + %252 = arith.andi %arg5, %c1_107 : index + %c8192_108 = arith.constant 8192 : index + %253 = arith.muli %252, %c8192_108 : index + %c6144 = arith.constant 6144 : index + %254 = arith.addi %253, %c6144 : index + %c2_i32_109 = arith.constant 2 : i32 + %255 = arith.muli %44, %c2_i32_109 : i32 + %256 = arith.index_cast %255 : i32 to index + %257 = arith.addi %254, %256 : index + %c7168_110 = arith.constant 7168 : index + %258 = arith.addi %257, %c7168_110 : index + %c4_111 = arith.constant 4 : index + %259 = arith.muli %258, %c4_111 : index + %260 = arith.index_cast %259 : index to i64 + %261 = pto.castptr %251 : !pto.ptr -> i64 + %262 = llvm.inttoptr %261 : i64 to !llvm.ptr<6> + %263 = llvm.getelementptr %262[%260] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %264 = llvm.load %263 : !llvm.ptr<6> -> vector<2xf32> + %c24 = arith.constant 24 : index + %c4_112 = arith.constant 4 : index + %265 = arith.muli %c24, %c4_112 : index + %266 = arith.index_cast %265 : index to i64 + %267 = llvm.getelementptr %42[%266] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %264, %267 : vector<2xf32>, !llvm.ptr + %268 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_113 = arith.constant 1 : index + %269 = arith.andi %arg5, %c1_113 : index + %c8192_114 = arith.constant 8192 : index + %270 = arith.muli %269, %c8192_114 : index + %c6656 = arith.constant 6656 : index + %271 = arith.addi %270, %c6656 : index + %c2_i32_115 = arith.constant 2 : i32 + %272 = arith.muli %44, %c2_i32_115 : i32 + %273 = arith.index_cast %272 : i32 to index + %274 = arith.addi %271, %273 : index + %c7168_116 = arith.constant 7168 : index + %275 = arith.addi %274, %c7168_116 : index + %c4_117 = arith.constant 4 : index + %276 = arith.muli %275, %c4_117 : index + %277 = arith.index_cast %276 : index to i64 + %278 = pto.castptr %268 : !pto.ptr -> i64 + %279 = llvm.inttoptr %278 : i64 to !llvm.ptr<6> + %280 = llvm.getelementptr %279[%277] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %281 = llvm.load %280 : !llvm.ptr<6> -> vector<2xf32> + %c26 = arith.constant 26 : index + %c4_118 = arith.constant 4 : index + %282 = arith.muli %c26, %c4_118 : index + %283 = arith.index_cast %282 : index to i64 + %284 = llvm.getelementptr %42[%283] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %281, %284 : vector<2xf32>, !llvm.ptr + %285 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_119 = arith.constant 1 : index + %286 = arith.andi %arg5, %c1_119 : index + %c8192_120 = arith.constant 8192 : index + %287 = arith.muli %286, %c8192_120 : index + %c7168_121 = arith.constant 7168 : index + %288 = arith.addi %287, %c7168_121 : index + %c2_i32_122 = arith.constant 2 : i32 + %289 = arith.muli %44, %c2_i32_122 : i32 + %290 = arith.index_cast %289 : i32 to index + %291 = arith.addi %288, %290 : index + %c7168_123 = arith.constant 7168 : index + %292 = arith.addi %291, %c7168_123 : index + %c4_124 = arith.constant 4 : index + %293 = arith.muli %292, %c4_124 : index + %294 = arith.index_cast %293 : index to i64 + %295 = pto.castptr %285 : !pto.ptr -> i64 + %296 = llvm.inttoptr %295 : i64 to !llvm.ptr<6> + %297 = llvm.getelementptr %296[%294] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %298 = llvm.load %297 : !llvm.ptr<6> -> vector<2xf32> + %c28 = arith.constant 28 : index + %c4_125 = arith.constant 4 : index + %299 = arith.muli %c28, %c4_125 : index + %300 = arith.index_cast %299 : index to i64 + %301 = llvm.getelementptr %42[%300] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %298, %301 : vector<2xf32>, !llvm.ptr + %302 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_126 = arith.constant 1 : index + %303 = arith.andi %arg5, %c1_126 : index + %c8192_127 = arith.constant 8192 : index + %304 = arith.muli %303, %c8192_127 : index + %c7680 = arith.constant 7680 : index + %305 = arith.addi %304, %c7680 : index + %c2_i32_128 = arith.constant 2 : i32 + %306 = arith.muli %44, %c2_i32_128 : i32 + %307 = arith.index_cast %306 : i32 to index + %308 = arith.addi %305, %307 : index + %c7168_129 = arith.constant 7168 : index + %309 = arith.addi %308, %c7168_129 : index + %c4_130 = arith.constant 4 : index + %310 = arith.muli %309, %c4_130 : index + %311 = arith.index_cast %310 : index to i64 + %312 = pto.castptr %302 : !pto.ptr -> i64 + %313 = llvm.inttoptr %312 : i64 to !llvm.ptr<6> + %314 = llvm.getelementptr %313[%311] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + %315 = llvm.load %314 : !llvm.ptr<6> -> vector<2xf32> + %c30 = arith.constant 30 : index + %c4_131 = arith.constant 4 : index + %316 = arith.muli %c30, %c4_131 : index + %317 = arith.index_cast %316 : index to i64 + %318 = llvm.getelementptr %42[%317] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %315, %318 : vector<2xf32>, !llvm.ptr + %c0_132 = arith.constant 0 : index + %cst = arith.constant 0.000000e+00 : f32 + %c4_133 = arith.constant 4 : index + %319 = arith.muli %c0_132, %c4_133 : index + %320 = arith.index_cast %319 : index to i64 + %321 = llvm.getelementptr %43[%320] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %cst, %321 : f32, !llvm.ptr + %c0_134 = arith.constant 0 : index + %c4_135 = arith.constant 4 : index + %322 = arith.muli %c0_134, %c4_135 : index + %323 = arith.index_cast %322 : index to i64 + %324 = llvm.getelementptr %43[%323] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %325 = llvm.load %324 : !llvm.ptr -> f32 + %c0_136 = arith.constant 0 : index + %c4_137 = arith.constant 4 : index + %326 = arith.muli %c0_136, %c4_137 : index + %327 = arith.index_cast %326 : index to i64 + %328 = llvm.getelementptr %42[%327] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %329 = llvm.load %328 : !llvm.ptr -> f32 + %c0_138 = arith.constant 0 : index + %c4_139 = arith.constant 4 : index + %330 = arith.muli %c0_138, %c4_139 : index + %331 = arith.index_cast %330 : index to i64 + %332 = llvm.getelementptr %42[%331] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %333 = llvm.load %332 : !llvm.ptr -> f32 + %334 = arith.mulf %329, %333 : f32 + %335 = arith.addf %325, %334 : f32 + %c0_140 = arith.constant 0 : index + %c4_141 = arith.constant 4 : index + %336 = arith.muli %c0_140, %c4_141 : index + %337 = arith.index_cast %336 : index to i64 + %338 = llvm.getelementptr %43[%337] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %335, %338 : f32, !llvm.ptr + %c0_142 = arith.constant 0 : index + %c4_143 = arith.constant 4 : index + %339 = arith.muli %c0_142, %c4_143 : index + %340 = arith.index_cast %339 : index to i64 + %341 = llvm.getelementptr %43[%340] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %342 = llvm.load %341 : !llvm.ptr -> f32 + %c1_144 = arith.constant 1 : index + %c4_145 = arith.constant 4 : index + %343 = arith.muli %c1_144, %c4_145 : index + %344 = arith.index_cast %343 : index to i64 + %345 = llvm.getelementptr %42[%344] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %346 = llvm.load %345 : !llvm.ptr -> f32 + %c1_146 = arith.constant 1 : index + %c4_147 = arith.constant 4 : index + %347 = arith.muli %c1_146, %c4_147 : index + %348 = arith.index_cast %347 : index to i64 + %349 = llvm.getelementptr %42[%348] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %350 = llvm.load %349 : !llvm.ptr -> f32 + %351 = arith.mulf %346, %350 : f32 + %352 = arith.addf %342, %351 : f32 + %c0_148 = arith.constant 0 : index + %c4_149 = arith.constant 4 : index + %353 = arith.muli %c0_148, %c4_149 : index + %354 = arith.index_cast %353 : index to i64 + %355 = llvm.getelementptr %43[%354] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %352, %355 : f32, !llvm.ptr + %c0_150 = arith.constant 0 : index + %c4_151 = arith.constant 4 : index + %356 = arith.muli %c0_150, %c4_151 : index + %357 = arith.index_cast %356 : index to i64 + %358 = llvm.getelementptr %43[%357] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %359 = llvm.load %358 : !llvm.ptr -> f32 + %c2_152 = arith.constant 2 : index + %c4_153 = arith.constant 4 : index + %360 = arith.muli %c2_152, %c4_153 : index + %361 = arith.index_cast %360 : index to i64 + %362 = llvm.getelementptr %42[%361] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %363 = llvm.load %362 : !llvm.ptr -> f32 + %c2_154 = arith.constant 2 : index + %c4_155 = arith.constant 4 : index + %364 = arith.muli %c2_154, %c4_155 : index + %365 = arith.index_cast %364 : index to i64 + %366 = llvm.getelementptr %42[%365] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %367 = llvm.load %366 : !llvm.ptr -> f32 + %368 = arith.mulf %363, %367 : f32 + %369 = arith.addf %359, %368 : f32 + %c0_156 = arith.constant 0 : index + %c4_157 = arith.constant 4 : index + %370 = arith.muli %c0_156, %c4_157 : index + %371 = arith.index_cast %370 : index to i64 + %372 = llvm.getelementptr %43[%371] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %369, %372 : f32, !llvm.ptr + %c0_158 = arith.constant 0 : index + %c4_159 = arith.constant 4 : index + %373 = arith.muli %c0_158, %c4_159 : index + %374 = arith.index_cast %373 : index to i64 + %375 = llvm.getelementptr %43[%374] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %376 = llvm.load %375 : !llvm.ptr -> f32 + %c3 = arith.constant 3 : index + %c4_160 = arith.constant 4 : index + %377 = arith.muli %c3, %c4_160 : index + %378 = arith.index_cast %377 : index to i64 + %379 = llvm.getelementptr %42[%378] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %380 = llvm.load %379 : !llvm.ptr -> f32 + %c3_161 = arith.constant 3 : index + %c4_162 = arith.constant 4 : index + %381 = arith.muli %c3_161, %c4_162 : index + %382 = arith.index_cast %381 : index to i64 + %383 = llvm.getelementptr %42[%382] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %384 = llvm.load %383 : !llvm.ptr -> f32 + %385 = arith.mulf %380, %384 : f32 + %386 = arith.addf %376, %385 : f32 + %c0_163 = arith.constant 0 : index + %c4_164 = arith.constant 4 : index + %387 = arith.muli %c0_163, %c4_164 : index + %388 = arith.index_cast %387 : index to i64 + %389 = llvm.getelementptr %43[%388] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %386, %389 : f32, !llvm.ptr + %c0_165 = arith.constant 0 : index + %c4_166 = arith.constant 4 : index + %390 = arith.muli %c0_165, %c4_166 : index + %391 = arith.index_cast %390 : index to i64 + %392 = llvm.getelementptr %43[%391] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %393 = llvm.load %392 : !llvm.ptr -> f32 + %c4_167 = arith.constant 4 : index + %c4_168 = arith.constant 4 : index + %394 = arith.muli %c4_167, %c4_168 : index + %395 = arith.index_cast %394 : index to i64 + %396 = llvm.getelementptr %42[%395] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %397 = llvm.load %396 : !llvm.ptr -> f32 + %c4_169 = arith.constant 4 : index + %c4_170 = arith.constant 4 : index + %398 = arith.muli %c4_169, %c4_170 : index + %399 = arith.index_cast %398 : index to i64 + %400 = llvm.getelementptr %42[%399] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %401 = llvm.load %400 : !llvm.ptr -> f32 + %402 = arith.mulf %397, %401 : f32 + %403 = arith.addf %393, %402 : f32 + %c0_171 = arith.constant 0 : index + %c4_172 = arith.constant 4 : index + %404 = arith.muli %c0_171, %c4_172 : index + %405 = arith.index_cast %404 : index to i64 + %406 = llvm.getelementptr %43[%405] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %403, %406 : f32, !llvm.ptr + %c0_173 = arith.constant 0 : index + %c4_174 = arith.constant 4 : index + %407 = arith.muli %c0_173, %c4_174 : index + %408 = arith.index_cast %407 : index to i64 + %409 = llvm.getelementptr %43[%408] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %410 = llvm.load %409 : !llvm.ptr -> f32 + %c5 = arith.constant 5 : index + %c4_175 = arith.constant 4 : index + %411 = arith.muli %c5, %c4_175 : index + %412 = arith.index_cast %411 : index to i64 + %413 = llvm.getelementptr %42[%412] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %414 = llvm.load %413 : !llvm.ptr -> f32 + %c5_176 = arith.constant 5 : index + %c4_177 = arith.constant 4 : index + %415 = arith.muli %c5_176, %c4_177 : index + %416 = arith.index_cast %415 : index to i64 + %417 = llvm.getelementptr %42[%416] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %418 = llvm.load %417 : !llvm.ptr -> f32 + %419 = arith.mulf %414, %418 : f32 + %420 = arith.addf %410, %419 : f32 + %c0_178 = arith.constant 0 : index + %c4_179 = arith.constant 4 : index + %421 = arith.muli %c0_178, %c4_179 : index + %422 = arith.index_cast %421 : index to i64 + %423 = llvm.getelementptr %43[%422] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %420, %423 : f32, !llvm.ptr + %c0_180 = arith.constant 0 : index + %c4_181 = arith.constant 4 : index + %424 = arith.muli %c0_180, %c4_181 : index + %425 = arith.index_cast %424 : index to i64 + %426 = llvm.getelementptr %43[%425] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %427 = llvm.load %426 : !llvm.ptr -> f32 + %c6_182 = arith.constant 6 : index + %c4_183 = arith.constant 4 : index + %428 = arith.muli %c6_182, %c4_183 : index + %429 = arith.index_cast %428 : index to i64 + %430 = llvm.getelementptr %42[%429] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %431 = llvm.load %430 : !llvm.ptr -> f32 + %c6_184 = arith.constant 6 : index + %c4_185 = arith.constant 4 : index + %432 = arith.muli %c6_184, %c4_185 : index + %433 = arith.index_cast %432 : index to i64 + %434 = llvm.getelementptr %42[%433] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %435 = llvm.load %434 : !llvm.ptr -> f32 + %436 = arith.mulf %431, %435 : f32 + %437 = arith.addf %427, %436 : f32 + %c0_186 = arith.constant 0 : index + %c4_187 = arith.constant 4 : index + %438 = arith.muli %c0_186, %c4_187 : index + %439 = arith.index_cast %438 : index to i64 + %440 = llvm.getelementptr %43[%439] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %437, %440 : f32, !llvm.ptr + %c0_188 = arith.constant 0 : index + %c4_189 = arith.constant 4 : index + %441 = arith.muli %c0_188, %c4_189 : index + %442 = arith.index_cast %441 : index to i64 + %443 = llvm.getelementptr %43[%442] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %444 = llvm.load %443 : !llvm.ptr -> f32 + %c7 = arith.constant 7 : index + %c4_190 = arith.constant 4 : index + %445 = arith.muli %c7, %c4_190 : index + %446 = arith.index_cast %445 : index to i64 + %447 = llvm.getelementptr %42[%446] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %448 = llvm.load %447 : !llvm.ptr -> f32 + %c7_191 = arith.constant 7 : index + %c4_192 = arith.constant 4 : index + %449 = arith.muli %c7_191, %c4_192 : index + %450 = arith.index_cast %449 : index to i64 + %451 = llvm.getelementptr %42[%450] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %452 = llvm.load %451 : !llvm.ptr -> f32 + %453 = arith.mulf %448, %452 : f32 + %454 = arith.addf %444, %453 : f32 + %c0_193 = arith.constant 0 : index + %c4_194 = arith.constant 4 : index + %455 = arith.muli %c0_193, %c4_194 : index + %456 = arith.index_cast %455 : index to i64 + %457 = llvm.getelementptr %43[%456] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %454, %457 : f32, !llvm.ptr + %c0_195 = arith.constant 0 : index + %c4_196 = arith.constant 4 : index + %458 = arith.muli %c0_195, %c4_196 : index + %459 = arith.index_cast %458 : index to i64 + %460 = llvm.getelementptr %43[%459] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %461 = llvm.load %460 : !llvm.ptr -> f32 + %c8_197 = arith.constant 8 : index + %c4_198 = arith.constant 4 : index + %462 = arith.muli %c8_197, %c4_198 : index + %463 = arith.index_cast %462 : index to i64 + %464 = llvm.getelementptr %42[%463] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %465 = llvm.load %464 : !llvm.ptr -> f32 + %c8_199 = arith.constant 8 : index + %c4_200 = arith.constant 4 : index + %466 = arith.muli %c8_199, %c4_200 : index + %467 = arith.index_cast %466 : index to i64 + %468 = llvm.getelementptr %42[%467] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %469 = llvm.load %468 : !llvm.ptr -> f32 + %470 = arith.mulf %465, %469 : f32 + %471 = arith.addf %461, %470 : f32 + %c0_201 = arith.constant 0 : index + %c4_202 = arith.constant 4 : index + %472 = arith.muli %c0_201, %c4_202 : index + %473 = arith.index_cast %472 : index to i64 + %474 = llvm.getelementptr %43[%473] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %471, %474 : f32, !llvm.ptr + %c0_203 = arith.constant 0 : index + %c4_204 = arith.constant 4 : index + %475 = arith.muli %c0_203, %c4_204 : index + %476 = arith.index_cast %475 : index to i64 + %477 = llvm.getelementptr %43[%476] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %478 = llvm.load %477 : !llvm.ptr -> f32 + %c9 = arith.constant 9 : index + %c4_205 = arith.constant 4 : index + %479 = arith.muli %c9, %c4_205 : index + %480 = arith.index_cast %479 : index to i64 + %481 = llvm.getelementptr %42[%480] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %482 = llvm.load %481 : !llvm.ptr -> f32 + %c9_206 = arith.constant 9 : index + %c4_207 = arith.constant 4 : index + %483 = arith.muli %c9_206, %c4_207 : index + %484 = arith.index_cast %483 : index to i64 + %485 = llvm.getelementptr %42[%484] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %486 = llvm.load %485 : !llvm.ptr -> f32 + %487 = arith.mulf %482, %486 : f32 + %488 = arith.addf %478, %487 : f32 + %c0_208 = arith.constant 0 : index + %c4_209 = arith.constant 4 : index + %489 = arith.muli %c0_208, %c4_209 : index + %490 = arith.index_cast %489 : index to i64 + %491 = llvm.getelementptr %43[%490] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %488, %491 : f32, !llvm.ptr + %c0_210 = arith.constant 0 : index + %c4_211 = arith.constant 4 : index + %492 = arith.muli %c0_210, %c4_211 : index + %493 = arith.index_cast %492 : index to i64 + %494 = llvm.getelementptr %43[%493] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %495 = llvm.load %494 : !llvm.ptr -> f32 + %c10_212 = arith.constant 10 : index + %c4_213 = arith.constant 4 : index + %496 = arith.muli %c10_212, %c4_213 : index + %497 = arith.index_cast %496 : index to i64 + %498 = llvm.getelementptr %42[%497] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %499 = llvm.load %498 : !llvm.ptr -> f32 + %c10_214 = arith.constant 10 : index + %c4_215 = arith.constant 4 : index + %500 = arith.muli %c10_214, %c4_215 : index + %501 = arith.index_cast %500 : index to i64 + %502 = llvm.getelementptr %42[%501] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %503 = llvm.load %502 : !llvm.ptr -> f32 + %504 = arith.mulf %499, %503 : f32 + %505 = arith.addf %495, %504 : f32 + %c0_216 = arith.constant 0 : index + %c4_217 = arith.constant 4 : index + %506 = arith.muli %c0_216, %c4_217 : index + %507 = arith.index_cast %506 : index to i64 + %508 = llvm.getelementptr %43[%507] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %505, %508 : f32, !llvm.ptr + %c0_218 = arith.constant 0 : index + %c4_219 = arith.constant 4 : index + %509 = arith.muli %c0_218, %c4_219 : index + %510 = arith.index_cast %509 : index to i64 + %511 = llvm.getelementptr %43[%510] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %512 = llvm.load %511 : !llvm.ptr -> f32 + %c11 = arith.constant 11 : index + %c4_220 = arith.constant 4 : index + %513 = arith.muli %c11, %c4_220 : index + %514 = arith.index_cast %513 : index to i64 + %515 = llvm.getelementptr %42[%514] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %516 = llvm.load %515 : !llvm.ptr -> f32 + %c11_221 = arith.constant 11 : index + %c4_222 = arith.constant 4 : index + %517 = arith.muli %c11_221, %c4_222 : index + %518 = arith.index_cast %517 : index to i64 + %519 = llvm.getelementptr %42[%518] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %520 = llvm.load %519 : !llvm.ptr -> f32 + %521 = arith.mulf %516, %520 : f32 + %522 = arith.addf %512, %521 : f32 + %c0_223 = arith.constant 0 : index + %c4_224 = arith.constant 4 : index + %523 = arith.muli %c0_223, %c4_224 : index + %524 = arith.index_cast %523 : index to i64 + %525 = llvm.getelementptr %43[%524] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %522, %525 : f32, !llvm.ptr + %c0_225 = arith.constant 0 : index + %c4_226 = arith.constant 4 : index + %526 = arith.muli %c0_225, %c4_226 : index + %527 = arith.index_cast %526 : index to i64 + %528 = llvm.getelementptr %43[%527] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %529 = llvm.load %528 : !llvm.ptr -> f32 + %c12_227 = arith.constant 12 : index + %c4_228 = arith.constant 4 : index + %530 = arith.muli %c12_227, %c4_228 : index + %531 = arith.index_cast %530 : index to i64 + %532 = llvm.getelementptr %42[%531] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %533 = llvm.load %532 : !llvm.ptr -> f32 + %c12_229 = arith.constant 12 : index + %c4_230 = arith.constant 4 : index + %534 = arith.muli %c12_229, %c4_230 : index + %535 = arith.index_cast %534 : index to i64 + %536 = llvm.getelementptr %42[%535] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %537 = llvm.load %536 : !llvm.ptr -> f32 + %538 = arith.mulf %533, %537 : f32 + %539 = arith.addf %529, %538 : f32 + %c0_231 = arith.constant 0 : index + %c4_232 = arith.constant 4 : index + %540 = arith.muli %c0_231, %c4_232 : index + %541 = arith.index_cast %540 : index to i64 + %542 = llvm.getelementptr %43[%541] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %539, %542 : f32, !llvm.ptr + %c0_233 = arith.constant 0 : index + %c4_234 = arith.constant 4 : index + %543 = arith.muli %c0_233, %c4_234 : index + %544 = arith.index_cast %543 : index to i64 + %545 = llvm.getelementptr %43[%544] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %546 = llvm.load %545 : !llvm.ptr -> f32 + %c13 = arith.constant 13 : index + %c4_235 = arith.constant 4 : index + %547 = arith.muli %c13, %c4_235 : index + %548 = arith.index_cast %547 : index to i64 + %549 = llvm.getelementptr %42[%548] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %550 = llvm.load %549 : !llvm.ptr -> f32 + %c13_236 = arith.constant 13 : index + %c4_237 = arith.constant 4 : index + %551 = arith.muli %c13_236, %c4_237 : index + %552 = arith.index_cast %551 : index to i64 + %553 = llvm.getelementptr %42[%552] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %554 = llvm.load %553 : !llvm.ptr -> f32 + %555 = arith.mulf %550, %554 : f32 + %556 = arith.addf %546, %555 : f32 + %c0_238 = arith.constant 0 : index + %c4_239 = arith.constant 4 : index + %557 = arith.muli %c0_238, %c4_239 : index + %558 = arith.index_cast %557 : index to i64 + %559 = llvm.getelementptr %43[%558] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %556, %559 : f32, !llvm.ptr + %c0_240 = arith.constant 0 : index + %c4_241 = arith.constant 4 : index + %560 = arith.muli %c0_240, %c4_241 : index + %561 = arith.index_cast %560 : index to i64 + %562 = llvm.getelementptr %43[%561] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %563 = llvm.load %562 : !llvm.ptr -> f32 + %c14_242 = arith.constant 14 : index + %c4_243 = arith.constant 4 : index + %564 = arith.muli %c14_242, %c4_243 : index + %565 = arith.index_cast %564 : index to i64 + %566 = llvm.getelementptr %42[%565] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %567 = llvm.load %566 : !llvm.ptr -> f32 + %c14_244 = arith.constant 14 : index + %c4_245 = arith.constant 4 : index + %568 = arith.muli %c14_244, %c4_245 : index + %569 = arith.index_cast %568 : index to i64 + %570 = llvm.getelementptr %42[%569] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %571 = llvm.load %570 : !llvm.ptr -> f32 + %572 = arith.mulf %567, %571 : f32 + %573 = arith.addf %563, %572 : f32 + %c0_246 = arith.constant 0 : index + %c4_247 = arith.constant 4 : index + %574 = arith.muli %c0_246, %c4_247 : index + %575 = arith.index_cast %574 : index to i64 + %576 = llvm.getelementptr %43[%575] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %573, %576 : f32, !llvm.ptr + %c0_248 = arith.constant 0 : index + %c4_249 = arith.constant 4 : index + %577 = arith.muli %c0_248, %c4_249 : index + %578 = arith.index_cast %577 : index to i64 + %579 = llvm.getelementptr %43[%578] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %580 = llvm.load %579 : !llvm.ptr -> f32 + %c15 = arith.constant 15 : index + %c4_250 = arith.constant 4 : index + %581 = arith.muli %c15, %c4_250 : index + %582 = arith.index_cast %581 : index to i64 + %583 = llvm.getelementptr %42[%582] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %584 = llvm.load %583 : !llvm.ptr -> f32 + %c15_251 = arith.constant 15 : index + %c4_252 = arith.constant 4 : index + %585 = arith.muli %c15_251, %c4_252 : index + %586 = arith.index_cast %585 : index to i64 + %587 = llvm.getelementptr %42[%586] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %588 = llvm.load %587 : !llvm.ptr -> f32 + %589 = arith.mulf %584, %588 : f32 + %590 = arith.addf %580, %589 : f32 + %c0_253 = arith.constant 0 : index + %c4_254 = arith.constant 4 : index + %591 = arith.muli %c0_253, %c4_254 : index + %592 = arith.index_cast %591 : index to i64 + %593 = llvm.getelementptr %43[%592] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %590, %593 : f32, !llvm.ptr + %c0_255 = arith.constant 0 : index + %c4_256 = arith.constant 4 : index + %594 = arith.muli %c0_255, %c4_256 : index + %595 = arith.index_cast %594 : index to i64 + %596 = llvm.getelementptr %43[%595] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %597 = llvm.load %596 : !llvm.ptr -> f32 + %c16_257 = arith.constant 16 : index + %c4_258 = arith.constant 4 : index + %598 = arith.muli %c16_257, %c4_258 : index + %599 = arith.index_cast %598 : index to i64 + %600 = llvm.getelementptr %42[%599] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %601 = llvm.load %600 : !llvm.ptr -> f32 + %c16_259 = arith.constant 16 : index + %c4_260 = arith.constant 4 : index + %602 = arith.muli %c16_259, %c4_260 : index + %603 = arith.index_cast %602 : index to i64 + %604 = llvm.getelementptr %42[%603] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %605 = llvm.load %604 : !llvm.ptr -> f32 + %606 = arith.mulf %601, %605 : f32 + %607 = arith.addf %597, %606 : f32 + %c0_261 = arith.constant 0 : index + %c4_262 = arith.constant 4 : index + %608 = arith.muli %c0_261, %c4_262 : index + %609 = arith.index_cast %608 : index to i64 + %610 = llvm.getelementptr %43[%609] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %607, %610 : f32, !llvm.ptr + %c0_263 = arith.constant 0 : index + %c4_264 = arith.constant 4 : index + %611 = arith.muli %c0_263, %c4_264 : index + %612 = arith.index_cast %611 : index to i64 + %613 = llvm.getelementptr %43[%612] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %614 = llvm.load %613 : !llvm.ptr -> f32 + %c17 = arith.constant 17 : index + %c4_265 = arith.constant 4 : index + %615 = arith.muli %c17, %c4_265 : index + %616 = arith.index_cast %615 : index to i64 + %617 = llvm.getelementptr %42[%616] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %618 = llvm.load %617 : !llvm.ptr -> f32 + %c17_266 = arith.constant 17 : index + %c4_267 = arith.constant 4 : index + %619 = arith.muli %c17_266, %c4_267 : index + %620 = arith.index_cast %619 : index to i64 + %621 = llvm.getelementptr %42[%620] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %622 = llvm.load %621 : !llvm.ptr -> f32 + %623 = arith.mulf %618, %622 : f32 + %624 = arith.addf %614, %623 : f32 + %c0_268 = arith.constant 0 : index + %c4_269 = arith.constant 4 : index + %625 = arith.muli %c0_268, %c4_269 : index + %626 = arith.index_cast %625 : index to i64 + %627 = llvm.getelementptr %43[%626] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %624, %627 : f32, !llvm.ptr + %c0_270 = arith.constant 0 : index + %c4_271 = arith.constant 4 : index + %628 = arith.muli %c0_270, %c4_271 : index + %629 = arith.index_cast %628 : index to i64 + %630 = llvm.getelementptr %43[%629] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %631 = llvm.load %630 : !llvm.ptr -> f32 + %c18_272 = arith.constant 18 : index + %c4_273 = arith.constant 4 : index + %632 = arith.muli %c18_272, %c4_273 : index + %633 = arith.index_cast %632 : index to i64 + %634 = llvm.getelementptr %42[%633] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %635 = llvm.load %634 : !llvm.ptr -> f32 + %c18_274 = arith.constant 18 : index + %c4_275 = arith.constant 4 : index + %636 = arith.muli %c18_274, %c4_275 : index + %637 = arith.index_cast %636 : index to i64 + %638 = llvm.getelementptr %42[%637] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %639 = llvm.load %638 : !llvm.ptr -> f32 + %640 = arith.mulf %635, %639 : f32 + %641 = arith.addf %631, %640 : f32 + %c0_276 = arith.constant 0 : index + %c4_277 = arith.constant 4 : index + %642 = arith.muli %c0_276, %c4_277 : index + %643 = arith.index_cast %642 : index to i64 + %644 = llvm.getelementptr %43[%643] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %641, %644 : f32, !llvm.ptr + %c0_278 = arith.constant 0 : index + %c4_279 = arith.constant 4 : index + %645 = arith.muli %c0_278, %c4_279 : index + %646 = arith.index_cast %645 : index to i64 + %647 = llvm.getelementptr %43[%646] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %648 = llvm.load %647 : !llvm.ptr -> f32 + %c19 = arith.constant 19 : index + %c4_280 = arith.constant 4 : index + %649 = arith.muli %c19, %c4_280 : index + %650 = arith.index_cast %649 : index to i64 + %651 = llvm.getelementptr %42[%650] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %652 = llvm.load %651 : !llvm.ptr -> f32 + %c19_281 = arith.constant 19 : index + %c4_282 = arith.constant 4 : index + %653 = arith.muli %c19_281, %c4_282 : index + %654 = arith.index_cast %653 : index to i64 + %655 = llvm.getelementptr %42[%654] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %656 = llvm.load %655 : !llvm.ptr -> f32 + %657 = arith.mulf %652, %656 : f32 + %658 = arith.addf %648, %657 : f32 + %c0_283 = arith.constant 0 : index + %c4_284 = arith.constant 4 : index + %659 = arith.muli %c0_283, %c4_284 : index + %660 = arith.index_cast %659 : index to i64 + %661 = llvm.getelementptr %43[%660] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %658, %661 : f32, !llvm.ptr + %c0_285 = arith.constant 0 : index + %c4_286 = arith.constant 4 : index + %662 = arith.muli %c0_285, %c4_286 : index + %663 = arith.index_cast %662 : index to i64 + %664 = llvm.getelementptr %43[%663] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %665 = llvm.load %664 : !llvm.ptr -> f32 + %c20_287 = arith.constant 20 : index + %c4_288 = arith.constant 4 : index + %666 = arith.muli %c20_287, %c4_288 : index + %667 = arith.index_cast %666 : index to i64 + %668 = llvm.getelementptr %42[%667] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %669 = llvm.load %668 : !llvm.ptr -> f32 + %c20_289 = arith.constant 20 : index + %c4_290 = arith.constant 4 : index + %670 = arith.muli %c20_289, %c4_290 : index + %671 = arith.index_cast %670 : index to i64 + %672 = llvm.getelementptr %42[%671] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %673 = llvm.load %672 : !llvm.ptr -> f32 + %674 = arith.mulf %669, %673 : f32 + %675 = arith.addf %665, %674 : f32 + %c0_291 = arith.constant 0 : index + %c4_292 = arith.constant 4 : index + %676 = arith.muli %c0_291, %c4_292 : index + %677 = arith.index_cast %676 : index to i64 + %678 = llvm.getelementptr %43[%677] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %675, %678 : f32, !llvm.ptr + %c0_293 = arith.constant 0 : index + %c4_294 = arith.constant 4 : index + %679 = arith.muli %c0_293, %c4_294 : index + %680 = arith.index_cast %679 : index to i64 + %681 = llvm.getelementptr %43[%680] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %682 = llvm.load %681 : !llvm.ptr -> f32 + %c21 = arith.constant 21 : index + %c4_295 = arith.constant 4 : index + %683 = arith.muli %c21, %c4_295 : index + %684 = arith.index_cast %683 : index to i64 + %685 = llvm.getelementptr %42[%684] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %686 = llvm.load %685 : !llvm.ptr -> f32 + %c21_296 = arith.constant 21 : index + %c4_297 = arith.constant 4 : index + %687 = arith.muli %c21_296, %c4_297 : index + %688 = arith.index_cast %687 : index to i64 + %689 = llvm.getelementptr %42[%688] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %690 = llvm.load %689 : !llvm.ptr -> f32 + %691 = arith.mulf %686, %690 : f32 + %692 = arith.addf %682, %691 : f32 + %c0_298 = arith.constant 0 : index + %c4_299 = arith.constant 4 : index + %693 = arith.muli %c0_298, %c4_299 : index + %694 = arith.index_cast %693 : index to i64 + %695 = llvm.getelementptr %43[%694] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %692, %695 : f32, !llvm.ptr + %c0_300 = arith.constant 0 : index + %c4_301 = arith.constant 4 : index + %696 = arith.muli %c0_300, %c4_301 : index + %697 = arith.index_cast %696 : index to i64 + %698 = llvm.getelementptr %43[%697] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %699 = llvm.load %698 : !llvm.ptr -> f32 + %c22_302 = arith.constant 22 : index + %c4_303 = arith.constant 4 : index + %700 = arith.muli %c22_302, %c4_303 : index + %701 = arith.index_cast %700 : index to i64 + %702 = llvm.getelementptr %42[%701] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %703 = llvm.load %702 : !llvm.ptr -> f32 + %c22_304 = arith.constant 22 : index + %c4_305 = arith.constant 4 : index + %704 = arith.muli %c22_304, %c4_305 : index + %705 = arith.index_cast %704 : index to i64 + %706 = llvm.getelementptr %42[%705] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %707 = llvm.load %706 : !llvm.ptr -> f32 + %708 = arith.mulf %703, %707 : f32 + %709 = arith.addf %699, %708 : f32 + %c0_306 = arith.constant 0 : index + %c4_307 = arith.constant 4 : index + %710 = arith.muli %c0_306, %c4_307 : index + %711 = arith.index_cast %710 : index to i64 + %712 = llvm.getelementptr %43[%711] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %709, %712 : f32, !llvm.ptr + %c0_308 = arith.constant 0 : index + %c4_309 = arith.constant 4 : index + %713 = arith.muli %c0_308, %c4_309 : index + %714 = arith.index_cast %713 : index to i64 + %715 = llvm.getelementptr %43[%714] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %716 = llvm.load %715 : !llvm.ptr -> f32 + %c23 = arith.constant 23 : index + %c4_310 = arith.constant 4 : index + %717 = arith.muli %c23, %c4_310 : index + %718 = arith.index_cast %717 : index to i64 + %719 = llvm.getelementptr %42[%718] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %720 = llvm.load %719 : !llvm.ptr -> f32 + %c23_311 = arith.constant 23 : index + %c4_312 = arith.constant 4 : index + %721 = arith.muli %c23_311, %c4_312 : index + %722 = arith.index_cast %721 : index to i64 + %723 = llvm.getelementptr %42[%722] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %724 = llvm.load %723 : !llvm.ptr -> f32 + %725 = arith.mulf %720, %724 : f32 + %726 = arith.addf %716, %725 : f32 + %c0_313 = arith.constant 0 : index + %c4_314 = arith.constant 4 : index + %727 = arith.muli %c0_313, %c4_314 : index + %728 = arith.index_cast %727 : index to i64 + %729 = llvm.getelementptr %43[%728] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %726, %729 : f32, !llvm.ptr + %c0_315 = arith.constant 0 : index + %c4_316 = arith.constant 4 : index + %730 = arith.muli %c0_315, %c4_316 : index + %731 = arith.index_cast %730 : index to i64 + %732 = llvm.getelementptr %43[%731] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %733 = llvm.load %732 : !llvm.ptr -> f32 + %c24_317 = arith.constant 24 : index + %c4_318 = arith.constant 4 : index + %734 = arith.muli %c24_317, %c4_318 : index + %735 = arith.index_cast %734 : index to i64 + %736 = llvm.getelementptr %42[%735] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %737 = llvm.load %736 : !llvm.ptr -> f32 + %c24_319 = arith.constant 24 : index + %c4_320 = arith.constant 4 : index + %738 = arith.muli %c24_319, %c4_320 : index + %739 = arith.index_cast %738 : index to i64 + %740 = llvm.getelementptr %42[%739] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %741 = llvm.load %740 : !llvm.ptr -> f32 + %742 = arith.mulf %737, %741 : f32 + %743 = arith.addf %733, %742 : f32 + %c0_321 = arith.constant 0 : index + %c4_322 = arith.constant 4 : index + %744 = arith.muli %c0_321, %c4_322 : index + %745 = arith.index_cast %744 : index to i64 + %746 = llvm.getelementptr %43[%745] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %743, %746 : f32, !llvm.ptr + %c0_323 = arith.constant 0 : index + %c4_324 = arith.constant 4 : index + %747 = arith.muli %c0_323, %c4_324 : index + %748 = arith.index_cast %747 : index to i64 + %749 = llvm.getelementptr %43[%748] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %750 = llvm.load %749 : !llvm.ptr -> f32 + %c25 = arith.constant 25 : index + %c4_325 = arith.constant 4 : index + %751 = arith.muli %c25, %c4_325 : index + %752 = arith.index_cast %751 : index to i64 + %753 = llvm.getelementptr %42[%752] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %754 = llvm.load %753 : !llvm.ptr -> f32 + %c25_326 = arith.constant 25 : index + %c4_327 = arith.constant 4 : index + %755 = arith.muli %c25_326, %c4_327 : index + %756 = arith.index_cast %755 : index to i64 + %757 = llvm.getelementptr %42[%756] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %758 = llvm.load %757 : !llvm.ptr -> f32 + %759 = arith.mulf %754, %758 : f32 + %760 = arith.addf %750, %759 : f32 + %c0_328 = arith.constant 0 : index + %c4_329 = arith.constant 4 : index + %761 = arith.muli %c0_328, %c4_329 : index + %762 = arith.index_cast %761 : index to i64 + %763 = llvm.getelementptr %43[%762] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %760, %763 : f32, !llvm.ptr + %c0_330 = arith.constant 0 : index + %c4_331 = arith.constant 4 : index + %764 = arith.muli %c0_330, %c4_331 : index + %765 = arith.index_cast %764 : index to i64 + %766 = llvm.getelementptr %43[%765] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %767 = llvm.load %766 : !llvm.ptr -> f32 + %c26_332 = arith.constant 26 : index + %c4_333 = arith.constant 4 : index + %768 = arith.muli %c26_332, %c4_333 : index + %769 = arith.index_cast %768 : index to i64 + %770 = llvm.getelementptr %42[%769] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %771 = llvm.load %770 : !llvm.ptr -> f32 + %c26_334 = arith.constant 26 : index + %c4_335 = arith.constant 4 : index + %772 = arith.muli %c26_334, %c4_335 : index + %773 = arith.index_cast %772 : index to i64 + %774 = llvm.getelementptr %42[%773] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %775 = llvm.load %774 : !llvm.ptr -> f32 + %776 = arith.mulf %771, %775 : f32 + %777 = arith.addf %767, %776 : f32 + %c0_336 = arith.constant 0 : index + %c4_337 = arith.constant 4 : index + %778 = arith.muli %c0_336, %c4_337 : index + %779 = arith.index_cast %778 : index to i64 + %780 = llvm.getelementptr %43[%779] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %777, %780 : f32, !llvm.ptr + %c0_338 = arith.constant 0 : index + %c4_339 = arith.constant 4 : index + %781 = arith.muli %c0_338, %c4_339 : index + %782 = arith.index_cast %781 : index to i64 + %783 = llvm.getelementptr %43[%782] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %784 = llvm.load %783 : !llvm.ptr -> f32 + %c27 = arith.constant 27 : index + %c4_340 = arith.constant 4 : index + %785 = arith.muli %c27, %c4_340 : index + %786 = arith.index_cast %785 : index to i64 + %787 = llvm.getelementptr %42[%786] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %788 = llvm.load %787 : !llvm.ptr -> f32 + %c27_341 = arith.constant 27 : index + %c4_342 = arith.constant 4 : index + %789 = arith.muli %c27_341, %c4_342 : index + %790 = arith.index_cast %789 : index to i64 + %791 = llvm.getelementptr %42[%790] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %792 = llvm.load %791 : !llvm.ptr -> f32 + %793 = arith.mulf %788, %792 : f32 + %794 = arith.addf %784, %793 : f32 + %c0_343 = arith.constant 0 : index + %c4_344 = arith.constant 4 : index + %795 = arith.muli %c0_343, %c4_344 : index + %796 = arith.index_cast %795 : index to i64 + %797 = llvm.getelementptr %43[%796] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %794, %797 : f32, !llvm.ptr + %c0_345 = arith.constant 0 : index + %c4_346 = arith.constant 4 : index + %798 = arith.muli %c0_345, %c4_346 : index + %799 = arith.index_cast %798 : index to i64 + %800 = llvm.getelementptr %43[%799] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %801 = llvm.load %800 : !llvm.ptr -> f32 + %802 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c39936 = arith.constant 39936 : index + %803 = pto.addptr %802, %c39936 : -> + %cst_347 = arith.constant 0.000000e+00 : f32 + %804 = pto.get_tid_x : i32 + %c32_i32_348 = arith.constant 32 : i32 + %805 = arith.floordivsi %804, %c32_i32_348 : i32 + %806 = pto.get_laneid : i32 + %807 = pto.redux_add %801 : f32 -> f32 + %c1_i32_349 = arith.constant 1 : i32 + %808 = arith.cmpi slt, %806, %c1_i32_349 : i32 + scf.if %808 { + %c1_i32_543 = arith.constant 1 : i32 + %1223 = arith.muli %805, %c1_i32_543 : i32 + %1224 = arith.addi %1223, %806 : i32 + %1225 = arith.index_cast %1224 : i32 to index + pto.store %807, %803[%1225] : !pto.ptr, f32 + } + pto.syncthreads + %c32_i32_350 = arith.constant 32 : i32 + %809 = arith.cmpi slt, %804, %c32_i32_350 : i32 + %810 = scf.if %809 -> (f32) { + %c8_i32 = arith.constant 8 : i32 + %1223 = arith.cmpi slt, %806, %c8_i32 : i32 + %1224 = arith.index_cast %806 : i32 to index + %1225 = pto.load %803[%1224] : !pto.ptr -> f32 + %1226 = arith.select %1223, %1225, %cst_347 : f32 + %1227 = pto.redux_add %1226 : f32 -> f32 + scf.yield %1227 : f32 + } else { + scf.yield %cst_347 : f32 + } + %c1_i32_351 = arith.constant 1 : i32 + %811 = arith.cmpi slt, %804, %c1_i32_351 : i32 + scf.if %811 { + %1223 = arith.index_cast %804 : i32 to index + pto.store %810, %803[%1223] : !pto.ptr, f32 + } + pto.syncthreads + %c1_i32_352 = arith.constant 1 : i32 + %812 = arith.remsi %804, %c1_i32_352 : i32 + %813 = arith.index_cast %812 : i32 to index + %814 = pto.load %803[%813] : !pto.ptr -> f32 + pto.syncthreads + %c0_353 = arith.constant 0 : index + %c4_354 = arith.constant 4 : index + %815 = arith.muli %c0_353, %c4_354 : index + %816 = arith.index_cast %815 : index to i64 + %817 = llvm.getelementptr %43[%816] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + llvm.store %814, %817 : f32, !llvm.ptr + %c0_355 = arith.constant 0 : index + %c4_356 = arith.constant 4 : index + %818 = arith.muli %c0_355, %c4_356 : index + %819 = arith.index_cast %818 : index to i64 + %820 = llvm.getelementptr %43[%819] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %821 = llvm.load %820 : !llvm.ptr -> f32 + %cst_357 = arith.constant 7.168000e+03 : f32 + %822 = arith.divf %821, %cst_357 : f32 + %823 = arith.addf %822, %arg4 : f32 + %824 = pto.sqrt %823 : f32 -> f32 + %cst_358 = arith.constant 1.000000e+00 : f32 + %825 = arith.divf %cst_358, %824 : f32 + %826 = pto.sqrt %823 : f32 -> f32 + %cst_359 = arith.constant 1.000000e+00 : f32 + %827 = arith.divf %cst_359, %826 : f32 + %828 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_360 = arith.constant 1 : index + %829 = arith.andi %arg5, %c1_360 : index + %c8_361 = arith.constant 8 : index + %830 = arith.muli %829, %c8_361 : index + pto.store %827, %828[%830] : !pto.ptr, f32 + %c0_362 = arith.constant 0 : index + %c4_363 = arith.constant 4 : index + %831 = arith.muli %c0_362, %c4_363 : index + %832 = arith.index_cast %831 : index to i64 + %833 = llvm.getelementptr %42[%832] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %834 = llvm.load %833 : !llvm.ptr -> vector<2xf32> + %835 = pto.sqrt %823 : f32 -> f32 + %cst_364 = arith.constant 1.000000e+00 : f32 + %836 = arith.divf %cst_364, %835 : f32 + %837 = llvm.mlir.undef : vector<2xf32> + %c0_i32 = arith.constant 0 : i32 + %838 = llvm.insertelement %836, %837[%c0_i32 : i32] : vector<2xf32> + %c1_i32_365 = arith.constant 1 : i32 + %839 = llvm.insertelement %836, %838[%c1_i32_365 : i32] : vector<2xf32> + %840 = arith.mulf %834, %839 : vector<2xf32> + %c0_366 = arith.constant 0 : index + %c4_367 = arith.constant 4 : index + %841 = arith.muli %c0_366, %c4_367 : index + %842 = arith.index_cast %841 : index to i64 + %843 = llvm.getelementptr %1[%842] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %844 = llvm.load %843 : !llvm.ptr -> vector<2xf32> + %845 = arith.mulf %840, %844 : vector<2xf32> + %846 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_368 = arith.constant 1 : index + %847 = arith.andi %arg5, %c1_368 : index + %c8192_369 = arith.constant 8192 : index + %848 = arith.muli %847, %c8192_369 : index + %c0_370 = arith.constant 0 : index + %849 = arith.addi %848, %c0_370 : index + %c2_i32_371 = arith.constant 2 : i32 + %850 = arith.muli %44, %c2_i32_371 : i32 + %851 = arith.index_cast %850 : i32 to index + %852 = arith.addi %849, %851 : index + %c23552_372 = arith.constant 23552 : index + %853 = arith.addi %852, %c23552_372 : index + %c4_373 = arith.constant 4 : index + %854 = arith.muli %853, %c4_373 : index + %855 = arith.index_cast %854 : index to i64 + %856 = pto.castptr %846 : !pto.ptr -> i64 + %857 = llvm.inttoptr %856 : i64 to !llvm.ptr<6> + %858 = llvm.getelementptr %857[%855] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %845, %858 : vector<2xf32>, !llvm.ptr<6> + %c2_374 = arith.constant 2 : index + %c4_375 = arith.constant 4 : index + %859 = arith.muli %c2_374, %c4_375 : index + %860 = arith.index_cast %859 : index to i64 + %861 = llvm.getelementptr %42[%860] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %862 = llvm.load %861 : !llvm.ptr -> vector<2xf32> + %863 = pto.sqrt %823 : f32 -> f32 + %cst_376 = arith.constant 1.000000e+00 : f32 + %864 = arith.divf %cst_376, %863 : f32 + %865 = llvm.mlir.undef : vector<2xf32> + %c0_i32_377 = arith.constant 0 : i32 + %866 = llvm.insertelement %864, %865[%c0_i32_377 : i32] : vector<2xf32> + %c1_i32_378 = arith.constant 1 : i32 + %867 = llvm.insertelement %864, %866[%c1_i32_378 : i32] : vector<2xf32> + %868 = arith.mulf %862, %867 : vector<2xf32> + %c2_379 = arith.constant 2 : index + %c4_380 = arith.constant 4 : index + %869 = arith.muli %c2_379, %c4_380 : index + %870 = arith.index_cast %869 : index to i64 + %871 = llvm.getelementptr %1[%870] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %872 = llvm.load %871 : !llvm.ptr -> vector<2xf32> + %873 = arith.mulf %868, %872 : vector<2xf32> + %874 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_381 = arith.constant 1 : index + %875 = arith.andi %arg5, %c1_381 : index + %c8192_382 = arith.constant 8192 : index + %876 = arith.muli %875, %c8192_382 : index + %c512_383 = arith.constant 512 : index + %877 = arith.addi %876, %c512_383 : index + %c2_i32_384 = arith.constant 2 : i32 + %878 = arith.muli %44, %c2_i32_384 : i32 + %879 = arith.index_cast %878 : i32 to index + %880 = arith.addi %877, %879 : index + %c23552_385 = arith.constant 23552 : index + %881 = arith.addi %880, %c23552_385 : index + %c4_386 = arith.constant 4 : index + %882 = arith.muli %881, %c4_386 : index + %883 = arith.index_cast %882 : index to i64 + %884 = pto.castptr %874 : !pto.ptr -> i64 + %885 = llvm.inttoptr %884 : i64 to !llvm.ptr<6> + %886 = llvm.getelementptr %885[%883] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %873, %886 : vector<2xf32>, !llvm.ptr<6> + %c4_387 = arith.constant 4 : index + %c4_388 = arith.constant 4 : index + %887 = arith.muli %c4_387, %c4_388 : index + %888 = arith.index_cast %887 : index to i64 + %889 = llvm.getelementptr %42[%888] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %890 = llvm.load %889 : !llvm.ptr -> vector<2xf32> + %891 = pto.sqrt %823 : f32 -> f32 + %cst_389 = arith.constant 1.000000e+00 : f32 + %892 = arith.divf %cst_389, %891 : f32 + %893 = llvm.mlir.undef : vector<2xf32> + %c0_i32_390 = arith.constant 0 : i32 + %894 = llvm.insertelement %892, %893[%c0_i32_390 : i32] : vector<2xf32> + %c1_i32_391 = arith.constant 1 : i32 + %895 = llvm.insertelement %892, %894[%c1_i32_391 : i32] : vector<2xf32> + %896 = arith.mulf %890, %895 : vector<2xf32> + %c4_392 = arith.constant 4 : index + %c4_393 = arith.constant 4 : index + %897 = arith.muli %c4_392, %c4_393 : index + %898 = arith.index_cast %897 : index to i64 + %899 = llvm.getelementptr %1[%898] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %900 = llvm.load %899 : !llvm.ptr -> vector<2xf32> + %901 = arith.mulf %896, %900 : vector<2xf32> + %902 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_394 = arith.constant 1 : index + %903 = arith.andi %arg5, %c1_394 : index + %c8192_395 = arith.constant 8192 : index + %904 = arith.muli %903, %c8192_395 : index + %c1024_396 = arith.constant 1024 : index + %905 = arith.addi %904, %c1024_396 : index + %c2_i32_397 = arith.constant 2 : i32 + %906 = arith.muli %44, %c2_i32_397 : i32 + %907 = arith.index_cast %906 : i32 to index + %908 = arith.addi %905, %907 : index + %c23552_398 = arith.constant 23552 : index + %909 = arith.addi %908, %c23552_398 : index + %c4_399 = arith.constant 4 : index + %910 = arith.muli %909, %c4_399 : index + %911 = arith.index_cast %910 : index to i64 + %912 = pto.castptr %902 : !pto.ptr -> i64 + %913 = llvm.inttoptr %912 : i64 to !llvm.ptr<6> + %914 = llvm.getelementptr %913[%911] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %901, %914 : vector<2xf32>, !llvm.ptr<6> + %c6_400 = arith.constant 6 : index + %c4_401 = arith.constant 4 : index + %915 = arith.muli %c6_400, %c4_401 : index + %916 = arith.index_cast %915 : index to i64 + %917 = llvm.getelementptr %42[%916] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %918 = llvm.load %917 : !llvm.ptr -> vector<2xf32> + %919 = pto.sqrt %823 : f32 -> f32 + %cst_402 = arith.constant 1.000000e+00 : f32 + %920 = arith.divf %cst_402, %919 : f32 + %921 = llvm.mlir.undef : vector<2xf32> + %c0_i32_403 = arith.constant 0 : i32 + %922 = llvm.insertelement %920, %921[%c0_i32_403 : i32] : vector<2xf32> + %c1_i32_404 = arith.constant 1 : i32 + %923 = llvm.insertelement %920, %922[%c1_i32_404 : i32] : vector<2xf32> + %924 = arith.mulf %918, %923 : vector<2xf32> + %c6_405 = arith.constant 6 : index + %c4_406 = arith.constant 4 : index + %925 = arith.muli %c6_405, %c4_406 : index + %926 = arith.index_cast %925 : index to i64 + %927 = llvm.getelementptr %1[%926] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %928 = llvm.load %927 : !llvm.ptr -> vector<2xf32> + %929 = arith.mulf %924, %928 : vector<2xf32> + %930 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_407 = arith.constant 1 : index + %931 = arith.andi %arg5, %c1_407 : index + %c8192_408 = arith.constant 8192 : index + %932 = arith.muli %931, %c8192_408 : index + %c1536_409 = arith.constant 1536 : index + %933 = arith.addi %932, %c1536_409 : index + %c2_i32_410 = arith.constant 2 : i32 + %934 = arith.muli %44, %c2_i32_410 : i32 + %935 = arith.index_cast %934 : i32 to index + %936 = arith.addi %933, %935 : index + %c23552_411 = arith.constant 23552 : index + %937 = arith.addi %936, %c23552_411 : index + %c4_412 = arith.constant 4 : index + %938 = arith.muli %937, %c4_412 : index + %939 = arith.index_cast %938 : index to i64 + %940 = pto.castptr %930 : !pto.ptr -> i64 + %941 = llvm.inttoptr %940 : i64 to !llvm.ptr<6> + %942 = llvm.getelementptr %941[%939] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %929, %942 : vector<2xf32>, !llvm.ptr<6> + %c8_413 = arith.constant 8 : index + %c4_414 = arith.constant 4 : index + %943 = arith.muli %c8_413, %c4_414 : index + %944 = arith.index_cast %943 : index to i64 + %945 = llvm.getelementptr %42[%944] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %946 = llvm.load %945 : !llvm.ptr -> vector<2xf32> + %947 = pto.sqrt %823 : f32 -> f32 + %cst_415 = arith.constant 1.000000e+00 : f32 + %948 = arith.divf %cst_415, %947 : f32 + %949 = llvm.mlir.undef : vector<2xf32> + %c0_i32_416 = arith.constant 0 : i32 + %950 = llvm.insertelement %948, %949[%c0_i32_416 : i32] : vector<2xf32> + %c1_i32_417 = arith.constant 1 : i32 + %951 = llvm.insertelement %948, %950[%c1_i32_417 : i32] : vector<2xf32> + %952 = arith.mulf %946, %951 : vector<2xf32> + %c8_418 = arith.constant 8 : index + %c4_419 = arith.constant 4 : index + %953 = arith.muli %c8_418, %c4_419 : index + %954 = arith.index_cast %953 : index to i64 + %955 = llvm.getelementptr %1[%954] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %956 = llvm.load %955 : !llvm.ptr -> vector<2xf32> + %957 = arith.mulf %952, %956 : vector<2xf32> + %958 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_420 = arith.constant 1 : index + %959 = arith.andi %arg5, %c1_420 : index + %c8192_421 = arith.constant 8192 : index + %960 = arith.muli %959, %c8192_421 : index + %c2048_422 = arith.constant 2048 : index + %961 = arith.addi %960, %c2048_422 : index + %c2_i32_423 = arith.constant 2 : i32 + %962 = arith.muli %44, %c2_i32_423 : i32 + %963 = arith.index_cast %962 : i32 to index + %964 = arith.addi %961, %963 : index + %c23552_424 = arith.constant 23552 : index + %965 = arith.addi %964, %c23552_424 : index + %c4_425 = arith.constant 4 : index + %966 = arith.muli %965, %c4_425 : index + %967 = arith.index_cast %966 : index to i64 + %968 = pto.castptr %958 : !pto.ptr -> i64 + %969 = llvm.inttoptr %968 : i64 to !llvm.ptr<6> + %970 = llvm.getelementptr %969[%967] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %957, %970 : vector<2xf32>, !llvm.ptr<6> + %c10_426 = arith.constant 10 : index + %c4_427 = arith.constant 4 : index + %971 = arith.muli %c10_426, %c4_427 : index + %972 = arith.index_cast %971 : index to i64 + %973 = llvm.getelementptr %42[%972] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %974 = llvm.load %973 : !llvm.ptr -> vector<2xf32> + %975 = pto.sqrt %823 : f32 -> f32 + %cst_428 = arith.constant 1.000000e+00 : f32 + %976 = arith.divf %cst_428, %975 : f32 + %977 = llvm.mlir.undef : vector<2xf32> + %c0_i32_429 = arith.constant 0 : i32 + %978 = llvm.insertelement %976, %977[%c0_i32_429 : i32] : vector<2xf32> + %c1_i32_430 = arith.constant 1 : i32 + %979 = llvm.insertelement %976, %978[%c1_i32_430 : i32] : vector<2xf32> + %980 = arith.mulf %974, %979 : vector<2xf32> + %c10_431 = arith.constant 10 : index + %c4_432 = arith.constant 4 : index + %981 = arith.muli %c10_431, %c4_432 : index + %982 = arith.index_cast %981 : index to i64 + %983 = llvm.getelementptr %1[%982] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %984 = llvm.load %983 : !llvm.ptr -> vector<2xf32> + %985 = arith.mulf %980, %984 : vector<2xf32> + %986 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_433 = arith.constant 1 : index + %987 = arith.andi %arg5, %c1_433 : index + %c8192_434 = arith.constant 8192 : index + %988 = arith.muli %987, %c8192_434 : index + %c2560_435 = arith.constant 2560 : index + %989 = arith.addi %988, %c2560_435 : index + %c2_i32_436 = arith.constant 2 : i32 + %990 = arith.muli %44, %c2_i32_436 : i32 + %991 = arith.index_cast %990 : i32 to index + %992 = arith.addi %989, %991 : index + %c23552_437 = arith.constant 23552 : index + %993 = arith.addi %992, %c23552_437 : index + %c4_438 = arith.constant 4 : index + %994 = arith.muli %993, %c4_438 : index + %995 = arith.index_cast %994 : index to i64 + %996 = pto.castptr %986 : !pto.ptr -> i64 + %997 = llvm.inttoptr %996 : i64 to !llvm.ptr<6> + %998 = llvm.getelementptr %997[%995] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %985, %998 : vector<2xf32>, !llvm.ptr<6> + %c12_439 = arith.constant 12 : index + %c4_440 = arith.constant 4 : index + %999 = arith.muli %c12_439, %c4_440 : index + %1000 = arith.index_cast %999 : index to i64 + %1001 = llvm.getelementptr %42[%1000] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1002 = llvm.load %1001 : !llvm.ptr -> vector<2xf32> + %1003 = pto.sqrt %823 : f32 -> f32 + %cst_441 = arith.constant 1.000000e+00 : f32 + %1004 = arith.divf %cst_441, %1003 : f32 + %1005 = llvm.mlir.undef : vector<2xf32> + %c0_i32_442 = arith.constant 0 : i32 + %1006 = llvm.insertelement %1004, %1005[%c0_i32_442 : i32] : vector<2xf32> + %c1_i32_443 = arith.constant 1 : i32 + %1007 = llvm.insertelement %1004, %1006[%c1_i32_443 : i32] : vector<2xf32> + %1008 = arith.mulf %1002, %1007 : vector<2xf32> + %c12_444 = arith.constant 12 : index + %c4_445 = arith.constant 4 : index + %1009 = arith.muli %c12_444, %c4_445 : index + %1010 = arith.index_cast %1009 : index to i64 + %1011 = llvm.getelementptr %1[%1010] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1012 = llvm.load %1011 : !llvm.ptr -> vector<2xf32> + %1013 = arith.mulf %1008, %1012 : vector<2xf32> + %1014 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_446 = arith.constant 1 : index + %1015 = arith.andi %arg5, %c1_446 : index + %c8192_447 = arith.constant 8192 : index + %1016 = arith.muli %1015, %c8192_447 : index + %c3072_448 = arith.constant 3072 : index + %1017 = arith.addi %1016, %c3072_448 : index + %c2_i32_449 = arith.constant 2 : i32 + %1018 = arith.muli %44, %c2_i32_449 : i32 + %1019 = arith.index_cast %1018 : i32 to index + %1020 = arith.addi %1017, %1019 : index + %c23552_450 = arith.constant 23552 : index + %1021 = arith.addi %1020, %c23552_450 : index + %c4_451 = arith.constant 4 : index + %1022 = arith.muli %1021, %c4_451 : index + %1023 = arith.index_cast %1022 : index to i64 + %1024 = pto.castptr %1014 : !pto.ptr -> i64 + %1025 = llvm.inttoptr %1024 : i64 to !llvm.ptr<6> + %1026 = llvm.getelementptr %1025[%1023] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1013, %1026 : vector<2xf32>, !llvm.ptr<6> + %c14_452 = arith.constant 14 : index + %c4_453 = arith.constant 4 : index + %1027 = arith.muli %c14_452, %c4_453 : index + %1028 = arith.index_cast %1027 : index to i64 + %1029 = llvm.getelementptr %42[%1028] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1030 = llvm.load %1029 : !llvm.ptr -> vector<2xf32> + %1031 = pto.sqrt %823 : f32 -> f32 + %cst_454 = arith.constant 1.000000e+00 : f32 + %1032 = arith.divf %cst_454, %1031 : f32 + %1033 = llvm.mlir.undef : vector<2xf32> + %c0_i32_455 = arith.constant 0 : i32 + %1034 = llvm.insertelement %1032, %1033[%c0_i32_455 : i32] : vector<2xf32> + %c1_i32_456 = arith.constant 1 : i32 + %1035 = llvm.insertelement %1032, %1034[%c1_i32_456 : i32] : vector<2xf32> + %1036 = arith.mulf %1030, %1035 : vector<2xf32> + %c14_457 = arith.constant 14 : index + %c4_458 = arith.constant 4 : index + %1037 = arith.muli %c14_457, %c4_458 : index + %1038 = arith.index_cast %1037 : index to i64 + %1039 = llvm.getelementptr %1[%1038] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1040 = llvm.load %1039 : !llvm.ptr -> vector<2xf32> + %1041 = arith.mulf %1036, %1040 : vector<2xf32> + %1042 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_459 = arith.constant 1 : index + %1043 = arith.andi %arg5, %c1_459 : index + %c8192_460 = arith.constant 8192 : index + %1044 = arith.muli %1043, %c8192_460 : index + %c3584_461 = arith.constant 3584 : index + %1045 = arith.addi %1044, %c3584_461 : index + %c2_i32_462 = arith.constant 2 : i32 + %1046 = arith.muli %44, %c2_i32_462 : i32 + %1047 = arith.index_cast %1046 : i32 to index + %1048 = arith.addi %1045, %1047 : index + %c23552_463 = arith.constant 23552 : index + %1049 = arith.addi %1048, %c23552_463 : index + %c4_464 = arith.constant 4 : index + %1050 = arith.muli %1049, %c4_464 : index + %1051 = arith.index_cast %1050 : index to i64 + %1052 = pto.castptr %1042 : !pto.ptr -> i64 + %1053 = llvm.inttoptr %1052 : i64 to !llvm.ptr<6> + %1054 = llvm.getelementptr %1053[%1051] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1041, %1054 : vector<2xf32>, !llvm.ptr<6> + %c16_465 = arith.constant 16 : index + %c4_466 = arith.constant 4 : index + %1055 = arith.muli %c16_465, %c4_466 : index + %1056 = arith.index_cast %1055 : index to i64 + %1057 = llvm.getelementptr %42[%1056] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1058 = llvm.load %1057 : !llvm.ptr -> vector<2xf32> + %1059 = pto.sqrt %823 : f32 -> f32 + %cst_467 = arith.constant 1.000000e+00 : f32 + %1060 = arith.divf %cst_467, %1059 : f32 + %1061 = llvm.mlir.undef : vector<2xf32> + %c0_i32_468 = arith.constant 0 : i32 + %1062 = llvm.insertelement %1060, %1061[%c0_i32_468 : i32] : vector<2xf32> + %c1_i32_469 = arith.constant 1 : i32 + %1063 = llvm.insertelement %1060, %1062[%c1_i32_469 : i32] : vector<2xf32> + %1064 = arith.mulf %1058, %1063 : vector<2xf32> + %c16_470 = arith.constant 16 : index + %c4_471 = arith.constant 4 : index + %1065 = arith.muli %c16_470, %c4_471 : index + %1066 = arith.index_cast %1065 : index to i64 + %1067 = llvm.getelementptr %1[%1066] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1068 = llvm.load %1067 : !llvm.ptr -> vector<2xf32> + %1069 = arith.mulf %1064, %1068 : vector<2xf32> + %1070 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_472 = arith.constant 1 : index + %1071 = arith.andi %arg5, %c1_472 : index + %c8192_473 = arith.constant 8192 : index + %1072 = arith.muli %1071, %c8192_473 : index + %c4096_474 = arith.constant 4096 : index + %1073 = arith.addi %1072, %c4096_474 : index + %c2_i32_475 = arith.constant 2 : i32 + %1074 = arith.muli %44, %c2_i32_475 : i32 + %1075 = arith.index_cast %1074 : i32 to index + %1076 = arith.addi %1073, %1075 : index + %c23552_476 = arith.constant 23552 : index + %1077 = arith.addi %1076, %c23552_476 : index + %c4_477 = arith.constant 4 : index + %1078 = arith.muli %1077, %c4_477 : index + %1079 = arith.index_cast %1078 : index to i64 + %1080 = pto.castptr %1070 : !pto.ptr -> i64 + %1081 = llvm.inttoptr %1080 : i64 to !llvm.ptr<6> + %1082 = llvm.getelementptr %1081[%1079] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1069, %1082 : vector<2xf32>, !llvm.ptr<6> + %c18_478 = arith.constant 18 : index + %c4_479 = arith.constant 4 : index + %1083 = arith.muli %c18_478, %c4_479 : index + %1084 = arith.index_cast %1083 : index to i64 + %1085 = llvm.getelementptr %42[%1084] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1086 = llvm.load %1085 : !llvm.ptr -> vector<2xf32> + %1087 = pto.sqrt %823 : f32 -> f32 + %cst_480 = arith.constant 1.000000e+00 : f32 + %1088 = arith.divf %cst_480, %1087 : f32 + %1089 = llvm.mlir.undef : vector<2xf32> + %c0_i32_481 = arith.constant 0 : i32 + %1090 = llvm.insertelement %1088, %1089[%c0_i32_481 : i32] : vector<2xf32> + %c1_i32_482 = arith.constant 1 : i32 + %1091 = llvm.insertelement %1088, %1090[%c1_i32_482 : i32] : vector<2xf32> + %1092 = arith.mulf %1086, %1091 : vector<2xf32> + %c18_483 = arith.constant 18 : index + %c4_484 = arith.constant 4 : index + %1093 = arith.muli %c18_483, %c4_484 : index + %1094 = arith.index_cast %1093 : index to i64 + %1095 = llvm.getelementptr %1[%1094] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1096 = llvm.load %1095 : !llvm.ptr -> vector<2xf32> + %1097 = arith.mulf %1092, %1096 : vector<2xf32> + %1098 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_485 = arith.constant 1 : index + %1099 = arith.andi %arg5, %c1_485 : index + %c8192_486 = arith.constant 8192 : index + %1100 = arith.muli %1099, %c8192_486 : index + %c4608_487 = arith.constant 4608 : index + %1101 = arith.addi %1100, %c4608_487 : index + %c2_i32_488 = arith.constant 2 : i32 + %1102 = arith.muli %44, %c2_i32_488 : i32 + %1103 = arith.index_cast %1102 : i32 to index + %1104 = arith.addi %1101, %1103 : index + %c23552_489 = arith.constant 23552 : index + %1105 = arith.addi %1104, %c23552_489 : index + %c4_490 = arith.constant 4 : index + %1106 = arith.muli %1105, %c4_490 : index + %1107 = arith.index_cast %1106 : index to i64 + %1108 = pto.castptr %1098 : !pto.ptr -> i64 + %1109 = llvm.inttoptr %1108 : i64 to !llvm.ptr<6> + %1110 = llvm.getelementptr %1109[%1107] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1097, %1110 : vector<2xf32>, !llvm.ptr<6> + %c20_491 = arith.constant 20 : index + %c4_492 = arith.constant 4 : index + %1111 = arith.muli %c20_491, %c4_492 : index + %1112 = arith.index_cast %1111 : index to i64 + %1113 = llvm.getelementptr %42[%1112] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1114 = llvm.load %1113 : !llvm.ptr -> vector<2xf32> + %1115 = pto.sqrt %823 : f32 -> f32 + %cst_493 = arith.constant 1.000000e+00 : f32 + %1116 = arith.divf %cst_493, %1115 : f32 + %1117 = llvm.mlir.undef : vector<2xf32> + %c0_i32_494 = arith.constant 0 : i32 + %1118 = llvm.insertelement %1116, %1117[%c0_i32_494 : i32] : vector<2xf32> + %c1_i32_495 = arith.constant 1 : i32 + %1119 = llvm.insertelement %1116, %1118[%c1_i32_495 : i32] : vector<2xf32> + %1120 = arith.mulf %1114, %1119 : vector<2xf32> + %c20_496 = arith.constant 20 : index + %c4_497 = arith.constant 4 : index + %1121 = arith.muli %c20_496, %c4_497 : index + %1122 = arith.index_cast %1121 : index to i64 + %1123 = llvm.getelementptr %1[%1122] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1124 = llvm.load %1123 : !llvm.ptr -> vector<2xf32> + %1125 = arith.mulf %1120, %1124 : vector<2xf32> + %1126 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_498 = arith.constant 1 : index + %1127 = arith.andi %arg5, %c1_498 : index + %c8192_499 = arith.constant 8192 : index + %1128 = arith.muli %1127, %c8192_499 : index + %c5120_500 = arith.constant 5120 : index + %1129 = arith.addi %1128, %c5120_500 : index + %c2_i32_501 = arith.constant 2 : i32 + %1130 = arith.muli %44, %c2_i32_501 : i32 + %1131 = arith.index_cast %1130 : i32 to index + %1132 = arith.addi %1129, %1131 : index + %c23552_502 = arith.constant 23552 : index + %1133 = arith.addi %1132, %c23552_502 : index + %c4_503 = arith.constant 4 : index + %1134 = arith.muli %1133, %c4_503 : index + %1135 = arith.index_cast %1134 : index to i64 + %1136 = pto.castptr %1126 : !pto.ptr -> i64 + %1137 = llvm.inttoptr %1136 : i64 to !llvm.ptr<6> + %1138 = llvm.getelementptr %1137[%1135] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1125, %1138 : vector<2xf32>, !llvm.ptr<6> + %c22_504 = arith.constant 22 : index + %c4_505 = arith.constant 4 : index + %1139 = arith.muli %c22_504, %c4_505 : index + %1140 = arith.index_cast %1139 : index to i64 + %1141 = llvm.getelementptr %42[%1140] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1142 = llvm.load %1141 : !llvm.ptr -> vector<2xf32> + %1143 = pto.sqrt %823 : f32 -> f32 + %cst_506 = arith.constant 1.000000e+00 : f32 + %1144 = arith.divf %cst_506, %1143 : f32 + %1145 = llvm.mlir.undef : vector<2xf32> + %c0_i32_507 = arith.constant 0 : i32 + %1146 = llvm.insertelement %1144, %1145[%c0_i32_507 : i32] : vector<2xf32> + %c1_i32_508 = arith.constant 1 : i32 + %1147 = llvm.insertelement %1144, %1146[%c1_i32_508 : i32] : vector<2xf32> + %1148 = arith.mulf %1142, %1147 : vector<2xf32> + %c22_509 = arith.constant 22 : index + %c4_510 = arith.constant 4 : index + %1149 = arith.muli %c22_509, %c4_510 : index + %1150 = arith.index_cast %1149 : index to i64 + %1151 = llvm.getelementptr %1[%1150] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1152 = llvm.load %1151 : !llvm.ptr -> vector<2xf32> + %1153 = arith.mulf %1148, %1152 : vector<2xf32> + %1154 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_511 = arith.constant 1 : index + %1155 = arith.andi %arg5, %c1_511 : index + %c8192_512 = arith.constant 8192 : index + %1156 = arith.muli %1155, %c8192_512 : index + %c5632_513 = arith.constant 5632 : index + %1157 = arith.addi %1156, %c5632_513 : index + %c2_i32_514 = arith.constant 2 : i32 + %1158 = arith.muli %44, %c2_i32_514 : i32 + %1159 = arith.index_cast %1158 : i32 to index + %1160 = arith.addi %1157, %1159 : index + %c23552_515 = arith.constant 23552 : index + %1161 = arith.addi %1160, %c23552_515 : index + %c4_516 = arith.constant 4 : index + %1162 = arith.muli %1161, %c4_516 : index + %1163 = arith.index_cast %1162 : index to i64 + %1164 = pto.castptr %1154 : !pto.ptr -> i64 + %1165 = llvm.inttoptr %1164 : i64 to !llvm.ptr<6> + %1166 = llvm.getelementptr %1165[%1163] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1153, %1166 : vector<2xf32>, !llvm.ptr<6> + %c24_517 = arith.constant 24 : index + %c4_518 = arith.constant 4 : index + %1167 = arith.muli %c24_517, %c4_518 : index + %1168 = arith.index_cast %1167 : index to i64 + %1169 = llvm.getelementptr %42[%1168] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1170 = llvm.load %1169 : !llvm.ptr -> vector<2xf32> + %1171 = pto.sqrt %823 : f32 -> f32 + %cst_519 = arith.constant 1.000000e+00 : f32 + %1172 = arith.divf %cst_519, %1171 : f32 + %1173 = llvm.mlir.undef : vector<2xf32> + %c0_i32_520 = arith.constant 0 : i32 + %1174 = llvm.insertelement %1172, %1173[%c0_i32_520 : i32] : vector<2xf32> + %c1_i32_521 = arith.constant 1 : i32 + %1175 = llvm.insertelement %1172, %1174[%c1_i32_521 : i32] : vector<2xf32> + %1176 = arith.mulf %1170, %1175 : vector<2xf32> + %c24_522 = arith.constant 24 : index + %c4_523 = arith.constant 4 : index + %1177 = arith.muli %c24_522, %c4_523 : index + %1178 = arith.index_cast %1177 : index to i64 + %1179 = llvm.getelementptr %1[%1178] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1180 = llvm.load %1179 : !llvm.ptr -> vector<2xf32> + %1181 = arith.mulf %1176, %1180 : vector<2xf32> + %1182 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_524 = arith.constant 1 : index + %1183 = arith.andi %arg5, %c1_524 : index + %c8192_525 = arith.constant 8192 : index + %1184 = arith.muli %1183, %c8192_525 : index + %c6144_526 = arith.constant 6144 : index + %1185 = arith.addi %1184, %c6144_526 : index + %c2_i32_527 = arith.constant 2 : i32 + %1186 = arith.muli %44, %c2_i32_527 : i32 + %1187 = arith.index_cast %1186 : i32 to index + %1188 = arith.addi %1185, %1187 : index + %c23552_528 = arith.constant 23552 : index + %1189 = arith.addi %1188, %c23552_528 : index + %c4_529 = arith.constant 4 : index + %1190 = arith.muli %1189, %c4_529 : index + %1191 = arith.index_cast %1190 : index to i64 + %1192 = pto.castptr %1182 : !pto.ptr -> i64 + %1193 = llvm.inttoptr %1192 : i64 to !llvm.ptr<6> + %1194 = llvm.getelementptr %1193[%1191] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1181, %1194 : vector<2xf32>, !llvm.ptr<6> + %c26_530 = arith.constant 26 : index + %c4_531 = arith.constant 4 : index + %1195 = arith.muli %c26_530, %c4_531 : index + %1196 = arith.index_cast %1195 : index to i64 + %1197 = llvm.getelementptr %42[%1196] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1198 = llvm.load %1197 : !llvm.ptr -> vector<2xf32> + %1199 = pto.sqrt %823 : f32 -> f32 + %cst_532 = arith.constant 1.000000e+00 : f32 + %1200 = arith.divf %cst_532, %1199 : f32 + %1201 = llvm.mlir.undef : vector<2xf32> + %c0_i32_533 = arith.constant 0 : i32 + %1202 = llvm.insertelement %1200, %1201[%c0_i32_533 : i32] : vector<2xf32> + %c1_i32_534 = arith.constant 1 : i32 + %1203 = llvm.insertelement %1200, %1202[%c1_i32_534 : i32] : vector<2xf32> + %1204 = arith.mulf %1198, %1203 : vector<2xf32> + %c26_535 = arith.constant 26 : index + %c4_536 = arith.constant 4 : index + %1205 = arith.muli %c26_535, %c4_536 : index + %1206 = arith.index_cast %1205 : index to i64 + %1207 = llvm.getelementptr %1[%1206] : (!llvm.ptr, i64) -> !llvm.ptr, i8 + %1208 = llvm.load %1207 : !llvm.ptr -> vector<2xf32> + %1209 = arith.mulf %1204, %1208 : vector<2xf32> + %1210 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_537 = arith.constant 1 : index + %1211 = arith.andi %arg5, %c1_537 : index + %c8192_538 = arith.constant 8192 : index + %1212 = arith.muli %1211, %c8192_538 : index + %c6656_539 = arith.constant 6656 : index + %1213 = arith.addi %1212, %c6656_539 : index + %c2_i32_540 = arith.constant 2 : i32 + %1214 = arith.muli %44, %c2_i32_540 : i32 + %1215 = arith.index_cast %1214 : i32 to index + %1216 = arith.addi %1213, %1215 : index + %c23552_541 = arith.constant 23552 : index + %1217 = arith.addi %1216, %c23552_541 : index + %c4_542 = arith.constant 4 : index + %1218 = arith.muli %1217, %c4_542 : index + %1219 = arith.index_cast %1218 : index to i64 + %1220 = pto.castptr %1210 : !pto.ptr -> i64 + %1221 = llvm.inttoptr %1220 : i64 to !llvm.ptr<6> + %1222 = llvm.getelementptr %1221[%1219] : (!llvm.ptr<6>, i64) -> !llvm.ptr<6>, i8 + llvm.store %1209, %1222 : vector<2xf32>, !llvm.ptr<6> + } + %c1_13 = arith.constant 1 : index + %18 = arith.andi %arg5, %c1_13 : index + pto.set_flag_dyn[, , %18] + %c1_14 = arith.constant 1 : index + %19 = arith.andi %arg5, %c1_14 : index + pto.set_flag_dyn[, , %19] + %c1_15 = arith.constant 1 : index + %20 = arith.andi %arg5, %c1_15 : index + pto.wait_flag_dyn[, , %20] + %21 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_16 = arith.constant 1 : index + %22 = arith.andi %arg5, %c1_16 : index + %c8192_17 = arith.constant 8192 : index + %23 = arith.muli %22, %c8192_17 : index + %c23552 = arith.constant 23552 : index + %24 = arith.addi %23, %c23552 : index + %25 = pto.addptr %21, %24 : -> + %c458752_18 = arith.constant 458752 : index + %26 = arith.muli %arg5, %c458752_18 : index + %27 = pto.get_block_idx + %c7168_i64_19 = arith.constant 7168 : i64 + %28 = arith.muli %27, %c7168_i64_19 : i64 + %29 = arith.index_cast %28 : i64 to index + %30 = arith.addi %26, %29 : index + %31 = pto.addptr %arg3, %30 : -> + %c1_i64_20 = arith.constant 1 : i64 + %c28672_i64_21 = arith.constant 28672 : i64 + %c28672_i64_22 = arith.constant 28672 : i64 + %c28672_i64_23 = arith.constant 28672 : i64 + %c0_i64_24 = arith.constant 0 : i64 + pto.mte_ub_gm %25, %31, %c28672_i64_23 nburst(%c1_i64_20, %c28672_i64_21, %c28672_i64_22) l2_cache_ctl(%c0_i64_24) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + %32 = pto.castptr %0 : !pto.ptr -> !pto.ptr + %c1_25 = arith.constant 1 : index + %33 = arith.andi %arg5, %c1_25 : index + %c8 = arith.constant 8 : index + %34 = arith.muli %33, %c8 : index + %35 = pto.addptr %32, %34 : -> + %c64_26 = arith.constant 64 : index + %36 = arith.muli %arg5, %c64_26 : index + %37 = pto.get_block_idx + %38 = arith.index_cast %37 : i64 to index + %39 = arith.addi %36, %38 : index + %40 = pto.addptr %arg0, %39 : -> + %c1_i64_27 = arith.constant 1 : i64 + %c4_i64 = arith.constant 4 : i64 + %c4_i64_28 = arith.constant 4 : i64 + %c4_i64_29 = arith.constant 4 : i64 + %c0_i64_30 = arith.constant 0 : i64 + pto.mte_ub_gm %35, %40, %c4_i64_29 nburst(%c1_i64_27, %c4_i64, %c4_i64_28) l2_cache_ctl(%c0_i64_30) {operandSegmentSizes = array} : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64 + %c1_31 = arith.constant 1 : index + %41 = arith.andi %arg5, %c1_31 : index + pto.set_flag_dyn[, , %41] + } + pto.wait_flag[, , ] + pto.wait_flag[, , ] + pto.wait_flag[, , ] + pto.wait_flag[, , ] + return + } + } +} diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/kernel.ptodsl.py b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/kernel.ptodsl.py new file mode 100644 index 0000000000..750ad27fe0 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/kernel.ptodsl.py @@ -0,0 +1,58 @@ +from ptodsl import pto, scalar +from ptodsl._ops import _coerce_i64 as _tl_coerce_i64 +from ptodsl._surface_values import wrap_surface_value as _tl_wrap_surface_value + +@pto.jit(name="rmsnorm_d7168_kernel", kernel_kind="vector", target="a5", mode="explicit") +def rmsnorm_d7168_kernel(RSTD: pto.ptr(pto.f32, "gm"), W: pto.ptr(pto.f32, "gm"), X: pto.ptr(pto.f32, "gm"), Y: pto.ptr(pto.f32, "gm"), eps: pto.f32): + buf_dyn_shmem = pto.castptr(pto.const(0, dtype=pto.i64), pto.ptr(pto.i8, "ub")) + w_frag = [None] * 32 + w_frag_1 = pto.alloc_buffer((32,), pto.f32) + pto.set_flag("MTE3", "V", event_id=0) + pto.set_flag("MTE3", "V", event_id=1) + pto.set_flag("V", "MTE2", event_id=0) + pto.set_flag("V", "MTE2", event_id=1) + pto.mte_gm_ub(W, pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), 0, 28672, nburst=(1, 28672, 28672)) + pto.set_flag("MTE2", "V", event_id=2) + pto.wait_flag("MTE2", "V", event_id=2) + with pto.simt(256, 1, 1): + simtvf_tx = pto.get_tid_x() + simtvf_ty = pto.get_tid_y() + simtvf_tz = pto.get_tid_z() + for i in pto.static_range(0, 16): + if i < 14: + scalar.store(scalar.load(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), (i * 512) + (simtvf_tx * 2), contiguous=2), w_frag_1, i * 2) + with pto.for_(0, 64, step=1) as t: + pto.wait_flag("V", "MTE2", event_id=t & 1) + pto.mte_gm_ub(pto.addptr(X, (t * 458752) + (pto.get_block_idx() * 7168)), pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((t & 1) * 8192) + 7168), 0, 28672, nburst=(1, 28672, 28672)) + pto.set_flag("MTE2", "V", event_id=t & 1) + pto.wait_flag("MTE3", "V", event_id=t & 1) + pto.wait_flag("MTE2", "V", event_id=t & 1) + with pto.simt(256, 1, 1): + x_frag = pto.alloc_buffer((32,), pto.f32) + sum_sq = pto.alloc_buffer((1,), pto.f32) + simtvf_tx_1 = pto.get_tid_x() + simtvf_ty_1 = pto.get_tid_y() + simtvf_tz_1 = pto.get_tid_z() + for i_1 in pto.static_range(0, 16): + scalar.store(scalar.load(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((((t & 1) * 8192) + (i_1 * 512)) + (simtvf_tx_1 * 2)) + 7168, contiguous=2), x_frag, i_1 * 2) + scalar.store(float.fromhex('0x0p+0'), sum_sq, 0) + for i_2 in pto.static_range(0, 32): + if i_2 < 28: + scalar.store(scalar.load(sum_sq, 0) + (scalar.load(x_frag, i_2) * scalar.load(x_frag, i_2)), sum_sq, 0) + scalar.store(pto.simt_allreduce_sum(scalar.load(sum_sq, 0), threads=256, scale=1, thread_offset=0, scratch=pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), 39936)), sum_sq, 0) + var = (scalar.load(sum_sq, 0) / float.fromhex('0x1.cp+12')) + eps + rstd_val = float.fromhex('0x1p+0') / pto.sqrt(var) + scalar.store(float.fromhex('0x1p+0') / pto.sqrt(var), pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), (t & 1) * 8) + for i_3 in pto.static_range(0, 16): + if i_3 < 14: + scalar.store((scalar.load(x_frag, i_3 * 2, contiguous=2) * pto.Vec(pto.f32, 2, init=(float.fromhex('0x1p+0') / pto.sqrt(var)))) * scalar.load(w_frag_1, i_3 * 2, contiguous=2), pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((((t & 1) * 8192) + (i_3 * 512)) + (simtvf_tx_1 * 2)) + 23552) + pto.set_flag("V", "MTE3", event_id=t & 1) + pto.set_flag("V", "MTE2", event_id=t & 1) + pto.wait_flag("V", "MTE3", event_id=t & 1) + pto.mte_ub_gm(pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), ((t & 1) * 8192) + 23552), pto.addptr(Y, (t * 458752) + (pto.get_block_idx() * 7168)), 28672, nburst=(1, 28672, 28672)) + pto.mte_ub_gm(pto.addptr(pto.castptr(buf_dyn_shmem, pto.ptr(pto.f32, "ub")), (t & 1) * 8), pto.addptr(RSTD, (t * 64) + pto.get_block_idx()), 4, nburst=(1, 4, 4)) + pto.set_flag("MTE3", "V", event_id=t & 1) + pto.wait_flag("MTE3", "V", event_id=0) + pto.wait_flag("MTE3", "V", event_id=1) + pto.wait_flag("V", "MTE2", event_id=0) + pto.wait_flag("V", "MTE2", event_id=1) diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/kernel_vpto.ll b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/kernel_vpto.ll new file mode 100644 index 0000000000..ed0f888f5a --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/kernel_vpto.ll @@ -0,0 +1,799 @@ +; ModuleID = 'ptoas.hivm.official.vector' +source_filename = "ptoas.hivm.official.vector" + +; Unknown intrinsic +declare void @llvm.hivm.SET.FLAG.IMM(i64, i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.MOV.OUT.TO.UB.ALIGN.V2.f32.DV(ptr addrspace(6), ptr addrspace(1), i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.WAIT.FLAG.IMM(i64, i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.store.vfsimt.info(i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.WAIT.FLAG.REG(i64, i64, i64) #0 + +; Unknown intrinsic +declare i64 @llvm.hivm.GET.BLOCK.IDX() #0 + +; Unknown intrinsic +declare void @llvm.hivm.SET.FLAG.REG(i64, i64, i64) #0 + +; Unknown intrinsic +declare void @llvm.hivm.MOV.UB.TO.OUT.ALIGN.V2.DV(ptr addrspace(1), ptr addrspace(6), i64, i64) #0 + +; Unknown intrinsic +declare i32 @llvm.hivm.get.TID.X() #0 + +; Unknown intrinsic +declare i32 @llvm.hivm.get.laneID() #0 + +; Unknown intrinsic +declare float @llvm.hivm.redux.add.f32(float) #0 + +; Unknown intrinsic +declare void @llvm.hivm.sync.workitems() #0 + +; Function Attrs: nocallback nofree nosync nounwind speculatable willreturn memory(none) +declare float @llvm.sqrt.f32(float) #1 + +define void @rmsnorm_d7168_kernel_mix_aiv(ptr addrspace(1) %0, ptr addrspace(1) %1, ptr addrspace(1) %2, ptr addrspace(1) %3, float %4) #2 { + call void @llvm.hivm.SET.FLAG.IMM(i64 5, i64 1, i64 0) + call void @llvm.hivm.SET.FLAG.IMM(i64 5, i64 1, i64 1) + call void @llvm.hivm.SET.FLAG.IMM(i64 1, i64 4, i64 0) + call void @llvm.hivm.SET.FLAG.IMM(i64 1, i64 4, i64 1) + call void @llvm.hivm.MOV.OUT.TO.UB.ALIGN.V2.f32.DV(ptr addrspace(6) null, ptr addrspace(1) %1, i64 962072674320, i64 31525197391622144) + call void @llvm.hivm.SET.FLAG.IMM(i64 4, i64 1, i64 2) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 4, i64 1, i64 2) + call void @llvm.hivm.store.vfsimt.info(i64 4295033088) + call simt_entry void @rmsnorm_d7168_kernel_simt_0(ptr addrspace(6) null) + br label %6 + +6: ; preds = %9, %5 + %7 = phi i64 [ %27, %9 ], [ 0, %5 ] + %8 = icmp slt i64 %7, 64 + br i1 %8, label %9, label %28 + +9: ; preds = %6 + %10 = and i64 %7, 1 + call void @llvm.hivm.WAIT.FLAG.REG(i64 1, i64 4, i64 %10) + %11 = mul i64 %7, 458752 + %12 = call i64 @llvm.hivm.GET.BLOCK.IDX() + %13 = mul i64 %12, 7168 + %14 = add i64 %11, %13 + %15 = getelementptr float, ptr addrspace(1) %2, i64 %14 + %16 = mul i64 %10, 8192 + %17 = add i64 %16, 7168 + %18 = getelementptr float, ptr addrspace(6) null, i64 %17 + call void @llvm.hivm.MOV.OUT.TO.UB.ALIGN.V2.f32.DV(ptr addrspace(6) %18, ptr addrspace(1) %15, i64 962072674320, i64 31525197391622144) + call void @llvm.hivm.SET.FLAG.REG(i64 4, i64 1, i64 %10) + call void @llvm.hivm.WAIT.FLAG.REG(i64 5, i64 1, i64 %10) + call void @llvm.hivm.WAIT.FLAG.REG(i64 4, i64 1, i64 %10) + call void @llvm.hivm.store.vfsimt.info(i64 4295033088) + call simt_entry void @rmsnorm_d7168_kernel_simt_1(i64 %16, ptr addrspace(6) null, i64 %17, float %4, i64 %10) + call void @llvm.hivm.SET.FLAG.REG(i64 1, i64 5, i64 %10) + call void @llvm.hivm.SET.FLAG.REG(i64 1, i64 4, i64 %10) + call void @llvm.hivm.WAIT.FLAG.REG(i64 1, i64 5, i64 %10) + %19 = add i64 %16, 23552 + %20 = getelementptr float, ptr addrspace(6) null, i64 %19 + %21 = getelementptr float, ptr addrspace(1) %3, i64 %14 + call void @llvm.hivm.MOV.UB.TO.OUT.ALIGN.V2.DV(ptr addrspace(1) %21, ptr addrspace(6) %20, i64 962072674320, i64 31525197391622144) + %22 = mul i64 %10, 8 + %23 = getelementptr float, ptr addrspace(6) null, i64 %22 + %24 = mul i64 %7, 64 + %25 = add i64 %24, %12 + %26 = getelementptr float, ptr addrspace(1) %0, i64 %25 + call void @llvm.hivm.MOV.UB.TO.OUT.ALIGN.V2.DV(ptr addrspace(1) %26, ptr addrspace(6) %23, i64 134217744, i64 4398046511108) + call void @llvm.hivm.SET.FLAG.REG(i64 5, i64 1, i64 %10) + %27 = add i64 %7, 1 + br label %6 + +28: ; preds = %6 + call void @llvm.hivm.WAIT.FLAG.IMM(i64 5, i64 1, i64 0) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 5, i64 1, i64 1) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 1, i64 4, i64 0) + call void @llvm.hivm.WAIT.FLAG.IMM(i64 1, i64 4, i64 1) + ret void +} + +; Function Attrs: noinline +define linkonce_odr simt_entry void @rmsnorm_d7168_kernel_simt_0(ptr addrspace(6) %0) #3 !annotation !8 !annotation !9 { + %2 = call i32 @llvm.hivm.get.TID.X() + %3 = mul i32 %2, 2 + %4 = sext i32 %3 to i64 + %5 = mul i64 %4, 4 + %6 = ptrtoint ptr addrspace(6) %0 to i64 + %7 = inttoptr i64 %6 to ptr addrspace(6) + %8 = getelementptr i8, ptr addrspace(6) %7, i64 %5 + %9 = load <2 x float>, ptr addrspace(6) %8, align 8 + %10 = extractelement <2 x float> %9, i32 0 + %11 = extractelement <2 x float> %9, i32 1 + %12 = add i32 %3, 512 + %13 = sext i32 %12 to i64 + %14 = mul i64 %13, 4 + %15 = getelementptr i8, ptr addrspace(6) %7, i64 %14 + %16 = load <2 x float>, ptr addrspace(6) %15, align 8 + %17 = extractelement <2 x float> %16, i32 0 + %18 = extractelement <2 x float> %16, i32 1 + %19 = add i32 %3, 1024 + %20 = sext i32 %19 to i64 + %21 = mul i64 %20, 4 + %22 = getelementptr i8, ptr addrspace(6) %7, i64 %21 + %23 = load <2 x float>, ptr addrspace(6) %22, align 8 + %24 = extractelement <2 x float> %23, i32 0 + %25 = extractelement <2 x float> %23, i32 1 + %26 = add i32 %3, 1536 + %27 = sext i32 %26 to i64 + %28 = mul i64 %27, 4 + %29 = getelementptr i8, ptr addrspace(6) %7, i64 %28 + %30 = load <2 x float>, ptr addrspace(6) %29, align 8 + %31 = extractelement <2 x float> %30, i32 0 + %32 = extractelement <2 x float> %30, i32 1 + %33 = add i32 %3, 2048 + %34 = sext i32 %33 to i64 + %35 = mul i64 %34, 4 + %36 = getelementptr i8, ptr addrspace(6) %7, i64 %35 + %37 = load <2 x float>, ptr addrspace(6) %36, align 8 + %38 = extractelement <2 x float> %37, i32 0 + %39 = extractelement <2 x float> %37, i32 1 + %40 = add i32 %3, 2560 + %41 = sext i32 %40 to i64 + %42 = mul i64 %41, 4 + %43 = getelementptr i8, ptr addrspace(6) %7, i64 %42 + %44 = load <2 x float>, ptr addrspace(6) %43, align 8 + %45 = extractelement <2 x float> %44, i32 0 + %46 = extractelement <2 x float> %44, i32 1 + %47 = add i32 %3, 3072 + %48 = sext i32 %47 to i64 + %49 = mul i64 %48, 4 + %50 = getelementptr i8, ptr addrspace(6) %7, i64 %49 + %51 = load <2 x float>, ptr addrspace(6) %50, align 8 + %52 = extractelement <2 x float> %51, i32 0 + %53 = extractelement <2 x float> %51, i32 1 + %54 = add i32 %3, 3584 + %55 = sext i32 %54 to i64 + %56 = mul i64 %55, 4 + %57 = getelementptr i8, ptr addrspace(6) %7, i64 %56 + %58 = load <2 x float>, ptr addrspace(6) %57, align 8 + %59 = extractelement <2 x float> %58, i32 0 + %60 = extractelement <2 x float> %58, i32 1 + %61 = add i32 %3, 4096 + %62 = sext i32 %61 to i64 + %63 = mul i64 %62, 4 + %64 = getelementptr i8, ptr addrspace(6) %7, i64 %63 + %65 = load <2 x float>, ptr addrspace(6) %64, align 8 + %66 = extractelement <2 x float> %65, i32 0 + %67 = extractelement <2 x float> %65, i32 1 + %68 = add i32 %3, 4608 + %69 = sext i32 %68 to i64 + %70 = mul i64 %69, 4 + %71 = getelementptr i8, ptr addrspace(6) %7, i64 %70 + %72 = load <2 x float>, ptr addrspace(6) %71, align 8 + %73 = extractelement <2 x float> %72, i32 0 + %74 = extractelement <2 x float> %72, i32 1 + %75 = add i32 %3, 5120 + %76 = sext i32 %75 to i64 + %77 = mul i64 %76, 4 + %78 = getelementptr i8, ptr addrspace(6) %7, i64 %77 + %79 = load <2 x float>, ptr addrspace(6) %78, align 8 + %80 = extractelement <2 x float> %79, i32 0 + %81 = extractelement <2 x float> %79, i32 1 + %82 = add i32 %3, 5632 + %83 = sext i32 %82 to i64 + %84 = mul i64 %83, 4 + %85 = getelementptr i8, ptr addrspace(6) %7, i64 %84 + %86 = load <2 x float>, ptr addrspace(6) %85, align 8 + %87 = extractelement <2 x float> %86, i32 0 + %88 = extractelement <2 x float> %86, i32 1 + %89 = add i32 %3, 6144 + %90 = sext i32 %89 to i64 + %91 = mul i64 %90, 4 + %92 = getelementptr i8, ptr addrspace(6) %7, i64 %91 + %93 = load <2 x float>, ptr addrspace(6) %92, align 8 + %94 = extractelement <2 x float> %93, i32 0 + %95 = extractelement <2 x float> %93, i32 1 + %96 = add i32 %3, 6656 + %97 = sext i32 %96 to i64 + %98 = mul i64 %97, 4 + %99 = getelementptr i8, ptr addrspace(6) %7, i64 %98 + %100 = load <2 x float>, ptr addrspace(6) %99, align 8 + %101 = extractelement <2 x float> %100, i32 0 + %102 = extractelement <2 x float> %100, i32 1 + %103 = bitcast float %10 to i32 + %104 = bitcast float %11 to i32 + %105 = bitcast float %17 to i32 + %106 = bitcast float %18 to i32 + %107 = bitcast float %24 to i32 + %108 = bitcast float %25 to i32 + %109 = bitcast float %31 to i32 + %110 = bitcast float %32 to i32 + %111 = bitcast float %38 to i32 + %112 = bitcast float %39 to i32 + %113 = bitcast float %45 to i32 + %114 = bitcast float %46 to i32 + %115 = bitcast float %52 to i32 + %116 = bitcast float %53 to i32 + %117 = bitcast float %59 to i32 + %118 = bitcast float %60 to i32 + %119 = bitcast float %66 to i32 + %120 = bitcast float %67 to i32 + %121 = bitcast float %73 to i32 + %122 = bitcast float %74 to i32 + %123 = bitcast float %80 to i32 + %124 = bitcast float %81 to i32 + %125 = bitcast float %87 to i32 + %126 = bitcast float %88 to i32 + %127 = bitcast float %94 to i32 + %128 = bitcast float %95 to i32 + %129 = bitcast float %101 to i32 + %130 = bitcast float %102 to i32 + %131 = call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "", "={TPER4},={TPER5},={TPER6},={TPER7},={TPER8},={TPER9},={TPER10},={TPER11},={TPER12},={TPER13},={TPER14},={TPER15},={TPER16},={TPER17},={TPER18},={TPER19},={TPER20},={TPER21},={TPER22},={TPER23},={TPER24},={TPER25},={TPER26},={TPER27},={TPER28},={TPER29},={TPER30},={TPER31},0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27"(i32 %103, i32 %104, i32 %105, i32 %106, i32 %107, i32 %108, i32 %109, i32 %110, i32 %111, i32 %112, i32 %113, i32 %114, i32 %115, i32 %116, i32 %117, i32 %118, i32 %119, i32 %120, i32 %121, i32 %122, i32 %123, i32 %124, i32 %125, i32 %126, i32 %127, i32 %128, i32 %129, i32 %130) + ret void +} + +; Function Attrs: noinline +define linkonce_odr simt_entry void @rmsnorm_d7168_kernel_simt_1(i64 %0, ptr addrspace(6) %1, i64 %2, float %3, i64 %4) #3 !annotation !8 !annotation !9 { + %6 = call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "", "={TPER4},={TPER5},={TPER6},={TPER7},={TPER8},={TPER9},={TPER10},={TPER11},={TPER12},={TPER13},={TPER14},={TPER15},={TPER16},={TPER17},={TPER18},={TPER19},={TPER20},={TPER21},={TPER22},={TPER23},={TPER24},={TPER25},={TPER26},={TPER27},={TPER28},={TPER29},={TPER30},={TPER31}"() + %7 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 0 + %8 = bitcast i32 %7 to float + %9 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 1 + %10 = bitcast i32 %9 to float + %11 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 2 + %12 = bitcast i32 %11 to float + %13 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 3 + %14 = bitcast i32 %13 to float + %15 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 4 + %16 = bitcast i32 %15 to float + %17 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 5 + %18 = bitcast i32 %17 to float + %19 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 6 + %20 = bitcast i32 %19 to float + %21 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 7 + %22 = bitcast i32 %21 to float + %23 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 8 + %24 = bitcast i32 %23 to float + %25 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 9 + %26 = bitcast i32 %25 to float + %27 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 10 + %28 = bitcast i32 %27 to float + %29 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 11 + %30 = bitcast i32 %29 to float + %31 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 12 + %32 = bitcast i32 %31 to float + %33 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 13 + %34 = bitcast i32 %33 to float + %35 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 14 + %36 = bitcast i32 %35 to float + %37 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 15 + %38 = bitcast i32 %37 to float + %39 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 16 + %40 = bitcast i32 %39 to float + %41 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 17 + %42 = bitcast i32 %41 to float + %43 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 18 + %44 = bitcast i32 %43 to float + %45 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 19 + %46 = bitcast i32 %45 to float + %47 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 20 + %48 = bitcast i32 %47 to float + %49 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 21 + %50 = bitcast i32 %49 to float + %51 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 22 + %52 = bitcast i32 %51 to float + %53 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 23 + %54 = bitcast i32 %53 to float + %55 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 24 + %56 = bitcast i32 %55 to float + %57 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 25 + %58 = bitcast i32 %57 to float + %59 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 26 + %60 = bitcast i32 %59 to float + %61 = extractvalue { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } %6, 27 + %62 = bitcast i32 %61 to float + %63 = alloca float, i32 32, align 4 + %64 = alloca float, align 4 + %65 = call i32 @llvm.hivm.get.TID.X() + %66 = mul i32 %65, 2 + %67 = sext i32 %66 to i64 + %68 = add i64 %0, %67 + %69 = add i64 %68, 7168 + %70 = mul i64 %69, 4 + %71 = ptrtoint ptr addrspace(6) %1 to i64 + %72 = inttoptr i64 %71 to ptr addrspace(6) + %73 = getelementptr i8, ptr addrspace(6) %72, i64 %70 + %74 = load <2 x float>, ptr addrspace(6) %73, align 8 + store <2 x float> %74, ptr %63, align 8 + %75 = add i64 %0, 512 + %76 = add i64 %75, %67 + %77 = add i64 %76, 7168 + %78 = mul i64 %77, 4 + %79 = getelementptr i8, ptr addrspace(6) %72, i64 %78 + %80 = load <2 x float>, ptr addrspace(6) %79, align 8 + %81 = getelementptr i8, ptr %63, i32 8 + store <2 x float> %80, ptr %81, align 8 + %82 = add i64 %0, 1024 + %83 = add i64 %82, %67 + %84 = add i64 %83, 7168 + %85 = mul i64 %84, 4 + %86 = getelementptr i8, ptr addrspace(6) %72, i64 %85 + %87 = load <2 x float>, ptr addrspace(6) %86, align 8 + %88 = getelementptr i8, ptr %63, i32 16 + store <2 x float> %87, ptr %88, align 8 + %89 = add i64 %0, 1536 + %90 = add i64 %89, %67 + %91 = add i64 %90, 7168 + %92 = mul i64 %91, 4 + %93 = getelementptr i8, ptr addrspace(6) %72, i64 %92 + %94 = load <2 x float>, ptr addrspace(6) %93, align 8 + %95 = getelementptr i8, ptr %63, i32 24 + store <2 x float> %94, ptr %95, align 8 + %96 = add i64 %0, 2048 + %97 = add i64 %96, %67 + %98 = add i64 %97, 7168 + %99 = mul i64 %98, 4 + %100 = getelementptr i8, ptr addrspace(6) %72, i64 %99 + %101 = load <2 x float>, ptr addrspace(6) %100, align 8 + %102 = getelementptr i8, ptr %63, i32 32 + store <2 x float> %101, ptr %102, align 8 + %103 = add i64 %0, 2560 + %104 = add i64 %103, %67 + %105 = add i64 %104, 7168 + %106 = mul i64 %105, 4 + %107 = getelementptr i8, ptr addrspace(6) %72, i64 %106 + %108 = load <2 x float>, ptr addrspace(6) %107, align 8 + %109 = getelementptr i8, ptr %63, i32 40 + store <2 x float> %108, ptr %109, align 8 + %110 = add i64 %0, 3072 + %111 = add i64 %110, %67 + %112 = add i64 %111, 7168 + %113 = mul i64 %112, 4 + %114 = getelementptr i8, ptr addrspace(6) %72, i64 %113 + %115 = load <2 x float>, ptr addrspace(6) %114, align 8 + %116 = getelementptr i8, ptr %63, i32 48 + store <2 x float> %115, ptr %116, align 8 + %117 = add i64 %0, 3584 + %118 = add i64 %117, %67 + %119 = add i64 %118, 7168 + %120 = mul i64 %119, 4 + %121 = getelementptr i8, ptr addrspace(6) %72, i64 %120 + %122 = load <2 x float>, ptr addrspace(6) %121, align 8 + %123 = getelementptr i8, ptr %63, i32 56 + store <2 x float> %122, ptr %123, align 8 + %124 = add i64 %0, 4096 + %125 = add i64 %124, %67 + %126 = add i64 %125, 7168 + %127 = mul i64 %126, 4 + %128 = getelementptr i8, ptr addrspace(6) %72, i64 %127 + %129 = load <2 x float>, ptr addrspace(6) %128, align 8 + %130 = getelementptr i8, ptr %63, i32 64 + store <2 x float> %129, ptr %130, align 8 + %131 = add i64 %0, 4608 + %132 = add i64 %131, %67 + %133 = add i64 %132, 7168 + %134 = mul i64 %133, 4 + %135 = getelementptr i8, ptr addrspace(6) %72, i64 %134 + %136 = load <2 x float>, ptr addrspace(6) %135, align 8 + %137 = getelementptr i8, ptr %63, i32 72 + store <2 x float> %136, ptr %137, align 8 + %138 = add i64 %0, 5120 + %139 = add i64 %138, %67 + %140 = add i64 %139, 7168 + %141 = mul i64 %140, 4 + %142 = getelementptr i8, ptr addrspace(6) %72, i64 %141 + %143 = load <2 x float>, ptr addrspace(6) %142, align 8 + %144 = getelementptr i8, ptr %63, i32 80 + store <2 x float> %143, ptr %144, align 8 + %145 = add i64 %0, 5632 + %146 = add i64 %145, %67 + %147 = add i64 %146, 7168 + %148 = mul i64 %147, 4 + %149 = getelementptr i8, ptr addrspace(6) %72, i64 %148 + %150 = load <2 x float>, ptr addrspace(6) %149, align 8 + %151 = getelementptr i8, ptr %63, i32 88 + store <2 x float> %150, ptr %151, align 8 + %152 = add i64 %0, 6144 + %153 = add i64 %152, %67 + %154 = add i64 %153, 7168 + %155 = mul i64 %154, 4 + %156 = getelementptr i8, ptr addrspace(6) %72, i64 %155 + %157 = load <2 x float>, ptr addrspace(6) %156, align 8 + %158 = getelementptr i8, ptr %63, i32 96 + store <2 x float> %157, ptr %158, align 8 + %159 = add i64 %0, 6656 + %160 = add i64 %159, %67 + %161 = add i64 %160, 7168 + %162 = mul i64 %161, 4 + %163 = getelementptr i8, ptr addrspace(6) %72, i64 %162 + %164 = load <2 x float>, ptr addrspace(6) %163, align 8 + %165 = getelementptr i8, ptr %63, i32 104 + store <2 x float> %164, ptr %165, align 8 + %166 = add i64 %2, %67 + %167 = add i64 %166, 7168 + %168 = mul i64 %167, 4 + %169 = getelementptr i8, ptr addrspace(6) %72, i64 %168 + %170 = load <2 x float>, ptr addrspace(6) %169, align 8 + %171 = getelementptr i8, ptr %63, i32 112 + store <2 x float> %170, ptr %171, align 8 + %172 = add i64 %0, 7680 + %173 = add i64 %172, %67 + %174 = add i64 %173, 7168 + %175 = mul i64 %174, 4 + %176 = getelementptr i8, ptr addrspace(6) %72, i64 %175 + %177 = load <2 x float>, ptr addrspace(6) %176, align 8 + %178 = getelementptr i8, ptr %63, i32 120 + store <2 x float> %177, ptr %178, align 8 + store float 0.000000e+00, ptr %64, align 4 + %179 = load float, ptr %64, align 4 + %180 = load float, ptr %63, align 4 + %181 = fmul float %180, %180 + %182 = fadd float %179, %181 + store float %182, ptr %64, align 4 + %183 = load float, ptr %64, align 4 + %184 = getelementptr i8, ptr %63, i32 4 + %185 = load float, ptr %184, align 4 + %186 = fmul float %185, %185 + %187 = fadd float %183, %186 + store float %187, ptr %64, align 4 + %188 = load float, ptr %64, align 4 + %189 = load float, ptr %81, align 4 + %190 = fmul float %189, %189 + %191 = fadd float %188, %190 + store float %191, ptr %64, align 4 + %192 = load float, ptr %64, align 4 + %193 = getelementptr i8, ptr %63, i32 12 + %194 = load float, ptr %193, align 4 + %195 = fmul float %194, %194 + %196 = fadd float %192, %195 + store float %196, ptr %64, align 4 + %197 = load float, ptr %64, align 4 + %198 = load float, ptr %88, align 4 + %199 = fmul float %198, %198 + %200 = fadd float %197, %199 + store float %200, ptr %64, align 4 + %201 = load float, ptr %64, align 4 + %202 = getelementptr i8, ptr %63, i32 20 + %203 = load float, ptr %202, align 4 + %204 = fmul float %203, %203 + %205 = fadd float %201, %204 + store float %205, ptr %64, align 4 + %206 = load float, ptr %64, align 4 + %207 = load float, ptr %95, align 4 + %208 = fmul float %207, %207 + %209 = fadd float %206, %208 + store float %209, ptr %64, align 4 + %210 = load float, ptr %64, align 4 + %211 = getelementptr i8, ptr %63, i32 28 + %212 = load float, ptr %211, align 4 + %213 = fmul float %212, %212 + %214 = fadd float %210, %213 + store float %214, ptr %64, align 4 + %215 = load float, ptr %64, align 4 + %216 = load float, ptr %102, align 4 + %217 = fmul float %216, %216 + %218 = fadd float %215, %217 + store float %218, ptr %64, align 4 + %219 = load float, ptr %64, align 4 + %220 = getelementptr i8, ptr %63, i32 36 + %221 = load float, ptr %220, align 4 + %222 = fmul float %221, %221 + %223 = fadd float %219, %222 + store float %223, ptr %64, align 4 + %224 = load float, ptr %64, align 4 + %225 = load float, ptr %109, align 4 + %226 = fmul float %225, %225 + %227 = fadd float %224, %226 + store float %227, ptr %64, align 4 + %228 = load float, ptr %64, align 4 + %229 = getelementptr i8, ptr %63, i32 44 + %230 = load float, ptr %229, align 4 + %231 = fmul float %230, %230 + %232 = fadd float %228, %231 + store float %232, ptr %64, align 4 + %233 = load float, ptr %64, align 4 + %234 = load float, ptr %116, align 4 + %235 = fmul float %234, %234 + %236 = fadd float %233, %235 + store float %236, ptr %64, align 4 + %237 = load float, ptr %64, align 4 + %238 = getelementptr i8, ptr %63, i32 52 + %239 = load float, ptr %238, align 4 + %240 = fmul float %239, %239 + %241 = fadd float %237, %240 + store float %241, ptr %64, align 4 + %242 = load float, ptr %64, align 4 + %243 = load float, ptr %123, align 4 + %244 = fmul float %243, %243 + %245 = fadd float %242, %244 + store float %245, ptr %64, align 4 + %246 = load float, ptr %64, align 4 + %247 = getelementptr i8, ptr %63, i32 60 + %248 = load float, ptr %247, align 4 + %249 = fmul float %248, %248 + %250 = fadd float %246, %249 + store float %250, ptr %64, align 4 + %251 = load float, ptr %64, align 4 + %252 = load float, ptr %130, align 4 + %253 = fmul float %252, %252 + %254 = fadd float %251, %253 + store float %254, ptr %64, align 4 + %255 = load float, ptr %64, align 4 + %256 = getelementptr i8, ptr %63, i32 68 + %257 = load float, ptr %256, align 4 + %258 = fmul float %257, %257 + %259 = fadd float %255, %258 + store float %259, ptr %64, align 4 + %260 = load float, ptr %64, align 4 + %261 = load float, ptr %137, align 4 + %262 = fmul float %261, %261 + %263 = fadd float %260, %262 + store float %263, ptr %64, align 4 + %264 = load float, ptr %64, align 4 + %265 = getelementptr i8, ptr %63, i32 76 + %266 = load float, ptr %265, align 4 + %267 = fmul float %266, %266 + %268 = fadd float %264, %267 + store float %268, ptr %64, align 4 + %269 = load float, ptr %64, align 4 + %270 = load float, ptr %144, align 4 + %271 = fmul float %270, %270 + %272 = fadd float %269, %271 + store float %272, ptr %64, align 4 + %273 = load float, ptr %64, align 4 + %274 = getelementptr i8, ptr %63, i32 84 + %275 = load float, ptr %274, align 4 + %276 = fmul float %275, %275 + %277 = fadd float %273, %276 + store float %277, ptr %64, align 4 + %278 = load float, ptr %64, align 4 + %279 = load float, ptr %151, align 4 + %280 = fmul float %279, %279 + %281 = fadd float %278, %280 + store float %281, ptr %64, align 4 + %282 = load float, ptr %64, align 4 + %283 = getelementptr i8, ptr %63, i32 92 + %284 = load float, ptr %283, align 4 + %285 = fmul float %284, %284 + %286 = fadd float %282, %285 + store float %286, ptr %64, align 4 + %287 = load float, ptr %64, align 4 + %288 = load float, ptr %158, align 4 + %289 = fmul float %288, %288 + %290 = fadd float %287, %289 + store float %290, ptr %64, align 4 + %291 = load float, ptr %64, align 4 + %292 = getelementptr i8, ptr %63, i32 100 + %293 = load float, ptr %292, align 4 + %294 = fmul float %293, %293 + %295 = fadd float %291, %294 + store float %295, ptr %64, align 4 + %296 = load float, ptr %64, align 4 + %297 = load float, ptr %165, align 4 + %298 = fmul float %297, %297 + %299 = fadd float %296, %298 + store float %299, ptr %64, align 4 + %300 = load float, ptr %64, align 4 + %301 = getelementptr i8, ptr %63, i32 108 + %302 = load float, ptr %301, align 4 + %303 = fmul float %302, %302 + %304 = fadd float %300, %303 + store float %304, ptr %64, align 4 + %305 = load float, ptr %64, align 4 + %306 = getelementptr float, ptr addrspace(6) %1, i64 39936 + %307 = sdiv i32 %65, 32 + %308 = mul i32 %307, 32 + %309 = icmp ne i32 %65, %308 + %310 = icmp slt i32 %65, 0 + %311 = icmp ne i1 %310, false + %312 = and i1 %309, %311 + %313 = add i32 %307, -1 + %314 = select i1 %312, i32 %313, i32 %307 + %315 = call i32 @llvm.hivm.get.laneID() + %316 = call float @llvm.hivm.redux.add.f32(float %305) + %317 = icmp slt i32 %315, 1 + br i1 %317, label %318, label %322 + +318: ; preds = %5 + %319 = add i32 %314, %315 + %320 = sext i32 %319 to i64 + %321 = getelementptr float, ptr addrspace(6) %306, i64 %320 + store float %316, ptr addrspace(6) %321, align 4 + br label %322 + +322: ; preds = %318, %5 + call void @llvm.hivm.sync.workitems() + %323 = icmp slt i32 %65, 32 + br i1 %323, label %324, label %331 + +324: ; preds = %322 + %325 = icmp slt i32 %315, 8 + %326 = sext i32 %315 to i64 + %327 = getelementptr float, ptr addrspace(6) %306, i64 %326 + %328 = load float, ptr addrspace(6) %327, align 4 + %329 = select i1 %325, float %328, float 0.000000e+00 + %330 = call float @llvm.hivm.redux.add.f32(float %329) + br label %332 + +331: ; preds = %322 + br label %332 + +332: ; preds = %324, %331 + %333 = phi float [ 0.000000e+00, %331 ], [ %330, %324 ] + br label %334 + +334: ; preds = %332 + %335 = icmp slt i32 %65, 1 + br i1 %335, label %336, label %339 + +336: ; preds = %334 + %337 = sext i32 %65 to i64 + %338 = getelementptr float, ptr addrspace(6) %306, i64 %337 + store float %333, ptr addrspace(6) %338, align 4 + br label %339 + +339: ; preds = %336, %334 + call void @llvm.hivm.sync.workitems() + %340 = getelementptr float, ptr addrspace(6) %306, i64 0 + %341 = load float, ptr addrspace(6) %340, align 4 + call void @llvm.hivm.sync.workitems() + store float %341, ptr %64, align 4 + %342 = load float, ptr %64, align 4 + %343 = fdiv float %342, 7.168000e+03 + %344 = fadd float %343, %3 + %345 = call float @llvm.sqrt.f32(float %344) + %346 = fdiv float 1.000000e+00, %345 + %347 = mul i64 %4, 8 + %348 = getelementptr float, ptr addrspace(6) %1, i64 %347 + store float %346, ptr addrspace(6) %348, align 4 + %349 = load <2 x float>, ptr %63, align 8 + %350 = insertelement <2 x float> undef, float %346, i32 0 + %351 = insertelement <2 x float> %350, float %346, i32 1 + %352 = fmul <2 x float> %349, %351 + %353 = insertelement <2 x float> poison, float %8, i32 0 + %354 = insertelement <2 x float> %353, float %10, i32 1 + %355 = fmul <2 x float> %352, %354 + %356 = add i64 %68, 23552 + %357 = mul i64 %356, 4 + %358 = getelementptr i8, ptr addrspace(6) %72, i64 %357 + store <2 x float> %355, ptr addrspace(6) %358, align 8 + %359 = load <2 x float>, ptr %81, align 8 + %360 = fmul <2 x float> %359, %351 + %361 = insertelement <2 x float> poison, float %12, i32 0 + %362 = insertelement <2 x float> %361, float %14, i32 1 + %363 = fmul <2 x float> %360, %362 + %364 = add i64 %76, 23552 + %365 = mul i64 %364, 4 + %366 = getelementptr i8, ptr addrspace(6) %72, i64 %365 + store <2 x float> %363, ptr addrspace(6) %366, align 8 + %367 = load <2 x float>, ptr %88, align 8 + %368 = fmul <2 x float> %367, %351 + %369 = insertelement <2 x float> poison, float %16, i32 0 + %370 = insertelement <2 x float> %369, float %18, i32 1 + %371 = fmul <2 x float> %368, %370 + %372 = add i64 %83, 23552 + %373 = mul i64 %372, 4 + %374 = getelementptr i8, ptr addrspace(6) %72, i64 %373 + store <2 x float> %371, ptr addrspace(6) %374, align 8 + %375 = load <2 x float>, ptr %95, align 8 + %376 = fmul <2 x float> %375, %351 + %377 = insertelement <2 x float> poison, float %20, i32 0 + %378 = insertelement <2 x float> %377, float %22, i32 1 + %379 = fmul <2 x float> %376, %378 + %380 = add i64 %90, 23552 + %381 = mul i64 %380, 4 + %382 = getelementptr i8, ptr addrspace(6) %72, i64 %381 + store <2 x float> %379, ptr addrspace(6) %382, align 8 + %383 = load <2 x float>, ptr %102, align 8 + %384 = fmul <2 x float> %383, %351 + %385 = insertelement <2 x float> poison, float %24, i32 0 + %386 = insertelement <2 x float> %385, float %26, i32 1 + %387 = fmul <2 x float> %384, %386 + %388 = add i64 %97, 23552 + %389 = mul i64 %388, 4 + %390 = getelementptr i8, ptr addrspace(6) %72, i64 %389 + store <2 x float> %387, ptr addrspace(6) %390, align 8 + %391 = load <2 x float>, ptr %109, align 8 + %392 = fmul <2 x float> %391, %351 + %393 = insertelement <2 x float> poison, float %28, i32 0 + %394 = insertelement <2 x float> %393, float %30, i32 1 + %395 = fmul <2 x float> %392, %394 + %396 = add i64 %104, 23552 + %397 = mul i64 %396, 4 + %398 = getelementptr i8, ptr addrspace(6) %72, i64 %397 + store <2 x float> %395, ptr addrspace(6) %398, align 8 + %399 = load <2 x float>, ptr %116, align 8 + %400 = fmul <2 x float> %399, %351 + %401 = insertelement <2 x float> poison, float %32, i32 0 + %402 = insertelement <2 x float> %401, float %34, i32 1 + %403 = fmul <2 x float> %400, %402 + %404 = add i64 %111, 23552 + %405 = mul i64 %404, 4 + %406 = getelementptr i8, ptr addrspace(6) %72, i64 %405 + store <2 x float> %403, ptr addrspace(6) %406, align 8 + %407 = load <2 x float>, ptr %123, align 8 + %408 = fmul <2 x float> %407, %351 + %409 = insertelement <2 x float> poison, float %36, i32 0 + %410 = insertelement <2 x float> %409, float %38, i32 1 + %411 = fmul <2 x float> %408, %410 + %412 = add i64 %118, 23552 + %413 = mul i64 %412, 4 + %414 = getelementptr i8, ptr addrspace(6) %72, i64 %413 + store <2 x float> %411, ptr addrspace(6) %414, align 8 + %415 = load <2 x float>, ptr %130, align 8 + %416 = fmul <2 x float> %415, %351 + %417 = insertelement <2 x float> poison, float %40, i32 0 + %418 = insertelement <2 x float> %417, float %42, i32 1 + %419 = fmul <2 x float> %416, %418 + %420 = add i64 %125, 23552 + %421 = mul i64 %420, 4 + %422 = getelementptr i8, ptr addrspace(6) %72, i64 %421 + store <2 x float> %419, ptr addrspace(6) %422, align 8 + %423 = load <2 x float>, ptr %137, align 8 + %424 = fmul <2 x float> %423, %351 + %425 = insertelement <2 x float> poison, float %44, i32 0 + %426 = insertelement <2 x float> %425, float %46, i32 1 + %427 = fmul <2 x float> %424, %426 + %428 = add i64 %132, 23552 + %429 = mul i64 %428, 4 + %430 = getelementptr i8, ptr addrspace(6) %72, i64 %429 + store <2 x float> %427, ptr addrspace(6) %430, align 8 + %431 = load <2 x float>, ptr %144, align 8 + %432 = fmul <2 x float> %431, %351 + %433 = insertelement <2 x float> poison, float %48, i32 0 + %434 = insertelement <2 x float> %433, float %50, i32 1 + %435 = fmul <2 x float> %432, %434 + %436 = add i64 %139, 23552 + %437 = mul i64 %436, 4 + %438 = getelementptr i8, ptr addrspace(6) %72, i64 %437 + store <2 x float> %435, ptr addrspace(6) %438, align 8 + %439 = load <2 x float>, ptr %151, align 8 + %440 = fmul <2 x float> %439, %351 + %441 = insertelement <2 x float> poison, float %52, i32 0 + %442 = insertelement <2 x float> %441, float %54, i32 1 + %443 = fmul <2 x float> %440, %442 + %444 = add i64 %146, 23552 + %445 = mul i64 %444, 4 + %446 = getelementptr i8, ptr addrspace(6) %72, i64 %445 + store <2 x float> %443, ptr addrspace(6) %446, align 8 + %447 = load <2 x float>, ptr %158, align 8 + %448 = fmul <2 x float> %447, %351 + %449 = insertelement <2 x float> poison, float %56, i32 0 + %450 = insertelement <2 x float> %449, float %58, i32 1 + %451 = fmul <2 x float> %448, %450 + %452 = add i64 %153, 23552 + %453 = mul i64 %452, 4 + %454 = getelementptr i8, ptr addrspace(6) %72, i64 %453 + store <2 x float> %451, ptr addrspace(6) %454, align 8 + %455 = load <2 x float>, ptr %165, align 8 + %456 = fmul <2 x float> %455, %351 + %457 = insertelement <2 x float> poison, float %60, i32 0 + %458 = insertelement <2 x float> %457, float %62, i32 1 + %459 = fmul <2 x float> %456, %458 + %460 = add i64 %160, 23552 + %461 = mul i64 %460, 4 + %462 = getelementptr i8, ptr addrspace(6) %72, i64 %461 + store <2 x float> %459, ptr addrspace(6) %462, align 8 + %463 = call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "", "={TPER4},={TPER5},={TPER6},={TPER7},={TPER8},={TPER9},={TPER10},={TPER11},={TPER12},={TPER13},={TPER14},={TPER15},={TPER16},={TPER17},={TPER18},={TPER19},={TPER20},={TPER21},={TPER22},={TPER23},={TPER24},={TPER25},={TPER26},={TPER27},={TPER28},={TPER29},={TPER30},={TPER31},0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27"(i32 %7, i32 %9, i32 %11, i32 %13, i32 %15, i32 %17, i32 %19, i32 %21, i32 %23, i32 %25, i32 %27, i32 %29, i32 %31, i32 %33, i32 %35, i32 %37, i32 %39, i32 %41, i32 %43, i32 %45, i32 %47, i32 %49, i32 %51, i32 %53, i32 %55, i32 %57, i32 %59, i32 %61) + ret void +} + +attributes #0 = { "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } +attributes #1 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } +attributes #2 = { "target-cpu"="dav-c310-vec" "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } +attributes #3 = { noinline "target-cpu"="dav-c310-vec" "target-features"="+ATOMIC,+ArchV130,+AregRedefinable,+ArithmeticBf16,+AtomicForB8 ,+F8e4m3,+F8e5m2,+F8e8m0,+FFTSBlk,+Fp4e1m2x2,+Fp4e2m1x2,+LDExtRefine,+MOVX8,+MSTX,+SPR7bits,+SyncV,+dav-c310-vec" } + +!llvm.module.flags = !{!0} +!hivm.annotations = !{!1, !2, !3, !4, !5, !6, !7} + +!0 = !{i32 2, !"Debug Info Version", i32 3} +!1 = !{ptr @rmsnorm_d7168_kernel_mix_aiv, !"kernel", i32 1} +!2 = !{ptr @rmsnorm_d7168_kernel_mix_aiv, !"kernel_with_simd", i32 1} +!3 = !{ptr @rmsnorm_d7168_kernel_mix_aiv, !"kernel_with_simt", i32 1} +!4 = distinct !{null, !"simt-max-threads", i32 256} +!5 = distinct !{null, !"simt-max-registers", i32 128} +!6 = distinct !{null, !"simt-max-threads", i32 256} +!7 = distinct !{null, !"simt-max-registers", i32 128} +!8 = !{!"simt-max-threads", i32 256} +!9 = !{!"simt-max-registers", i32 128} diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/launch.cpp b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/launch.cpp new file mode 100644 index 0000000000..4e18d4bb57 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/kernels/d7168/launch.cpp @@ -0,0 +1,12 @@ +#ifndef AICORE +#define AICORE [aicore] +#endif +extern "C" __global__ AICORE void rmsnorm_d7168_kernel( + __gm__ void*, __gm__ void*, __gm__ void*, __gm__ void*, float); +extern "C" int call( + float* X, float* Y, float* W, float* RSTD, float eps, void* stream) { + rmsnorm_d7168_kernel<<<64, 160768, stream>>>( + (__gm__ float*)RSTD, (__gm__ float*)W, (__gm__ float*)X, + (__gm__ float*)Y, eps); + return 0; +} diff --git a/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/run.sh b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/run.sh new file mode 100755 index 0000000000..93a183d1d0 --- /dev/null +++ b/test/vpto/cases/onboard-only/rmsnorm-persistent-fragment-sequence-repro/run.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# Copyright (c) 2026 Huawei Technologies Co., Ltd. +# This program is free software, you can redistribute it and/or modify it under the terms and conditions of +# CANN Open Software License Agreement Version 2.0 (the "License"). +# Please refer to the License for details. You may not use this file except in compliance with the License. +# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +# See LICENSE in the root of the software repository for the full text of the License. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +BUILD_DIR="${BUILD_DIR:-${SCRIPT_DIR}/build}" +CANN_HOME="${CANN_HOME:-/home/qukelin/tools/CANN_9.1/cann-9.1.T530}" +RUNNER="${BUILD_DIR}/sequence_runner" + +[[ -x "${RUNNER}" ]] || { + echo "ERROR: missing ${RUNNER}; run ./build.sh first" >&2 + exit 2 +} +for d in 4096 5120 7168; do + [[ -f "${BUILD_DIR}/d${d}/libkernel.so" ]] || { + echo "ERROR: missing ${BUILD_DIR}/d${d}/libkernel.so; run ./build.sh first" >&2 + exit 2 + } +done + +set +u +source "${CANN_HOME}/set_env.sh" >/dev/null 2>&1 +set -u + +export ASCEND_RT_VISIBLE_DEVICES="${ASCEND_RT_VISIBLE_DEVICES:-0}" +export ACL_DEVICE_ID="${ACL_DEVICE_ID:-0}" +export LD_LIBRARY_PATH="${BUILD_DIR}/d4096:${BUILD_DIR}/d5120:${BUILD_DIR}/d7168:${CANN_HOME}/lib64:${LD_LIBRARY_PATH:-}" + +exec "${RUNNER}" \ + "${BUILD_DIR}/d4096/libkernel.so" \ + "${BUILD_DIR}/d5120/libkernel.so" \ + "${BUILD_DIR}/d7168/libkernel.so" diff --git a/tools/ptoas/ptoas.cpp b/tools/ptoas/ptoas.cpp index 1f6f3a43fe..0041cbe743 100644 --- a/tools/ptoas/ptoas.cpp +++ b/tools/ptoas/ptoas.cpp @@ -2710,6 +2710,11 @@ static void prepareVPTOForEmission(PassManager &pm) { kernelModulePM.addPass(createSCCPPass()); kernelModulePM.addPass(createCanonicalizerPass()); kernelModulePM.addPass(createCSEPass()); + kernelModulePM.addNestedPass( + pto::createPTOAnalyzeSIMTPersistentFragmentPass()); + kernelModulePM.addNestedPass( + pto::createPTOMaterializeSIMTPersistentFragmentPass()); + kernelModulePM.addPass(pto::createPTOOutlineSIMTSectionsPass()); kernelModulePM.addPass(pto::createVPTOPtrNormalizePass()); kernelModulePM.addPass(pto::createVPTOPtrCastCleanupPass()); kernelModulePM.addPass(createReconcileUnrealizedCastsPass());