[Performance] VMI rotate-partner RoPE form (fewer muls) regresses

[MirkoDeVita98]

Summary

The f16 RoPE interleave (GPT-J) kernel has two mathematically equivalent VMI formulations:

• Even/odd form (current, rope backend.py fp16 kernel): deinterleave x/cos/sin, compute
  y_even = x_even*cos - x_odd*sin, y_odd = x_odd*cos + x_even*sin, re-interleave.
  4 muls + 1 sub + 1 add. RVEC 4077.
• Rotate-partner form (matches CCE ComputeF16): build rot(x) = [-x_odd, x_even]
  via vdintlv → vneg → vintlv, then y = x*cos + rot(x)*sin.
  2 muls + 1 add (+1 neg). RVEC 4639 — slower.

The rotate-partner form has half the muls (960 vs 1920) but is ~14% slower, because the
vneg on the deinterleaved half (x_odd) followed by re-vintlv makes the VMI layout pipeline
insert RV_VPACK (480) + RV_VZUNPACK (480) layout-conversion ops. The even/odd form avoids
this by keeping every value in the deinterleaved layout until a single final vintlv.

CCE's hand-written MI uses exactly the rotate-partner form (negate-via-×-1, 3 muls) and reaches
RVEC 3036 with no pack/unpack — because it controls the physical layout directly. VMI can't
express the same efficiently: any elementwise op (vneg/vmul) on a deinterleaved-layout operand
that later feeds vintlv triggers pack/unpack.

Request: let the VMI layout system carry a value through vdintlv → elementwise → vintlv
without materializing VPACK/VZUNPACK (i.e., keep the deinterleaved layout across the
elementwise op). That would let the 3-mul rotate-partner kernel match CCE, and would generally make
the deinterleave→compute→interleave idiom free of layout churn.

Command line

# kernel-test cannsim transport (cce reference + vmi kernel):
scripts/run_sim.sh --output sim_outputs/manual/rope-cce-f16-il run.py -- \
  --op rope --workflow cycle --backend cce --case f16_interleave
scripts/run_sim.sh --output sim_outputs/manual/rope-vmi-f16-il  run.py -- \
  --op rope --workflow cycle --backend vmi --case f16_interleave
# correctness: --workflow correctness  (both forms PASS, maxDiff=0.000000)

Reproduction input

kernels/rope/vmi/backend.py, rope_vmi_f16, interleave branch (MODE != 0). Both variants below
are correct (maxDiff=0); only the arithmetic form differs.

Even/odd form (current — faster, 4077):

cos = pto.vmi.vload(cos_ptr, cs_off, size=64)
sin = pto.vmi.vload(sin_ptr, cs_off, size=64)
cos_even, cos_odd = pto.vmi.vdintlv(cos, cos, full_mask)
sin_even, sin_odd = pto.vmi.vdintlv(sin, sin, full_mask)
for n in range(0, n_count, 1):
    x = pto.vmi.vload(x_ptr, row_off, size=64)
    x_even, x_odd = pto.vmi.vdintlv(x, x, full_mask)
    y_even = pto.vmi.vsub(pto.vmi.vmul(x_even, cos_even, half_mask),
                          pto.vmi.vmul(x_odd, sin_even, half_mask), half_mask)
    y_odd  = pto.vmi.vadd(pto.vmi.vmul(x_odd, cos_odd, half_mask),
                          pto.vmi.vmul(x_even, sin_odd, half_mask), half_mask)
    y, _ = pto.vmi.vintlv(y_even, y_odd, full_mask)
    pto.vmi.vstore(y, y_ptr, row_off, full_mask)

Rotate-partner form (fewer muls but slower, 4639 — the regression):

cos = pto.vmi.vload(cos_ptr, cs_off, size=64)   # duplicated per pair; NOT deinterleaved
sin = pto.vmi.vload(sin_ptr, cs_off, size=64)
for n in range(0, n_count, 1):
    x = pto.vmi.vload(x_ptr, row_off, size=64)
    x_even, x_odd = pto.vmi.vdintlv(x, x, full_mask)
    neg_odd = pto.vmi.vneg(x_odd, full_mask)              # <-- triggers VPACK/VZUNPACK
    rot, _  = pto.vmi.vintlv(neg_odd, x_even, full_mask)  #     around this neg + re-intlv
    y = pto.vmi.vadd(pto.vmi.vmul(x, cos, full_mask),
                     pto.vmi.vmul(rot, sin, full_mask), full_mask)
    pto.vmi.vstore(y, y_ptr, row_off, full_mask)

Expected performance

The rotate-partner form does strictly less arithmetic (2 muls vs 4), so it should be ≤ the
even/odd form and ideally reach the CCE reference (~3036). At minimum it should not be slower.

Actual performance

• Even/odd form: RVEC 4077
• Rotate-partner form: RVEC 4639 (~14% slower despite half the muls)
• CCE reference: RVEC 3036

The regression is entirely the injected RV_VPACK (480) + RV_VZUNPACK (480) conversions.

Profiling data (optional)

Per-op counts / avg duration from cannsim trace_core0.json (tile s=15, n=32; 480-iter loop):

Op	Even/odd (4077)	Rotate-partner (4639)	CCE (3036)
RV_VMUL	1920 @8.0	960 @8.0	1440 @8.0
RV_VSUB	480 @7.0	0	0
RV_VADD	480 @7.0	480 @7.0	480 @7.0
RV_VDINTLV	510 @11.0	480 @11.0	480 @11.0
RV_VINTLV	480 @11.0	480 @11.0	480 @11.0
RV_VPACK	0	480 @11.0	0
RV_VZUNPACK	0	480 @11.0	0
load	RV_VLDS 510 @10.6	RV_VLDS 510 @10.8	RV_VLD 510 @9.0
store	RV_VSTS 480 @9.0	RV_VSTS 480 @9.0	RV_VST 480 @9.0

Note: the load is already fine on VMI (RV_VLDS@~10.6, essentially CCE's RV_VLD@9.0) once the
kernel uses size=64 vectors — so the load is not the issue here. The remaining VMI↔CCE gap is
(a) even/odd doing 4 muls vs CCE's 3, and (b) rotate-partner's pack/unpack blocking the 3-mul path.

Git commit

vmi-v0.1.1

[mouliangyu]

how can i get the cce handwrite codes

[MirkoDeVita98]

how can i get the cce handwrite codes

See this PR please #543

[mouliangyu]

Thanks for the detailed report. The main issue described here has now been addressed in commit 392609d1a3cf02211a2d13265cb16b54a27aebf0.

The VMI layout assignment now preserves the lane-stride layout through the vdintlv -> elementwise -> vintlv chain. As a result, the RV_VPACK and RV_VZUNPACK operations previously inserted before/around vdintlv are no longer generated.

Results for the original f16_interleave case (s=15, n=32, d=64) are:

Implementation	RVEC	Correctness
VMI rotate-partner, before the fix	4639	max_diff=0
VMI rotate-partner, after the fix	3446	max_diff=0
Hand-written CCE	3036	max_diff=0

The relevant instruction counts after the fix are:

Instruction	VMI after fix	CCE
RV_VMUL	960	1440
RV_VNEG_FP	480	0
RV_VADD	480	480
RV_VDINTLV	480	480
RV_VINTLV	480	480
RV_VPACK	0	0
RV_VZUNPACK	0	0
load	RV_VLDS 510, avg. 10.40 cycles	RV_VLD 510, avg. 9.03 cycles
store	RV_VSTS 480, avg. 9.09 cycles	RV_VST 480, avg. 9.00 cycles

This removes most of the original regression. The remaining work is to lower the 64xf16 load through NORM memory access instead of the current UNPK-style path. That optimization requires memory-overlap safety analysis; we need to prove that the wider access does not overlap an unsafe neighboring region or cross an invalid boundary. Therefore, I suggest keeping this issue open for that follow-up optimization.

One additional question about the provided case: the d=64 path processes 64 f16 values, i.e. 128 bytes, so it does not appear to fully utilize a 256-byte vector width. Is that under-utilization inherent to the real workload, or could the case shape/kernel configuration be unintended?

[learning-chip]

the d=64 path processes 64 f16 values, i.e. 128 bytes, so it does not appear to fully utilize a 256-byte vector width. Is that under-utilization inherent to the real workload, or could the case shape/kernel configuration be unintended?

This RoPE kernel intends to match ops-nn apply_rotary_pos_emb, the last dimension D / head_dim supports both 64 and 128, from the API doc:

输入shape最后一维必须等于128或者64。

[mouliangyu]

挂着：64xf16 访存模式由 unpack 访存优化为连续访存