fix: use PIPE_ALL for vector GM fences

[zhangstevenunity]

Summary

• lower vector-kernel GM/ALL fences to pipe_barrier(PIPE_ALL) followed by dsb(DSB_DDR)
• retain the conservative MTE2/MTE3/FIX drain for non-vector kernels
• add a TPUT-to-fence regression for the device hang and update the existing TNotify fence expectations

Validation

• LLVM 21.1.8: ptoas_runtime_staging build passed with -j1
• issue1026_vector_gm_fence_no_fix.pto: passed
• issue711_tnotify_mte_drain.pto: passed
• comm_p2p_emitc.pto: passed
• generated A3 vector C++ compiled with Bisheng for dav-c220-vec
• full lit was attempted; PTODSL daemon tests are unavailable because the local LLVM 21 build has MLIR_ENABLE_BINDINGS_PYTHON=OFF

Closes #1026

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[reedhecre]

A5 板测失败

• 触发方式：merged
• 源码提交：195fdb8c5354
• 结果汇总：OK 12 / FAIL 9 / SKIP 0
• 日志：/root/ptoas-board-monitor-a5/logs/20260729_102007_merged_pr1033.log
• 失败阶段：board-validation-qwen / exit=1

失败用例

• out_proj_residual (run, exit=1)
• down_proj_residual (run, exit=1)
• sparse_attn_test_incore_7 (run, exit=1)
• decode_swa_test_incore_40 (gen, exit=1)
• decode_hca_test_incore_54 (gen, exit=1)
• decode_csa_test_incore_81 (gen, exit=1)
• attention_swa_test_incore_40 (gen, exit=1)
• attention_hca_test_incore_54 (gen, exit=1)
• attention_csa_test_refresh_incore_81 (gen, exit=1)

[reedhecre]

A5 板测成功

• 触发方式：manual
• 源码提交：195fdb8c5354
• 结果汇总：OK 9 / FAIL 0 / SKIP 0
• 日志：/root/ptoas-board-monitor-a5/logs/20260729_103728_manual_pr1033.log
• 结果 TSV：/root/ptoas-board-monitor-a5/logs/20260729_103728_manual_pr1033.tsv
• 手动指令：/run a5 out_proj_residual down_proj_residual sparse_attn_test_incore_7 decode_swa_test_incore_40 decode_hca_test_incore_54 decode_csa_test_incore_81 attention_swa_test_incore_40 attention_hca_test_incore_54 attention_csa_test_refresh_incore_81
• 触发人：root
• 指定用例：out_proj_residual,down_proj_residual,sparse_attn_test_incore_7,decode_swa_test_incore_40,decode_hca_test_incore_54,decode_csa_test_incore_81,attention_swa_test_incore_40,attention_hca_test_incore_54,attention_csa_test_refresh_incore_81

[reedhecre]

A3 板测失败

• 触发方式：merged
• 源码提交：195fdb8c5354
• 结果汇总：OK 217 / FAIL 3 / SKIP 3
• 日志：/home/zhongxuan/ptoas-board-monitor/runtime/logs/20260728_194907_merged_pr1033.log
• 失败阶段：board-validation / exit=1

失败用例

• syncall_binding (run, exit=2)
• comm_p2p_binding_variants (run, exit=1)
• comm_p2p (run, exit=1)

[reedhecre]

A3 板测失败详情：PR #1033

syncall_binding

stage=run info=exit=2

/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260728_194907_merged_pr1033/npu_validation/SyncAll/syncall_binding/syncall_binding_kernel.cpp:96:3: error: no matching function for call to 'SYNCALL'
  SYNCALL<SyncAllMode::Soft, SyncCoreType::AIVOnly>(v8, v10, v2);
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260728_194907_merged_pr1033/payload/pto-isa/include/pto/common/pto_instr.hpp:67:15: note: candidate function template not viable: requires at most 2 arguments, but 3 were provided
PTO_INST void SYNCALL(GlobalData& gmWorkspace, int32_t usedCores = 0)
              ^
/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260728_194907_merged_pr1033/payload/pto-isa/include/pto/common/pto_instr.hpp:53:15: note: candidate function template not viable: requires 0 arguments, but 3 were provided
PTO_INST void SYNCALL()
              ^
1 error generated.
gmake[2]: *** [CMakeFiles/syncall_binding_kernel.dir/build.make:76: CMakeFiles/syncall_binding_kernel.dir/syncall_binding_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/syncall_binding_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-07-28 20:08:03] ERROR: testcase failed (exit 2): syncall_binding

comm_p2p_binding_variants

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507035 (/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260728_194907_merged_pr1033/npu_validation/CommSync/comm_p2p_binding_variants/main.cpp:122)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 405656] 2026-07-28-20:20:30.361.171 (EZ9999):  The error from device(chipId:4, dieId:1), serial number is 162, there is an exception of aivec error, core id is 39, error code = 0x10, dump info: pc start: 0x125200000000, current: 0x125200000164, vec error info: 0x7600000062, mte error info: 0x430300408d, ifu error info: 0x204b0205b4ac0, ccu error info: 0x4f, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100000000.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:645]
        TraceBack (most recent call last):
       The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0x300408d, fixp_error1 info: 0x43, fsmId:1, tslot:0, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:658]
       Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1729]
       AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [DFX_INFO]Aicore kernel execute failed, device_id=9, stream_id=46, report_stream_id=46, task_id=0, flip_num=0, fault kernel_name=comm_p2p_binding_variants_kernel, fault kernel info ext=comm_p2p_binding_variants_kernel, program id=0, hash=16541545288638093324.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       rtStreamSynchronize execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-07-28 20:20:32] ERROR: testcase failed (exit 1): comm_p2p_binding_variants

comm_p2p

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507035 (/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260728_194907_merged_pr1033/npu_validation/CommSync/comm_p2p/main.cpp:122)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 423812] 2026-07-28-20:20:50.058.621 (EZ9999):  The error from device(chipId:4, dieId:1), serial number is 163, there is an exception of aivec error, core id is 44, error code = 0x10, dump info: pc start: 0x125200000000, current: 0x125200000168, vec error info: 0x7600000062, mte error info: 0x200300405d, ifu error info: 0x204b0205b4ac0, ccu error info: 0x4f, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100000000.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:645]
        TraceBack (most recent call last):
       The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0x300405d, fixp_error1 info: 0x20, fsmId:1, tslot:0, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:658]
       Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1729]
       AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [DFX_INFO]Aicore kernel execute failed, device_id=9, stream_id=46, report_stream_id=46, task_id=0, flip_num=0, fault kernel_name=comm_p2p_kernel, fault kernel info ext=comm_p2p_kernel, program id=0, hash=6260346515940177617.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       rtStreamSynchronize execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-07-28 20:20:51] ERROR: testcase failed (exit 1): comm_p2p