ci: warn PR authors about slow ci-sim runs

[mouliangyu]

背景

ci-sim 运行时间增长时，目前只能由作者主动查看 workflow，缺少一个不会阻塞门禁的持续提醒机制。

修改

• 在 ci-sim 完成后由受信任的 workflow_run watchdog 统计 vpto-sim-validation 从 started_at 到 completed_at 的实际执行时间，不包含 runner 排队时间。
• 超过软预算时添加 ci-slow label，并在 PR Conversation 中创建或更新一条 bot 评论提醒作者；后续恢复到预算内时移除 label，并将同一条评论更新为 resolved，避免多次提交产生重复评论。
• 默认软预算为 90 分钟，可通过仓库变量 CI_SIM_SOFT_TIMEOUT_MINUTES 调整。
• 校验 artifact、workflow run 和当前 PR 的 head SHA，跳过已经过期的运行、已关闭 PR 和未实际执行 SIM 的 workflow。
• 将评论权限限制在默认分支上的 watchdog job，显式 checkout 受信任的默认分支脚本；告警 job 使用 best-effort 语义，不影响原有 ci-sim 门禁结果。
• 使用 actions/github-script 注入的 Octokit 调用 GitHub API，观察逻辑保留在独立 JavaScript 文件中，避免手动维护认证、REST URL、分页和重试。

这是任务完成后的软告警，不会在运行达到预算时中止任务或实时发表评论。

验证

• node --check .github/scripts/ci_sim_duration_warning.js
• .work/tools/actionlint .github/workflows/watchdog.yml
• git diff --check
• 使用真实 Octokit 和已完成的 GitHub workflow run 执行 dry-run，正确识别 2h 27m 43s 超过 90 分钟预算，并计划添加 ci-slow、创建评论，未写入任何状态。
• 使用内存 fake client 验证首次超时添加 label、重复超时保持 label、恢复后移除 label，以及 dry-run 零写入。

[reedhecre]

A5 板测失败

• 触发方式：merged
• 源码提交：6a4ebb144935
• 结果汇总：OK 20 / FAIL 1 / SKIP 0
• 日志：/root/ptoas-board-monitor-a5/logs/20260729_113505_merged_pr1031.log
• 失败阶段：board-validation-qwen / exit=1

失败用例

• qwen3_decode_incore_5 (run, exit=139)

[reedhecre]

A3 板测失败

• 触发方式：merged
• 源码提交：6a4ebb144935
• 结果汇总：OK 217 / FAIL 3 / SKIP 3
• 日志：/home/zhongxuan/ptoas-board-monitor/runtime/logs/20260728_203808_merged_pr1031.log
• 失败阶段：board-validation / exit=1

失败用例

• syncall_binding (run, exit=2)
• comm_p2p_binding_variants (run, exit=1)
• comm_p2p (run, exit=1)

[reedhecre]

A3 板测失败详情：PR #1031

syncall_binding

stage=run info=exit=2

/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260728_203808_merged_pr1031/npu_validation/SyncAll/syncall_binding/syncall_binding_kernel.cpp:96:3: error: no matching function for call to 'SYNCALL'
  SYNCALL<SyncAllMode::Soft, SyncCoreType::AIVOnly>(v8, v10, v2);
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260728_203808_merged_pr1031/payload/pto-isa/include/pto/common/pto_instr.hpp:67:15: note: candidate function template not viable: requires at most 2 arguments, but 3 were provided
PTO_INST void SYNCALL(GlobalData& gmWorkspace, int32_t usedCores = 0)
              ^
/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260728_203808_merged_pr1031/payload/pto-isa/include/pto/common/pto_instr.hpp:53:15: note: candidate function template not viable: requires 0 arguments, but 3 were provided
PTO_INST void SYNCALL()
              ^
1 error generated.
gmake[2]: *** [CMakeFiles/syncall_binding_kernel.dir/build.make:76: CMakeFiles/syncall_binding_kernel.dir/syncall_binding_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/syncall_binding_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-07-28 20:57:10] ERROR: testcase failed (exit 2): syncall_binding

comm_p2p_binding_variants

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507035 (/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260728_203808_merged_pr1031/npu_validation/CommSync/comm_p2p_binding_variants/main.cpp:122)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 3295255] 2026-07-28-21:10:07.994.752 (EZ9999):  The error from device(chipId:0, dieId:1), serial number is 2327, there is an exception of aivec error, core id is 11, error code = 0x10, dump info: pc start: 0x124200000000, current: 0x124200000164, vec error info: 0x4000000b7, mte error info: 0x6000099, ifu error info: 0x22980fa9992c0, ccu error info: 0x40e008000000004f, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100000000.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:645]
        TraceBack (most recent call last):
       The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0x6000099, fixp_error1 info: 0, fsmId:0, tslot:6, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:658]
       Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1729]
       AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=41, report_stream_id=41, task_id=0, flip_num=0, fault kernel_name=comm_p2p_binding_variants_kernel, fault kernel info ext=comm_p2p_binding_variants_kernel, program id=0, hash=16541545288638093324.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       rtStreamSynchronize execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-07-28 21:10:10] ERROR: testcase failed (exit 1): comm_p2p_binding_variants

comm_p2p

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507035 (/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260728_203808_merged_pr1031/npu_validation/CommSync/comm_p2p/main.cpp:122)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 3316172] 2026-07-28-21:10:27.759.788 (EZ9999):  The error from device(chipId:0, dieId:1), serial number is 2328, there is an exception of aivec error, core id is 16, error code = 0x10, dump info: pc start: 0x124200000000, current: 0x124200000168, vec error info: 0xdf00005f59, mte error info: 0xfd060000d3, ifu error info: 0x212c0bbbb5900, ccu error info: 0x21b59000000004f, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100000000.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:645]
        TraceBack (most recent call last):
       The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0x60000d3, fixp_error1 info: 0xfd, fsmId:0, tslot:6, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:658]
       Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1729]
       AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=41, report_stream_id=41, task_id=0, flip_num=0, fault kernel_name=comm_p2p_kernel, fault kernel info ext=comm_p2p_kernel, program id=0, hash=6260346515940177617.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       rtStreamSynchronize execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-07-28 21:10:29] ERROR: testcase failed (exit 1): comm_p2p