Skip to content

Fix: cleanup_retired frees only the retiring task's entries (a2a3 + a5) - #1511

Open
yanghaoran29 wants to merge 1 commit into
hw-native-sys:mainfrom
yanghaoran29:scheme-d-cleanup-retired-cap
Open

Fix: cleanup_retired frees only the retiring task's entries (a2a3 + a5)#1511
yanghaoran29 wants to merge 1 commit into
hw-native-sys:mainfrom
yanghaoran29:scheme-d-cleanup-retired-cap

Conversation

@yanghaoran29

@yanghaoran29 yanghaoran29 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Selective free in cleanup_retired (extracted from #906)

Bug. When cleanup_retired (pto_tensormap.h, a2a3 + a5) retires a
local_id, it frees the slot's entire task-entry chain. But when a slot is
reused by a later task and cleanup lags behind ring advancement, that chain
also holds entries from a still-live task (link_entry only resets the
chain head on the first touch of a slot per run epoch, not on slot reuse).
Cleanup therefore frees the live entries too; the only guard was a
debug_assert, which compiles out in release → silent tensormap corruption.

Fix. Free only entries whose producer_task_id == retired_task, unlinking
each from the task chain (prev_in_task / next_in_task) and fixing up the
head pointer as it goes; entries from other tasks stay linked. No change to the
reclaim_retired_all / sync_tensormap call contract. Applied to both a2a3
and a5.

Regression barrier. Add CleanupRetiredSparesLaterTaskReusingSlot
(tests/ut/cpp/{a2a3,a5}/test_tensormap.cpp): task 0 and task 0+WINDOW share a
slot; after retiring task 0, asserts task 0+WINDOW's entry survives. Before the
fix this throws AssertionError in debug and silently frees in release → fail;
after the fix it passes. Both suites green (33 cases each).

Onboard before/after (a5, task-submit device 0, 100 rounds, one run each):

Sample: paged_attention_unroll/Case1

metric before after Δ
Device(trim) 141.6 µs 142.5 µs +0.6%
Effective 117.3 µs 117.1 µs −0.2%
Orch 85.8 µs 84.7 µs −1.3%
Sched 116.0 µs 115.3 µs −0.6%

All |Δ| < 2%, mixed signs → within noise: no regression, no gain, consistent
with this being a correctness fix that is a no-op on the common path.

Extracted from #906.

@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: da67d287-158c-437a-ad07-72a37fd8fe1c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The change selectively cleans retired tensor-map entries by producer task, adds slot-reuse coverage for A2A3 and A5, and documents A5 cross-cluster last_task_alive investigation results and scheme decisions.

Changes

A5 last_task_alive cleanup

Layer / File(s) Summary
Selective retired-task cleanup
src/a2a3/.../pto_tensormap.h, src/a5/.../pto_tensormap.h, tests/ut/cpp/a2a3/test_tensormap.cpp, tests/ut/cpp/a5/test_tensormap.cpp
cleanup_retired unlinks only entries belonging to the retired task and preserves newer entries that reuse the same slot; unit tests cover this behavior.
Investigation results and index
docs/investigations/2026-07-a5-last-task-alive-cross-cluster.md, docs/investigations/README.md
The investigation records measurement conventions, Scheme D results, the rejected Scheme B placement change, deferred Scheme C work, and the new index entry.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Poem

I’m a rabbit guarding each task-chain thread,
Newer slots stay safely fed.
Old links hop out, one by one,
Tests confirm the cleanup’s done.
B rests, C waits beneath the sun.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title accurately summarizes the main change: cleanup_retired now frees only the retiring task's entries in a2a3 and a5.
Description check ✅ Passed The description matches the code changes, tests, benchmarks, and investigation notes in the pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/investigations/2026-07-a5-last-task-alive-cross-cluster.md`:
- Line 1: Update the document title to remove the stale “待上机测量” wording, while
preserving the existing A5 cross-cluster last_task_alive optimization plan and
its completed measurement and B/C decision details.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c97bc815-840d-407a-957b-d594169286a2

📥 Commits

Reviewing files that changed from the base of the PR and between f7909e2 and 5173d50.

📒 Files selected for processing (6)
  • docs/investigations/2026-07-a5-last-task-alive-cross-cluster.md
  • docs/investigations/README.md
  • src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto_tensormap.h
  • src/a5/runtime/tensormap_and_ringbuffer/runtime/pto_tensormap.h
  • tests/ut/cpp/a2a3/test_tensormap.cpp
  • tests/ut/cpp/a5/test_tensormap.cpp

Comment thread docs/investigations/2026-07-a5-last-task-alive-cross-cluster.md Outdated
@yanghaoran29
yanghaoran29 force-pushed the scheme-d-cleanup-retired-cap branch 5 times, most recently from 5997fde to ac38165 Compare July 29, 2026 02:45
cleanup_retired walked a slot's whole task_entry_head chain and freed
every entry, assuming — only via a debug_assert, which compiles out in
release — that the chain belonged to one task. When cleanup lagged ring
advancement, a reused slot's chain mixed entries from a later, still-live
task, and those were freed too: silent tensormap corruption in release
builds.

Free only entries whose producer_task_id matches the retiring local_id,
unlinking each from the task chain; entries from other tasks stay linked.
Applied to both a2a3 and a5 tensormap_and_ringbuffer.

- Add CleanupRetiredSparesLaterTaskReusingSlot unit test (a2a3 + a5) as
  the regression barrier; both test_tensormap suites pass (33 each),
  a2a3sim/a5sim runtimes rebuild clean, a5sim mixed_example passes.
- Onboard before/after (a5, spmd_multiblock_mix, 100 rounds, task-submit
  device 0): Device/Effective/Orch/Sched all within +-2% — no regression,
  as expected for a correctness fix that is a no-op on the common path.

Extracted from hw-native-sys#906.
@yanghaoran29
yanghaoran29 force-pushed the scheme-d-cleanup-retired-cap branch 2 times, most recently from 760db31 to bae4564 Compare July 29, 2026 03:09
@yanghaoran29

Copy link
Copy Markdown
Contributor Author

来自 #906 的相关原文(对照)

本 PR 从 #906 的调查中抽出;#906M5_DESIGN.md 当时把根因归因于 cleanup 范围跨 task_window_size 的 wrap/double-free,并提出 「范围 cap」 作为解法。原文如下:

M5_DESIGN.md — 解法 B:cleanup_retired 范围 cap(#906

解法 B:cleanup_retired 范围 cap

仅靠 A 还不够。cleanup_retired 在范围 ≥ task_window_size会 wrap

for (int32_t local_id = old; local_id < new; ++local_id) {
    int32_t slot = local_id & (task_window_size - 1);
    // 当 (new - old) >= window,多个 local_id 哈希到同一 slot
    // task_entry_heads[slot] 在前一次循环已被清空 → 但被当成新 local 的入口
    // 同一组 entry 被 free 两次 → free_entry 的 always_assert 挂
}

M5 下 SM 一次可能跳很大……Cap 后 cleanup 每次只清 task_window_size - 1,多次调用消化:

int32_t max_step = task_window_sizes[r] > 0 ? task_window_sizes[r] - 1 : 0;
if (max_step > 0 && target - last_cleanup[r] > max_step) {
    target = last_cleanup[r] + max_step;
}
cleanup_retired(r, last_cleanup[r], target);
last_cleanup[r] = target;

M5 v6(终版)

v4 + cleanup_retired 范围 cap。所有路径都正确。

当时的 cleanup_retired 实现原文(main / #906 头,修复前)

while (cur_entry != nullptr) {
    PTO2TensorMapEntry *next_entry = cur_entry->next_in_task;  // Save before clearing
    // Only remove if this entry belongs to the retiring task
    // (slot may have been reused by a newer task)
    debug_assert(
        cur_entry->producer_task_id ==
        PTO2TaskId::make(static_cast<uint8_t>(ring_id), static_cast<uint32_t>(local_id))
    );
    free_entry(*cur_entry);   // ← release 下 debug_assert 被编译掉,整条链无条件 free
    cur_entry = next_entry;
}
task_entry_heads[ring_id][task_slot] = nullptr;

为何 cap 不够、本 PR 才是修法

slot reuse + cleanup lag 时,同一条 task_entry_head 链会混入仍存活的后续 task 的 entry。cap 只限制清理的 local_id 跨度,第一次走到混合链仍会整链 free;需要的是按 producer_task_id == retired_task 选择性 unlink/free(本 PR),而不是只 cap 范围。详见 #906 上的 follow-up comment(Side find 段)。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant