fix(runtime): Scope the L3 swimlane name map to each dispatch's own L2 program - #2173
Merged
lyfne123 merged 2 commits intoJul 29, 2026
Merged
Conversation
…2 program Fixes hw-native-sys#2169 `_collect_l3_swimlane` merged every `next_levels/*/kernel_config.py` into one flat `func_id -> name` table and wrote it into every `rank*/d{k}` dir. But `func_id` is a per-L2-program namespace — each program numbers its kernels from 0 — so the alphabetically-later program won ids 0..N and every other dispatch was rendered with its names. On a two-program L3 kernel, `lm_head_dispatch_wait` (a cross-card spin-wait) printed as `mtp_projection_norm`: a 24 ms barrier read as a 24 ms RMS-norm kernel, in the console table and in the Perfetto trace alike. The converter's `-k` fallback had the same flaw (`work_dir` was always `chip_dirs[0]`). Nothing on disk recorded which program produced which dispatch. `_submit_chip` is the only place that sees both the dispatch directory and the callable being dispatched, so it now stamps a `dispatch_program.json` marker there; the offline post-pass reads it back and resolves names from that program's `kernel_config.py` alone, keeping the name map and the `-k` fallback in agreement. The marker goes through the filesystem so the two halves stay independent of where the runtime places the L3 orchestrator, and both swimlane passes rewrite it identically. A dispatch whose program cannot be resolved (multi-program build, no marker) is converted with anonymous labels instead of guessed ones — a plausible wrong name reads as a real measurement. That path also drops a stale sibling `name_map.json` the converter would otherwise auto-discover, and `_generate_swimlane` now omits `-k` when the config is absent rather than handing the converter a path it rejects. Also consolidates the two duplicated `orch._dfx_dispatch_idx = {}` reset sites into `_reset_dfx_dispatch_state`, which publishes the callable -> program names alongside the per-card dispatch counter.
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughL3 distributed DFX processing now records each dispatch’s originating program, resolves kernel names using that program’s kernel configuration, anonymizes unresolved dispatches, and conditionally passes kernel configuration to the swimlane converter. Tests and English/Chinese documentation cover the updated behavior. ChangesL3 swimlane labeling
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant DistributedRunner
participant DispatchDirectory
participant SwimlaneCollector
participant SwimlaneConverter
DistributedRunner->>DispatchDirectory: write dispatch_program.json
SwimlaneCollector->>DispatchDirectory: read dispatch_program.json
SwimlaneCollector->>SwimlaneCollector: create program-scoped name_map.json
SwimlaneCollector->>SwimlaneConverter: convert records with selected kernel names
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/pypto/runtime/distributed_runner.py`:
- Around line 803-832: Update the resolved-program branch around
_write_dispatch_name_map so that when it returns None, all stale name_map*.json
files in disp_dir are removed before _generate_swimlane runs, preserving
anonymous-label behavior. Keep the existing mapped-name path unchanged when a
valid map is produced, and add a regression test alongside
test_unresolvable_dispatch_drops_a_stale_name_map for a resolved program with an
empty or unloadable kernel table.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: cb293613-9730-4d82-a7bc-835af9bc28f2
📒 Files selected for processing (6)
docs/en/dev/03-runtime-dfx.mddocs/zh-cn/dev/03-runtime-dfx.mdpython/pypto/runtime/distributed_runner.pypython/pypto/runtime/runner.pytests/ut/runtime/test_distributed_worker.pytests/ut/runtime/test_swimlane_two_pass.py
- Hoist the stale ``name_map*.json`` cleanup out of the unresolvable branch so it covers every dispatch: a resolved program whose ``kernel_config.py`` names no kernels also writes no map, and an earlier run's map must not be picked up in its place. The invariant is now unconditional — a dispatch only ever holds a name map the current run wrote — instead of depending on the converter's internal precedence between ``--func-names``, ``-k`` and sibling discovery. - Add a regression test for the resolved-but-empty-table case.
lyfne123
approved these changes
Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_collect_l3_swimlanemerged everynext_levels/*/kernel_config.pyinto one flatfunc_id -> nametable and wrote it into everyrank*/d{k}dispatch dir.func_idis a per-L2-program namespace — each program numbers its kernels from 0 — so the alphabetically-later program won ids0..Nand every other dispatch was rendered with its names. The converter's-kfallback had the same flaw (work_dirwas alwayschip_dirs[0]).On a two-program L3 kernel this printed
lm_head_dispatch_wait(a cross-card spin-wait) asmtp_projection_norm— a 24 ms barrier reading as a 24 ms RMS-norm kernel, in the console table and in the Perfetto trace alike. Silent, and actively misleading for performance work.Nothing on disk recorded which program produced which dispatch.
_submit_chipis the only place that sees both the dispatch directory and the callable being dispatched, so it now stamps adispatch_program.jsonmarker there; the offline post-pass reads it back and resolves names from that program'skernel_config.pyalone, keeping--func-namesand the-kfallback in agreement. The marker goes through the filesystem so the two halves stay independent of where the runtime places the L3 orchestrator, and both swimlane passes rewrite it identically.name_map.jsonthe converter would otherwise auto-discover._generate_swimlaneomits-kwhen the config is absent rather than handing the converter a path it rejects (it errors out on a missing-k, losing the swimlane entirely).orch._dfx_dispatch_idx = {}reset sites into_reset_dfx_dispatch_state, which publishes the callable -> program names alongside the per-card dispatch counter.Testing
tests/ut7981 passed / 2 skipped;tests/ut/runtime+tests/lintgreen after rebase; pre-commit (ruff, pyright, markdownlint, en-zh parity) cleandocs/{en,zh-cn}/dev/03-runtime-dfx.md: the marker in the L3 dispatch-dir layout, why names must be resolved per dispatch, and the new implementation-map rowsNew coverage in
tests/ut/runtime/test_distributed_worker.pyandtest_swimlane_two_pass.py:func_idfrom 0, asserting each dispatch'sname_map.jsonand-kdir name its own program (today the two maps are byte-identical)next_levels/does not hide it-k, stale map dropped_generate_swimlanepassing / omitting-kby config presenceBoth key cases were verified to be genuine discriminators: they fail against
mainand pass with the fix. Thesimpler_setupconverter is mocked in tests (optional runtime dep), not vendored.Related Issues
Fixes #2169