[RFC] docs: distributed programming guides — collectives, benchmarking, and API reference - #2162
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis documentation-focused change adds English and Chinese distributed-programming, API-reference, and benchmarking guides, updates operation references and getting-started pages, and expands Python API docstrings. No executable logic or public signatures change. ChangesDistributed documentation
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e2172dbf6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 11
🧹 Nitpick comments (1)
docs/en/user/06-benchmarking-guide.md (1)
28-28: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winFix the mirrored Markdownlint warnings.
Add appropriate language identifiers to fenced blocks and remove or prefix blank lines inside blockquotes in both language versions.
docs/en/user/06-benchmarking-guide.md#L28-L28: tag the output fence.docs/en/user/06-benchmarking-guide.md#L58-L58: tag the API-signature fence.docs/en/user/06-benchmarking-guide.md#L114-L114: tag the span-tree fence.docs/en/user/06-benchmarking-guide.md#L193-L193: tag the variance-output fence.docs/en/user/06-benchmarking-guide.md#L264-L264: tag the formula fence.docs/en/user/06-benchmarking-guide.md#L285-L285: tag the worked-example fence.docs/en/user/06-benchmarking-guide.md#L324-L324: fix the blank blockquote line.docs/en/user/06-benchmarking-guide.md#L329-L329: fix the blank blockquote line.docs/en/user/06-benchmarking-guide.md#L334-L334: fix the blank blockquote line.docs/en/user/06-benchmarking-guide.md#L363-L363: tag the span-tree fence.docs/zh-cn/user/06-benchmarking-guide.md#L27-L27: tag the output fence.docs/zh-cn/user/06-benchmarking-guide.md#L57-L57: tag the API-signature fence.docs/zh-cn/user/06-benchmarking-guide.md#L113-L113: tag the span-tree fence.docs/zh-cn/user/06-benchmarking-guide.md#L188-L188: tag the variance-output fence.docs/zh-cn/user/06-benchmarking-guide.md#L257-L257: tag the formula fence.docs/zh-cn/user/06-benchmarking-guide.md#L278-L278: tag the worked-example fence.docs/zh-cn/user/06-benchmarking-guide.md#L312-L312: fix the blank blockquote line.docs/zh-cn/user/06-benchmarking-guide.md#L316-L316: fix the blank blockquote line.docs/zh-cn/user/06-benchmarking-guide.md#L320-L320: fix the blank blockquote line.docs/zh-cn/user/06-benchmarking-guide.md#L346-L346: tag the span-tree fence.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/en/user/06-benchmarking-guide.md` at line 28, Fix the mirrored Markdownlint issues in docs/en/user/06-benchmarking-guide.md at lines 28, 58, 114, 193, 264, 285, and 363 by adding appropriate language identifiers to the output, API-signature, span-tree, variance-output, formula, worked-example, and span-tree fences; prefix the blank blockquote lines at 324, 329, and 334. Apply the same corresponding fixes in docs/zh-cn/user/06-benchmarking-guide.md at lines 27, 57, 113, 188, 257, 278, and 346 for fences, and 312, 316, and 320 for blank blockquote lines, preserving the existing bilingual content.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/en/user/02-operation_reference.md`:
- Around line 289-298: The capability matrices in
docs/en/user/02-operation_reference.md lines 289-298 and
docs/zh-cn/user/02-operation_reference.md lines 289-298 incorrectly place Put’s
AtomicType.None_ / Add values in the ReduceOps column. Update both matrices to
represent Put’s atomic types in a separate, appropriate capability column while
leaving ReduceOps distinct and preserving the equivalent English and Chinese
content.
In `@docs/en/user/04-distributed-api-reference.md`:
- Around line 243-244: Update the descriptions of
rt.release_inherited_host_tensor_refs() in both
docs/en/user/04-distributed-api-reference.md (lines 243-244) and
docs/zh-cn/user/04-distributed-api-reference.md (lines 240-241) to state that
runtime-held host references are dropped after fork, without promising
unconditional destruction of the host tensor.
- Around line 157-160: Update the benchmark() documentation in
docs/en/user/04-distributed-api-reference.md lines 157-160 to replace the
incorrect RuntimeError claim with the documented zero-sample fallback when
SIMPLER_HOST_STRACE is disabled. Apply the same corrected benchmark contract in
docs/zh-cn/user/04-distributed-api-reference.md lines 155-158, keeping both API
references consistent.
- Around line 220-227: The DeviceTensor examples are not executable or
worker-safe because they omit torch, allocate outside the prepared runtime, and
invoke compiled directly. In docs/en/user/04-distributed-api-reference.md lines
220-227 and docs/zh-cn/user/04-distributed-api-reference.md lines 217-224,
import torch, move rt.alloc_tensor into the compiled.prepare() lifecycle, and
dispatch the compiled call through rt(...), preserving the documented
DeviceTensor usage in both language versions.
- Around line 231-236: Correct the documented compiled.prepare() signature to
remove inherited_host_tensors and list the actual supported parameters in
docs/en/user/04-distributed-api-reference.md lines 231-236 and
docs/zh-cn/user/04-distributed-api-reference.md lines 226-233, keeping both
bilingual API references consistent.
In `@docs/en/user/05-distributed-guide.md`:
- Around line 35-45: Define SIZE before the type annotations in both mirrored
HelloAllReduce examples, using the same documented extent such as 256:
docs/en/user/05-distributed-guide.md lines 35-45 and
docs/zh-cn/user/05-distributed-guide.md lines 31-42 each require this direct
change.
- Around line 155-164: Update both handshake examples in
docs/en/user/05-distributed-guide.md (lines 155-164) and
docs/zh-cn/user/05-distributed-guide.md (lines 149-159) to document and enforce
the precondition tag >= 1 before the notify/wait sequence; keep the existing
signal cell and wait behavior unchanged.
In `@docs/en/user/06-benchmarking-guide.md`:
- Line 87: The benchmarking documentation inconsistently describes non-STRACE
behavior. Update docs/en/user/06-benchmarking-guide.md lines 87 and 320-328 and
docs/zh-cn/user/06-benchmarking-guide.md lines 85 and 308-315 so both languages
consistently state that non-STRACE benchmarking raises before returning
statistics, removing the implication that zero-valued statistics are returned.
In `@docs/zh-cn/user/00-getting_started.md`:
- Around line 407-459: 将 HelloAllReduce 示例引导语中的“可直接复制运行”改为明确表明这是 InCore
kernel/sample 的措辞，避免暗示无需额外配置即可执行；保留现有内核代码及分布式指南链接不变，除非补充完整的 host、COMPILER 和
distributed 运行设置。
In `@python/pypto/language/distributed/op/tensor_ops.py`:
- Around line 541-544: Update the affected Python docstrings near the
signal-shape documentation to replace the Unicode multiplication and minus
symbols with ASCII `x` and `-`, including the additional occurrences around the
referenced lines. Do not add inline Ruff suppressions; preserve the documented
shape expressions and wording otherwise.
In `@python/pypto/language/dsl_api.py`:
- Around line 715-724: Update the “Three usage forms at a glance” table to show
the capture form as `with pl.spmd(n) as tid:` and indicate that optional
`deps=[...]` may be added, rather than presenting deps as required. Keep the
existing descriptions of dispatch/inline behavior and captured producer TaskId.
---
Nitpick comments:
In `@docs/en/user/06-benchmarking-guide.md`:
- Line 28: Fix the mirrored Markdownlint issues in
docs/en/user/06-benchmarking-guide.md at lines 28, 58, 114, 193, 264, 285, and
363 by adding appropriate language identifiers to the output, API-signature,
span-tree, variance-output, formula, worked-example, and span-tree fences;
prefix the blank blockquote lines at 324, 329, and 334. Apply the same
corresponding fixes in docs/zh-cn/user/06-benchmarking-guide.md at lines 27, 57,
113, 188, 257, 278, and 346 for fences, and 312, 316, and 320 for blank
blockquote lines, preserving the existing bilingual content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6354b3a2-bc3b-4fe9-ac31-591dc7526bdf
📒 Files selected for processing (17)
docs/en/user/00-getting_started.mddocs/en/user/02-operation_reference.mddocs/en/user/04-distributed-api-reference.mddocs/en/user/05-distributed-guide.mddocs/en/user/06-benchmarking-guide.mddocs/zh-cn/user/00-getting_started.mddocs/zh-cn/user/02-operation_reference.mddocs/zh-cn/user/04-distributed-api-reference.mddocs/zh-cn/user/05-distributed-guide.mddocs/zh-cn/user/06-benchmarking-guide.mdpython/pypto/ir/distributed_compiled_program.pypython/pypto/language/distributed/op/system_ops.pypython/pypto/language/distributed/op/tensor_ops.pypython/pypto/language/distributed/op/tile_ops.pypython/pypto/language/dsl_api.pypython/pypto/pypto_core/ir.pyipython/pypto/runtime/bench.py
Fixes from CodeRabbit:
- CR1: Separate Put's AtomicType (None_/Add) from ReduceOps column in
capability matrix, adding a dedicated "Atomic" column.
- CR2: Fix all_zero_device description to remove "or non-STRACE builds"
(non-STRACE raises RuntimeError, does not return zero-valued stats).
- CR3: Make DeviceTensor example executable and worker-safe — add import
torch, wrap in compiled.prepare() context, dispatch via rt(...).
- CR5: Clarify release_inherited_host_tensor_refs() — drops runtime-held
references, not unconditional host-tensor destruction.
- CR6: Define SIZE=256 in HelloAllReduce kernel examples in both EN/zh-cn
distributed guides, so the code compiles as-written.
- CR7: Document handshake tag>=1 precondition — passing tag=0 causes
permanent hang with the Ge expected=1 wait.
- CR9: Clarify zh-cn HelloAllReduce wording from "可直接复制运行" to note
this is an InCore kernel sample requiring additional host/runtime setup.
- CR10: Replace Unicode math symbols (×, −) with ASCII (x, -) in
tensor_ops.py docstrings.
- CR11: Update spmd table to show optional deps, capture form as
`with pl.spmd(n) as tid:`.
Nit: Add language tags (text, python) to bare fenced code blocks in
EN benchmarking guide.
AI Review Comment Fixes — All AddressedAll 17 review comments from CodeRabbit and chatgpt-codex-connector have been addressed in the latest commit. Here's a summary: chatgpt-codex-connector (6 comments — already fixed in previous commits, verified clean):
CodeRabbit (11 comments — all fixed in this commit):
Nit:
|
711c32b to
e0d3ce5
Compare
… API reference Add comprehensive user-facing documentation for PyPTO distributed programming: New guides (EN + zh-cn mirror): - 04-distributed-api-reference.md — API reference for pld.tensor/system/tile operations, DeviceTensor, DistributedWorker, environment variables and macros - 05-distributed-guide.md — Tutorial covering HelloAllReduce, window buffers, signal handshake patterns, mesh vs ring AllReduce, all-to-all, reduce-scatter, persistent workers, debugging hangs - 06-benchmarking-guide.md — L2/L3 benchmarking usage, BenchmarkStats fields, span-tree rendering, ring sizing/prewarm, OSU-style bus bandwidth, complete runnable example, caveats Updated existing guides: - 00-getting_started.md — Table formatting normalized - 02-operation_reference.md — Dtype support table split into InCore/HOST columns; Put AtomicType moved to dedicated Atomic column Code docstring improvements: - distributed_compiled_program.py — prepare() parameters documented - distributed/op/system_ops.py — Unicode symbols replaced with ASCII - distributed/op/tensor_ops.py — Unicode symbols replaced with ASCII - distributed/op/tile_ops.py — remote_load/store signatures documented - dsl_api.py — pl.spmd table improved - ir.pyi — ir.compile() distributed parameters documented - bench.py — benchmark() full signature and BenchmarkStats documented Fixes: markdownlint MD028/MD040/MD060 across all user-facing docs. RFC: review for comment.
e0d3ce5 to
ee56f06
Compare
YunjiQin
left a comment
YunjiQin
left a comment
There was a problem hiding this comment.
Thanks for putting these together — the distributed and benchmarking surface has needed user-facing documentation for a while, and the structure here is good. I checked the prose against the implementation rather than reading for style; most of it holds up.
Verified correct (spot-checking the claims that are easiest to get wrong):
- C1
pl.Level.HOST+pl.Role.Orchestrator— matchesdecorator.py:776andir.pyi:837/896. - C2
SIMPLER_HOST_STRACE/SIMPLER_DFXas compile-time macros defaulting to1—profiling_config.h:15-24;SIMPLER_DEVICE_STRACE_ENABLEreally is agetenv(c_api_shared.cpp:474). Good catch, this one was genuinely misdocumented before. benchmark()signature — byte-for-byte withbench.py:840-849; everyBenchmarkStatsfield and property in the guide exists.- Ring signal
[2(NR-1), NR]+ the factory-function pattern — matchestest_l3_allreduce_ring.py:71-84,205. - allreduce-inside-a-loop rejection —
synthesize_allreduce_signals_pass.cpp:190. pld.*short-form / no-short-form table — exactly matchesdistributed/__init__.py's__all__.RunConfig(ring_task_window=256, ring_heap=1024)— both satisfy the power-of-2 constraints inrunner.py:307-312.
Blocking, in rough priority order (details inline):
05— the "InCore Hand-Rolled vs Host-Level Builtin" section is built on a dichotomy that does not exist; InCore composites are a first-class path with their own ST suite.05— the Hello World HOST orchestrator will not compile (missing chip-orch layer, rank mismatch, tensor-into-pl.store,pl.dynamicused as a runtime shape).05vs06— contradictory claims about a missing.share_memory_(); the runtime raisesTypeError.tensor_ops.pyx6 +04/05— the "1-D host / 2-D InCore" signal-shape rule; the host rail accepts both.04—window/alloc_window_bufferfiled underpld.system.*; they arepld.tensor.*.06—prepare(ring_task_window=...)and the "prewarm" paragraph describe an API that does not exist.04/05—cmake -DSIMPLER_HOST_STRACE=1is a no-op; nothing in the build reads it.
Scope / hygiene:
04— thepl.*DSL-constructs andpl.system.*cross-core sections are L2 / intra-chip material and do not belong in a distributed API reference.04(and to a lesser degree06) — thePYPTO_BENCH*table documents pypto-lib-owned variables that have no effect in this repo.04—ScopeModeispl.ScopeMode;bench.py— the new RST table does not parse;05is 581 lines against the 500-line cap in.claude/rules/documentation-length.md.
All of the above apply equally to docs/zh-cn/user/, which mirrors the English text faithfully — including the errors. Worth fixing both sides in the same pass.
| | `window` | `(buf: Ptr, shape: Sequence[IntLike], *, dtype: DataType) -> DT` | Materialise a window-buffer `Ptr` as a `DistributedTensor` view. `buf` comes from `alloc_window_buffer`. | | ||
| | `alloc_window_buffer` | `(size: IntLike, *, name: str = "") -> Ptr` | Allocate a per-rank HCCL window buffer. **Size is in bytes.** The `name` kwarg is injected by the parser from the LHS assignment — never pass it explicitly. | | ||
| | `alloc_window_buffer` | `(shape: Sequence[IntLike], *, dtype: DataType, name: str = "") -> Ptr` | Convenience overload. `size = prod(shape) × dtype.get_byte()` computed automatically. | |
There was a problem hiding this comment.
Wrong namespace. window and alloc_window_buffer live in pld.tensor.*, not pld.system.*:
# python/pypto/language/distributed/op/system_ops.py:170
__all__ = ["get_comm_ctx", "notify", "nranks", "rank", "wait", "world_size"]
# python/pypto/language/distributed/op/tensor_ops.py:806
__all__ = ["all_to_all", "alloc_window_buffer", "allgather", "allreduce",
"barrier", "broadcast", "get", "put", "reduce_scatter", "window"]This also contradicts this page's own short-form table further down, which correctly maps pld.alloc_window_buffer(...) -> pld.tensor.alloc_window_buffer(...). Please move these three rows into the "Tensor Collectives (pld.tensor.*)" section (or give them their own pld.tensor.* window-management subsection).
| | `AtomicType` | `None_`, `Add` | Remote-store combine mode. `None_` (alias for `pld.AtomicType.None_`): plain store. `Add`: atomically accumulate into peer's destination. | | ||
| | `DistributedTensor` | — | A tensor view bound to a comm-domain window buffer. Every collective and RMA op requires this type on the window side. | | ||
| | `CommCtx` | — | Communication context handle. Produced by `get_comm_ctx()`; consumed by `rank()` and `nranks()`. | | ||
| | `ScopeMode` | `AUTO`, `MANUAL` | Dependency-tracking mode for `pl.scope()`. `AUTO`: OverlapMap tracks deps. `MANUAL`: user declares every edge with `pl.submit(..., deps=...)`. | |
There was a problem hiding this comment.
ScopeMode is pl.ScopeMode, not pld.ScopeMode — it is exported from python/pypto/language/__init__.py:228 (from .scope import ScopeMode, manual_scope, scope, spmd_submit, submit) and is absent from pypto/language/distributed/__init__.py's __all__.
It is already listed correctly in the pl.* table below, so this row can just be dropped.
| | `NotifyOp` | `AtomicAdd`, `Set` | Signal deposit mode. `AtomicAdd`: atomically increment the peer's signal slot (use for multi-rank barriers). `Set`: overwrite the peer's signal slot (use for 1:1 handshakes). | | ||
| | `WaitCmp` | `Eq`, `Ge` | Wait predicate. `Eq`: block until signal slot equals expected value. `Ge`: block until signal slot >= expected value. | | ||
| | `ReduceOp` | `Sum`, `Max`, `Min`, `Prod` | Reduction operator for collective operations. First-version lowering supports `Sum` only; `Max`/`Min`/`Prod` are reserved and currently rejected. | | ||
| | `AtomicType` | `None_`, `Add` | Remote-store combine mode. `None_` (alias for `pld.AtomicType.None_`): plain store. `Add`: atomically accumulate into peer's destination. | |
There was a problem hiding this comment.
Minor: `None_` (alias for `pld.AtomicType.None_`) is self-referential — it defines the symbol in terms of itself. Presumably the intent was to note that None_ is spelled with a trailing underscore because None is a Python keyword.
| | `put` | `(dst: DT, peer: IntLike, src: DT \| Tensor, dst_offsets=None, src_offsets=None, shape=None, *, atomic=AtomicType.None_, chunk_rows=0, chunk_cols=0, pipeline=False) -> Call` | `dst: InOut`, `src: In` | Write local `src` into peer rank's `dst`. `dst` **must** be window-bound; `src` may be plain `Tensor`. With no offsets/shape, writes the full local slice. `atomic=Add` accumulates instead of overwriting. `chunk_rows`/`chunk_cols` size the VEC staging tile (0 = full). `pipeline=True` enables ping-pong double-buffering (requires both chunk dims > 0). | | ||
| | `get` | `(dst: DT \| Tensor, peer: IntLike, src: DT, dst_offsets=None, src_offsets=None, shape=None, *, chunk_rows=0, chunk_cols=0, pipeline=False) -> Call` | `dst: Out`, `src: In` | Read peer rank's `src` into local `dst`. `src` **must** be window-bound; `dst` may be plain `Tensor`. With no offsets/shape, reads the full peer slice. | | ||
| | `allreduce` | `(target: DT, *, op: ReduceOp = Sum, mode: str = "mesh") -> DT` | `target: InOut` | In-place cross-rank allreduce. `mode="mesh"` (default): all-to-all exchange — O(N) remote traffic per step, one global barrier. `mode="ring"`: 2(P−1)-step reduce-scatter + allgather — O(N/P) remote traffic per step, more barriers. See the [Distributed Guide](05-distributed-guide.md) for when to use each. Host-orchestrator code may omit `signal` outside loops; the compiler synthesizes one of shape `[world_size(), 1]` (mesh only). | | ||
| | `allreduce` | `(target: DT, signal: DT, *, op: ReduceOp = Sum, mode: str = "mesh") -> DT` | `target: InOut`, `signal: InOut` | Explicit-signal form. Mesh signal shape: `[NR, 1]`. Ring signal shape: `[2 × (NR − 1), NR]`. Signal is single-shot — allocate a fresh buffer per call. | |
There was a problem hiding this comment.
The host-side signal rank is not 1-D only. LowerHostTensorCollectives accepts both rank-1 and rank-2 signals:
// src/ir/transforms/lower_host_tensor_collectives_pass.cpp:101-110
CHECK_SPAN(signal_type->shape_.size() == 1 || signal_type->shape_.size() == 2, call->span_)
<< "LowerHostTensorCollectives: collective signal must be rank-1 [world_size] "
"or rank-2 [world_size, 1]";and the compiler-synthesized signal on the host rail is itself rank-2 (synthesize_allreduce_signals_pass.cpp:243-247 builds MakeTuple{world_size_var, 1}) — consistent with the "synthesizes one of shape [world_size(), 1]" sentence in the row above, but inconsistent with the "host builtins use 1-D" claim repeated in six tensor_ops.py docstrings.
Suggest re-framing by mode / layer rather than host-vs-InCore:
- host builtins: rank-1
[world_size]or rank-2[world_size, 1](synthesized form is rank-2) - InCore mesh composite:
[nranks, 1]— rank count may be dynamic - InCore ring composite:
[2*(NR-1), NR]—NRmust be a compile-time constant
See also the thread on tensor_ops.py.
| ## Distributed DSL Constructs (`pl.*`) | ||
|
|
||
| These are host-orchestrator constructs in the `pl` namespace (not `pld`). | ||
|
|
||
| | Name | Signature | Description | | ||
| | ---- | --------- | ----------- | | ||
| | `spmd` | `(core_num: RangeArg, *, sync_start=False, name_hint="", optimizations=None, deps=None, allow_early_resolve=False, predicate=None) -> SpmdContext` | SPMD multi-block dispatch. Three forms: `with pl.spmd(n):`, `for i in pl.spmd(n):`, and `with pl.spmd(n, deps=[...]) as tid:`. | | ||
| | `cluster` | `(*, name_hint: str = "") -> ClusterContext` | Group co-scheduled AIC (Cube) and AIV (Vector) kernels sharing physical cluster resources. | | ||
| | `submit` | `(*args: Any, deps=None, dumps=None, allow_early_resolve=False, predicate=None) -> Any` | Submit a kernel and capture its producer TaskId. Parser construct — only valid inside `@pl.function`. Returns a 2-tuple `(result, task_id)`. | | ||
| | `spmd_submit` | `(*args: Any, core_num: int, sync_start=False, deps=None, allow_early_resolve=False, predicate=None) -> Any` | SPMD variant of `submit`. Launches a kernel across `core_num` blocks. Returns `(result, task_id)`. | | ||
| | `scope` | `(mode: ScopeMode = AUTO) -> scope` | Runtime scope context manager. `AUTO`: OverlapMap tracks deps (default). `MANUAL`: user must declare every edge. | | ||
| | `manual_scope` | `() -> manual_scope` | Alias for `pl.scope(mode=pl.ScopeMode.MANUAL)`. Turns off auto dep-tracking for the region. | | ||
| | `ScopeMode` | enum | `AUTO`, `MANUAL` | |
There was a problem hiding this comment.
This whole section looks out of scope for a distributed API reference. spmd / cluster / submit / spmd_submit / scope / manual_scope / ScopeMode are L2 orchestration-layer task-graph constructs; none of them are rank-aware or related to cross-rank communication. They already belong to 01-language_guide.md / 02-operation_reference.md.
Duplicating them here creates a second source of truth that will drift — the ScopeMode-under-pld.* error above is exactly that drift already happening. Suggest deleting the section and cross-referencing the language guide instead.
| The `prepare(ring_task_window=...)` call prewarms the ring scheduler with the | ||
| specified window size, so the first dispatch avoids cold-setup overhead. | ||
| Subsequent dispatches reuse the prewarmed state. |
There was a problem hiding this comment.
prepare() has no ring_task_window kwarg, and there is no prewarm.
# python/pypto/ir/distributed_compiled_program.py:366-374
def prepare(self, config=None, *, extra_compiled=(), persistent=False,
callbacks=None, sub_worker_overrides=None) -> "DistributedWorker":ring_task_window / ring_heap are RunConfig fields forwarded per invocation into CallConfig.runtime_env:
# python/pypto/runtime/runner.py:776-779
if run_config.ring_task_window is not None:
call_config.runtime_env.ring_task_window = run_config.ring_task_window
if run_config.ring_heap is not None:
call_config.runtime_env.ring_heap = run_config.ring_heapThe code block at 345-351 is correct; the paragraph after it describes behaviour that does not exist. Suggest replacing it with the actual constraints, which are enforced and worth documenting (runner.py:307-312): ring_task_window must be a power of 2 >= 4, ring_heap a power of 2 >= 1024 bytes; both also accept a per-ring list.
The section title "Ring Sizing and Prewarm" would need updating too.
| ### Environment Variables | ||
|
|
||
| | Variable | Default | Description | | ||
| | -------- | ------- | ----------- | | ||
| | `PYPTO_BENCH` | — | Set to `1` to enable the benchmark harness. | | ||
| | `PYPTO_BENCH_ROUNDS` | `100` | Number of measured launches. | | ||
| | `PYPTO_BENCH_WARMUP` | `3` | Leading launches discarded for cache/page warm-up. | |
There was a problem hiding this comment.
These four variables are defined and consumed by the pypto-lib golden harness, not by anything in this repository:
$ grep -rn "PYPTO_BENCH_ROUNDS\|PYPTO_BENCH_WARMUP\|PYPTO_BENCH_RAW" --include=*.py .
(no matches)
Transcribing another repo's defaults and expected stdout here guarantees drift. Suggest replacing the table with a one-line pointer ("provided by the pypto-lib benchmark harness — see ") and keeping this guide focused on pypto.runtime.benchmark(), which is this repo's contract. The same table is duplicated in 04-distributed-api-reference.md:168-176 and should be dropped there outright.
|
|
||
| | Operation | API | Modes | ReduceOp | Atomic | Supported dtypes | Notes | | ||
| | --------- | --- | ----- | -------- | ------ | ---------------- | ----- | | ||
| | AllReduce | `pld.tensor.allreduce` | `mesh` (InCore + HOST), `ring` (InCore only) | `Sum` only | — | FP32 (HOST builtin). FP16, FP32, BF16 (InCore hand-rolled) | Mesh: O(N) remote traffic. Ring: O(N/P) traffic, 2(P−1) steps. HOST ring builtin on a separate branch — not yet in this checkout. | |
There was a problem hiding this comment.
Two things in this row:
-
"HOST ring builtin on a separate branch — not yet in this checkout" — branch/worktree state does not belong in canonical user documentation; it is stale the moment the branch lands or is abandoned. If the limitation needs stating, phrase it as a capability ("
ringis currently InCore-only") without referencing a branch. -
The dtype column — I could not find backing for the
FP16, FP32, BF16 (InCore hand-rolled)restriction.LowerCompositeOpshandles the mesh target generically by dtype width (lower_composite_ops_pass.cpp:699-705: rejects only dtypes with no storage width, wider than the chunk budget, or not dividing the tile alignment) — no dtype allowlist. Either point at the check that enforces this list, or soften it to "not restricted by the lowering; validated per-dtype in ".
Note the "Modes" column here (ring (InCore only)) is correct and contradicts the InCore-vs-Host framing in 05-distributed-guide.md:361-371 — see that thread.
|
|
||
| sig = pld.tensor.barrier(sig) | ||
|
|
||
| **Signal shape:** 1-D ``[world_size]`` for host-level builtins; |
There was a problem hiding this comment.
This "1-D for host builtins / 2-D for InCore composites" sentence is added in six places in this file (barrier 646, broadcast 679, allgather 727, reduce_scatter 766, all_to_all 807 + its Args: signal at 816, plus the allreduce variant at 541-544), and mirrored into 04/05. The host half is not accurate:
// src/ir/transforms/lower_host_tensor_collectives_pass.cpp:101-110
CHECK_SPAN(signal_type->shape_.size() == 1 || signal_type->shape_.size() == 2, call->span_)
<< "LowerHostTensorCollectives: collective signal must be rank-1 [world_size] "
"or rank-2 [world_size, 1]";Both ranks are accepted, and the signal the compiler synthesizes for a host-orchestrator allreduce is itself rank-2 (synthesize_allreduce_signals_pass.cpp:243-247). Written as an either/or rule, this reads as "passing a 2-D signal on the host rail is illegal", which it is not.
The InCore half is also finer-grained than [NR, 1] suggests — mesh never reads the signal shape and loops on the runtime pld.nranks(ctx), so the rank count may be dynamic; ring requires a compile-time-constant NR (lower_composite_ops_pass.cpp:919, 948-966).
Suggested replacement (applies to all six sites):
**Signal shape:** host builtins accept rank-1 ``[world_size]`` or rank-2
``[world_size, 1]`` (the compiler-synthesized signal is rank-2). InCore
composites take rank-2 ``[nranks, 1]`` for mesh -- the rank count may be
dynamic -- and ``[2*(NR-1), NR]`` for ring, where ``NR`` must be a
compile-time constant.
Side note, not for this PR: ring's static-NR requirement has no CHECK_SPAN behind it. A symbolic NR falls into the MakeFloorDiv branch at lower_composite_ops_pass.cpp:966 and then fails inside InitMemRef rather than with a message pointing at the ring limitation.
| ``stats.samples`` Alias for ``device_wall_us`` (the raw list). | ||
| ``stats.per_round("device")`` List[float]: device wall per round. |
There was a problem hiding this comment.
This RST simple table is malformed — the column-1 marker is 26 characters, but two cells overrun it:
========================== =========================================================
<- col 2 starts at column 28
``stats.per_round("device")`` <- 29 chars
``stats.per_rank("device")`` <- 28 chars
docutils reports "text in column margin" for cells that cross the boundary, so the table will not render. Widening the first column marker to 30 = (and re-aligning the rows) fixes it.
The pl.spmd table added in dsl_api.py is fine — its 30-char first column clears the longest cell.
…rectories - Create distributed/ (6 files) and performance/ (5 files) in EN + zh-cn - Merge content from 04-distributed-api-reference, 05-distributed-guide, and 06-benchmarking-guide into the new chapter structure - Remove old flat files in EN and zh-cn - All new files <= 210 lines; 7 docstring backfill files unchanged
2 participants
Summary
Three new and expanded user-facing documentation guides for PyPTO distributed
programming, plus code docstring improvements across 7 modules.
New / substantially expanded guides
docs/en/user/05-distributed-guide.md— Distributed programming fromfirst principles: Hello World allreduce, notify/wait handshake, ring vs mesh
algorithm, InCore-vs-host collective comparison, all collectives with
worked examples, cluster launch, and a debugging table (6 common failure
patterns with symptoms, causes, and fixes).
docs/en/user/04-distributed-api-reference.md— Full API surface:types/enums (
NotifyOp,WaitCmp,ReduceOp,AtomicType),pld.system.*substrate,pld.tensor.*collectives with mutationcontracts,
pld.tile.*RMA, correctness contract, env var reference,put/get examples, DeviceTensor and DistributedWorker reference.
docs/en/user/06-benchmarking-guide.md—PYPTO_BENCH=1quick start,programmatic
benchmark()API,BenchmarkStatsfields and aggregates,span-tree rendering (
print_tree/print_mean_tree), L3 distributedbenchmarking (preparation, metrics, per-rank breakdown), bus-bandwidth
formulas with correction factors, ring sizing, L3 span-tree format, and
variance interpretation with warmup tuning advice.
Critical accuracy fixes across all guides
pl.FunctionType.Hostwith canonicalpl.Level.HOST+pl.Role.Orchestratordecorator.SIMPLER_HOST_STRACE/SIMPLER_DFXarecompile-time C preprocessor macros, not runtime env vars; documented
SIMPLER_DEVICE_STRACE_ENABLEas the runtime toggle.ir.compile()call signature — removed unsupportedconfig=RunConfig, useplatform=+distributed_config=directly.alloc_window_bufferexamples(
pld.world_size()→pld.world_size() * pl.INT32.get_byte()).import torch,from pypto import ir, andfrom pypto.runtime import DistributedWorkerin code examples.Code docstring improvements
Enhanced docstrings in
tensor_ops.py(allreduce restructure, signal shapes,cross-references),
system_ops.py(InCore warnings, naming notes),tile_ops.py(code examples),bench.py(BenchmarkStats quick-ref table),distributed_compiled_program.py(DistributedConfig fields, dispatchexamples),
dsl_api.py(spmd three-form table),ir.pyi(ReduceOp versionnote).
Translation
All changes mirrored in
docs/zh-cn/user/.Why this is an RFC
the repository — requesting review for factual accuracy, API correctness,
tone, and completeness before they become canonical reference.
alloc_window_bufferbyte-sizing formulasDistributedWorker/DeviceTensorAPI surfaceWhat this does NOT cover
pypto-lib/docs/performance-tuning.md)