[WIP] Add: partial-good-aware AICPU thread resolution (a2a3/a5) - #5
Draft
yanghaoran29 wants to merge 78 commits into
Draft
[WIP] Add: partial-good-aware AICPU thread resolution (a2a3/a5)#5yanghaoran29 wants to merge 78 commits into
yanghaoran29 wants to merge 78 commits into
Conversation
…git Worker.submit() constructs the DAG synchronously and returns a RunHandle that owns the run's completion, so L3 device work may still be active after submit() returns. Worker.run() is submit(...).wait(), so existing callers keep their blocking semantics and None return. - RunHandle exposes done, wait(timeout) and result(timeout); the terminal result is cached, so repeated and concurrent waits observe the same outcome and exactly one waiter performs fence-owned cleanup and release_run - wait_run_for binds a timed native fence wait that releases the GIL - the handle retains the callback, args and config until the fence fires, so run-owned resources outlive submit() - done reports false while a concurrent waiter is publishing cleanup, so it never queries a run id that waiter is about to release - graph-construction errors stay synchronous in submit(); device and endpoint errors belong to the originating handle and do not poison a later submission - fail_run_submission closes a run out from whatever state submission reached instead of rejecting one that is no longer building: the submission failure path waits on that fence, so refusing it left the waiter blocked forever - close() fences admission and drains accepted handles before tearing down the worker tree - L2 submission stays blocking and returns an already-completed handle Only one live device run is admitted: a later submission drains the previous handle before building another DAG, so preparation of one run does not overlap execution of another. Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
yanghaoran29
force-pushed
the
aicpu-pg-a2a3-a5
branch
from
July 25, 2026 09:52
d86ef48 to
bdd0454
Compare
yanghaoran29
changed the base branch from
rt-available-aic-aiv-counts-a2a3
to
pr1309-blockdim-auto
July 25, 2026 09:53
Split the A2/A3 onboard DeviceRunner::run() kernel submission boundary into launch_run(), and move stream synchronization, device-error recovery, profiling export and timing collection into reap_run(). run() stays strictly synchronous as launch_run() followed immediately by reap_run(). The effective statement sequence, every error return code and every recovery action are unchanged, including the collector teardown on the sync-failure path and print_handshake_results() on the success path only. reap_run() takes no arguments and reads only members, so a later change can defer it without re-splitting its inputs. Correct the run() step list, which named the AICPU kernel before the AICore kernel while the launch has submitted AICore first. No ACK states, resource banks, background reaping, or execution overlap are added here. Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
…tive-sys#1472) The scene test derived its expected block slots from the count the runtime reported, so a count that was too small was self-consistent and passed. It also ran on tensormap_and_ringbuffer only, leaving the host_build_graph read path — the one that was returning zero — uncovered. - A Pinned case fixes block_dim, giving the host an expectation independent of what the runtime reported. It is the only case that can catch an under-count; the Auto case, which takes whatever the platform resolves, cannot. The pinned width differs from every platform's auto width, so a count sourced from the ceiling rather than from this run fails it too. - The same case runs on host_build_graph. Against the pre-fix ordering it fails at `block_num >= 1` in submit_task, because host orchestration read a cluster count of zero and asked for a cohort that wide. - The shape tensor no longer takes a dummy_task producer. host_build_graph runs its orchestrator to completion on the host before the device executes anything, so a producer's task_state can never reach COMPLETED and set_tensor_data's wait_for_tensor_ready spins out to PTO2_TENSOR_DATA_TIMEOUT_CYCLES. The MIX cohort already keeps the graph non-empty, and shape has neither producer nor consumer.
…-native-sys#1477) CoreTracker packs three state bits per cluster (AIC, AIV0, AIV1) into a single uint64_t, so it held floor(64/3) = 21 clusters — and the two literals that encoded that, MAX_CORE_PER_THREAD = 63 and core_id_map_[63], came from the width of the backing word rather than from any device. a2a3 has 24 clusters and a5 has 36, so neither fits. assign_cores_to_threads() checks the ceiling; assign_own_clusters(), the barrier-free path the decoupled scheduler actually takes, does not. A run that gives one scheduler thread more than 21 clusters therefore writes past core_id_map_ into the next CoreTracker, which on that path is the orchestrator's. For 24 clusters the store lands exactly on core_trackers_[1].cluster_count_, so the orchestrator — whose tracker is never assigned and whose shutdown() documents "core_num() == 0 -> no-op" — reports 210 cores, walks a zeroed id map, and sends AICORE_EXIT to core 0 two hundred times. If the scheduler still has a task dispatched there it polls that core's COND for a FIN that can no longer arrive, and the run dies on the scheduler's forward-progress timeout. Nothing reaches this today because every scene test pins block_dim low enough to stay under 21 clusters per thread, but the bound is a property of the hardware, not of the test suite. - MAX_CLUSTERS now derives from PLATFORM_MAX_BLOCKDIM and MAX_CORE_PER_THREAD from it, so one scheduler thread can own a whole device on either arch. A static_assert ties the pair to the storage. - BitStates is backed by unsigned __int128 (a2a3 needs 72 bits, a5 108). The hot path shifts the whole mask by 1 and 2 to test cluster co-residency, and letting the compiler generate those crossings is what keeps the widening honest; only popcount and ctz split by hand. - Callers can no longer build a mask with `1ULL << offset`, undefined once offset reaches 64. BitStates::bit() owns the shift. - assign_own_clusters() gets the guard its serial sibling already had, and init()/set_cluster() assert the bound at both ends, so exceeding it fails by name instead of corrupting a neighbour. The lower bound matters as much as the upper one: a negative cluster_idx would index core_id_map_ backwards, and a negative cluster_count would leave every dispatch loop visiting no clusters at all.
…-sys#1481) The a5 self-hosted runner is out for repair, so the job has no host to run on and sits waiting for one instead of reporting a result. Short- circuiting its existing gate makes it report skipped: nothing declares `needs: st-onboard-a5` and main has no required status checks, so the job's absence blocks no merge. Coverage for a5 changes falls back to st-sim-a5, which runs on GitHub-hosted runners. ut-a5 targets the same self-hosted runner and is left enabled.
…hw-native-sys#1482) Fixes hw-native-sys#1479 RunHandle.wait() elects one waiter to cross the native fence and publish cleanup. That waiter claimed the election with _wait_in_progress, then called Worker._finalize_run_handle() unguarded and only afterwards marked the handle terminal. An asynchronous exception (KeyboardInterrupt / SystemExit) landing anywhere in that window propagated out with the election still claimed and the handle still un-terminal. Nothing else ever clears that state, so every later wait() blocked or timed out forever, done reported False permanently, and the handle was never retired from _accepted_run_handles — leaving Worker.close(), which drains accepted handles with an untimed wait(), unable to ever complete. Publication is now unconditional and resilient, in the shape close() already uses for its _CloseAttempt: - an exception from _finalize_run_handle() is cached as the handle's error instead of being lost, and still propagates out of wait() - terminal state is published with plain attribute assigns (_error before _terminal) BEFORE acquiring the CV, whose only remaining work is notify_all(); the two election-abandonment paths publish the same way, so an async exception in the interruptible acquire cannot strand the handle - a parked waiter re-checks on a bounded interval, so a skipped notify_all() is recovered instead of blocking forever; wait(timeout) semantics are unchanged - _finalize_run_handle() retries its trailing discard/notify once, so an interrupt during that acquire cannot skip retiring the handle from _accepted_run_handles Three regression tests cover the interrupted-finalize paths: the elected waiter's terminal publication and cached error, a second waiter parked behind it, and retirement when the finalize CV acquire itself is interrupted.
…nsumed (hw-native-sys#1485) Fixes hw-native-sys#1480 `TensorMap::erase_task_outputs` erased every key unconditionally, so a consumed task's cleanup could drop a mapping that a later same-run task already owns. Two `OUTPUT` / `OUTPUT_EXISTING` writers of one key carry no edge between them — that path inserts without looking up — so the first writer can reach CONSUMED while the second is still running. Its cleanup then removed the entry pointing at the second writer, and a subsequent `INPUT` consumer looked up an empty slot, inferred no dependency, and could be dispatched before the second writer had written the buffer. `INOUT` is unaffected: it looks up before inserting, so the writers are ordered and the erase is harmless. The erase now takes the consumed slot and drops a key only while it still maps to that slot, matching how the L2 TensorMap reclaims a retired task's entries (`pto_tensormap.h::cleanup_retired`). Whether two unordered `OUTPUT` writers of one key should instead be a diagnosed error is left open.
…hw-native-sys#1486) dep_gen_aicpu_init() resolves the collector's buffer state and pops its first buffer, and it ran from the scheduler cold path after the AICore handshake, next to pmu_aicpu_init(). pmu belongs there — it needs the physical_core_ids_ the handshake produces. dep_gen needs nothing from it: init only reads dep_gen_data_base, which kernel.cpp publishes before aicpu_execute() starts. The orchestrator, meanwhile, deliberately skips that handshake: if (decouple_orch && is_orchestrator) { return 0; // do NOT touch the handshake or hs_arrived_ barrier } so it goes straight to building the graph while the schedulers are still handshaking cores. Every submit_task it issues in that window reaches dep_gen_aicpu_record_submit() with s_dep_gen_state still null and is dropped. Whether any records survive is a race against handshake wall time, which scales with core count: at three clusters the schedulers usually win, at a full device they usually do not, and in between runs capture a partial graph missing only its first tasks. Initialising dep_gen on the thread that feeds it removes the race — there is nothing left to order it against. Both cold-path call sites go, not just one: the free_queue is SPSC and the orchestrator is now its only device-side consumer. The failure was silent because the early return skipped the accounting one line below it, the line whose comment reads "Account every attempted record so total == collected + dropped on host". The host's reconcile_counters() saw 0/0/0, called that consistent, and wrote an empty deps.json instead of refusing. A pre-init record cannot be accounted — the counter lives in the BufferState that is still null — so it now reports itself once instead. Once, because record_submit is a per-task path and the AICPU log is not free. a5 carried the identical arrangement and is fixed with it.
Define the versioned PipelineContract C ABI and the HOST_PER_RUN / DEVICE_SCRATCH / EXEC_HANDLE resource classes, and let the A2/A3 host_build_graph and tensormap_and_ringbuffer runtimes declare the resources that cross KernelLaunch. A resource's copy count follows from its class: HOST_PER_RUN and EXEC_HANDLE need one instance per in-flight run, DEVICE_SCRATCH needs exactly one because it is not rewritten per run and device ops run one at a time. That leaves pipeline_depth as the only replication count, so a runtime that wants a small resource replicated and a large one shared says so by classifying them rather than by carrying a second count. host_build_graph materializes this run's own graph into the image it uploads, so every device region it declares is HOST_PER_RUN. tensormap_and_ringbuffer orchestrates on the device, so only its task args carry per-run content; its heap, SM and runtime image are built and uploaded once per callable sizing and then served from the prebuilt-arena cache, so they are DEVICE_SCRATCH. Zero is reserved as an unspecified kind. An entry a runtime never filled in stays zeroed, so a declaration that overstates resource_count is rejected whatever its filled entries are, and a kind may repeat for a runtime that needs several resources of one type. ChipWorker loads the declaration as optional runtime metadata and rejects one it cannot honor: wrong ABI version, too many resources, a depth other than 1, an unspecified or out-of-range kind, an out-of-range class, or a non-zero reserved bytes_per_copy. A runtime that exports no contract keeps the default depth-1 shape. The contract is published only once the runtime is up, so a failed init never reports the shape of a runtime this worker is not bound to. One runtime buffer and one pipeline slot are kept throughout. A depth above 1, slot selection, asynchronous acceptance and overlap remain later changes; the existing synchronous simpler_run -> launch_run -> reap_run path is unchanged. Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
…w-native-sys#1478) The six spmd_sync_start scene tests hardcoded their cohort widths and cache-line bases against a 24-cluster device. Those literals silently encode one device geometry, so the cases only ever exercised the shape they were written for and would mis-slice a device of any other width. Each orchestration now derives its cohort widths from rt_available_cluster_count() / rt_available_aiv_count() as fractions of the run's actual width, clamped to at least one block, and reports the geometry it used (block_num and base cache line per task) in a new `layout` output tensor. The host sizes `output` for the widest device the platform allows and rebuilds the golden from the reported layout in a compare_outputs override, so the check follows the device instead of assuming it. Every case keeps its existing block_dim pin, and on a 24-cluster device the same widths fall out of the divisors — with one correction. a5 spmd_sync_start_mix_spill pinned block_dim 24, which is 48 AIV cores, but asked for a 72-block producer under the comment "fill all a5 AIV cores"; 72 is the count for a 36-cluster device. Deriving the width makes the producer 48 there, which is what the case says it wants. Co-authored-by: majin0824 <majin15@huawei.com> Also skips the bgemm host_build_graph case. Its golden comparison reads memory the run never wrote — max_diff between 1e25 and 1e36 on inputs of magnitude 1e-2 — roughly one full-suite run in two or three, and never when the case runs alone. It reproduces on unmodified main with a from-scratch rebuild, so it is not this change and not a stale build; tracked in hw-native-sys#1483. Skipping it keeps that noise out of the signal here.
block_dim let a caller pin how many AICore blocks a task occupied, but the value is a property of the device, not of the task: every in-repo caller either left it at the "auto" sentinel or pinned the width of the one device it was written for. Pinning it below the device width simply wasted cores, and pinning it above was rejected at run time, so the knob only ever expressed what the runtime already knows. - Drop CallConfig::block_dim and its plumbing through the task interface, the packed mailbox wire layout, the remote-L3 protocol, and both platform device runners; the runner now always resolves the block count from the stream's capacity. - Strip the pinned "block_dim" entries from the scene tests and examples that carried them, and update the fixtures that constructed a CallConfig with one. The wire contract was asserted in three places — the header's static_assert, a C++ unit test that restated the same sizeof expression, and a remote-wire round-trip that set the field — and all three move with the layout. - The two spmd_sync_start_mix_spill docstrings stop quoting core counts. They named a5's 72 AIV / 24 clusters and a2a3's 48 / 24, which were already inconsistent with the block_dim those cases pinned and are meaningless once the width is the device's own. (The a5 one also carried a paragraph twice.) - available_aicore_counts loses its Pinned case: it fixed block_dim so the host held an expectation independent of the reported count, which was the only way to catch an under-report. Nothing replaces it, and the docstrings say so rather than implying coverage that is gone. - Sweep the docs and skills that documented the knob. Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
…-native-sys#1488) Fixes hw-native-sys#1483 `tests/st/a2a3/host_build_graph/bgemm` declared its `C` argument `D.OUT`, but the task graph read-modify-writes it: `p_add.add_inout(c_view)` feeds `kernel_tile_add`, whose first `k` iteration `TLOAD`s `C` before any task has written it. The runtime stages only IN/INOUT tensors host->device (`runtime_maker.cpp`, "pure write-only OUTPUT buffers are never read by the kernel"), and since hw-native-sys#1365 it no longer zero-fills OUT buffers either. So `C` accumulated onto whatever the device allocation happened to hold. On sim that allocation is a plain `malloc` block on a process-lifetime worker: fresh zeros when the case runs alone, a previous case's bytes in a full-suite run — hence a golden mismatch of 1e25-1e36 that reproduced only under the full suite. Same class as hw-native-sys#1449. Declare `C` INOUT, matching the two sibling bgemm variants (`examples/a5/tensormap_and_ringbuffer/bgemm`, `examples/a2a3/tensormap_and_ringbuffer/benchmark_bgemm`), which already tag their accumulator that way. Seed `C` with a non-zero base (0.25) so the staging is load-bearing and the case is deterministic: golden is now `C_init + A @ B`, which a zeroed or unstaged device buffer cannot match. Without the signature fix this fails every run instead of one in two or three — verified, it reproduces the reported 1e35 magnitude on the first attempt. Drop the `pytest.mark.skip` that quarantined the case: the underlying defect is fixed, so the case runs in CI again. Verified: bgemm passes 3/3 in isolation on a2a3sim and onboard on a2a3; the full `pytest examples tests/st --platform a2a3sim` suite is green 3/3 runs.
…patch (hw-native-sys#1490) `_increment_counter` is a non-atomic read-modify-write on one shared 4-byte cell, so a slot tolerates at most one writing process. `test_l4_sub_and_l3_dispatch` is the only case in the file whose L4 parent has both its own sub worker and an L3 child, and its orch submits to both in one run. L4's SubWorker and the L3 child's SubWorker are distinct forked processes dispatched from independent WorkerThreads with no ordering between them, so when both read the cell before either writes, one increment is lost and the total reads 1 instead of 2. Every other counter test in the file funnels its increments through a single `num_sub_workers=1` process, which is why only this one is affected; `TestL4L3WithMultipleSubs` documents that workaround in its docstring, but it cannot apply here because the two incrementers belong to two different workers by construction. This is a lost update, not a dropped dispatch: `Worker.run()` waits on `active_tasks == 0` with no timeout, so a task that never ran would hang rather than under-count. Give `_make_shared_counter` / `_read_counter` / `_increment_counter` an optional slot index (default 0, so the six single-writer call sites are unchanged) and give each path its own slot. Asserting the two slots separately also names which path was lost when it fails, where the previous `assert 1 == 2` did not. Observed as a macOS-only CI flake — 2 failures in 218 sampled `ut (macos-latest, 3.10)` jobs, 0 in 101 Ubuntu jobs. The race is real on both platforms; the macOS runner's narrower core count widens the window between the read and the write.
…w-native-sys#1492) Fixes hw-native-sys#1487 host_build_graph called dep_gen_aicpu_init() from its scheduler cold path but never called dep_gen_aicpu_set_orch_thread_idx() or _flush(), leaving the subsystem initialised and half-wired. It cannot be finished the tensormap_and_ringbuffer way: host_build_graph orchestrates on the host, so there is no device orchestrator thread to own a ready queue, nothing on the device to flush, and the AICPU never submits a task. Capture the graph where this runtime actually builds it. compute_task_fanin takes an Annotate hook that fires alongside its existing emit, so creator retention and tensormap lookups are recorded as the runtime resolves them, and submit_task_common opens the task entry and records declared dependencies. What lands in deps.json is therefore the runtime's own dependency resolution rather than a replay's reconstruction of it, and it cannot drift from compute_task_fanin semantics. No ring, no collector, no reconcile: the orchestration completes before any scheduler thread starts, so nothing can be dropped under back-pressure. The capture state is thread-local, matching the per-thread runner both c_api_shared.cpp files already key off a pthread_key_t — two runners on two threads build two independent graphs, the same per-runner isolation the device-orch shape gets from DeviceRunner::dep_gen_collector_ being a member. emit() refuses to write when no capture ran on its thread, so a mis-wired arm reports instead of producing an empty-but-valid deps.json. deps.json keeps the schema the device-orch replay emits, so deps_viewer and the swimlane join read both runtimes identically. Two st cases cover it: one runs the same vector_example orchestration as the tensormap_and_ringbuffer dep_gen test and asserts the same 6 edges (which is what would catch the two shapes diverging), the other runs predicated_dispatch for the tensormap and explicit edge sources and their producer-side slice annotation. Both are wired into the a2a3 dep_gen smoke steps, sim and onboard — the default st sweep passes no --enable-dep-gen, so a capture regression is only visible there. The runner picks the shape via dep_gen_host_graph_active() and skips device collector init/start/reconcile for the host-orch one; host_build_graph's copy of dep_gen_replay.{h,cpp} is removed, matching what the device runners' weak stubs already claimed. Also guard the shared AICPU writer: a record arriving before dep_gen_aicpu_set_orch_thread_idx() has no ready queue to reach, so it is now charged to dropped_record_count instead of filling a buffer nothing can publish and surfacing later as a misleading "ready_queue full" error.
…ys#1464) Give the A2/A3 device runner its own AICPU/AICore stream set per pipeline slot and submit every run on the selected set, so two in-flight runs would never share a stream. The persistent pair created by ensure_device_initialized() stays, now reserved for bootstrap and control work: the Init and RegisterCallable payloads and the block-dim query. A stream set carries no per-run content, so a slot's set is created on first use and reused for every later run on that slot. The pipeline contract classifies both streams as EXEC_HANDLE, which asks for one instance per in-flight run — that is one set per slot, not one per run invocation — so kRunStreamSetCount is PTO_PIPELINE_MAX_DEPTH and only slot 0 is selected while the contract is K=1. destroy_run_stream_sets() in finalize() is the sole release point, running while RTS is live and before the device reset, which is the window finalize_common() uses for the bootstrap pair. It is idempotent, so the destructor's finalize() call is a safe backstop. As there, no pre-destroy sync: rtStreamDestroy is the supported teardown for a stream an op-timeout left in the error state. sync_run_streams() is split so the wait can name a stream pair; DeviceRunnerBase keeps the no-argument form for A5, whose runs still use the persistent pair. Report the number of sets a runner has built through the C API and ChipWorker, alongside the dlopen counters that already serve this purpose, so the reuse invariant is assertable. A new onboard scene test runs one worker four times and requires exactly one set; rebuilding per run makes it report four. Rebuilding the set per run costs two rtStreamCreate plus two rtStreamDestroy, ~300 us each, on the synchronous host path around KernelLaunch. Measured on a2a3 silicon with vector_example at 40 rounds, three alternating repetitions of each binary under one device lock, host time per round (p50 of per-repetition p50, warmup rounds dropped): main 10316.8 us, per-run rebuild 12607.6 us (+22.2%), reuse 10481.1 us (+1.6%, inside main's own 2.7% repetition spread). Device time is unchanged. Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
…w-native-sys#1494) `_sub_worker_loop`, `_child_worker_loop` and `_run_chip_main_loop` each carried their own copy of the same protocol — read the mailbox state word, dispatch on TASK_READY / CONTROL_REQUEST / SHUTDOWN, write the error region, publish the matching DONE state. All three docstrings described themselves as the "unified TASK_READY / CONTROL_REQUEST state machine", three separate times. Only two things ever differed: what a task does (a Python callable, a nested `Worker.run`, or a device `run_from_blob`) and which control sub-commands are accepted (2, 6 and 15 respectively). Both are now closures the caller passes to `_run_mailbox_loop`, which owns the state machine. This does not shrink the code — 389 lines becomes 386, because the handlers were always the bulk. What it buys is single ownership of the protocol: the "publish the error report before the DONE state" ordering that the parent depends on is enforced in one place instead of being restated three times, and there is now exactly one site that waits on a mailbox. Replace `_child_worker_loop`'s nested-ternary control-op naming with a `_CTRL_OP_NAMES` lookup beside the sub-command constants. `_run_chip_main_loop` keeps its PLR0913/PLR0915 suppressions: it is a fork-child entry point, so every dependency its handlers close over has to arrive as an explicit argument, and PLR0915 counts the nested control handler's per-sub-command statements against it.
…s#1495) A forked sub/child/chip worker never checked whether its parent was still alive. When the parent died without `Worker.close()` — SIGKILL from a `timeout`, an OOM kill, a cancelled CI job, an editor killing a wedged pytest — every child was reparented and kept polling a mailbox nobody would ever write to. Because that poll is an unbounded busy-wait, each survivor held a full core indefinitely. This is not theoretical. On a shared dev box six such processes belonging to two different users had been alive 5-7 days each at ~99.6% CPU, six cores permanently consumed. Four of the six carried `tests/ut/py/test_worker/test_startup_readiness.py` on their command line: runs whose parent was killed rather than allowed to finish. `_run_mailbox_loop` now samples `getppid()` on its idle path and leaves by the same SHUTDOWN route once it changes, so a child tears down exactly as it would on a clean shutdown. Compare against the pid captured at loop entry rather than testing for pid 1: a subreaper (container init, a systemd user session) adopts orphans instead of init, so the pid changes but never becomes 1. A live parent's pid cannot change, so the check cannot fire spuriously. Sampling every `_PARENT_LIVENESS_POLL_INTERVAL` idle polls puts a `getppid()` roughly every 100 us against a ~0.1 us poll — fast enough that an orphan goes away before it is noticeable, cheap enough to be lost in the noise of the poll itself. Residual gap: a parent that dies between `os.fork()` and the child reaching the loop is already gone when the pid is captured, so that child is not detected. Closing it needs the pid handed in from before the fork, or `PR_SET_PDEATHSIG`, which is Linux-only and therefore an optimisation on top of this rather than the mechanism. `test_orphan_child_reaping.py` SIGKILLs a subprocess parent and asserts every child is gone within 20 s. Three details are load-bearing: - It reads the pids from the Worker's own `_sub_pids` rather than `pgrep -P`. Enumerating children externally also catches the transient shell that runs pgrep, and inside a container's shallow pid namespace it picks up unrelated low pids: on CI that produced "1 of 6 children outlived ... [3]", waiting on and then SIGKILLing a container process that was never ours. - It writes to a file and never reads a pipe. Orphans inherit the parent's stdout, so on regression they hold a pipe's write end open and reading one would hang the test for as long as the bug survives instead of failing it. - Its `finally` kills whatever survived, so a regression does not hand spinning processes to the next test. - It asserts the parent died from its own SIGKILL. A parent that fails before forking otherwise surfaces as an unexplained "expected 3 pids, got []", which reads like a defect in the code under test rather than an environment problem in the harness. It runs on macOS as well as Linux: reparenting on parent death is POSIX behaviour, and nothing here is platform-specific once the pid list comes from the Worker. Also drop `src/a2a3/runtime/host_build_graph/docs/GRAPH_EXECUTION.md`, which hw-native-sys#1494 added by accident: it is documentation from the open hw-native-sys#1444 (`feat/graph-execution`) that happened to be sitting untracked in the working tree, and a `git add -A` swept it into that commit. Removing it restores main to the state hw-native-sys#1444 expects to merge into. It is unrelated to everything else here and can be reviewed independently of the fix
…hw-native-sys#1499) Codestyle rule 5 banned `yield()` in AICPU spin-waits but said nothing about the host, so a sleep-based backoff on a mailbox poll read as acceptable. It is not: a sleep quantum lands on every dispatch that arrives mid-quantum, and no tuning recovers it, because a sleep interval on a general-purpose OS overshoots what was asked for — a 50 us request measures ~102 us on this box. Generalize the rule. A wait that a task's latency passes through may only spin or block on a wakeup primitive (semaphore, futex, condition variable, pipe/eventfd read). Initialization and teardown are exempt and may poll with a sleep, since a bring-up handshake or a shutdown drain costs a one-off wait rather than per-task latency. The boundary is "does a task wait behind this?" rather than which tier or file the wait lives in, so a host completion poll lands on the same side as an AICPU ticket lock while `close()` reaping lands on the other. `LocalMailboxEndpoint::run` was the one place in the hierarchical worker that broke this: it slept 50 us between `TASK_DONE` polls, on the critical path of every dispatch. Its sibling `CONTROL_DONE` wait in the same file already spins and samples child liveness against a `kChildLivenessPollPeriod` wall clock, so the dispatch wait now matches it. Liveness has to move off the iteration count for the same reason the old comment gave for using one — "~10 ms at the 50 us sleep below" — that mapping only held while the loop slept. Measured, 3 sub workers with no-op callables: before after run() median 150.2 us 50.7 us run() p90 152.3 us 51.6 us run() max 175.9 us 94.1 us burst, per task 106.0 us 11.0 us `kChildLivenessPollInterval` loses its last caller and goes with it. The trade is CPU: a parent thread now spins for as long as its child holds a task, so W concurrently-dispatched workers keep W threads busy (measured 2.98 cores for 3 outstanding dispatches). On the dispatch path that is what the rule asks for, and it does not degrade under core pressure — `tests/ut/py/test_worker` plus `test_hostsub_fork_shm.py` run 342 passed in 54.3 s pinned to 3 cores against 52.6 s unpinned. Removing the CPU cost as well needs a blocking wakeup primitive on both ends of the mailbox, which is tracked separately and is what the rule's "spin *or* block" is pointing at.
…ive-sys#1500) comm_alloc_domain_windows wiped a rank's freshly allocated window after the alloc path's subset barrier — on HCCL after the file barrier that gates the peer imports, on sim after the ready_count barrier. Publication is the point from which a peer may reach the window: a peer that clears the barrier can return, launch its kernel and store a barrier signal into this rank's window, so a wipe issued any later erases a signal the owner has not yet waited on and leaves that owner's TWAIT spinning forever. Move the wipe ahead of publication on all three backends — before the shareable-handle announce on a2a3 (both the Fabric V2 and the VMM IPC paths) and on a5, before the ready_count barrier on sim. No peer can import a window before that point, so the existing barriers now order the wipe too and no extra synchronization is needed. a5's DomainAllocation::alloc_size loses its only reader with the caller-side wipe and goes with it. Orchestrator.allocate_domain joins every rank's chip child before any kernel is submitted, so no kernel can reach a peer's window before that peer's wipe and the hazard is unreachable through today's callers: this removes an ordering footgun, not a live failure. Refs hw-native-sys#1489
…w-native-sys#1497) The AICore loader copies the literal `.text` bytes and jumps to offset 0, so the payload has to be self-contained. Nothing made it so: the loader rejected any object carrying `.rela.text` or `.text._Z*` and told the author to annotate the call chain with `always_inline`. That is not always available — CANN AscendC declares `g_vecTPipePtr` / `g_cubeTPipePtr` and `g_kfcClient` as block-local globals in separate `.bl.uninit.*` sections, so every kernel built on AscendC carries data relocations no amount of inlining removes. `compile_incore` now links each object with `ld.lld -e kernel_entry`. The linker applies `.rela.text` and folds `.text.*` COMDAT groups into the output `.text` — the work issue hw-native-sys#900 recorded as missing. `extract_text_section` keeps both checks as post-link assertions and gains a third: `kernel_entry` must sit at the start of `.text`, because merging COMDAT groups lets the linker place another function first and the loader would enter that instead. Leaving these relocations unapplied is not a benign shortcut. pypto's forked copy of this parser has no such check and uploads the raw `.text`; for the qwen3 attention extern that leaves `g_kfcClient` resolved to offset 0, aliasing onto `g_vecTPipePtr`'s slot. It survives only because that kernel synchronizes through an FFTS barrier and never touches the KFC mailbox. All 202 incore kernels in the repo (121 a2a3, 55 a5, 26 under examples/workers and the non-arch st dirs) produce byte-identical payloads before and after, so no currently-working kernel changes. A kernel with a noinline template that was previously rejected now loads: `.text` 80 B + `.text._Z4bumpIlET_S0_` 8 B links to an 88 B payload.
…log (hw-native-sys#1504) `test_orphan_child_reaping` scraped the forked children's pids out of the parent subprocess's combined stdout+stderr, taking every digit-only token it found. That stream also carries log lines and warnings, so any bare number in them was read as a pid. On CI it collected a fourth entry: expected exactly 3 forked sub-worker pids, parent reported [6778, 6779, 6780, 3] The three real pids were correct — `Worker._sub_pids` is not at fault — and the run was otherwise healthy, so the test failed on main for a parsing artifact rather than a defect. It also meant the `finally` block would have sent SIGKILL to whatever process that stray number named. Give the parent a dedicated output file, named on its command line, and keep stdout+stderr as a separate log used only for diagnostics when an assertion fails. Parsing a channel that carries nothing but the pids removes the class of failure rather than filtering harder against it, which is the same mistake in a different shape. Also drop the `isdigit()` filter, now that a non-numeric token in the pid file would be a real defect worth surfacing rather than noise to skip. 15/15 local runs pass; red-to-green still holds against the pre-hw-native-sys#1495 `worker.py` ("3 of 3 children outlived their SIGKILLed parent by 20s", now with exactly three pids and no phantom entry).
…ive-sys#1502) After hw-native-sys#1499 took a single `Worker.run()` from 150 us to ~51 us, the obvious follow-up was whether another factor of three was available in what remained, and whether a blocking wakeup primitive (hw-native-sys#1498) was how to get it. Measured, and the answer to both is no. Recording it so the next person does not re-derive it. A bare fork+shm mailbox round trip with no simpler runtime in it costs 3.5 us against `Worker.run()`'s 53.8 us, which reads as "94% of dispatch latency is our code, not the IPC". That headline is an artifact of benchmarking a single task. Sweeping tasks per `run()` from 1 to 256 splits it into **43.3 us fixed per run() and 8.08 us marginal per task**: the cost is entering and leaving an orchestration, not dispatching work, and a production run submits a graph per `run()` rather than one task. At 256 tasks the fixed part is 2% of the invocation. The same numbers settle hw-native-sys#1498 against itself on latency grounds. A pipe wake measures 4.4 us one way on this box, so replacing the poll with a blocking wait adds more than the entire 3.5 us IPC floor to every dispatch, in exchange for CPU, while the latency it was meant to recover is not in the poll at all. What survives is narrower and unrelated: a child spinning while nothing is outstanding holds a core for as long as its parent lives — a CPU question for narrow-core hosts, already smaller since hw-native-sys#1495 made orphans exit with their parent. Also records the reconsideration triggers, since both conclusions are conditional: a workload that issues many small `run()` calls stops amortizing the fixed cost, and the CPU argument for a spin-then-block hybrid remains open.
…s#1501) The guide covered capacity codes and stalls but had nothing for an AICore addressing fault, which is the third way a run reaches the same generic 507018. hw-native-sys#1489 spent an investigation re-deriving the procedure from scratch and still could not close, so record it. Three steps, ordered so the cheap disqualifications come first: - F1 separates "the kernel computed a bad address" from "the core was made to execute something that is not the kernel". `binSize` in GetBinAndKernelNameExceptionArgs matching the runtime's own aicore_kernel.o means the report is naming the polling-dispatch executor, so the fault is a dispatch-payload problem and no amount of reading the kernel's arithmetic will find it. hw-native-sys#1036 is the worked example. - F2 rules the kernel in or out statically. Constant TASSIGN bases plus template tile extents do not make an out-of-range address impossible — the constants can overrun on their own — but they make it decidable on paper, because runtime values that only shrink a tile move no base and grow no extent. Summing the highest byte reached and comparing against the UB size settles it either way without touching the device. This is what cleared the allreduce collectives in hw-native-sys#1489. - F3 separates a real fault from a post-mortem register dump of a core reaped mid-spin, by counting which detector actually fired. hw-native-sys#1489 has since been closed as very likely fixed by hw-native-sys#1477 — a CoreTracker whose uint64_t state overran past 21 clusters while those cases ran at the device's full 24. F2's verdict held: the kernels were never at fault, and the answer was in F1's second case all along. The section says so, because a worked example that names its own outcome is worth more than one that trails off. Also state where the device log has to be redirected and why the harness will not do it: outputs/<case>_<ts>/ exists only when a DFX flag is on, so a plain onboard run leaves its log in the shared ~/ascend/log/debug/ with every other user's, and the evidence is unattributable once the run ends. That is how hw-native-sys#1489's only real occurrence lost the one artifact that would have decided F1. Renumber the trailing section and fix the in-page link that pointed at its old number.
…w-native-sys#1452) * host_build_graph: all AICPU threads schedule (drop orch/sched split) host_build_graph builds the task graph on the host, so no AICPU thread ever orchestrates on device. The runtime nonetheless reserved thread N-1 as an "orchestrator" that owned zero AICore cores and did not dispatch, leaving only aicpu_thread_num-1 scheduling threads (3 of 4 in every scene test). The PTO2_ORCH_TO_SCHED / orch_to_sched_ "orchestrator-to-scheduler transition" was a tensormap_and_ringbuffer (device-orch) concept that, in host_build_graph, only made the core-less boot thread enter the dispatch loop with nothing to dispatch. Remove the split entirely (matching a5 host_build_graph, which already runs all threads as schedulers): - Delete orch_to_sched_, runtime.orch_to_sched, and the PTO2_ORCH_TO_SCHED env read. - Delete sched_thread_num_; assign cores over all aicpu_thread_num threads (active_sched_threads_ = aicpu_thread_num_). The drain barrier, completion counting, and cluster-id math already key off active_sched_threads_, so they scale to the full thread count. - The last thread (aicpu_thread_num_ - 1) still performs the one-time host-orch boot preamble, then falls through and schedules its own cores like every other thread. At aicpu_thread_num=4 this yields 4 scheduling threads (was 3). NUMA locality is unchanged: compute_allowed_cpus still pins all 4 threads to one AICPU cluster (cpu_ids {4,5,6,7} on the observed 0xfc pool) because active_count stays 4 — do NOT raise aicpu_thread_num past a single cluster's core count or threads spill across the NUMA boundary. * host_build_graph docs: drop stale "3 sched + 1 orch" references The all-threads-schedule change falsifies several docs/comments that still described the old split (one reserved orchestrator thread + aicpu_thread_num-1 schedulers). Update them to the current model (every thread schedules; the highest-index thread also runs the one-time host-orch boot): - runtime.h: aicpu_thread_num comment (no on-device orchestrator; all threads schedule), the aicpu_allowed_cpus exec_idx comment ("orchestrator slot" -> boot thread), and the ready-queue-shards comment (drop "minus orchestrator"). - RUNTIME_LOGIC.md: §8.1 core-assignment table (thread 3 now a scheduler with cores, not "Orchestrator | none"; cores split across all 4 threads) and the PLATFORM_MAX_AICPU_THREADS constants-table note. - device_log_profiling.md: scheduler-summary block now lists all threads (Thread 0-3), matching the 4-summary onboard output. Addresses review feedback on hw-native-sys#1452. Docs/comments only; no behavior change. * host_build_graph docs: fix §8.1 table alignment (markdownlint MD060) The prior doc commit widened the §8.1 header to "Cores (block_dim=24)" but left the 5-dash delimiter, breaking markdownlint's aligned_delimiter table style. Move the block_dim note into the intro line so the header stays "Cores" and aligns with the delimiter. Verified: pre-commit markdownlint-cli2 passes.
hw-native-sys#1447) Trim low-signal LOG_* call sites across the runtime and platform layers, keeping the diagnostics that trace program flow or failures: - Flow / lifecycle / summary logs kept at INFO; per-item (per-tensor, per-arg) detail demoted to LOG_DEBUG; failure paths stay LOG_ERROR / LOG_WARN. Loader / device-runner load-path diagnostics kept at their original levels. - AICPU hot paths (per-task / per-scope dispatch loops) carry no logs (codestyle rule 7); DFX collectors keep their buffer-level init/flush diagnostics and telemetry counters. - STRACE, LOG_RUNTIME_FAILURE and ACL logging left untouched. - Log-only support removed with its logs (bookkeeping counters, dump scaffolding); genuine out-of-scope strip collateral restored (unused-parameter removals, blank-line deletions). Rebased onto upstream/main; dense dependency diagnostics follow upstream's debug-only / removed decisions (hw-native-sys#1448), completion follows the polling mechanism (hw-native-sys#1435). Co-authored-by: vegetabledoww <114059112+vegetabledoww@users.noreply.github.com>
…e-sys#1512) fetch-comments queried only reviewThreads, so review summary bodies and PR conversation comments were never fetched. A CHANGES_REQUESTED rationale or a plain "please rebase" comment could therefore sit unread while the fix-pr loop reported the PR clean. - Fetch reviewThreads, reviews[].body, and issue comments in one query - Track handled review bodies and conversation comments in a per-PR ledger, since neither surface carries an isResolved bit and both would otherwise re-present as unaddressed every iteration - Filter bot status/walkthrough noise, and dedup summaries that only restate their own inline threads - Answer non-thread feedback with a batched PR conversation comment; resolveReviewThread stays thread-only, as it rejects review and issue-comment node IDs - Require no unhandled feedback on any surface before the loop exits
…-native-sys#1506) Fixes hw-native-sys#1484 The example carried pypto codegen last refreshed on 2026-06-25, cited a provenance file upstream had deleted (`models/qwen3/14b/decode_layer.py`), and covered 2 of Qwen3-14B's 40 layers. Re-harvested from `decode_fwd.py`'s `decode_fwd_layers` at pypto-lib `45be52c` with `_CHUNK_NLAYERS = 40`, and the README now records that commit plus the ptoas and pto-isa pins so the next drift is detectable rather than archaeological. Scaling to the full model is nearly free. The generated orchestration loops over layers (`for (int64_t i = 0; i < 40; i += 1)`) instead of unrolling them, so going from 2 to 40 moves a trip count and leaves the kernel set the same size. The lib chunks its dispatches because a 40-layer one used to exceed a 2000 ms stream-sync timeout; simpler raised that to 50 s in hw-native-sys#1175 and a 40-layer step is ~40 ms, so the constraint is gone. What did change is attention. Since pypto-lib hw-native-sys#765 and hw-native-sys#796 it is a CANN FusedInferAttentionScore extern rather than generated kernels, folding per-head QK-norm, RoPE, the paged KV write, the flash inner loop and the online softmax into one mixed AIC+2xAIV task behind an FFTS barrier. Seven kernels go away (`rope_qkv`, `qk_gamma`, `qk_recip`, `fa_work_build`, `fa_fused_aic`/`_aiv`, `online_softmax`, `attn_fence`) and the harvested extern arrives under `kernels/vendor/paged_attention_cce/`. That extern needs CANN devkit headers, which nothing could express: every incore compiled against the same fixed include set, even though `KernelCompiler.compile_incore` has always taken `extra_include_dirs`. The scene-test `CALLABLE` now carries a per-incore `extra_include_dirs`, resolved when the kernel is compiled rather than when its module is imported — sim and macOS runners collect a case declaring a CANN dependency without having CANN. Entries may use `$VAR` and may name directories that do not exist here, so one candidate list spans SDK layouts; an unset variable or a set where nothing survived raises, which beats a "file not found" from four includes deep in vendored code. The harvested tree is copied verbatim and must stay that way: `fai_body.hpp` reaches its dependencies by relative include, and reformatting it would make the next refresh diff against our copy instead of against upstream — the exact drift this example exists to expose. It therefore lives under `kernels/vendor/`, and the repo's header, formatting and language lint skip anything below a `vendor/` directory. Keying the carve-out on the directory rather than on this operator's name means the next harvested extern needs no lint change at all. The extern also moves the paged KV layout from NSND to vLLM's BSND (`[page, token, kv_head, dim]`, so `slot_mapping` is directly the row index), which the golden now models for both the attention read and the KV write-back. The previous harvest needed one hand-edit, replacing a `[[block_local]] static` in `fa_fused_aiv` that the AICore loader rejected. That kernel no longer exists and the current codegen emits no such construct, so this harvest is unmodified codegen throughout. At 40 layers the case takes ~5 min against a 38.05 GiB fixture (24.61 weights + 13.44 paged KV, bf16) — over half the onboard sweep's 600 s session budget. It runs as its own CI step on a dedicated device instead, keeping that sweep's short timeout meaningful as a hang detector. Verification: passes on device with output and all 40 layers' KV caches matching the torch reference at RTOL=5e-2 / ATOL=1e-1, with no ring-sizing overrides — per-layer intermediates live inside that iteration's scope, so the live set does not grow with layer count. The l0-swimlane docs used this example as their "real SPMD workload", shrinking the generated fa_fused via `--set-arg 0=96` on its work-item count. The extern takes its work distribution from runtime tiling metadata instead, so no `--set-arg` shrinks it and those references are removed rather than repointed. The dfx buffer-capacity row measured at 2 layers is dropped for the same reason: it would now be a stale number rather than a wrong one.
…sys#1509) The parallel-execution case asserted elapsed < 0.8*serial — a wall-clock magnitude proxy for "the workers ran concurrently". It flaked ~2% on macOS CI (0/101 Ubuntu), failing at 0.49-0.57 s. A magnitude bound is the wrong shape for a concurrency property: it conflates concurrency with host timer accuracy and scheduling latency. The macOS time.sleep inflation measured under load (0.2 s -> 0.24-0.31 s, 1.2-1.55x) is not enough on its own to reach 0.49-0.57 s for three parallel sleeps (that needs ~2.5-2.9x), and the failure did not reproduce on bare metal — so whether the runner sees worse inflation or partial serialisation remains open. Either way the magnitude bound is the wrong assertion, so it is replaced, not widened. - Each worker records its [begin, end] work window in its mailbox; assert the three windows mutually overlap (latest begin < earliest end). Relational, not a bound — holds whether a sleep takes 0.2 s or 0.6 s, so immune to timer inflation, yet fails if the pool serialises. - Distinct per-worker result tags so result routing/cross-talk is actually verified (previously all returned 200, so misrouting was silent). - Codify the principle in docs/testing.md: off-hardware tests must not assert wall-clock magnitude as a proxy for a non-temporal property. If the overlap assertion itself flakes on the macOS runner, that is direct evidence of serialisation — the open question in hw-native-sys#1496 — a stronger result than the deleted bound gave. Refs hw-native-sys#1496
…ive-sys#1543) The hand-written Python reference promises that the source is authoritative when the two disagree, which only holds if the source is actually published. This builds docs/ into a searchable site and adds a generated reference for simpler.worker, simpler.task_interface and simpler.orchestrator alongside the curated page, which keeps the things a generator cannot state: the **config keys Worker accepts, CallConfig defaults, and the argument-order footguns. Two build details carry the design. Out-of-tree links. A quarter of the relative links in docs/ leave the tree (132 of 523, most of them citing a source file as evidence), and MkDocs only serves docs_dir, so all of them would 404 on the site. docs/_hooks/ repo_links.py resolves each link against its page and rewrites the ones that escape into permalinks at the ref being built, leaving intra-docs links for MkDocs to validate. The docs stay readable on GitHub, where those relative paths still resolve, with no rewriting of the prose. Strictness. The build runs --strict so a dead intra-docs link, a page missing from nav, or a bad anchor fails CI. griffe also warns about parameters that a docstring describes but the signature does not annotate; those report a source typing gap rather than a doc defect, and annotating them means first establishing what a parameter such as ChipWorker.init's `bins` accepts. docs/_hooks/griffe_filter.py attaches a filter to the strict counter alone, so that one message still prints but does not abort, and every other warning still fails the build. The workflow installs docs/requirements.txt rather than the project. Installing the project would run scikit-build-core and compile the C++ runtime; mkdocstrings reads the API through griffe, which parses statically, so the docs need neither a CANN toolkit nor the built nanobind extension. Verified by building the site in an environment where simpler is not installed at all. A pull request builds the site to prove it still compiles; only main deploys. Pages is enabled for the repository with GitHub Actions as the source, so the site publishes at https://hw-native-sys.github.io/simpler/ on the first push to main carrying this workflow. workflow_dispatch is accepted too, so the site can be republished, or the deploy path exercised, without a code change.
yanghaoran29
force-pushed
the
aicpu-pg-a2a3-a5
branch
from
July 28, 2026 07:48
4bb36f8 to
83d054d
Compare
) Two gaps left the docs build weaker than it looked. A page absent from nav still built cleanly under --strict, showing up as one INFO line. So the rule docs/README.md states — a new doc needs a nav entry — was unenforceable, and a missing entry silently dropped the page out of the site's navigation. validation.nav.omitted_files now warns, which --strict treats as a failure. Verified both ways: a page left out of nav fails the build, and removing it passes again. Twenty-six existing pages are out of nav on purpose: the investigations log, the remote-L3 design set, and the per-code error pages are reached through their own index pages, and listing every entry would bury the rest of the tree. They are declared in not_in_nav so the check stays strict for everything else rather than being switched off. The nine parameters that griffe reported as documented-but-unannotated are annotated, so docs/_hooks/griffe_filter.py — which existed only to keep that one message from failing --strict — is deleted. The build now passes --strict with zero warnings and nothing suppressed. Two of those annotations needed a decision rather than a transcription. ChipWorker.init's `bins` is structurally typed: the docstring accepts any object exposing the *_path attributes, and the body reads dispatcher_path through getattr, so it is Any. Naming RuntimeBinaries would be both narrower than the contract and a dependency from simpler onto simpler_setup. ChipWorker.run's `handle` is a CallableHandle, imported under TYPE_CHECKING because the runtime import is deliberately lazy at its use site; PEP 563 is already on in both files, so the annotations cost no import.
…#1547) Three inconsistencies, each of which makes the tree harder to read than the code in it warrants. None changes behaviour. Test module names now match their directory. Four scene tests were named after a sibling example rather than their own: paged_attention_manual_scope and paged_attention_unroll_manual_scope, on both arches, each carried the plain paged_attention test name, so a failure report named a directory that was not the one running. scalar_data_test is renamed to scalar_data, since the directory suffix only duplicated what test_ already says; the file inside keeps its name and now matches. vector_add keeps two test modules on purpose: test_run_timing.py asserts the [STRACE] markers and is a second test, not a mismatched one. Four L3 example directories gained the __init__.py their siblings all have. Without it the package-relative `from .main import run` that every one of these tests uses depends on rootdir inference rather than being a package. allreduce had no test at all, so the only L3 example without regression protection was a collective. It now has one, scoped to a2a3: the demo's own README and CLI default document that platform, and while the algorithm is covered on a5 by tests/st/worker/collectives/allreduce, nobody has run this demo there. Widening it is a separate, verifiable step. The test documents that run() takes (device_ids, platform), the reverse of the single-device demos beside it.
…1549) Kernel sources carry two conventions. The collectives kernels qualify (pto::Stride<...>) and use no using-directive; the older scene-test kernels open with `using namespace pto;` and use bare names. Each is internally consistent, and nothing recorded which one new code should follow, so the split propagates by whichever file an author copied from. Rule 11 names qualification as the direction, because it is the only spelling that compiles whether or not a using-directive is in scope, and because a file-scope `using namespace` is what most style guides steer away from. New kernels qualify and do not add the directive; existing bare files are not defects and need no churn. The load-bearing part is the constraint that a file is qualified completely or not at all. Shape, Stride, Tile and GlobalTensor all live in namespace pto, so qualifying one and leaving its siblings bare yields adjacent lines that read worse than either consistent state. That is a real failure mode rather than a hypothetical: an earlier revision of PR hw-native-sys#1547 rewrote Stride across 24 files and left Shape alone, and the resulting mixture is why it was reverted. A repo-wide sweep is ruled out with the measurement that rules it out: the 142 files carrying the directive hold roughly 3,200 bare uses, of which Tile alone is 2080. Rewriting them would touch every kernel, collide with every in-flight branch, and risk regex damage for no functional gain. Unifying the tree in one go needs its own decision plus a lint rule to hold the line.
- Align a5 runtime_maker with a2a3 hw-native-sys#1322: shared build_and_cache_prebuilt_arena, strong prewarm_config_impl, decouple ensure_static_arenas from Runtime - Add a5 prewarm regression test and onboard cold/hot benchmark Onboard effect (a5, ring_task_window=64, dummy_task): cold first-run bind.prebuilt ~9.58 ms -> hot ~0.002 ms; ~9.97 ms build cost shifts to init simpler_prewarm.build (cache hit on first run).
…sharing alignment fix and batch updates. (hw-native-sys#1550) - Extract the repeated completion_flags[id & task_window_mask] load/store into is_completion_flag_set() / set_completion_flag() on PTO2SharedMemoryRingHeader; route all 6 call sites through them (orchestrator, scheduler fanin hot path, scheduler cold path). - Replace the single-slot completed_watermark CAS loop with update_completed_watermark(): scan the full contiguous completed prefix locally, then issue one CAS to jump the watermark forward. Atomic RMWs on completed_watermark drop from O(advance distance) to O(contention events). - Give task_window_size its own cache line (alignas(64)), separating the cold read-mostly layout metadata from the completed_watermark atomic CAS'd on every completion — removes false sharing between the two. task_descriptors_offset shifts 160 -> 216; struct stays 256 bytes (static_assert updated). Signed-off-by: raphaelsteiner <raphael.steiner@huawei.com>
Register the MX block-scale host dtypes so torch FP8/FP4 tensors can flow through the task interface (host<->device byte transfer and compute): - DataType::FP8E4M3FN / FP8E8M0 (1 byte each) and FP4E2M1 (1 byte = 2 packed E2M1 values, matches torch float4_e2m1fn_x2). FP8E5M2 is intentionally omitted (MX paths use E4M3FN + E8M0 scale). - get_element_size / get_dtype_name entries + nanobind enum bindings. - torch_interop maps float8_e4m3fn / float8_e8m0fnu / float4_e2m1fn_x2 -> DataType, guarded by getattr so an older torch silently skips the dtypes it lacks instead of failing at import. - UT: enum/size/name/make_tensor_arg; plus MX FP8/FP4 decode tables, MXFP8/MXFP4 golden with E8M0 block scales and ZZ/NN packing helpers under simpler_setup/goldens/mx_fp_gemm.py (shared golden package). Skip the torch-interop case only when none of the optional dtypes exist. - ST (a5 only) mx_fp_gemm: AIC kernel calls pto-isa TMATMUL_MX for host-prequant MXFP8 / MXFP4 (E8M0 scales, MX_A_ZZ / MX_B_NN) and checks FP32 output against the host golden. a5sim leftover: CPU stub has no GetScaleAddr and TLOAD does not support MX_A_ZZ / MX_B_NN, so this ST is onboard-a5 only for now. Unifying the sim path belongs in pto-isa (or a dual kernel), not this PR. Verified (torch 2.13): related UT passed; a5 MXFP8_TMATMUL_MX / MXFP4_TMATMUL_MX passed.
…1544) * host_build_graph: dedicated resolution thread (3S+1P) Split the AICPU threads into (aicpu_thread_num - 1) core-owning schedulers (S) plus one core-less resolution thread (P); enabled whenever there are at least two threads. Each S thread detects FIN on its own cores, frees the core immediately, and hands the finished task to P over a per-S SPSC queue. P runs on_task_complete for every finished task — publish completion_flags, drain the wake list, advance the watermark — and owns completed_tasks_ and the terminal completed_ flip, so the schedulers keep dispatching until P has resolved the whole graph and the host's wait_for_consumers never sees a stranded prefix. Because P is the sole producer of the ready queues, the enqueue CAS never contends and the wake-list drain is single-threaded. Async deferred completions and dependency-only retirement (dummy tasks and predicate-failed tasks, both routed to dummy_ready_queue) also move to P, so the scheduler loop stays purely core-local — poll own COND, dispatch own cores — with no per-iteration polling of the shared mailbox or dummy queue. Measured on a2a3, paged_attention batch=256 (65536 tasks) on one die: 29.5ms (4S) -> 25.4ms (3S+1P), -14%. The scheduler-thread completion phase roughly halves as the resolution contention leaves the hot path. The dependency-bound A/B (matmul/add, tile size held constant) confirms the direction splits with graph shape: a wide parallel graph speeds up, a single serial chain costs a few percent from the extra S->P hop. * host_build_graph: address 3S+1P review — P stall guard, DFX flush, queue sizing - P's idle timeout now only latches when work is outstanding AND no thread owns a running task (mirrors resolve_and_dispatch); a task that legitimately runs longer than the budget refreshes last_progress_ts instead of false-latching PTO2_ERROR_SCHEDULER_TIMEOUT. - P resets its per-thread swimlane state on entry and flushes swimlane / dump-args / PMU buffers on exit, so DFX records written under p_thread_idx_ are not dropped. - P mirrors completed_tasks_ into the SIMPLER_SCHED_PROFILING tasks_completed counter, which the scheduler threads' completed_this_turn no longer feeds. - Size each CompletedTaskQueue to min(total_tasks, ring task window) instead of total_tasks capped at 1<<20 — the window already bounds in-flight completions, so there is no ceiling that could clamp capacity below the working set and engage the backpressure spin. Assert power-of-two capacity; note the atomic cursors already make the queue non-copyable. * host_build_graph: make the resolution thread mandatory, drop the fallback branches The dedicated resolution (P) thread is not optional: like tmr's scheduler + orchestrator split, host_build_graph needs at least two AICPU threads (one core-owning scheduler + one resolution thread). A single thread cannot both own cores and resolve on a dedicated thread, and no config runs fewer than two threads (scene tests use 3 or 4; the default is 3). Require aicpu_thread_num >= 2 in assign_cores_to_threads and remove the `p_thread_mode_` flag and every branch on it. This deletes the retained inline completion path from complete_slot_task and the per-iteration async-poll and dummy/predicate-drain blocks from resolve_and_dispatch — those were dead in the only supported configuration and only added predicted branches to the scheduler's hot loop. Async, dummy and predicate resolution live solely in run_resolution_thread now; the scheduler loop goes straight from completion detection to dispatch. Behavior is unchanged for the >= 2 thread configs (all of them). matmul, bgemm, paged_attention, vector_example and predicated_dispatch (the dummy / predicate path, now P-only) all pass.
…e-sys#1554) This repo is worked through many concurrent worktrees, and a branch ref is shared by all of them — `HEAD`, the index, and the working tree are what is per worktree. So `git checkout -B <name>` does not merely move "your" branch: if another worktree has that name checked out, it is taken out from under it. Rule 6 states the general form: only ever create a new branch, or reset one you exclusively own. Park a worktree on a detached HEAD between tasks rather than on a shared branch, and read merged content with `git show upstream/main:<path>` instead of checking anything out. The rule names `-B` as the specific trap, because git already refuses the obvious forms. Both `git checkout <branch>` and `git branch -f <branch>` are rejected when another worktree holds the branch, but `git checkout -B <branch> <ref>` is not (git 2.39.1). That one call creates the dual-checkout state the other two guards exist to prevent, and afterwards every further force-move is legal, since moving a branch your own worktree has checked out is an ordinary reset. The guard never fires again. It also records what the victim sees, because the symptom does not look like a ref problem. The other worktree's HEAD is a symref, so it follows the branch to the new commit while its index and files stay at the old one. "Staged" being the HEAD-to-index diff by definition, `git status` there reports the net tree difference between the two commits, in reverse, as staged modifications — a file you added appears as a staged deletion; anything a later commit undid cancels out and never shows. So long as that index still matches the old tree, a commit made there would revert the whole net difference and look routine to whoever ran it. Written from an incident in this repo: eight such force-moves of `main` from a linked worktree, which left the primary worktree reporting 96 phantom staged entries.
yanghaoran29
force-pushed
the
aicpu-pg-a2a3-a5
branch
from
July 29, 2026 02:56
83d054d to
59d5824
Compare
…threads (hw-native-sys#1553) The device boot classify — seed the whole graph's ready queues and wake lists (route roots to the ready queues, register every other task on its first unmet producer's wake list) — used to run O(total_tasks) serially on the boot leader while the other AICPU threads idle-waited on runtime_init_ready_. On a 65792-task paged_attention graph that is ~2.26 ms of one thread working and three spinning. Split it: on_orchestration_done keeps only the leader-only setup (attach, task count, queue allocation) and the classify moves to classify_partition(), which each thread runs over a disjoint contiguous slice of the submitted-task range. The boot leader publishes classify_ready_ once its setup is visible; every thread then classifies its slice and increments the classify_arrived_ barrier; the leader publishes runtime_init_ready_ only once all slices are done, so no thread dispatches against a half-seeded graph. push_ready_routed and register_wake are the same lock-free primitives the scheduler threads already use concurrently, and at boot no producer has completed (wake-list heads are nullptr, never SENTINEL), so registration never re-classifies — the parallel scan needs no new locking. Measured on a2a3, paged_attention batch=256 (65792 tasks), 4 AICPU threads: the classify phase drops from 2.26 ms (serial) to 1.58 ms (-30%). It does not reach the 4x of perfect scaling because concurrent register_wake on a high-fanout producer's wake list and push_ready_routed on the shared ready queues re-introduce the CAS contention the run-time 3S+1P split removes, plus the barrier waits on the slowest slice — but it still returns ~0.68 ms of otherwise-idle serial boot time to the run. matmul, bgemm, paged_attention, vector_example and predicated_dispatch all pass (the barrier introduces no deadlock and the seeded graph is correct).
`examples/workers/l3/` had a README for 4 of its 10 examples. The other six were reachable only by reading `main.py`, and two of them — `l3_l2_message_queue` and `l3_l2_orch_comm_stream` — were missing from the directory index entirely, so nothing in the tree pointed at them at all. Each new README follows the shape of the existing ones: what the example demonstrates that its predecessors do not, how to run it (both the `main.py` CLI and the scene test, with the device count each requires), and what the golden check actually proves. Two facts worth stating explicitly, because neither is guessable from the call site: - The integer in `children=[(id, core_callable)]` is the `func_id` the orchestration passes to `rt_submit_aic_task` / `rt_submit_aiv_task`. It selects which child kernel runs, not which core type — the core type comes from `core_type=` at compile time and from which `rt_submit_*` the orchestration calls. - `ep_dispatch_combine` verifies only `routed_y` on the simulator, because sim keeps intermediate child outputs device-local when they feed a later child task, so `recv_y` is not host-visible there. The directory index is regrouped into reading order — first steps, communication domains, then the two examples that drive a task already in flight — and gains rows for `allreduce`, `per_task_runtime_env`, and the two L3-L2 examples that were absent.
The a5 runner is back in service, so both jobs targeting it have a host to run on again. Restore their gates to match their a2a3 counterparts: ut-a5 gated on docs_only like ut-a2a3, st-onboard-a5 gated on a5_changed like st-onboard-a2a3. a5 changes get silicon coverage again instead of falling back to st-sim-a5 alone. Drops the notes about the repair along with the short-circuits they described — the condition they explained no longer exists.
Permit A2/A3 host runtimes to declare `pipeline_depth = 2` and give every
resource class the copy count its declaration implies, without admitting a
second run into device execution. `HOST_PER_RUN` and `EXEC_HANDLE` gain one
copy per slot, `DEVICE_SCRATCH` keeps one, and `pipeline_resource_copy_count`
/ `pipeline_resource_slot` derive both from the contract so no caller
hardcodes a replication rule.
GM heap, GM shared memory, and the runtime image are committed together into
one arena bank, so a contract that classifies them inconsistently — or repeats
one of them, where only the first entry is ever read — describes a layout no
runner can build. `has_serviceable_arena_topology` rejects it when the runtime
loads rather than at the first launch that would need the second bank.
Repeated stream kinds select no bank and stay legal.
A run reaches its resources through a `PipelineSlotLease {slot_id,
generation}`. `PipelineSlotPool` is the only mint and the only authority on
ownership: it hands out one lease per free slot, bumps the generation on
reuse, and makes release idempotent for the current generation while
rejecting a stale one. Consumers downstream of the pool cannot re-derive
ownership, so `ChipWorker` carries a `PipelineSlotGenerationFilter` that only
refuses generations older than the newest that slot has presented. The filter
mints nothing: an unleased run selects slot 0 and advances no generation, so a
pool's first lease is still admitted after any number of ordinary runs.
Each slot owns a host `Runtime` staging buffer. That buffer is not the
`RUNTIME_IMAGE` resource — the contract classifies the device-resident image,
while this holds the host-side object a run constructs in place, with its
tensor leases, launch arguments, and validate/finalize state. That is per-run
whatever the device image sharing is, so a runtime whose image is
`DEVICE_SCRATCH` still needs its own buffer per slot.
The A2/A3 runner owns its slot and arena-bank selection as member state, so
two contexts on one thread cannot see each other's choice — a fresh worker's
prewarm lays out its arenas in the bank it will actually serve runs from. The
three pooled device regions become one bank per slot, and the single-entry
prebuilt-arena cache stays owned by bank 0.
AICore streams are outside that lease and outside the slot: each run creates
its own and retires it on every exit path. The instruction cache belongs to the
cores, every slot publishes its image to the same GM code address, and the
platform offers no cache invalidation for code replaced there. Creating a
stream is the only operation known to leave a core free of the previous image's
instructions — selecting an existing one is not — so no record of which image a
stream last ran can make reuse safe, and none is kept. The AICPU stream carries
no such state and stays with its slot for the worker's lifetime.
Simulation implements the same depth, so the contract means the same thing on
both platforms: its runner owns one arena bank and one retained temp buffer per
slot, and a leased slot-1 run there exercises the second copy rather than being
refused.
The generation filter is a replay filter, not an ownership check, and says so:
it rejects a lease whose successor has already presented itself, but one that
was released while its successor has not yet reached the consumer still passes.
Closing that window needs dispatch gated on `PipelineSlotPool::owns`, which
belongs to whoever admits runs.
The ordinary synchronous path stays on slot 0 and the chip child's mailbox
loop passes no lease, so every production run is unleased. No prepared FIFO,
second mailbox frame, or backend publication overlap is enabled here.
A run owns its AICore stream, so a destroy it cannot complete is that run's
failure: the retire path returns the driver's error and keeps the handle, which
both leaves the slot unusable for the next run — the stream may still hold the
previous image's instructions — and leaves teardown something to retry.
Reporting success there would have told the caller a slot is reusable that the
next acquire now refuses. That state machine lives in `RunStreamSlots`, whose
stream creation and destruction are injected, so the failure paths are covered
without a device.
`get_arena_bank_gm_heap_base_ctx`, `get_retained_temp_addr_ctx`, and
`ChipWorker::runtime_buffer_addrs` report which storage a bank and a slot
actually own. Sequential runs re-stage their arguments every round, so correct
output alone survives two slots sharing one buffer; the tests compare addresses
instead.
Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
hw-native-sys#1557) The child-pointer provenance guard keyed every allocation by its exact (worker_id, base) and admitted a copy only when the address hit that base. A partial update of a persistent device buffer — copy_to(base + offset) into one page of a resident cache — was rejected as though the interior address were unknown, because the guard stored no allocation extent to range-check against. Record each allocation's byte extent in provenance and validate copies by containment instead of exact base: base <= ptr && ptr + nbytes <= base + extent Changes: - _ChildProvEntry carries malloc_size and maps each CommDomain allocation id to the extent of the window / buffer recorded at that base; live_extent() returns the widest live role. - copy_to / copy_from (Worker L2 path and Orchestrator L3 path) call the new _child_prov_require_live_range, which admits any range wholly contained in one live allocation on the worker. A copy to the exact base resolves in O(1); only an interior address falls back to scanning the worker's live allocations. Python ints are unbounded, so the ptr + nbytes bound is exact. - A CommDomain carves its first buffer at offset 0, so that buffer aliases the window base under the same allocation id. Record the max of the prior and new extent per allocation id so the smaller buffer never narrows the window's copy range. - free() and kind4 dispatch are unchanged — they still require the exact base, since they act on the whole allocation, not an interior slice. - Document the interior-range behavior in the Python API reference. Regression coverage in tests/ut/py/test_worker/test_child_addr_guard.py, at both the L3 orchestrator and L2 worker entry points: interior range accepted, out-of-range / negative-size / wrong-worker / stale rejected, free still requires the exact base, and a domain-window chunk is not narrowed by its aliasing buffer. Fixes hw-native-sys#1537.
…w-native-sys#1564) Of the three branches under `examples/`, only `workers/` had an index. The 22 `@scene_test` examples under `a2a3/tensormap_and_ringbuffer/` and `a5/tensormap_and_ringbuffer/` were reachable only by listing the directory, and nothing said what distinguished one from the next. `examples/README.md` states the organizing principle, which is not visible from the tree: `workers/` teaches how the runtime works through the raw `Worker` API and is split by level, while `a2a3/` and `a5/` teach how to write and validate a kernel through `@scene_test` and are split by architecture. The two per-architecture indexes group their examples by what each one adds — first kernel, compute, asynchronous completion and cross-card transfer — and record which platforms each is marked for and how many dies it needs, since most are onboard-only and that is invisible until a run silently deselects. The exceptions are named rather than generalized over: `deferred_notify_demo` runs the simulator path on both architectures, and `prefetch_async_demo` is the one async demo that needs a single die. Two facts that were previously undiscoverable: - `docs/INCORE_ORCHESTRATION_GUIDE.md` explains the AICPU-side orchestration model every example in both trees depends on, and had no inbound link from anywhere in the repo. Both indexes now open by pointing at it. - Seven examples exist under both architectures with the same name and are ports of each other, differing mainly in tile shapes and platform strings (`vector_example` differs by two lines). Each index names the seven and the ones unique to it, so a change to one side is not made without seeing the other. The root README's `examples/a2a3/` reference is retargeted at the new index.
…sys#1569) The `mkdocs build --strict` step pointed readers at `docs/_hooks/griffe_filter.py` for "the one warning class that is deliberately not counted". That hook was removed once the annotations it worked around landed, so `docs/_hooks/` holds only `repo_links.py` and `mkdocs.yml` registers only that one. The strict build has no exemption left; the comment now says so.
…-native-sys#1568) Fixes hw-native-sys#1561 Four `*_notify_demo` tests were plain pytest functions that hardcoded both the platform string and the device IDs, so `--platform` and `--device` were ignored. The tests grabbed devices 0 and 1 whatever `task-submit` had allocated, and the a5 variants built an a5 callable on a2a3-only silicon, where the forked chip child segfaults in `ChipWorker.init`. They also never ran in CI. `_collect_resource_jobs` in `conftest.py` dispatches a standalone function only when it declares both `@pytest.mark.device_count` and `@pytest.mark.runtime`; carrying neither, these four were collected but never selected. They fired only on a hand-scoped `--runtime`/`--level` invocation, which enters child mode and bypasses the dispatcher — so the failure looked like a regression in whatever branch was being validated. Adopt the marker + fixture pattern the sibling demos in the same directories already use. That both honours the CLI options and enrols the tests in resource-phase dispatch. `deferred_notify_demo` stays sim-only, matching its docstring and the platform it has been validated on. Add `tests/lint/check_test_platform_literals.py` and wire it into pre-commit. It AST-walks test bodies under `examples/` and `tests/st/` and rejects a platform name passed as a call argument; a literal compared against the `st_platform` fixture is correct usage and is not flagged. Platform names come from `PLATFORM_MAP` in `simpler_setup/platform_info.py`, parsed rather than imported so the hook runs in pre-commit's isolated environment. The hook's file filter matches test files both directly under those roots and nested within them.
…s#1571) Four sibling directories share a name prefix and nothing tells you what separates them. Diffing establishes it, and each README now leads with the answer: - `paged_attention` is the baseline: four tasks per KV block (QK / SF / PV / UP) inside a plain scope, with TensorMap deriving the ordering from the tensors the tasks share. - `paged_attention_manual_scope` has **byte-identical kernels** — all four — and differs only in orchestration, 45 lines of 292. It is the cheapest way to see what MANUAL scope costs, and it deliberately uses both dependency APIs in one file: the primitive `set_dependencies(deps, n)` for tasks with a fixed predecessor set, and `L0TaskArgsWithDeps<>::add_dep` for `UP`, whose edges are conditional. - `paged_attention_unroll_manual_scope` is **not** the manual-scope variant plus an unroll. Every file differs; `aiv_softmax_prepare.cpp` changes 312 lines against a 156-line original. It batches KV blocks into groups of `N_UNROLL` and submits four tasks per group rather than per block. - `paged_attention_ringbuffer` contains **no kernels at all** — it points at the `tests/st` batch-paged-attention sources and exists to stress ring rotation at `ring_task_window: 64`, 256x below the 16384 default. The directory index carried a wrong one-liner for the unroll variant ("manual scope plus loop unrolling"); it now describes a second implementation, and the manual-scope row states the byte-identical kernels.
…e-sys#1511) cleanup_retired walked a slot's whole task_entry_head chain and freed every entry, assuming — only via a debug_assert, which compiles out in release — that the chain belonged to one task. A slot is shared by every task at local_id + N * task_window_size, and link_entry prepends to whatever chain is already there rather than resetting the head on slot reuse. When cleanup lagged ring advancement, a reused slot's chain mixed entries from a later, still-live task, and those were freed too: silent tensormap corruption in release builds, with a live producer vanishing from the map and the dependency computation losing its edges. Free only entries whose producer_task_id matches the retiring local_id, unlinking each from the task chain and fixing up the head as it goes; entries from other tasks stay linked. The unconditional head reset is gone because the chain can legitimately outlive the task being retired. free_entry clears only the freed node's task pointers, so every caller must unlink first; the other caller, remove_entry, already does via remove_from_task. Applied to all three copies that carry the defect: a2a3 and a5 tensormap_and_ringbuffer, and a2a3 host_build_graph (single-ring, no entry epochs, and no exposure difference that would spare it). - Add CleanupRetiredSparesLaterTaskReusingSlot to the a2a3 + a5 test_tensormap suites as the regression barrier. - Add tests/ut/cpp/a2a3/test_hbg_tensormap.cpp for the host_build_graph copy, which had no ut-cpp coverage: the same reuse case plus two adjacent cleanup_retired cases. It reuses add_a2a3_hbg_runtime_test and pulls in the runtime's shared/pto_tensormap.cpp for the out-of-line PTO2TensorMap members. The reuse case fails before this change and passes after; ut-cpp is 65/65 green. - Correct the cleanup description in all three RUNTIME_LOGIC.md copies: cleanup is O(entries_in_slot), not O(entries_per_task), and a slot's chain can hold more than one task's entries. - Onboard before/after (a5, spmd_multiblock_mix, 100 rounds, task-submit device 0): Device/Effective/Orch/Sched all within +-2% — no regression, as expected for a correctness fix that is a no-op on the common path. Extracted from hw-native-sys#906. Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
…ers (hw-native-sys#1572) Four copies of the unroll paged-attention orchestration open with "Paged Attention Orchestration Function V2 - N_UNROLL=8, 4 Tasks Per Group" while `#define N_UNROLL 64` sits twenty lines below. The comment has been wrong since the constant was raised, and it is the first line a reader sees. The value is dropped rather than corrected to 64. The line below already says "Batches up to N_UNROLL blocks per group" without a number, and the `#define` is the only place the value belongs — restating it in prose is what let the two drift apart. All four copies land together: `examples/{a2a3,a5}/.../ paged_attention_unroll_manual_scope/` and `tests/st/{a2a3,a5}/.../ paged_attention_unroll/`. `grep -rn "N_UNROLL=8"` now returns nothing. The `paged_attention_unroll_manual_scope` README noted the discrepancy; that sentence goes with it.
…hw-native-sys#1573) The three remaining a2a3 examples whose point is not guessable from the directory listing. Each one inverts something the rest of the tree takes for granted, and each README leads with that. `merge_pipeline_barrier` replaces the scheduler with a barrier: three stages merged into one block_num=8 SPMD task, ordered by an intra-task cross-core barrier. The stages are deliberately unbalanced — stage A runs on block 0 alone, C on all eight — and idle blocks still arrive at every barrier, so each one waits for the full block count. Two implementations sit behind MERGE_BARRIER_COUNTER: a per-slot st_dev poll costing O(n) non-cacheable reads, and a single-counter st_atomic arrival polled with one ld_dev, costing O(1). Its `compare_outputs` override is also the pattern for carrying a diagnostic output that has no golden. `scalar_data` has the orchestration read and write tensor elements directly, interleaved with submits. The load-bearing fact is where the automatic waits stop: `add_input` on an *external* tensor registers no TensorMap entry, so a following `set_tensor_data` finds no producer and overwrites the buffer under a running reader. There is no diagnostic — use `add_inout` on external tensors whose later writes must not race. `benchmark_bgemm` is a shape sweep rather than a kernel demo. Its cases are single-axis moves from Case1 over task count, tile size, work per task, and accumulation depth, which is what makes Case2 (4x the tasks) and Case3 (4x the work per task) comparable. Four of the six carry `"manual": True`, so a default run executes only Case0 and Bgemm64 — `--manual include` gets the rest. Its `C` is INOUT rather than OUT because the zero-initialised accumulator is read before it is written and its host zeros must be staged. The index rows for `scalar_data` and `benchmark_bgemm` are sharpened to match: the first now names the external-tensor exception rather than implying the waits are universal, the second the sweep structure and the manual-case default.
- Collapse V0-V9 call sites to INFO and reserve TIMING for STRACE - Add DEBUG, TIMING, WARN, and ERROR severity APIs - Default to TIMING and align Python, host, sim, and onboard thresholds - Gate onboard TIMING independently from CANN's coarser WARN level - Forward the threshold into RTLD_LOCAL AICPU simulation libraries - Add CANN configuration and host/device propagation regressions - Update affected call sites and documentation Fixes hw-native-sys#1429 Co-authored-by: zm <zhaomin88@huawei.com> Co-authored-by: ChaoZheng109 <zhengchao47@huawei.com>
…native-sys#1577) The scalar_data orchestration migrated LOG_INFO_V0 to LOG_INFO when the verbose V0..V9 tiers were replaced by severity levels (hw-native-sys#1475), but its README still named the removed macro. Align the doc with the code. Co-authored-by: zm <zhaomin88@huawei.com>
…e-sys#1576) The three a5 examples with no a2a3 counterpart to read instead. `bgemm` keeps its kernel in `kernels/mix/` rather than `kernels/aic/` plus `kernels/aiv/`, and that directory name is the example. One source is registered as two incores with different core_types; `__DAV_CUBE__` compiles the TLOAD/TMATMUL/TPUSH half and `__DAV_VEC__` the TPOP/TADD/TSTORE half, both over the same three-tensor args[]. The matmul product moves between them through VEC_FIFO and never reaches GM — only the accumulator C round-trips. a2a3's `benchmark_bgemm` computes the same thing with separate sources and a GM round trip, so the two are not ports of each other. `urma_deferred_completion_demo` and `sdma_async_completion_demo` run an identical two-rank protocol over different transports; `kernel_consumer.cpp` is byte-identical between them, leaving the transport as the only variable. Both are gated behind a workspace overlay that defaults OFF, and the two overlays are mutually exclusive in one build — `CommContext` exposes a single workSpace/workSpaceSize pair, so the CMakeLists raises a FATAL_ERROR if both are on. A stock build therefore skips both tests even on a5 silicon: a green CI run is not coverage of either path. Each README gives the two-step invocation, since the env var is read once by runtime_builder.py to compile the overlay in and once by the test's skipif. The a5 index gains those facts, and its `paged_attention_unroll_manual_scope` row is corrected the same way the a2a3 index was in hw-native-sys#1571: the kernels are rewritten (softmax_prepare changes 269 lines against a 149-line original), so it is a second implementation rather than manual scope plus an unroll. The manual_scope row now records that its four kernels are byte-identical to the baseline's and only the orchestration differs, 52 lines of 288.
…w-native-sys#1578) RemoteSocketTransport tests start a helper server thread, then construct a RemoteL3SocketTransport whose constructor throws on a connect or HELLO timeout — routine on a loaded box. The unwind destroyed the local std::thread while it was still joinable, so std::terminate aborted the whole binary mid-suite: #5 std::terminate () #6 std::thread::~thread() () hw-native-sys#7 RemoteSocketTransport_ClosedPeerWriteDoesNotRaiseSigpipe_Test::TestBody() Reproduced at 2/96 with 32 concurrent copies on a loaded 320-core box; the run that aborts takes every other case in the binary with it, and the message ("terminate called without an active exception", printed because no handler has caught the in-flight exception yet) reads like a hang rather than a timeout. start_stalling_server also captured the test's stack std::atomic<bool> stop flag by reference, so the same unwind left a running thread polling a dead object. ScopedServerThread owns both the thread and the stop flag, and joins in its destructor, which closes both holes: the flag now outlives the thread, and no unwind can reach a joinable std::thread. Joining alone would swap the abort for a hang, because a thread parked in accept() never returns when the client's connect is exactly what failed. accept_until_stop polls the listener in 20 ms slices and rechecks the stop flag, so stop_and_join always makes progress. It reads the flag only after a poll slice, so a connection already pending still wins: ClosedPeerWriteDoesNotRaiseSigpipe calls stop_and_join right after a successful connect and depends on that connection being accepted and RST. Add ServerThreadIsJoinedWhenTestBodyUnwinds, which throws out of a scope holding a live server thread that never sees a client. It covers both halves — reverted to the old bare-std::thread idiom it aborts with the production message, and a join that could not reap a parked accept() would trip its elapsed bound instead. ut-cpp 65/65 green; 192 concurrent runs of the binary now abort 0 times.
…native-sys#1580) The skill told the reader to read `ci.yml` only for `--pto-session-timeout`, then handed them pytest examples tests/st --platform a2a3 --device <range> ... which is not what CI runs. The onboard sweep carries `SDMA_IGNORE` and `HEAVY_IGNORE`, and the excluded tests get their own step on their own devices because they are only correct in isolation. Following the skill therefore reports failures CI never sees — `sdma_async_completion_demo` fails every time it shares a sweep (measured 4/4) and passes alone (measured 1/1), which reads as a regression in whatever branch is being validated. Record the quarantine table with its `ci.yml` line anchors, show the sweep command carrying the ignore sets, show how to run the quarantined tests the way CI does, and state the triage rule: run a failing onboard test alone before calling it a regression, and do not "fix" an isolation requirement by reordering — moving such a test earlier only relocates the pollution onto whatever runs after it.
Use zero as the automatic AICPU thread sentinel and resolve it against the local hardware topology. - A2A3 selects every usable CPU in the fullest cluster. - A5 selects every usable CPU in the fullest die and excludes the OS-reserved SMT pair from the launch pool. - Keep runtime, affinity, launch, and DFX counts consistent; reject one-thread normal runs and guard A5 register allocations on failure. - Add deterministic placement tests for healthy, partial-good, tie, and insufficient-capacity layouts.
yanghaoran29
force-pushed
the
aicpu-pg-a2a3-a5
branch
from
July 29, 2026 09:47
59d5824 to
dfe7e8d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
aicpu_thread_numis now derived from the probed AICPU usable count and an arch-default topology instead of a fixed user value:PLATFORM_DEFAULT_AICPU_THREAD_NUM: a2a3 = 1 orch + 3 sched (=4), a5 = 1 orch + 4 sched (=5).validate_launch_aicpu_num/ simprepare_launch_shapeallow 0 (auto).Note: a known test failure is pre-existing on hw-native-sys#1309, not from this PR
tests/st/a5/tensormap_and_ringbuffer/available_aicore_countsfails on this branch withinit_l2_swimlane: Invalid number of AICPU threads: 0. Checked out hw-native-sys#1309 alone (without this PR) and the same test fails identically — hw-native-sys#1309's sim path doesn't propagateruntime.aicpu_thread_numtoinit_l2_swimlane. Not introduced here; track/fix in hw-native-sys#1309.Verified
available_aicore_counts+spmd_basicvia task-submit.Not yet
hardware.md(a2a3/a5) documents the policy and aligns the "user-accessible 6" note with the active cap (4 a2a3 / 7 a5).