[Code Health] Mailbox dispatch waits should block on a wakeup primitive, not poll

[ChaoWao]

Category

Performance / robustness of a hot path

Component

Host runtime (src/common/hierarchical/, python/simpler/worker.py)

Description

Both ends of the parent-child mailbox wait by polling, and neither can be made both low-latency and cheap that way.

side	mechanism	cost
parent (LocalMailboxEndpoint::run)	sleep_for(50us) per TASK_DONE poll	up to a 50 us quantum added to every dispatch
child (_run_mailbox_loop)	unbounded busy-spin	a full core held for the child's whole lifetime

The parent's sleep is on the critical path of every dispatch and dominates the round trip. Measured on aarch64 Linux, 3 sub workers, no-op callables:

sequential run():  median 150 us   p90 152 us   max 176 us
burst (200 subs):  106 us per task

Note sleep_for(50us) does not cost 50 us — time.sleep(50e-6) measures ~102 us on this box, so the quantum is roughly double what the constant says.

A blocking wakeup primitive fixes both sides at once: zero CPU while waiting, single-digit-microsecond wake, and no interval to tune. This is now what codestyle.md rule 5 requires of any dispatch-path wait, so worker_manager.cpp is a recorded deviation rather than a style preference.

Location

• src/common/hierarchical/worker_manager.cpp — LocalMailboxEndpoint::run, the sleep_for(std::chrono::microseconds(50)) in the TASK_DONE poll loop
• python/simpler/worker.py — _run_mailbox_loop (single site since Refactor: fold the three child mailbox loops into one state machine #1494; all three child loops route through it)

Priority

medium

Additional context

Candidate primitive. A pipe per mailbox direction looks better than a PTHREAD_PROCESS_SHARED condvar in the mailbox shm, despite slightly worse wake latency:

	pipe	pshared condvar
wake latency	~5-20 us (unverified on this box)	~3-10 us
macOS support	POSIX, identical both ends	supported, but needs ctypes into libc from Python
GIL	os.read releases it	pthread_cond_wait needs care
peer death	poll() with a timeout	a child dying while holding the mutex hangs the parent; no robust mutexes on macOS

The repo has no blocking primitive today — futex, sem_wait, PTHREAD_PROCESS_SHARED, eventfd, pthread_cond are all zero hits across src/, python/ and docs/.

Constraint to preserve. The parent currently samples child liveness every 200 poll iterations via waitpid(WNOHANG), so a child that dies without publishing TASK_DONE cannot hang the loop forever. poll() with a timeout keeps that behaviour; a bare blocking read() would lose it.

Measure before implementing. The 5-20 us figure is a general expectation, not measured here. The gating experiment is one number: pipe write-to-wake latency on this host versus the current 50 us quantum. If the gain is not real on this hardware, the design should change before any code does.

What this is not. Not a fix for #1493 (children surviving parent death). With a blocking wait an orphan would cost 0% CPU, but it would still be a leaked process holding fds and shm, so the two are complementary.

An earlier attempt to pace the child's poll with a sleep instead (#1491) was closed: measurement showed it traded 17% of burst dispatch throughput for the CPU saving, which is the trade the rule now forbids.

[ChaoWao]

Ran the gating measurement this issue asked for, and it inverts the original framing. Re-scoping rather than closing.

Measured on this box (aarch64 Linux, os.pipe + os.fork, 2000 round trips):

	latency
spin poll (one mailbox load)	~0.1-0.3 us
pipe wake, one way	4.4 us
pipe wake, round trip	8.85 us median, 9.62 us p99

So 4.4 us is the real figure, well inside the 5-20 us the issue guessed.

What changed. #1499 removed the parent's 50 us TASK_DONE sleep, so both ends now spin and dispatch latency is already 50.7 us median / 11.0 us per burst task. Against that, replacing the polls with blocking waits adds ~8.8 us of wake per dispatch — +17% on a single run(), +80% on burst throughput. As originally written, this issue is now a latency regression bought with CPU, which is the wrong direction for this codebase.

What survives. The waste was never the wait that has a task behind it — it is a child spinning while nothing is outstanding, which holds a core for as long as its parent lives (~80% of a core each, measured). The fix for that is a spin window followed by a blocking wait, not a sleep:

• Spin while a dispatch is plausibly imminent → burst and steady-state latency unchanged, because the wait never reaches the blocking path.
• Block once genuinely idle → 0% CPU, and the first dispatch after an idle stretch pays 4.4 us rather than the 102 us a sleep(50us) actually costs. That is a 23x better fallback, and it is what makes the trade worth making at all.

This is the shape #1491 attempted and got wrong twice over: its fallback was a sleep, and its window was 19 us against a 110 us turnaround, so a burst slept between every task and lost 17% of throughput. Both mistakes are now measurable rather than guessed — the turnaround is ~50 us post-#1499, so the window has to comfortably exceed that, and the fallback costs 4.4 us instead of 102 us.

codestyle.md rule 5 already sanctions exactly this: "spin or block on a wakeup primitive". A hybrid that only ever blocks off the latency path satisfies it.

Revised acceptance criteria:

1. run() median and burst per-task latency unchanged versus main (within noise). This is the hard gate — the change is not worth any latency.
2. Idle child CPU drops from ~80% of a core to ~0%.
3. Child-liveness detection preserved: poll() with a kChildLivenessPollPeriod timeout, not a bare blocking read(), so a dead peer still cannot hang the parent.
4. Parent side unchanged. It already spins, is already optimal for latency, and its threads only spin while a dispatch is outstanding — there is no idle waste to reclaim there.

Item 4 shrinks the scope significantly: this is now a child-side change only, at the single _run_mailbox_loop wait site.

[ChaoWao]

Closing: measured, and the premise does not hold. The verdict and its reconsideration triggers now live in docs/investigations/2026-07-host-dispatch-latency-budget.md (#1502), which is where a considered-and-dropped optimisation belongs per the investigations README — an open issue would read as "someone should do this".

Why it is dropped.

This issue was written when the parent slept 50 us per TASK_DONE poll, so a blocking primitive looked like it would cut latency and CPU. #1499 removed that sleep, and the measurements afterwards invert the case:

bare fork+shm mailbox round trip (no runtime)	3.5 us
pipe wake, one way	4.4 us (8.85 us round trip, p99 9.62 us)

Blocking would add more than the entire IPC floor to every dispatch. And the latency it was meant to recover is not in the poll: a Worker.run() decomposes into 43.3 us fixed per run() + 8.08 us marginal per task, so the cost is entering and leaving an orchestration, which a production run amortizes over a whole graph (2% at 256 tasks/run).

What survives, and why it is not enough to keep this open.

The real waste was never the wait with a task behind it — it is a child spinning while nothing is outstanding, holding a core for as long as its parent lives. Two things shrank that:

• Fix: exit a forked worker child once its parent is gone #1495 made orphans exit with their parent, so the unbounded case (six processes at ~99.6% CPU for 5-7 days) is gone.
• What is left is idle sub workers inside a live pool. On a 320-core box that is unmeasurable; it matters on a narrow-core host or a laptop.

That is a real but small CPU-efficiency question with no current workload asking for it, and codestyle.md rule 5 already sanctions the shape it would take (spin, then block — never sleep). The investigation entry records the trigger to reopen: a workload where idle sub workers outnumber available cores, or many small run() calls that stop amortizing the fixed cost.

One correction to this issue's own text, so it does not mislead if reopened: it lists the parent side as in scope. That is now wrong — the parent already spins after #1499, its threads only spin while a dispatch is outstanding, and there is no idle waste to reclaim there. Any future attempt is child-side only, at the single _run_mailbox_loop wait site.