From 0573493d78ed40dbafab332d2cc448fb010d2cd1 Mon Sep 17 00:00:00 2001 From: Kacy Fortner Date: Mon, 27 Jul 2026 01:41:18 +0000 Subject: [PATCH] put the cross-language scoreboard up front, and say why both backends exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit two things a reader had to assemble themselves. the first is a straight comparison. the numbers were spread across five sections and three documents, so docs/performance.md now opens with two tables — coordination and compute — that put pith next to go, rust, and zig on every benchmark this repo runs, with the backend named on the rows where it matters. the summary underneath is three sentences: green beats rust and zig on coordination and beats go pinned to one worker, spawn is ~2x go at go's memory, and compute is competitive with go and well behind rust and zig with string building the remaining gap. the second is which backend to use, which the docs never actually answered. green wins every measured shape, so the interesting half is why os threads are still there, and it is not inertia. blocking calls stall a green worker and everything pinned to it — the reactor covers sockets, but dns, file i/o, and native calls do not yield — where on os threads a blocking call costs only the task making it. the reactor is also epoll-only, so macos and the bsds run green with no reactor at all and block a worker on every socket wait; "green by default" is a linux-server claim rather than a universal one. and preemption is free on os threads but needs a build flag on green. there is a fourth reason that matters to the repo rather than to a user, and it is worth writing down: the os-thread backend is the oracle the green one is checked against. verify-green-corpus diffs the whole corpus under green against the output os threads produce, which is how the single-worker deadline bug surfaced. two implementations that have to agree is a testing asset worth keeping past the point where one is faster. --- docs/concurrency.md | 28 ++++++++++++++++++++++++++++ docs/limitations.md | 10 ++++++++++ docs/performance.md | 30 ++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+) diff --git a/docs/concurrency.md b/docs/concurrency.md index e12f7a91..5af6fb1f 100644 --- a/docs/concurrency.md +++ b/docs/concurrency.md @@ -236,6 +236,34 @@ many run in total — see the green fan-out row in `docs/performance.md`. (the d os-thread backend still keeps a record per task it has run; teaching it the same reclamation is the next step.) +### which backend to use + +green is faster on every shape this repo measures, so the short answer is to +use it on linux for anything that coordinates a lot of tasks. the longer +answer is why the os-thread backend is still here, because it is not just +waiting to be deleted. + +a blocking call on a green worker stalls the worker, not only the task making +it. the epoll reactor covers sockets, so the whole net stack yields, but +nothing else does — dns at dial, file reads and writes, and any slow native +call hold the thread they run on, and every task pinned to that worker waits +behind them. on os threads that same call costs one task, because the task is +a thread and the kernel just runs someone else. + +the reactor is also linux-only. it is epoll and eventfd; macos and the bsds +compile a stand-in with no reactor at all, so a green task waiting on a socket +there blocks its worker for as long as the wait lasts. green is still correct +on those platforms, but an i/o-heavy server on macos wants os threads. and +preemption, which the kernel gives os threads for free, is a build-time opt-in +under green (see `PITH_GREEN_PREEMPT` below). + +there is one more reason, which matters to this repo rather than to your +program: os threads are the reference green gets checked against. +`make verify-green-corpus` runs the whole regression corpus under green and +diffs it against what the os-thread backend produces, which is how a +single-worker deadline bug turned up. two implementations that have to agree +is worth keeping around past the point where one of them is faster. + it is off by default and still experimental. the known rough edges: - dns resolution at dial still blocks the worker (`getaddrinfo` is diff --git a/docs/limitations.md b/docs/limitations.md index 79ed86fa..510527d0 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -132,6 +132,16 @@ the scheduler; it is the list below. 200-million-iteration arithmetic loop. that is cheap enough — but the flag must flip *with* the green default, not before it, or os-thread builds pay for a check that never fires. +- **the reactor is linux-only** — it is built on epoll and eventfd. macos and + the bsds compile a stand-in with no reactor, where a green task waiting on a + socket blocks its worker outright. so "green by default" is a linux-server + claim, not a universal one; on other platforms the default would have to stay + os threads or the reactor would need a kqueue sibling. +- **blocking calls stall a worker, not just their task** — file i/o and native + calls have no yield point, so they hold the worker and everything pinned to + it. on os threads a blocking call costs only the task making it. this is the + structural difference that keeps the os-thread backend worth having + regardless of the benchmark numbers. - **dns still blocks a worker** — `green_connect` resolves with a synchronous `getaddrinfo` before its non-blocking connect, so a dial that waits on dns parks the whole worker. the tcp handshake and everything after it already diff --git a/docs/performance.md b/docs/performance.md index 7eeddcaf..045190e6 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -9,6 +9,36 @@ the helpers in `bench/` before trusting them on different hardware. ## where pith stands +the short version, all on the same 2-core machine. the concurrency rows use +the green backend (`PITH_GREEN=1`) where marked, since that is where the +2026-07-26 scheduler work landed: + +| coordination | pith | go | rust | zig | +|---|---:|---:|---:|---:| +| chan_fanout, 1m msgs | **171 ms** (green) | 69 ms | 94-135 ms | 135-204 ms | +| chan_fanout, pinned to 1 worker | **~46 ms** (green) | 69 ms | — | — | +| chan_fanout, os threads | ~438 ms | 69 ms | — | — | +| context switches over the run | 2.5k (green) | 258 | 4.8k | 60k | +| 20k spawn + join | **~50 ms** (green) | ~27 ms | — | — | +| 20k spawn, peak rss | 10 mb (green) | 11 mb | — | — | +| 20k spawn, os threads | ~1450 ms / 174 mb | — | — | — | + +| compute | pith | go | rust | zig | +|---|---:|---:|---:|---:| +| event_ledger, 200k events | 955 ms (1.16x go) | 820 ms | 178 ms | 199 ms | +| std_pipeline, 50k records | 476 ms (1.54x go) | 310 ms | 135 ms | — | +| grpc unary echo, conc=8 | 8075 calls/s (green, 1w) | 13417 | 11012 | — | + +on coordination the green backend beats rust and zig outright, sits ~2.3x +behind go at the default worker count, and beats go pinned to one worker. on +spawn it is ~2x go at go's memory, down from ~29x before the coroutine stack +pool. compute is the older story: competitive with go, well behind rust and +zig, and what is left of that gap is string building rather than the runtime. + +the rows that oversubscribe os threads swing run to run on a 2-core box (see +the caveat on the fan-out table below); the green and go rows are the stable +ones. + all numbers from one 2-core machine, medians of 5 where quick enough to repeat; the tables below were fully rerun 2026-07-15 (after the arc reclamation and weak-reference work), the grpc table again on 2026-07-19