Skip to content

os/open: stop the stderr drain from spinning and leaking zombies - #161

Merged
lawrencecchen merged 1 commit into
mainfrom
fix-open-stderr-spin
Jul 28, 2026
Merged

os/open: stop the stderr drain from spinning and leaking zombies#161
lawrencecchen merged 1 commit into
mainfrom
fix-open-stderr-spin

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Jul 28, 2026

Copy link
Copy Markdown

openThread drained a spawned child's stderr with std.Io.Reader.takeDelimiterExclusive, which advances up to but not past the delimiter. Once the seek position sits on a \n it returns a zero-length slice forever — no progress, no error. Verified on zig 0.15.2:

input "aaa\nbbb\nccc\n"
iter 0: len=3 'aaa'
iter 1: len=0 ''      <- parked on the newline
iter 2..inf: len=0 ''

So any single line of stderr wedges the thread permanently. The loop spun at 100% CPU emitting empty open stderr= records until macOS throttled the process-wide logging firehose (__FIREHOSE_CLIENT_THROTTLED_DUE_TO_HEAVY_LOGGING__, which then makes every os_log call in the process expensive). exe.wait() was only reachable by exiting that loop, so the child was never reaped either.

Measured on cmux 0.64.20 after ~1 day uptime: 11 zombie children matched one-for-one by 11 threads at ~12.4% CPU each — ~95% of the process's 500-600% total, every one 94-97% of its stack inside zig_os_log_with_type. The pairing is causal: all 11 zombies spawned in a 16-second burst, and thread-id against zombie-pid regresses at R² = 0.999927, the signature of exe.spawn() immediately followed by std.Thread.spawn() eleven times.

Change

  • Use takeDelimiter, which consumes the delimiter and reports end-of-stream as null, so the loop advances and terminates.
  • defer _ = exe.wait() catch {} so the child is reaped unconditionally, including on early exits.
  • Cap reporting at 32 lines while still draining, so a child blocked writing into a full pipe can finish and exit.
  • Extract the loop as drainStderr and test it: every line reported, all-blank-lines input terminates (the case that used to hang), a final line with no trailing newline, and cap-still-drains-to-EOF.

Affects Linux too — the GTK apprt reaches this via OpenURI.zig whenever the XDG portal is unavailable, which is a more common path than the macOS one.

zig build clean and zig build test -Dtest-filter=drainStderr green on zig 0.15.2.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Fixes a bug in os/open where draining a child’s stderr could spin at 100% CPU and leave zombie processes. Always reaps the child and limits log spam while still draining to EOF.

  • Bug Fixes
    • Switched to takeDelimiter so the loop advances and terminates on EOF.
    • Reap the child unconditionally via defer _ = exe.wait() catch {} to prevent zombies.
    • Cap stderr reporting at 32 lines while continuing to drain to avoid pipe backpressure and log throttling.
    • Extracted drainStderr and added tests for blank lines, no trailing newline, full-cap drain, and normal line reporting.

Written for commit 8f31fb5. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of application launch errors to prevent hangs when reading process output.
    • Ensured launched processes are properly completed and cleaned up.
    • Added safeguards to limit excessive error messages while continuing to process all output.
    • Improved handling of blank lines and final messages without trailing line breaks.

openThread drained a child's stderr with takeDelimiterExclusive, which
advances only *up to* the delimiter. Once the seek position sits on a '\n'
it returns a zero-length slice forever, without progressing and without
erroring, so the loop spun at 100% CPU after the very first stderr line,
emitting empty 'open stderr=' records until macOS throttled the
process-wide logging firehose. wait() was only reachable by exiting that
loop, so the child was never reaped either.

Measured on cmux 0.64.20 after ~1 day uptime: 11 zombie children matched
one-for-one by 11 threads burning ~12.4% CPU each (~132.8% of the process's
148%), each with 94-97% of its stack inside zig_os_log_with_type and
__FIREHOSE_CLIENT_THROTTLED_DUE_TO_HEAVY_LOGGING__ in the trace.

Switch to takeDelimiter, which consumes the delimiter and reports
end-of-stream as null, so the loop genuinely advances and terminates. Reap
via defer so wait() is unconditional. Cap reporting at 32 lines while still
draining, so a child blocked writing into a full pipe can finish and exit.

Extract the loop as drainStderr and test that it consumes its input and
terminates, including the all-blank-lines case that used to hang.
@lawrencecchen
lawrencecchen merged commit 0b1734f into main Jul 28, 2026
146 of 147 checks passed
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 21b62013-f8ed-4704-b06a-fb9530c9962d

📥 Commits

Reviewing files that changed from the base of the PR and between 4a6c443 and 8f31fb5.

📒 Files selected for processing (1)
  • src/os/open.zig

📝 Walkthrough

Walkthrough

src/os/open.zig now uses a dedicated, capped stderr-draining helper, continues consuming stderr after the reporting limit, avoids delimiter-related spinning, and always reaps child processes. Unit tests cover termination, blank lines, unterminated final lines, and capped logging.

Changes

Open stderr handling

Layer / File(s) Summary
Capped stderr draining
src/os/open.zig
Adds drainStderr, caps logged stderr lines, emits a truncation warning, continues draining to end-of-stream, and tests termination and line-count behavior.
openThread process lifecycle
src/os/open.zig
Routes stderr through the helper and guarantees child-process reaping with deferred wait().

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested reviewers: jcollie

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-open-stderr-spin

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant