os/open: stop the stderr drain from spinning and leaking zombies - #161
Merged
Conversation
openThread drained a child's stderr with takeDelimiterExclusive, which advances only *up to* the delimiter. Once the seek position sits on a '\n' it returns a zero-length slice forever, without progressing and without erroring, so the loop spun at 100% CPU after the very first stderr line, emitting empty 'open stderr=' records until macOS throttled the process-wide logging firehose. wait() was only reachable by exiting that loop, so the child was never reaped either. Measured on cmux 0.64.20 after ~1 day uptime: 11 zombie children matched one-for-one by 11 threads burning ~12.4% CPU each (~132.8% of the process's 148%), each with 94-97% of its stack inside zig_os_log_with_type and __FIREHOSE_CLIENT_THROTTLED_DUE_TO_HEAVY_LOGGING__ in the trace. Switch to takeDelimiter, which consumes the delimiter and reports end-of-stream as null, so the loop genuinely advances and terminates. Reap via defer so wait() is unconditional. Cap reporting at 32 lines while still draining, so a child blocked writing into a full pipe can finish and exit. Extract the loop as drainStderr and test that it consumes its input and terminates, including the all-blank-lines case that used to hang.
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthrough
ChangesOpen stderr handling
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
openThreaddrained a spawned child's stderr withstd.Io.Reader.takeDelimiterExclusive, which advances up to but not past the delimiter. Once the seek position sits on a\nit returns a zero-length slice forever — no progress, no error. Verified on zig 0.15.2:So any single line of stderr wedges the thread permanently. The loop spun at 100% CPU emitting empty
open stderr=records until macOS throttled the process-wide logging firehose (__FIREHOSE_CLIENT_THROTTLED_DUE_TO_HEAVY_LOGGING__, which then makes everyos_logcall in the process expensive).exe.wait()was only reachable by exiting that loop, so the child was never reaped either.Measured on cmux 0.64.20 after ~1 day uptime: 11 zombie children matched one-for-one by 11 threads at ~12.4% CPU each — ~95% of the process's 500-600% total, every one 94-97% of its stack inside
zig_os_log_with_type. The pairing is causal: all 11 zombies spawned in a 16-second burst, and thread-id against zombie-pid regresses at R² = 0.999927, the signature ofexe.spawn()immediately followed bystd.Thread.spawn()eleven times.Change
takeDelimiter, which consumes the delimiter and reports end-of-stream asnull, so the loop advances and terminates.defer _ = exe.wait() catch {}so the child is reaped unconditionally, including on early exits.drainStderrand test it: every line reported, all-blank-lines input terminates (the case that used to hang), a final line with no trailing newline, and cap-still-drains-to-EOF.Affects Linux too — the GTK apprt reaches this via
OpenURI.zigwhenever the XDG portal is unavailable, which is a more common path than the macOS one.zig buildclean andzig build test -Dtest-filter=drainStderrgreen on zig 0.15.2.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by cubic
Fixes a bug in
os/openwhere draining a child’s stderr could spin at 100% CPU and leave zombie processes. Always reaps the child and limits log spam while still draining to EOF.takeDelimiterso the loop advances and terminates on EOF.defer _ = exe.wait() catch {}to prevent zombies.drainStderrand added tests for blank lines, no trailing newline, full-cap drain, and normal line reporting.Written for commit 8f31fb5. Summary will update on new commits.
Summary by CodeRabbit