fix(with-lease): kill the child and release the lease on a signal - #3
Open
KalebKE wants to merge 1 commit into
Open
fix(with-lease): kill the child and release the lease on a signal#3KalebKE wants to merge 1 commit into
KalebKE wants to merge 1 commit into
Conversation
`cmdWithLease` ran the child with a bare `exec.Command`: no process group, no
signal handling, no deferred release. `kill -TERM` on roboranch therefore killed
only roboranch. Measured on a real pool:
roboranch alive? no
child alive? YES <- orphaned
lease files after: 2 <- never released
Both halves are bad, and together they are worse than either. The orphan keeps
driving the device. The lease is retained — but its holder PID is now dead, so
`stale()` classifies it as reclaimable and the next checkout's GC hands that
same device to another consumer. Two builds, one simulator: exactly the
double-booking the pool exists to prevent.
Now:
* The child runs in its own process group (`Setpgid`), so a signal can be
forwarded to it *and anything it spawned* — xcodebuild spawns plenty.
* SIGINT/SIGTERM/SIGHUP are forwarded to that group, with a 10s grace period
before SIGKILL. Long enough for xcodebuild to tear down a test run, short
enough that a wedged child cannot pin a lease.
* Release runs exactly once via `sync.Once` behind a `defer`, so it happens on
every exit path including a panic.
* An interrupted run reports 128+signum, so callers can distinguish
"interrupted" from "the command failed".
Note `Setpgid` detaches the child from the terminal's process group, so it no
longer receives ^C on its own — the forwarding is mandatory, not a nicety.
After the fix, the same scenario:
roboranch alive? no
child alive? no <- killed with parent
lease files after: 0 <- released
exit code: 143 (128+15)
Regression test asserts a grandchild dies too, since signalling only the direct
child is precisely the pre-fix behaviour. Verified by mutation: reverting to
`syscall.Kill(pid, ...)` fails the test with "grandchild was orphaned".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014DwDn2W4hVENxD7XZm5naw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cmdWithLeaseran the child with a bareexec.Command— no process group, no signal handling, no deferred release. Measured against a real pool:Both halves are bad; together they're worse than either. The orphan keeps driving the device. The lease is retained — but its holder PID is now dead, so
stale()classifies it reclaimable and the next checkout's GC hands that same device to another consumer. Two builds, one simulator: the exact double-booking the pool exists to prevent.Change
Setpgid), so a signal reaches it and anything it spawned — xcodebuild spawns plenty.sync.Oncebehind adefer— every exit path, including a panic.128+signum, so callers can distinguish "interrupted" from "command failed".Setpgiddetaches the child from the terminal's process group, so it no longer receives ^C on its own — the forwarding is mandatory, not a nicety.After
Testing
The regression test asserts a grandchild dies too. Signalling only the direct child is precisely the pre-fix behaviour, so a test checking the child alone would pass against the bug.
Verified by mutation: reverting to
syscall.Kill(pid, …)fails withgrandchild N was orphaned — the signal did not reach the process group.Also covers signal-0 as a no-op (the un-interrupted path's zero value) and an already-exited child (the race between
selectand the signal).Full suite green,
go vetclean,gofmtclean.Why this matters now
Every leasing consumer inherits it.
sim-pool-run,verify-build.shand both repos'xcbuild-serialwrap builds inwith-lease; without this, cancelling any of them leaks a device to a live orphan.The fix is currently running on the Mac Studio as a hand-installed binary built from this branch — so
go installfrommainsilently reverts the fleet to the broken behaviour. That's the reason to merge rather than leave it.🤖 Generated with Claude Code
https://claude.ai/code/session_014DwDn2W4hVENxD7XZm5naw