Skip to content

fix(with-lease): kill the child and release the lease on a signal - #3

Open
KalebKE wants to merge 1 commit into
mainfrom
fix/with-lease-signal-handling
Open

fix(with-lease): kill the child and release the lease on a signal#3
KalebKE wants to merge 1 commit into
mainfrom
fix/with-lease-signal-handling

Conversation

@KalebKE

@KalebKE KalebKE commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Problem

cmdWithLease ran the child with a bare exec.Command — no process group, no signal handling, no deferred release. Measured against a real pool:

kill -TERM <roboranch>
  roboranch alive?   no
  child alive?       YES  <- orphaned
  lease files after: 2    <- never released

Both halves are bad; together they're worse than either. The orphan keeps driving the device. The lease is retained — but its holder PID is now dead, so stale() classifies it reclaimable and the next checkout's GC hands that same device to another consumer. Two builds, one simulator: the exact double-booking the pool exists to prevent.

Change

  • Child runs in its own process group (Setpgid), so a signal reaches it and anything it spawned — xcodebuild spawns plenty.
  • SIGINT/SIGTERM/SIGHUP forwarded to that group, 10s grace, then SIGKILL. Long enough for xcodebuild to tear down a test run; short enough that a wedged child can't pin a lease.
  • Release runs exactly once via sync.Once behind a defer — every exit path, including a panic.
  • Interrupted runs report 128+signum, so callers can distinguish "interrupted" from "command failed".

Setpgid detaches the child from the terminal's process group, so it no longer receives ^C on its own — the forwarding is mandatory, not a nicety.

After

kill -TERM <roboranch>
  roboranch alive?   no
  child alive?       no   <- killed with parent
  lease files after: 0    <- released
  exit code:         143  (128+15)

Testing

The regression test asserts a grandchild dies too. Signalling only the direct child is precisely the pre-fix behaviour, so a test checking the child alone would pass against the bug.

Verified by mutation: reverting to syscall.Kill(pid, …) fails with grandchild N was orphaned — the signal did not reach the process group.

Also covers signal-0 as a no-op (the un-interrupted path's zero value) and an already-exited child (the race between select and the signal).

Full suite green, go vet clean, gofmt clean.

Why this matters now

Every leasing consumer inherits it. sim-pool-run, verify-build.sh and both repos' xcbuild-serial wrap builds in with-lease; without this, cancelling any of them leaks a device to a live orphan.

The fix is currently running on the Mac Studio as a hand-installed binary built from this branch — so go install from main silently reverts the fleet to the broken behaviour. That's the reason to merge rather than leave it.

🤖 Generated with Claude Code

https://claude.ai/code/session_014DwDn2W4hVENxD7XZm5naw

`cmdWithLease` ran the child with a bare `exec.Command`: no process group, no
signal handling, no deferred release. `kill -TERM` on roboranch therefore killed
only roboranch. Measured on a real pool:

    roboranch alive?   no
    child alive?       YES  <- orphaned
    lease files after: 2    <- never released

Both halves are bad, and together they are worse than either. The orphan keeps
driving the device. The lease is retained — but its holder PID is now dead, so
`stale()` classifies it as reclaimable and the next checkout's GC hands that
same device to another consumer. Two builds, one simulator: exactly the
double-booking the pool exists to prevent.

Now:

  * The child runs in its own process group (`Setpgid`), so a signal can be
    forwarded to it *and anything it spawned* — xcodebuild spawns plenty.
  * SIGINT/SIGTERM/SIGHUP are forwarded to that group, with a 10s grace period
    before SIGKILL. Long enough for xcodebuild to tear down a test run, short
    enough that a wedged child cannot pin a lease.
  * Release runs exactly once via `sync.Once` behind a `defer`, so it happens on
    every exit path including a panic.
  * An interrupted run reports 128+signum, so callers can distinguish
    "interrupted" from "the command failed".

Note `Setpgid` detaches the child from the terminal's process group, so it no
longer receives ^C on its own — the forwarding is mandatory, not a nicety.

After the fix, the same scenario:

    roboranch alive?   no
    child alive?       no   <- killed with parent
    lease files after: 0    <- released
    exit code:         143  (128+15)

Regression test asserts a grandchild dies too, since signalling only the direct
child is precisely the pre-fix behaviour. Verified by mutation: reverting to
`syscall.Kill(pid, ...)` fails the test with "grandchild was orphaned".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014DwDn2W4hVENxD7XZm5naw
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants