fix(test): stop the suite signalling pids it does not own — SIGKILL kills systemd --user, SIGTERM logs you out - #352
Conversation
Four times in one afternoon this suite took a developer's whole
desktop down. The kernel audit named the sender: a test-runner
process SIGKILLed pid 1076694, which was `systemd --user`, and pid 1
then tore down the session cgroup — gnome-shell, Xwayland, browsers,
terminals, every editor, mid-work.
ROOT CAUSE, proxy-held-port.test.mjs, the holder sweep:
let parent = Number(ps -p <pid> -o ppid=);
if (parent > 1) process.kill(parent, "SIGKILL");
`parent > 1` is a liveness test wearing an ownership test's clothes.
An orphan does not reparent to pid 1 on a machine running a systemd
USER manager: that manager is a child subreaper, so it inherits the
orphan and `ps -o ppid=` returns ITS pid. On the affected machine 63
live processes name it as their parent, this repo's own proxy among
them. CI never saw it because a container really does reparent to 1 —
which is exactly why the assumption survived review.
THE FIX is a choke point, not a patch at the site that fired: every
signal to a pid this process did not spawn now goes through
killOurs(), which resolves the command line and refuses anything that
is not ours. Three outcomes, each proven before this commit landed:
alive-and-not-ours throws loudly and names the pid and its command
line; already-gone returns false quietly, so ordinary reaping races
do not turn every sweep red; alive-and-ours is signalled as before.
That last one matters — a guard that over-fires would have broken
the suite instead of the desktop.
Also routed through the choke point: an lsof sweep in
stdio-epipe-survival.test.mjs that killed by port with no ownership
filter at all (raw `lsof -iTCP:<port>` without `@127.0.0.1`, so any
interface), and the remaining SIGKILL-by-number sites in
proxy-held-port and proxy-holder-handover.
NOT changed, and flagged rather than silently touched: the two
negative-pgid reaps on self-spawned handles
(stdio-epipe-survival.test.mjs, shutdown-exit-code.test.mjs). They
signal a group id this process did not necessarily create, which is
the same class of assumption; changing cleanup semantics without
being able to run the suite would be an unverified fix on top of a
verified one.
Verification: every touched file parses (`node --check`); killOurs
proven on all three outcomes against real processes, including the
actual `systemd --user` pid; static sweep confirms no unguarded
SIGKILL-by-number remains. The suite itself was NOT run — running it
is what breaks the machine, and it should not be run until this is
confirmed. The audit rule the operator armed is the confirmation:
after a run, `sudo ausearch -k sigkill9` showing no `ocomm=systemd`
record is the proof, and nothing weaker is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKbpDn7noZ1ryPtSRScXH9
My own fix in this branch was incomplete, and the gap has a name: I
swept for SIGKILL. The commit message even says so — "the remaining
SIGKILL-by-number sites". Two sites signalled the same walked-to
parent with SIGTERM and were never looked at, and they are the worse
half.
let target = pid;
try { target = Number(ps -o ppid= -p <pid>) || pid; } catch {}
if (target <= 1) target = pid;
try { process.kill(target, "SIGTERM"); } catch {}
listeners() had already been introduced above these lines, and its
comment says that is where the ours-only predicate lives. It filters
the LISTENER. Nothing filters what `ps -o ppid=` returns next, and on a
machine whose orphans reparent to a systemd USER manager that pid IS
the manager.
SIGTERM to a user manager is not a stop, it is a logout. systemd(1):
"systemd user managers will start the exit.target unit when this signal
is received. This is mostly equivalent to systemctl --user start
exit.target."
Measured 2026-08-20, five seconds into a suite run: the journal shows
"Activating special unit Exit the Session", then Main User Target and
GNOME Session stopping, then gnome-shell faulting inside
JSRuntime::destroyRuntime — the shutdown path, not a crash that caused
one. The kernel audit rule armed for this had nothing, because it
watches SIGKILL and this is SIGTERM. An instrument keyed to one route
returns, for every route it does not watch, exactly what a true absence
returns; I read that silence as exoneration for most of an hour.
Both sites now go through killOurs(target, "SIGTERM"), and NOT inside a
try/catch — the throw is the mechanism, and catching it restores the
silence that let this run unattributed. A cleanup loop that meets a
stranger now goes red naming the pid and its command line.
Expect this to make the suite RED where it was previously green and
destructive, on any host where the walked-to parent is not one of ours.
That is the trade, stated rather than discovered.
NOT changed, and reported instead: proxy-held-port.test.mjs:256 is the
same false premise in its silent direction. Its orphan sweep reads
`if (ppid !== 1) continue`, and its comment asserts "a leaked successor
is detached and always reparented to init" — false under a subreaper,
so on every desktop that sweep matches nothing and reaps nothing. It
fails quiet rather than loud, and proc-helpers.mjs already records the
consequence: ten invisible orphans at once, the oldest 788 s. Fixing it
changes what gets reaped, and I cannot run this suite to find out.
Verification: node --check passes. The suite was NOT run — running it
is what takes the desktop down. Static sweep of the remaining
process.kill sites in this file: 505/550/2407 signal listeners()/
onPort() output (ours-filtered), 723 signals proxyPid() (filtered on
/server\.mjs/ against the cmdline), 816 is a signal-0 liveness probe,
256 is the site named above.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKbpDn7noZ1ryPtSRScXH9
This PR was incomplete when I opened it. New commit, and the reason is worth stating.The first version fixed the SIGKILL sites. It ran on my machine tonight and the
let target = pid;
try { target = Number(execFileSync("ps", ["-o","ppid=","-p",String(pid)], …).trim()) || pid; } catch {}
if (target <= 1) target = pid;
try { process.kill(target, "SIGTERM"); } catch {}Same defect, worse consequence. From
SIGTERM to a user manager is a logout. The journal, five seconds into the run: The kernel audit rule I recommended in #351 saw none of it. It watches SIGKILL. How I missed it, since it is the same shape twiceI swept for SIGKILL. My own commit message says so — "the remaining The line above them is the sharper lesson, and it is not mine: // THROUGH listeners(), which is where the ours-only predicate lives.
This commitBoth sites now go through Expect this to make the suite red where it was previously green and One sibling I did not touch
Fixing it changes what gets reaped and I cannot run this suite to find out. An Verification, unchanged in kind
If you arm an audit rule to confirm this after merging, arm both signals — the sudo auditctl -a always,exit -F arch=b64 -S kill -F a1=9 -k sigkill9
sudo auditctl -a always,exit -F arch=b64 -S kill -F a1=15 -k sigterm15🤖 Generated with Claude Code — Gunther's Claude Code session |
systemd --user)systemd --user, SIGTERM logs you out
`no test file asks lsof who holds a port` matched the literal
`execFileSync("` + lsof. The one file that had gone around listeners()
spells it `spawn(` + lsof, so the guard reported clean on a tree
carrying exactly the defect it was written for — and its own comment
says the class was closed: "the only thing left to police is a NEW
inline call".
That comment is also where this bug is described, in the past tense:
"two of them then walked UP to the listener's parent and sent SIGTERM;
a stranger's parent is this runner." The hazard was known. The
predicate covered one spelling of it.
Discriminating pair, same input both times — the frozen upstream/main
tree, which still carries the raw call:
OLD guard: 17 pass, 0 fail (the violation is invisible)
NEW guard: 16 pass, 1 fail ("these files ask lsof directly and
never reach killOurs()")
Three changes:
1. The needle becomes every call form — execFileSync, execSync,
spawnSync, spawn, exec — with optional whitespace and any quote
style, assembled so this file (itself swept) cannot match itself.
2. The rule is re-pointed at what actually protects the machine. A raw
lsof call is legitimate where a DEADLINE is wanted, which the shared
helper's execFileSync cannot give; what must never be skipped is the
filter. So the predicate is now "asks lsof AND never reaches
killOurs()", not "asks lsof".
3. A second arm: the call must be loopback-scoped. `-iTCP:<port>`
matches any interface, so the answer can name a process that merely
holds the same port number on another address.
Both arms proven to fire alone, so neither rides the other: dropping
only the scope from an otherwise-filtered call goes red naming
@127.0.0.1 while the filter arm stays quiet; the upstream tree goes red
naming killOurs with the scope arm quiet. Baseline before each: 17
pass, 0 fail.
Also fixes the scope of that call in stdio-epipe-survival.test.mjs. The
fixture binds 127.0.0.1, so the narrow query was always the intended
one; the wide one was how the answer could name a stranger at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKbpDn7noZ1ryPtSRScXH9
Third commit: the guard for this class was green on the tree that had itA fix without the check that would have caught it just waits to be re-made, and Its own comment is where the hazard is described, in the past tense:
…followed by "the only thing left to police is a NEW inline call." The hazard was Discriminating pair, same input both times — the frozen
What changed:
Both arms are proven to fire alone, so neither rides the other: dropping only
🤖 Generated with Claude Code — Gunther's Claude Code session |
…n red since
Five reds in CI run 32409699496, one cause. Every one of them is killOurs()
refusing to kill a fake proxy the case had spawned itself moments earlier:
refusing to SIGKILL pid 3599: it is alive and its command line is not one
of ours.
command: node /tmp/ccf-fake-proxy-3290-3-PDRL3r/scratch-fake-server-3290-3.mjs
ours: /\/(?:bin|proxy)\/[\w.-]+\.mjs\b/
That is this branch's own guard firing on legitimate work, which is not a
safe failure: it is precisely what trains the `catch {}` around killOurs()
that the guard's own message forbids. The error even says "filter the pid at
its source" — and the call site had already done so twice, taking only
children of a launcher it spawned and then only those whose command line
names its own stand-in. The predicate then overrode that with a weaker,
path-shaped test and threw.
The two stand-in families differed only by accident of LOCATION. The launcher
copy lands in bin/, so it matched; its server is written to a fresh temp dir,
so it did not.
The fix widens ownership with evidence that is STRONGER than a path segment,
not weaker. Stand-in names are built from `${process.pid}-${++fakeSeq}`, so a
command line carrying OUR pid inside a filename WE generated cannot belong to
a stranger — where a bare `scratch-fake-server-` match could, and is
deliberately not what this does. It is rebuilt per call, so a forked runner
cannot inherit its parent's claim, and it reaches killOurs() only: listeners()
and ours() answer "which proxy holds this port", where a stand-in is unwanted.
The predicate had no test of its own, which is why this shipped red. It has
one now, and the arms come in pairs because both directions are load-bearing:
a wrong NO is the failure above, a wrong YES is the outage this branch exists
to stop. The heaviest arm is a DIFFERENT runner's stand-in — same shape, must
still be refused — because without it a bare name match passes everything
else. The first case is the real command line from that CI run.
Red-first, staged so the arms are observable rather than lost behind a missing
export: the old predicate re-exported under the new name gives 1 pass / 2 fail,
the failure naming that exact command line; with the fix, 3 of 3. The
"OURS itself is unchanged" arm is green in BOTH, so it is not riding on the
change.
Not run locally, deliberately: running this suite on a host with a systemd
user manager is what this branch exists to stop, and the fix has not yet been
exercised on such a host. CI is the verifier.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014eneLs4v99wKd1S5hmaard
…body knew Operator trigger: the sense that PR work keeps dangling. The survey run the same hour says the sense is right. Of five open PRs authored here, cnighswonger#352 had been CI-red since 08-20 and cnighswonger#276 both conflicting and CI-red since 08-06 with its head unmoved for sixteen days. Neither redness had reached anyone. The cause is structural rather than anyone's lapse. The session that pushed cnighswonger#352 could not run the suite — running it is what takes the desktop down — so it shipped a red it had no way to observe, and after that nothing ever looked again. The existing runbook is an INTENT workflow: a procedure a session sets out to run for a round. Rot is not an intent, it is a passage of time. The repo's own taxonomy already names the missing kind, an event lane entered because something fired, and nothing fires here today. Booked as RECORD rather than READY because the trigger is genuinely undecided — schedule versus event — and a sweep that runs when nobody is present produces findings with no reader, which the closing gate's recurring-producer clause covers in full. The design and both red-first arms are written down, including the one that matters most: a CI pass whose run head_sha is not the PR head must report unverified, not green. That case is available in the live data rather than constructed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014eneLs4v99wKd1S5hmaard
|
CI has been red on this branch since it opened, and the cause was the branch's own guard. Fixed in What was failingFive failures in run That is this branch's guard firing on legitimate work, and it is not a safe direction to fail in: it is precisely what trains the The two stand-in families differed only by accident of location. The launcher copy is written into The fixOwnership is widened with evidence that is stronger than a path segment, which is what makes this a widening rather than a softening. Stand-in names are built from
The predicate now has a testIt had none, which is why this shipped red. Arms come in pairs because both directions are load-bearing: a wrong NO is the failure above, a wrong YES is the outage this branch exists to stop. The heaviest arm is a different runner's stand-in — identical shape, must still be refused — because without it a bare name match passes everything else. The first case is the real command line from the failing run. Red-first, staged so the arms are observable rather than lost behind a missing export: the old predicate re-exported under the new name gives 1 pass / 2 fail, the failure naming that exact command line; with the fix, 3 of 3. The "OURS itself is unchanged" arm is green in both, so it is not riding on the change. What I did not doI did not run the suite locally. Running it on a host with a systemd user manager is what this branch exists to prevent, and the fix has not yet been exercised on such a host — so CI is the verifier here, not a local green I could not honestly claim. Prior CI on this head: 1941 pass / 5 fail. Now: Worth stating for sequencing, since it affects more than this PR: until this lands, no contributor on such a host can run the suite for any branch cut from 🤖 Generated with Claude Code — Gunther's Claude Code session |
…pt measured Two things, both from the same hour's work. THE LANE. There was a runbook for answering a review round and nothing that ever asked after the round ended. That is the right shape for an intent — a session sets out to run a round — and the wrong shape for rot, which is elapsed time and fires nothing. So PRs sat: two of five red, one of them for sixteen days with its head unmoved, and neither redness had reached anybody. The lane sweeps the SET, because acting on the PR that came to mind is how the other four keep rotting. It carries the traps this session hit rather than a generic checklist: read state from the API and never from the backlog's table (that table was seven days stale, listing a closed PR and missing two open ones); compare a CI run's head_sha against the PR head, because a pass on a superseded head reads exactly like a real one and is not one; read the thread before rebasing, since a green mergeable PR can be waiting on a maintainer's sequencing answer and rebasing it is work done twice. Seven terminal dispositions, and waiting is one of them — a nudge on a PR whose ball is upstream's is noise. It also records the constraint that currently orders the whole board: while cnighswonger#352 is unmerged, a tree cut from upstream main cannot have its suite run on this host at all, so CI is the only verifier for every upstream-facing branch. That makes cnighswonger#352 the sequencing head rather than merely the oldest red. THE XDG PR is parked, and the attempt is what produced the reason. Of 66 files, 47 exist upstream; the 48-file slice still lands 9 files and 13 hunks in conflict, because the change sits on other unmerged fork changes in the same files. It is a re-author against upstream, not a cherry-pick, and saying so before starting is cheaper than grinding the conflicts as if they were mechanical. Verification is blocked besides — demonstrated live, the gate denied the slice worktree's suite. Both missing pieces are named and neither is dissolvable from here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014eneLs4v99wKd1S5hmaard
Four times in one afternoon, running this suite took my whole desktop down.
The kernel audit subsystem named the sender: a node test-runner process
SIGKILLed
systemd --user, and pid 1 then tore down the session cgroup —shell, editors, browser, terminals, mid-work.
Full write-up, evidence and the audit recipe: #351
The defect
test/proxy-held-port.test.mjs, the holder sweep:parent > 1is a liveness test wearing an ownership test's clothes. An orphandoes not reparent to pid 1 on a machine running a systemd user manager:
that manager sets
PR_SET_CHILD_SUBREAPER, so it inherits the orphan andps -o ppid=returns its pid. The sweep reads that as a parent it created.CI never saw it because a container really does reparent to pid 1 — which is
exactly why the assumption survived review.
The fix
A choke point, not a patch at the site that fired. Every signal to a pid this
process did not spawn now goes through
killOurs()intest/proc-helpers.mjs,beside the existing
OURSpredicate thatlisteners()andours()alreadyuse. It resolves the target's command line and refuses anything that is not one
of this repo's own binaries.
Three outcomes, each proven against real processes before this landed:
falsequietly — reaping races stay greenThe middle row is what keeps the guard from being the next bug: a guard that
over-fires would have broken the suite instead of the desktop. The first row
throws rather than skipping because a silent skip is what let this run four
times before anyone knew where it came from.
Also routed through the choke point in this change:
test/stdio-epipe-survival.test.mjs— anlsof -nP -t -iTCP:<port> -sTCP:LISTENsweep with no ownership filter and no
@127.0.0.1, so it matched listenerson any interface;
n > 1 && n !== holder.pidwas the only thing between thatloop and a stranger's pid. (Its local variable is named
listeners, whichreads like the shared helper of that name but is a raw
lsof— worth a lookif you'd prefer it renamed.)
test/proxy-holder-handover.test.mjs— two double-filtered kill sites.proxy-held-port.test.mjs.Deliberately not changed
The two negative-pgid reaps on self-spawned handles
(
stdio-epipe-survival.test.mjs,shutdown-exit-code.test.mjs). They signal agroup id this process did not necessarily create — the same class of
assumption — but changing cleanup semantics without being able to run the suite
would be an unverified fix stacked on a verified one. Flagged here rather than
silently touched.
Verification, and its honest limit
node --check)killOurs()exercised on all three outcomes against real processes, includingthe actual
systemd --userpid — it refuses it loudlytest/The suite itself was not run. Running it is what breaks the machine, and it
should not be run on a systemd-user desktop until this is in. The confirmation I
can offer afterwards is the audit rule: a run followed by
ausearch -k sigkill9showing no record whose target is
ocomm=systemd. Nothing weaker proves it, andI'd rather say so than claim a green I don't have. CI will exercise the changed
paths normally, since containers were never affected.
🤖 Generated with Claude Code