Introduced by #1495. Observed failing on an unrelated PR's ut job and reproduced on plain main (1 failure in 20 local runs).
Symptom
FAILED tests/ut/py/test_worker/test_orphan_child_reaping.py::test_children_exit_when_parent_is_killed
E AssertionError: expected exactly 3 forked sub-worker pids, parent reported [6511, 6512, 6513, 3]
E parent output:
E 6511 6512 6513
E .../multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 3 leaked shared_memory objects to clean up at shutdown
E assert 4 == 3
Cause
The test collects the parent's pids by scanning its whole combined output for digit tokens:
reported = [int(tok) for tok in out_path.read_text().split() if tok.isdigit()]
assert len(reported) == NUM_SUB_WORKERS
stdout and stderr of the parent share one file. The parent is SIGKILLed by design, so its shared-memory segments are never unlinked and multiprocessing.resource_tracker — a separate process holding the same fd — emits
resource_tracker: There appear to be 3 leaked shared_memory objects to clean up at shutdown
That sentence contributes a bare 3, which isdigit() accepts as a fourth pid.
It is intermittent because the tracker is a separate process: the failure happens only when its warning reaches the file before the test reads it, after parent.wait() returns. The leaked segments themselves are inherent to the test (it kills the parent with SIGKILL), so the warning is expected — only the parsing is wrong.
Coincidentally the leak count equals NUM_SUB_WORKERS, so the stray token is always 3 and always pushes the count to exactly 4.
Reproduction
for i in $(seq 1 20); do
python -m pytest tests/ut/py/test_worker/test_orphan_child_reaping.py -q || echo "FAILED on iter $i"
done
1/20 on main at 3dc617aa (aarch64, Python 3.10). CI hit it on ubuntu-latest and macos-latest in the same run, so it is not arch-specific.
Suggested fix
Do not scan free text for pids. Either
- have the parent print the pids on a single delimited line and parse only that line, or
- redirect the parent's
stderr to a separate file so tracker noise cannot reach the pid stream, or
- keep one file but bound the parse to tokens that are plausible pids and to the first line of output.
The first is the most robust: the parse then does not depend on what else the interpreter decides to write at shutdown.
Introduced by #1495. Observed failing on an unrelated PR's
utjob and reproduced on plainmain(1 failure in 20 local runs).Symptom
Cause
The test collects the parent's pids by scanning its whole combined output for digit tokens:
stdoutandstderrof the parent share one file. The parent is SIGKILLed by design, so its shared-memory segments are never unlinked andmultiprocessing.resource_tracker— a separate process holding the same fd — emitsThat sentence contributes a bare
3, whichisdigit()accepts as a fourth pid.It is intermittent because the tracker is a separate process: the failure happens only when its warning reaches the file before the test reads it, after
parent.wait()returns. The leaked segments themselves are inherent to the test (it kills the parent with SIGKILL), so the warning is expected — only the parsing is wrong.Coincidentally the leak count equals
NUM_SUB_WORKERS, so the stray token is always3and always pushes the count to exactly 4.Reproduction
1/20 on
mainat3dc617aa(aarch64, Python 3.10). CI hit it on ubuntu-latest and macos-latest in the same run, so it is not arch-specific.Suggested fix
Do not scan free text for pids. Either
stderrto a separate file so tracker noise cannot reach the pid stream, orThe first is the most robust: the parse then does not depend on what else the interpreter decides to write at shutdown.