Five tests in tests/ut/py/test_callable_identity.py start a simpler.remote_l3_worker subprocess on a free TCP port, time.sleep(0.3), then connect. Under load the daemon is not listening yet and the connect is refused.
Observed
ut (macos-latest, 3.10) on PR #1466 head c5c50ca6:
FAILED tests/ut/py/test_callable_identity.py::test_remote_sim_buffer_copy_roundtrip
- ConnectionRefusedError: [Errno 61] Connection refused
and locally on aarch64 in a full pytest tests/ut run, a sibling in the same family:
FAILED tests/ut/py/test_callable_identity.py::test_remote_sim_health_lane_stays_live_during_long_task
- ConnectionRefusedError: [Errno 111] Connection refused
python/simpler/worker.py:2486 sock.connect(sockaddr)
Both cleared on re-run. Each test passes 6/6 when run alone on both main and the PR branch, so the trigger is contention during the full suite, not the code under test.
Cause
daemon = subprocess.Popen([sys.executable, "-m", "simpler.remote_l3_worker", ...])
worker = Worker(level=4, ...)
try:
time.sleep(0.3) # <- hope the daemon bound the port
worker_id = worker.add_remote_worker(...)0.3s is a guess at interpreter start + import + bind() + listen(). It holds on an idle machine and fails when the runner is busy. The five call sites are at test_callable_identity.py:1026, 1059, 1094, 1128, 1161.
Note the failure surfaces inside Worker._connect_within_deadline, whose own bounded-deadline logic is working correctly — it reports the refusal immediately because the port is closed, not because the deadline expired. Raising the worker-side timeout would not help; nothing is listening yet.
Suggested fix
Wait for readiness rather than for a duration: poll connect() on the port until it succeeds or a generous deadline expires, and fail with the daemon subprocess's captured output if it exits first. A small shared helper would cover all five call sites.
This is the same class as #1504 (that test guessed at how to identify children; these guess at when a daemon is up) — replacing the guess removes the failure mode rather than widening the margin.
Five tests in
tests/ut/py/test_callable_identity.pystart asimpler.remote_l3_workersubprocess on a free TCP port,time.sleep(0.3), then connect. Under load the daemon is not listening yet and the connect is refused.Observed
ut (macos-latest, 3.10)on PR #1466 headc5c50ca6:and locally on aarch64 in a full
pytest tests/utrun, a sibling in the same family:Both cleared on re-run. Each test passes 6/6 when run alone on both
mainand the PR branch, so the trigger is contention during the full suite, not the code under test.Cause
0.3sis a guess at interpreter start + import +bind()+listen(). It holds on an idle machine and fails when the runner is busy. The five call sites are attest_callable_identity.py:1026, 1059, 1094, 1128, 1161.Note the failure surfaces inside
Worker._connect_within_deadline, whose own bounded-deadline logic is working correctly — it reports the refusal immediately because the port is closed, not because the deadline expired. Raising the worker-side timeout would not help; nothing is listening yet.Suggested fix
Wait for readiness rather than for a duration: poll
connect()on the port until it succeeds or a generous deadline expires, and fail with the daemon subprocess's captured output if it exits first. A small shared helper would cover all five call sites.This is the same class as #1504 (that test guessed at how to identify children; these guess at when a daemon is up) — replacing the guess removes the failure mode rather than widening the margin.