Skip to content

Fix macOS psutil failures and add a macOS CI runner - #1349

Open
brianegge wants to merge 2 commits into
morganstanley:mainfrom
brianegge:macos-ci-support
Open

Fix macOS psutil failures and add a macOS CI runner#1349
brianegge wants to merge 2 commits into
morganstanley:mainfrom
brianegge:macos-ci-support

Conversation

@brianegge

@brianegge brianegge commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Two psutil calls assume capabilities macOS does not provide. Both are reachable from production code, not only from tests, so this is a pair of real macOS bugs rather than only a CI-enablement change. With them fixed, the unit suite is green on macOS and this adds a runner to keep it that way.

Independent of #1348 — either can merge first.

The bugs

1. psutil.net_connectionsAccessDenied

TestRunner._is_remote_process_alive (testplan/runnable/base.py) calls psutil.net_connections(kind="tcp"). Enumerating system-wide sockets requires root on macOS; otherwise psutil raises AccessDenied. Nothing caught it, so it propagated out through _check_pidfilemake_runpath_dirs — meaning an unprivileged macOS run aborts outright whenever the runpath holds a pidfile in the {host}:{port};{pid} form.

Now caught and logged at debug, returning False. That is not a silent swallow: _check_pidfile already falls through to the local psutil.pid_exists check when the remote instance can't be confirmed, so the remaining protection is the one that path was already designed around.

2. Process.io_counters does not exist on macOS

testplan/monitor/resource.py asked proc.as_dict() for io_counters. psutil does not define that attribute on macOS at all, and as_dict() raises ValueError: invalid attr name 'io_counters' for an unknown attribute — killing the collector subprocess on every poll, so resource monitoring silently produced nothing on macOS.

The attribute is now only requested where psutil implements it (hasattr(psutil.Process, "io_counters"), true on Linux/Windows, false on macOS). The existing except (AttributeError, KeyError) branch already reports the per-process IO metrics as zero, so that path is unchanged.

Test determinism (a fix beyond macOS)

The two pidfile tests drove the real system connection table. Besides needing root on macOS, that makes them dependent on whatever the host is doing — test_check_pidfile_active_remote_process looked for any ESTABLISHED TCP connection and called pytest.skip if it found none, so it was silently skipping on Linux CI too, on any idle runner.

Both now stub psutil.net_connections with a minimal record carrying only the fields _is_remote_process_alive reads. Visible in the counts below: Linux goes from 1025 passed, 14 skipped to 1026 passed, 13 skipped — that test now actually runs.

The stub is a local namedtuple rather than psutil's own, whose location is private and moved between psutil 6 and 7 (psutil._common.sconn is gone in 7.x).

The macOS runner

Added as a separate test_macos job, following the separate macos-build job in morganstanley/hobbes rather than folding macOS into the main matrix — the main matrix installs Zookeeper/Kafka and runs the full suite, neither of which fits macOS today.

Scope: tests/unit on 3.10 and 3.13 — the oldest and newest versions this branch's matrix covers. Both were run locally on macOS before proposing them. If #1348 lands, adding 3.14 here is a one-line follow-up.

Deliberately not the functional suite, which needs Zookeeper/Kafka and the built UI bundle, and parts of which additionally depend on the host resolving its own FQDN — on macOS socket.getfqdn() can return a reverse-DNS artifact (I get 1.0.0.127.in-addr.arpa where gethostname() is correct), which breaks the resource-monitor server address. Widening this is follow-up work, and I did not want to hand you a job that goes red for reasons unrelated to these fixes.

Verification

Suite Before After
tests/unit, macOS 3.10 / 3.12 / 3.13 / 3.14 1024 passed, 2 failed (each version) 1026 passed, 0 failed
tests/unit, Linux 3.14 1025 passed, 14 skipped 1026 passed, 13 skipped
test_resource_monitor.py, macOS failed (invalid attr name 'io_counters') passes
test_resource_monitor.py, Linux passed passes

Linux runs were in python:3.14-slim containers, macOS natively on arm64.

One thing I did not touch

The success job's condition references needs.Test.result, but the job id is test (lowercase), so that term looks like it never evaluates as intended. It predates this PR and changing it would alter your merge gating, so I only added test_macos to that job's needs list and left the condition alone — flagging it in case it is not deliberate.

constraints.txt is likewise untouched; the uv-export hook rewrites it on every commit because releaseherald is pinned to a moving main rev, which is unrelated churn.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GWfe48Svf5j2TtgrryJfBK

@brianegge
brianegge requested a review from a team as a code owner August 21, 2026 15:25
Two psutil calls assumed capabilities that macOS does not provide, and both
were reachable from production code rather than only from tests.

`TestRunner._is_remote_process_alive` calls `psutil.net_connections`, which
needs root on macOS and raises `AccessDenied` otherwise. The exception
propagated out through `_check_pidfile` / `make_runpath_dirs`, so an
unprivileged run would abort outright whenever the runpath held a pidfile in
the `{host}:{port};{pid}` form. It is now caught and logged, and the check
falls through to the local PID check that already runs when the remote
instance cannot be confirmed.

The resource monitor asked `Process.as_dict()` for `io_counters`, an
attribute psutil simply does not define on macOS; requesting it raises
`ValueError: invalid attr name 'io_counters'` and killed the collector
subprocess on every poll. The attribute is now only requested where psutil
implements it, and the existing `except (AttributeError, KeyError)` path
reports the per-process IO metrics as zero elsewhere.

The two pidfile tests covering this drove the real system connection table,
which is privileged on macOS and depends on host state everywhere else --
`test_check_pidfile_active_remote_process` skipped entirely on an idle
machine, including on Linux CI. Both now stub `psutil.net_connections`, so
they are deterministic and run on every platform.

Adds a macOS job to the PR workflow, following the separate macos-build job
in morganstanley/hobbes. It is scoped to `tests/unit` on 3.10 and 3.13: the
functional suite needs Zookeeper/Kafka and the built UI bundle, and parts of
it depend on the host resolving its own FQDN, which macOS does not do
reliably.

Verified: unit suite green on macOS 3.10/3.12/3.13/3.14 (1026 passed, was
1024 passed / 2 failed) and on Linux 3.14 (1026 passed, was 1025 passed / 14
skipped -- the formerly skipped test now runs). Resource monitor functional
test passes on both macOS and Linux.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GWfe48Svf5j2TtgrryJfBK
`TestDriverTiming::test_driver_timings` allowed only 100ms of headroom over
the DummyDriver's own sleeps: the teardown sleeps 100ms and the assertion
required the recorded interval to be under 200ms, and setup sleeps 200ms
against a 300ms ceiling.

That is too tight for a loaded GitHub macOS runner. The macOS 3.13 job
recorded a 243ms teardown and failed, while every other job in the same run
passed, including macOS 3.10 with identical bounds.

The test checks that an interval was recorded around the sleep, not how fast
the host is, so the upper bounds move to 600ms for setup and 500ms for
teardown. The lower bounds are unchanged and still catch a timer that never
measured the sleep. This mirrors the widening already done for the driver
report tests in morganstanley#1287 and for test_scheduling_2 in morganstanley#1295.

Verified on macOS 3.10, 3.13 and 3.14: 6 passed in each.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GWfe48Svf5j2TtgrryJfBK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant