feat(sys): capture and log the NCCL environment on run start

[asaiacai]

Records each run's NCCL configuration automatically, with no code change from the user.

What changed

• collect_nccl_env() (pluto/sys.py) is now the single source of truth for the NCCL environment. It widens the previous NCCL_-only scan to the variables that actually shape NCCL behaviour:

  Matched	Examples
  NCCL_*	NCCL_DEBUG, NCCL_SOCKET_IFNAME, NCCL_IB_HCA, NCCL_ALGO
  TORCH_NCCL_*	TORCH_NCCL_ASYNC_ERROR_HANDLING, TORCH_NCCL_BLOCKING_WAIT
  FI_*, OFI_*	FI_PROVIDER, FI_EFA_USE_DEVICE_RDMA (libfabric / aws-ofi-nccl)
  UCX_*	UCX_TLS, UCX_NET_DEVICES
  Extras	TORCH_DISTRIBUTED_DEBUG, TORCH_CPP_LOG_LEVEL, TORCH_SHOW_CPP_STACKTRACES, GLOO_SOCKET_IFNAME

  Results are sorted, and values whose key looks credential-bearing (*_TOKEN, *_SECRET, *_PASSWORD, *_API_KEY, *_ACCESS_KEY, …) are replaced with MASKED_VALUE before anything is stored, sent, or logged.

• System.get_nccl_info() uses it, so the environment continues to ship in systemMetadata.nccl.nccl_env on run create — queryable through the existing systemMetadata. filter prefix.

• Op._log_nccl_env() (new) emits the environment as one console line from Op.start():

  Operation: NCCL environment (4 vars): NCCL_DEBUG=INFO, NCCL_IB_DISABLE=0, ...
  

  It runs after setup_logger() and the sync manager are up, so the line is captured and uploaded like any other console output.

Why both sinks

They cover different ranks. Resuming ranks go through /api/runs/resume, whose payload carries no system info (make_compat_resume_v1), so in a multi-node run systemMetadata only ever describes the rank that created the run. The console line is emitted per rank, which is where a misconfigured worker node actually shows up.

Safety / noise

• Suppressed under mode == 'noop' and disable_system_metrics (backfills via pluto.migrate — the importing host's environment says nothing about the run being written).
• Collection failures are swallowed at debug level; nothing here can break a run.
• The rendered line is truncated at NCCL_ENV_LOG_MAX_CHARS (2048) so a host with a large FI_*/UCX_* set cannot push a multi-KB line into the console stream. The full set still goes to systemMetadata.
• Helper naming (_is_masked_key, not _is_secret_key) follows the CodeQL note already in CLAUDE.md: py/clear-text-logging-sensitive-data classifies a call's result by callee name, and this result is logged.

Known limitation (documented): the environment is read at pluto.init() time, so variables exported later — e.g. by a launcher that configures NCCL immediately before init_process_group() — are not recorded.

Docs: this branch originally added the user-facing description to docs/docs/advanced/01-debugging.md. #145 then deleted the Docusaurus site, so merging main produced a modify/delete conflict there. Resolved by accepting the deletion — the site is gone deliberately and live docs are Mintlify, built outside this repo — and folding the user-facing content (matched-prefix table, list_runs filter example, the read-at-init-time limitation) into the CLAUDE.md section alongside the implementation notes, so nothing is lost and no dangling path reference remains. Happy to move it wherever the Mintlify content is authored instead.

Also in this PR: a fix for a pre-existing race in tests/test_fork_e2e.py::test_fork_e2e_log_metrics, which CI hit on this branch. It polled only until the metric name was queryable — which happens as soon as the first point lands — then read the series and asserted all five values were present (assert 1 == 5 on 3.13, while 3.10/3.11/3.12 passed on the same commit). It now polls for the full series the way the module's parent fixture already does via _poll_max_step; the assertions are unchanged and still have to hold at the deadline.

Tested (run the relevant ones):

• Code formatting: ruff check / ruff format clean under the pinned ruff 0.4.10; python scripts/gen_api_docs.py --check reports docs-api/ up to date.
• Any manual or new tests for this PR (please specify below)

New/updated tests, all passing (re-run post-merge under main's new pytest 9 pin: 103 passed, 4 skipped across the NCCL/system-info/console/fd-capture suites):

• tests/test_nccl_env.py (new, 8 tests) — the log line: emitted at INFO, silent when no vars are set, credentials masked, long environments truncated, suppressed under disable_system_metrics and noop, collection failure never raises, and the hook is actually wired into Op.start().
• tests/test_system_info.py — added coverage for TORCH_NCCL_*/FI_*/UCX_*/extra-key capture, unrelated vars staying out, sorting, case-insensitive masking, explicit-mapping input, and the env reaching the get_info() payload.

Manually verified against a process with NCCL_DEBUG, NCCL_SOCKET_IFNAME, TORCH_NCCL_ASYNC_ERROR_HANDLING, FI_PROVIDER and a decoy NCCL_AUTH_TOKEN set: all five appear in get_nccl_info(), the token redacted, and the startup line renders as expected.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 050f784d-0075-484e-b880-2ca171a9c543

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}