feat(sc): NeMo-Gym path

[yuki-97]

Part 4/4 of the #2819 split. Stack on #3266.

#3219 (1/4) -> #3220 (2/4) -> #3266 (3/4) -> #3267 (4/4)

Summary

Wires NeMo-Gym rollouts into the SC (Single Controller) entrypoint by introducing an SC-facing gym-actor spinup helper and wiring it into the SC setup path. The existing grpo.py path retains its local _spinup_nemo_gym implementation for now because it also supplies GRPO-specific timing, routed-experts dtype, and fastokens configuration.

• spinup_nemo_gym_actor (nemo_rl/environments/nemo_gym.py): factors the NeMo-Gym actor construction needed by SC (config assembly, uv/venv cache reuse, image-baked venv reuse, and soft NodeAffinitySchedulingStrategy when num_gpu_nodes > 0) into a reusable helper. Consolidating the remaining GRPO-specific startup path is left for follow-up work.
• SC setup wiring (nemo_rl/algorithms/single_controller_utils/setup.py): setup_single_controller branches on _should_use_nemo_gym — skips env spinup in setup_response_data, brings up generation first, then calls spinup_nemo_gym_actor with the vLLM OpenAI URLs. Raises NotImplementedError on non-vllm backends.
• examples/run_grpo_single_controller.py: restores the setup_nemo_gym_config call (removed in feat(sc): setup + entrypoint #3266 while the SC path had no gym support).
• Tests: tests/unit/single_controller/test_single_controller_setup.py gains env-handle wiring + non-vllm backend guard coverage (replaces the previous "not supported yet" trip-wire). New tests/functional/grpo_async_gym_single_controller.sh covers SC + NeMo-Gym end-to-end.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[terrykong]

Re-review at the rebased head (dd3cb538, now MERGEABLE/CLEAN). The spinup_nemo_gym_actor extraction and SC wiring check out: the 2-tuple/4-tuple setup_response_data unpacking is correct (now with an explicit assert len(...) == 2), dp_openai_server_base_urls is guaranteed real by the _should_use_nemo_gym asserts, and configure_vllm_for_router_replay is a safe no-op when router replay is off. Note: the rebase dropped the original PR's grpo.py refactor — grpo.py keeps its own inline _spinup_nemo_gym (still importing NemoGym, never referencing spinup_nemo_gym_actor), so the new helper is currently a near-duplicate used only by the SC path. grpo's sync path is untouched (no behavior change), but the description's "shared function so both grpo.py and the SC setup can call it" no longer holds — worth either re-landing the grpo call-site swap or noting the duplication. No correctness bugs.

Six inline comments: one medium (the new e2e functional test is registered in no CI lane, so it never runs — regressed during the rebase), and five low/non-blocking (guard hoist, a test-assertion tightening, a docstring, a smoke-threshold note, and a router-replay tracking note). Enabling router replay (R3) on the SC async+TransferQueue path is only half-wired and fails silently — filed as #3327 (assigned) and added to the SC loose-ends tracker #2625.

Cross-checked against merged #3219/#3220 and base #3266 threads to avoid duplication.

Reviewed by an agent team (rl-expert, bug-finder, test-agent, comment-reviewer, devil's-advocate), re-audited after rebase and cross-referenced against the PR stack. Generated by Claude Code

[RayenTian]

Re-review at the rebased head (dd3cb538, now MERGEABLE/CLEAN). The spinup_nemo_gym_actor extraction and SC wiring check out: the 2-tuple/4-tuple setup_response_data unpacking is correct (now with an explicit assert len(...) == 2), dp_openai_server_base_urls is guaranteed real by the _should_use_nemo_gym asserts, and configure_vllm_for_router_replay is a safe no-op when router replay is off. Note: the rebase dropped the original PR's grpo.py refactor — grpo.py keeps its own inline _spinup_nemo_gym (still importing NemoGym, never referencing spinup_nemo_gym_actor), so the new helper is currently a near-duplicate used only by the SC path. grpo's sync path is untouched (no behavior change), but the description's "shared function so both grpo.py and the SC setup can call it" no longer holds — worth either re-landing the grpo call-site swap or noting the duplication. No correctness bugs.

Six inline comments: one medium (the new e2e functional test is registered in no CI lane, so it never runs — regressed during the rebase), and five low/non-blocking (guard hoist, a test-assertion tightening, a docstring, a smoke-threshold note, and a router-replay tracking note). Enabling router replay (R3) on the SC async+TransferQueue path is only half-wired and fails silently — filed as #3327 (assigned) and added to the SC loose-ends tracker #2625.

Cross-checked against merged #3219/#3220 and base #3266 threads to avoid duplication.

Reviewed by an agent team (rl-expert, bug-finder, test-agent, comment-reviewer, devil's-advocate), re-audited after rebase and cross-referenced against the PR stack. Generated by Claude Code

Update the PR doc describe.

[RayenTian]

/ok to test 1d6a09c

[RayenTian]

/ok to test 1d6a09c

[RayenTian]

/ok to test b2bd34d