Skip to content

feat(slurm): support multi-node Dynamo RL jobs - #4

Draft
furionw wants to merge 2 commits into
feat/dynamo-slurm-single-nodefrom
feat/dynamo-slurm-multi-node
Draft

feat(slurm): support multi-node Dynamo RL jobs#4
furionw wants to merge 2 commits into
feat/dynamo-slurm-single-nodefrom
feat/dynamo-slurm-multi-node

Conversation

@furionw

@furionw furionw commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Based on #3.

Why

The single-node Dynamo launcher owns every process on one host, but Prime's multi-node Slurm template previously could not coordinate one frontend, role-specific workers, shared discovery, and service teardown across ranks. Multi-node aggregate and P/D RL need the parent allocation to own those processes without nested Slurm submissions, while standalone Dynamo inference remains unsupported. Shared-file discovery and explicit completion markers keep worker identity stable and prevent long-lived services from holding a finished allocation.

What Change

  • Launch one frontend and role-indexed Dynamo worker per inference node.
  • Validate shared-filesystem and supported aggregate/P/D topology shapes.
  • Coordinate clean cross-node completion and document runnable examples.

Test Plan

  • Single-node gate: aggregate and P/D passed three steps with zero effective errors.
  • Run multi-node aggregate and P/D E2Es before marking ready.

Gate evidence: 20260717T181440Z/portable-evidence/summary.json passed both topologies with zero fatal signatures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant