slurm-accounting: fix restart Job ordering, document controller-restart pitfalls - #96
Merged
Merged
Conversation
…itfalls The restart Job (restartPods.includeWorkers=true) previously deleted the controller and worker pods at the same time. Deleting workers while slurmctld is down strands them: the nodeset controller can't sync its Slurm node cache, errors into exponential backoff, and doesn't recreate the deleted slurmd pods. And once workers do come back, the freshly booted slurmctld has already parsed topology.conf without the (dynamic) nodes, so jobs fail with "Requested topology configuration is not available" until a reconfigure. The Job now waits for the replacement controller pod to be Ready before deleting workers, waits for the replacement workers, then runs scontrol reconfigure in the controller pod so the nodes rejoin the topology tree. README known issues added for the above plus the statesave fsGroup mount failure that any controller restart can hit on a cluster with accumulated state (prevention one-liner + helper-pod recovery). Chart 0.1.1.
martin-cala1
requested review from
chinmaybaikar,
datadoc24,
smazigh and
youngjeong46
as code owners
August 19, 2026 23:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
restartPods.includeWorkers=true): the Job previously deleted controller and worker pods simultaneously. Deleting workers while slurmctld is down strands them — the nodeset controller errors into backoff and doesn't recreate them — and the freshly booted slurmctld parses topology.conf without the dynamic nodes, so jobs fail with "Requested topology configuration is not available". The Job now: waits for the replacement controller pod to be Ready → deletes workers → waits for replacement workers → runsscontrol reconfigure. NewrestartPods.waitForControllerReadyTimeoutSeconds/waitForWorkersReadyTimeoutSecondsvalues; restart Role gainswatchon pods andcreateon pods/exec.applyFSGroup ... hash.0: permission denied) — fsGroup walk vs 0700 state dirs on a root-squashed NFS export; prevention one-liner and helper-pod recovery. Platform-level; any controller restart on a cluster with accumulated state is exposed.scontrol reconfigurefix.Testing
Verified end to end on a live Managed Slurm cluster (1 controller, 2 logins, 1 worker):
helm upgradewith the new Job completes with hooks sequencing exactly as intended (worker deleted only after controller Ready; reconfigure after workers Ready), the worker was recreated within seconds instead of stranding, andsrunsucceeds immediately afterhelm upgradereturns with no manual steps.helm lintand template rendering verified for bothincludeWorkersmodes.