Add cmk-node-remediation: automated node cordon, drain & vm-reset for B200 NVLink bug - #88
Merged
Conversation
chekolyn
requested review from
chinmaybaikar,
datadoc24,
smazigh and
youngjeong46
as code owners
August 13, 2026 05:40
chekolyn
force-pushed
the
cmk-node-remediation
branch
4 times, most recently
from
August 17, 2026 22:33
9e61e14 to
21165cf
Compare
youngjeong46
left a comment
youngjeong46
left a comment
Contributor
There was a problem hiding this comment.
@chekolyn - we just added a simple pre-commit check. You just need to add a link to your repo readme to the top-level readme per Contributing guide.
chekolyn
force-pushed
the
cmk-node-remediation
branch
6 times, most recently
from
August 21, 2026 04:13
329d532 to
185ce39
Compare
… NVLink bug Kubernetes CronJob that automatically cordons, drains, and remediates nodes approaching critical uptime thresholds. Built for Crusoe Managed Kubernetes (CMK) clusters with nodes affected by the NVIDIA Blackwell NVLink bug. Features: - Two-phase threshold design (cordon → remediate) with configurable cooldown - Guardrails: global + per-pool limits (absolute count or percentage) - Self-node exclusion, max retries, dry-run mode - Actions: vm-reset, vm-stop, vm-start, vm-delete, noop, multi-step sequences - PDB-aware drain with optional force eviction fallback - Helm chart with built-in crusoe-secrets support for CMK clusters - E2E test harness with 10 phases (build, deploy, verify, vm-reset, empirical) - RemediationReport CRD: per-nodepool status via kubectl get remediationreport - Progressive CRD updates during each run (running → per-pool → final) - Serial processing with guardrail-based concurrency limits - Journald logging support (dual stdout + journald via Crusoe Watch Agent)
chekolyn
force-pushed
the
cmk-node-remediation
branch
from
August 21, 2026 17:46
185ce39 to
3b9d296
Compare
Contributor
Author
Thank you @youngjeong46 . I've updated the main README.md with the required entry. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kubernetes CronJob that automatically cordons, drains, and remediates nodes approaching critical uptime thresholds. Built for Crusoe Managed Kubernetes (CMK) clusters with B200 GPU nodes affected by the NVIDIA 60-day NVLink firmware bug.
Background
NVIDIA Blackwell B200/B300 systems experience NVLink hangs after ~60 days of continuous uptime due to a firmware issue. NVIDIA has released patched VBIOS and system tray firmware (DGX firmware release 26.03.1+) that fixes the root cause. For systems that have not yet been patched, scheduled preventative reboots before the 60-day threshold avoid sudden production hangs.
See the NVIDIA DGX B300 firmware update guide for details.
Features
vm-reset,vm-stop,vm-start,vm-delete,noop, multi-step sequencescrusoe-secretssupport for CMK clustersTesting
go test ./internal/config/...)cordonThreshold: 10m,remediationThreshold: 20m,dryRun: false,action: vm-reset— cordon, drain, vm-reset, boot verification, cooldown, and guardrails all functioning correctly