[CP 222] fix(gpu): wait for GPU device release before partitioning (GPUOP-909) - #38
Merged
Conversation
… (#222) * fix(gpu): wait for GPU device release before partitioning (GPUOP-909) CPX/compute partition failed with "Device busy" (AMDSMI_STATUS_BUSY) when a user GPU workload pod was still holding the device during eviction. DCM issued the AMD SMI partition command before the kernel finished releasing the GPU. Add a best-effort gate (waitForGPUIdle) that polls amdsmi_get_gpu_process_list and waits up to GPUIdleWaitTimeout for the process count to drain to zero before partitioning. On API error, under-report, or timeout it proceeds, so the existing 30-min retry loop remains the backstop and behavior never regresses. Also fix state-label semantics: retryable busy errors now leave the node label at "in-progress" instead of prematurely writing "failure"; the authoritative "failure" write moves to the 30-min retry-loop expiry. Non-retryable errors (invalid/unsupported profile) still fail immediately. Verified on MI350P hardware: process-list API reports the workload, busy path holds in-progress and recovers to success on release, expiry writes failure, and no-workload fast path partitions immediately. * docs(gpuop909): add design doc for CPX device-busy fix (cherry picked from commit d1648bb4263bc71b7b43ba35c74596c0d42e5673)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cp of pensando/device-config-manager#222
Source PR Description (pensando/device-config-manager#222):
Summary
Fixes GPUOP-909: CPX/compute partition failed with
Device busy(AMDSMI_STATUS_BUSY) when a user GPU workload pod was still holding the device during eviction. DCM issued the AMD SMI partition command before the kernel finished releasing the GPU, and prematurely marked the nodestate=failure.Changes
waitForGPUIdle): before issuing a partition command, pollamdsmi_get_gpu_process_listand wait up toGPUIdleWaitTimeout(60s) for the GPU process count to drain to zero. On API error / under-report / timeout it proceeds anyway, so the existing 30-min retry loop remains the backstop and behavior never regresses.in-progressinstead of writingfailureon the first busy error; the authoritativefailurewrite moves to the 30-min retry-loop expiry. Non-retryable errors (invalid/unsupported profile) still fail immediately.PartitionRetryingevent, not a new label value — thegpu-config-profile-statelabel contract keeps its three values (in-progress / success / failure).Design doc:
docs-internal/plans/2026-07-08-gpuop909-cpx-device-busy-design.md.Chose the SMI process-list API over the ticket's
/proc/*/fdscan on/dev/kfd, which would requirehostPID: trueon the DaemonSet (a privilege escalation) and cannot see other pods' PIDs from DCM's namespace. The SMI API queries the driver directly and works from inside the pod as-is.Dependency
Important
Depends on #221 (build-infra split). Please merge #221 first, then merge this. The two changesets are independent (verified this builds on the current
mainbase), but this branch is cut frommainand should land after the infra PR to avoid churn.Test plan
Verified on MI350P hardware (amdgpu driver 6.19.4, GPU Operator v1.5.0):
/dev/kfd(core premise of the fix)in-progressacross retries (previously flipped tofailure)state=success; HW confirmedcurrent_compute_partition=CPXfailurelabel +PartitionFailureeventfailure, no retry churnCherrypick triggered by: ACP-Automation