Add the vendor VFIO vGPU device backend - #364
Conversation
2bb8e86 to
7fc3b49
Compare
7fc3b49 to
f661e63
Compare
f661e63 to
d1207d0
Compare
d1207d0 to
5b47670
Compare
5b47670 to
ffbf8a0
Compare
ffbf8a0 to
78801e7
Compare
78801e7 to
b4651f3
Compare
b4651f3 to
d2bf3fa
Compare
d2bf3fa to
495ab62
Compare
|
added 7f5233f: report |
7f5233f to
0baacb3
Compare
-->
✱ stlc build✅ go code · compare
✅ python code · compare
✅ typescript code · compare
Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note
Build metadata
This comment is auto-generated by stlc and is kept up to date as you push. |
c3b37b1 to
dae5383
Compare
1f1f24f to
c2bcb69
Compare
c2bcb69 to
1f1f24f
Compare
Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's nvidia/current_vgpu_type and passed to QEMU as a plain VFIO PCI device. Add a vendor VFIO backend behind the existing framework dispatch: profile discovery from the capacity-dependent creatable catalogs, least-loaded VF placement, create/verify/rollback, and release. Because the same VF path is reused across assignments (unlike mdev UUIDs), release is guarded: an in-process owner map covers the window before QEMU opens the device, and an open-VFIO-handle scan refuses to clear a VF a running VM still holds. Reconciliation clears orphaned assignments on startup, skipping VFs protected by the caller and failing closed when the protected set is unavailable. Branch the vGPU integration test by discovered framework and extend it to cover release on stop and reacquisition on start.
Sort GPUs with unaccountable load last instead of rejecting placement, and stop reporting passthrough capacity when vGPU discovery fails.
…nership through releases
The instance lifecycle already routes create/start/stop/delete through CreateVGPU/DestroyVGPU, so dispatching vendor VFIO creates here would activate the backend before assignment durability and release guards exist. Reject vendor VFIO creates for now; destroy stays wired so existing assignments remain releasable. The integration test skips on vendor VFIO hosts at this layer and no longer asserts the transitional stop-retention behavior.
Counting every free VF advertising a type overreports concurrent capacity: sibling VFs share their parent GPU's framebuffer, so one 48Q assignment revokes the type from every other VF on that GPU. Bound each GPU's contribution by both its free VFs and how many times the profile framebuffer fits into the GPU's remaining framebuffer, using the largest still-creatable profile as a lower bound on what remains.
This reverts commit d6adc4c.
A single unreadable current_vgpu_type failed discoverVFs wholesale, and GetGPUStatus turns a discovery error into a host with no GPU, so one flaky sysfs read blanked out the host's entire GPU capacity for admission and monitoring. Skip unreadable VFs with a warning and keep the readable inventory: a skipped VF is never selected for placement and never reconciled, both safe directions. When no VF is readable, discovery still fails so a wholesale sysfs outage cannot demote a vGPU host to passthrough while assignments exist. Also document that vendor VFIO vGPUs are known broken on Cloud Hypervisor upstream and QEMU is the required hypervisor for GPU instances.
listProfiles failed wholesale when one VF's creatable_vgpu_types read failed, blanking every advertised profile while discoverVFs directly above it already skips unreadable VFs for exactly that reason. Skip and warn instead; underreporting is the safe direction for status and admission. Also document why openVFIOPaths stays strict where mdev's scan is lax (it authorizes clearing a reused VF path), and the 0Q/0B parsing caveat in framebufferFromProfileName.
listProfiles skips an unreadable VF but create still failed placement wholesale when profileMetadata or selectLeastLoadedVF hit the same VF, so /resources could advertise capacity a create then failed to use. Skip the VF in both loops; it simply stops being a placement candidate.
1f1f24f to
d28e3d0
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d28e3d0. Configure here.
| log.WarnContext(ctx, "preserving vendor VFIO vGPU held open without a live instance claim", "vf", vf.PCIAddress) | ||
| continue | ||
| } | ||
| if err := s.destroyWithOpenPaths(ctx, vf.PCIAddress, "", openPaths); err != nil { |
There was a problem hiding this comment.
Stale open-handle scan race
Medium Severity
reconcile takes one openVFIOPaths snapshot and passes it into destroyWithOpenPaths, which skips a fresh scan under vendorVFIOMu when that map is non-nil. A VMM can open the VF after the snapshot and before the current_vgpu_type clear, so reconcile can wipe an assignment a running guest already holds. The normal destroy path avoids this by passing nil and rescanning under the lock.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d28e3d0. Configure here.


Summary
Layer 2 of the vendor VFIO vGPU stack (
generalize-vgpu-device←hypervisor-liveness← this ←vendor-vfio-vgpu). Self-contained inlib/devices+lib/resources; nothing in the instance lifecycle calls it yet (that's the top layer).Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's
nvidia/current_vgpu_typeand passed to QEMU as a plain VFIO PCI device. This adds that backend behind the framework dispatch introduced in #322:creatable_vgpu_typescatalogs, least-loaded-GPU VF selection, create/verify/rollback. Profile availability counts free VFs currently advertising each type as a best-effort snapshot; creating one assignment may change sibling catalogs.Testing
go build ./...,go vetcleango test -race ./lib/devices/ ./lib/resources/passNote
High Risk
Touches GPU assignment, sysfs writes, orphan reconciliation, and resource admission. A discovery or release bug can steal VFs, blank capacity, or demote a vGPU host to passthrough.
Overview
Adds NVIDIA vendor VFIO vGPU support for hosts that assign profiles via
current_vgpu_typeinstead of mdev. Discovery prefers mdev, then vendor VFIO; resource status uses that inventory and refuses to fall through to passthrough on a failed vGPU probe.The new backend places on the least-loaded GPU, verifies the write, and rolls back on mismatch. Release is guarded by an in-process owner map plus a strict open-VFIO-handle scan so a reused VF path cannot be cleared while another VM holds it. Reconciliation clears orphans except protected or in-use VFs.
Create is still rejected for vendor VFIO until instance lifecycle integration. Profile
availableis documented as a best-effort VF snapshot. Unreadable VFs are skipped rather than wiping advertised capacity. Integration tests cover stop/start reassignment on mdev; vendor VFIO E2E is skipped until create is wired.Reviewed by Cursor Bugbot for commit d28e3d0. Bugbot is set up for automated code reviews on this repo. Configure here.