Driver init fails in air-gapped clusters due to hard-coded mount of Red Hat subscription repo config

[changhyuni]

Summary

When deploying GPU Operator in an air-gapped (offline) cluster the nvidia-driver-daemonset init container fails to start.
Root cause: the driver image ships with a public YUM repo enabled by default, which triggers yum errors in offline environments.
Additionally, the pod spec tries to mount /etc/yum.repos.d/redhat.repo (HostPath, File) but the file is absent, so the kubelet rejects the volume with hostPath type check failed.

Node OS: RHEL 8.10

We rebuilt the driver image and **removed /etc/yum.repos.d/redhat.repo

gpu-operator/internal/state/driver_volumes.go

Line 203 in cc4abab

volMountSubscriptionName := fmt.Sprintf("subscription-config-%d", num)

Related issue

• helm instal gpu-operator was in Init stage for a long time #980

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[cdesiniotis]

@changhyuni thanks for filing this issue. Would you be able to provide more information, like the k8s flavor / version, the container runtime used, etc. Also, did you follow our procedure for installing GPU Operator in air-gapped environments? Which part of that procedure did you need to tweak in order to get things to work?

Root cause: the driver image ships with a public YUM repo enabled by default, which triggers yum errors in offline environments.

Have you followed https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-air-gapped.html#local-package-repository for provisioning a local package repository?

[konstantin-kornienko]

Can confirm this.
Nvidia Operator 23.9.1 fails on Google Cloud VMs with RHEL OS 8.10.
k8s: k3s 1.32.6

Operator's pod nvidia-driver-daemonset tries to mount /etc/pki/entitlement and /etc/yum.repos.d/rhel.repo, but there are no files like this. It makes pod to stuck forever.

MountVolume.SetUp failed for volume "subscription-config-1" : hostPath type check failed: /etc/yum.repos.d/redhat.repo is not a file
  Warning  FailedMount  3m55s (x23 over 34m)  kubelet
MountVolume.SetUp failed for volume "subscription-config-0" : hostPath type check failed: /etc/pki/entitlement is not a directory

[changhyuni]

@cdesiniotis While building the nvidia-driver image, we removed the license-related folder from the image. After that change, the deployment succeeded in our offline cluster.