Failed to call webhook error fpga.mutator.webhooks.intel.com and sgx.mutator.webhooks.intel.com repeated in kube-apiserver logs

[jtmusson]

I am seeing repeated errors in the kube-apiserver logs (will paste below), being repeated every 5 minutes. This looks like similar behaviour to issue #46.

I am using charts intel-device-plugins-gpu and intel-device-plugins-operator, both are version 0.36.0 and installed as HelmRelease via Flux.

I would note that I have the gpu resource being made available to pods; however I would want to ensure there is not a configuration error that may cause issues down the line. Thank you in advance for any assistance.

W0717 08:25:00.146602       1 dispatcher.go:210] Failed calling webhook, failing open fpga.mutator.webhooks.intel.com: failed calling webhook "fpga.mutator.webhooks.intel.com": failed to call webhook: the server could not find the requested resource
E0717 08:25:00.146626       1 dispatcher.go:214] "Unhandled Error" err="failed calling webhook \"fpga.mutator.webhooks.intel.com\": failed to call webhook: the server could not find the requested resource" logger="UnhandledError"
W0717 08:25:00.146681       1 dispatcher.go:210] Failed calling webhook, failing open fpga.mutator.webhooks.intel.com: failed calling webhook "fpga.mutator.webhooks.intel.com": failed to call webhook: the server could not find the requested resource
E0717 08:25:00.146700       1 dispatcher.go:214] "Unhandled Error" err="failed calling webhook \"fpga.mutator.webhooks.intel.com\": failed to call webhook: the server could not find the requested resource" logger="UnhandledError"
W0717 08:25:00.146747       1 dispatcher.go:210] Failed calling webhook, failing open fpga.mutator.webhooks.intel.com: failed calling webhook "fpga.mutator.webhooks.intel.com": failed to call webhook: the server could not find the requested resource
E0717 08:25:00.146764       1 dispatcher.go:214] "Unhandled Error" err="failed calling webhook \"fpga.mutator.webhooks.intel.com\": failed to call webhook: the server could not find the requested resource" logger="UnhandledError"
W0717 08:25:00.149478       1 dispatcher.go:210] Failed calling webhook, failing open sgx.mutator.webhooks.intel.com: failed calling webhook "sgx.mutator.webhooks.intel.com": failed to call webhook: the server could not find the requested resource
E0717 08:25:00.149496       1 dispatcher.go:214] "Unhandled Error" err="failed calling webhook \"sgx.mutator.webhooks.intel.com\": failed to call webhook: the server could not find the requested resource" logger="UnhandledError"
W0717 08:25:00.150090       1 dispatcher.go:210] Failed calling webhook, failing open sgx.mutator.webhooks.intel.com: failed calling webhook "sgx.mutator.webhooks.intel.com": failed to call webhook: the server could not find the requested resource
E0717 08:25:00.150104       1 dispatcher.go:214] "Unhandled Error" err="failed calling webhook \"sgx.mutator.webhooks.intel.com\": failed to call webhook: the server could not find the requested resource" logger="UnhandledError"
W0717 08:25:00.150166       1 dispatcher.go:210] Failed calling webhook, failing open sgx.mutator.webhooks.intel.com: failed calling webhook "sgx.mutator.webhooks.intel.com": failed to call webhook: the server could not find the requested resource
E0717 08:25:00.150179       1 dispatcher.go:214] "Unhandled Error" err="failed calling webhook \"sgx.mutator.webhooks.intel.com\": failed to call webhook: the server could not find the requested resource" logger="UnhandledError"

[jtmusson]

If helpful, my helm-release specs are available here:

• https://github.com/jtmusson/hydra/blob/main/clusters/main/kubernetes/kube-system/intel-device-plugins-gpu/app/helm-release.yaml
• https://github.com/jtmusson/hydra/blob/main/clusters/main/kubernetes/kube-system/intel-device-plugins-operator/app/helm-release.yaml

[mythi]

ack. I believe the error is that MutatingWebhookConfigurations are applied but the webhooks are not running due to --devices=gpu in your case.

btw, have you seen:

helm-charts/charts/device-plugin-operator/values.yaml

Lines 10 to 16 in 110e39e

	# supported devices by the operator
	devices:
	# dsa: true
	# gpu: true
	# iaa: true
	# qat: true
	# sgx: true

[jtmusson]

Ahh, I see. Thank you for the prompt on the device-plugin-operator values, I have updated my chart.

Is there anything to be done to deal with the not running MutatingWebhookConfigurations?

[mythi]

Is there anything to be done to deal with the not running MutatingWebhookConfigurations?

you can remove them manually until we change the chart not to create those when fpga/sgx device values are not enabled.

[jtmusson]

you can remove them manually until we change the chart not to create those when fpga/sgx device values are not enabled.

Ah I understand, thank you for your help with this.