Fix controller metrics server race with serving-cert volume sync - #1010
Conversation
Since the operator no longer exits and restarts when the metrics serving-cert secret is missing (PR openshift#845), the custom :8585 metrics server can start before the kubelet has synced the secret into the container's volume mount. ListenAndServeTLS then fails once with "open /var/run/secrets/serving-cert/tls.crt: no such file or directory", the error is swallowed, and the metrics endpoint never comes up — seen as curl exit 7 (connection refused) in TestMetricsHTTPVersion and cascading e2e failures. Wait for the cert key pair to appear in the mount with exponential backoff before serving, propagate errors so the manager restarts the pod as a last resort instead of silently running without metrics, and shut the server down cleanly when the manager context is canceled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
openshift-ci
Bot
added
the
approved
After the operator pod is killed, the node_failed gauges are restored
asynchronously by the configmap controller as each node's status is
re-reconciled (OCPBUGS-42807 path). The test polled only until any
file_integrity_operator_node metric appeared, slept a fixed 5 seconds,
and then asserted all nodes in one snapshot - failing when a node's
gauge had not been restored yet ("expected 1, got 0"). Poll for the
expected values instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@Vincent056: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Vincent056, yuumasato The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/verified bypass |
openshift-ci-robot
added
the
verified
|
@vickeybrown: The DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/verified remove |
openshift-ci-robot
removed
the
verified
|
@vickeybrown: The DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/verified by e2e-aws test |
openshift-ci-robot
added
the
verified
|
@vickeybrown: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Manually merging so we can move forward with FIO v1.4.0 release. |
|
/assign docs-approved |
|
@GroceryBoyJr: GitHub didn't allow me to assign the following users: docs-approved. Note that only openshift members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
5 participants
What
Makes the custom
:8585controller metrics server wait (with exponential backoff, up to 5 minutes) for the serving-cert key pair to be synced into the container's volume mount before callingListenAndServeTLS, propagates server failures to the manager instead of swallowing them, and shuts the server down cleanly when the manager context is canceled.Why
TestMetricsHTTPVersionfails deterministically on e2e-aws (e.g. on #1003, and its skipped cleanup cascades intoclusterrole already existsfailures for every subsequent test). The failing curl gets exit 7 (connection refused) againsthttps://metrics.<ns>.svc:8585/metrics-fio, and the operator log shows why:This is a regression from #845: before it, the operator exited and restarted when the serving-cert secret was missing, and by the time the container came back the kubelet had synced the cert files into the mount. Now the operator waits in-process — but only for the secret to exist in the API.
Metrics.Startthen does a one-shotListenAndServeTLSbefore the kubelet has synced the files into the volume, fails, logs, and returnsnil, so the metrics server never comes up and nothing restarts.The fix mirrors the backoff pattern #845 introduced for the API-side checks, applied to the mounted files. Returning the error after the backoff is exhausted restores the old exit-and-restart behavior as a last resort instead of silently running without metrics.
🤖 Generated with Claude Code