Fix e2e-aws flake cascade and metrics-pod connection race - #1005
Fix e2e-aws flake cascade and metrics-pod connection race#1005parametalol wants to merge 2 commits into
Conversation
InitializeClusterResources hard-failed if a cluster-scoped RBAC object
(e.g. the file-integrity-operator ClusterRole) already existed, which
happened whenever a prior test failed and E2E_SKIP_CLEANUP_ON_ERROR left
it behind. That turned one failing test into every remaining test in the
serial run failing at setup. Pass skipIfExists=true, which createFromYAML
already supports, so setup tolerates a leaked object instead.
Separately, runOCandGetOutput (used to curl the metrics endpoint from an
ephemeral pod) failed the test on the first error, but curl exit 7
("connection refused") happens routinely while the metrics service is
still coming up -- the codebase already tolerates this elsewhere via a
wait.Poll retry loop. Add the same tolerance directly in
runOCandGetOutput: retry a few times before failing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: parametalol The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
needs-ok-to-test
|
Hi @parametalol. Thanks for your PR. I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
openshift-ci
Bot
added
ok-to-test
The first fix (skipIfExists=true) avoided the hard "already exists" failure, but silently left a leaked ClusterRole/ClusterRoleBinding bound to a deleted namespace's ServiceAccount in place. On the live e2e-aws run, this meant the *next* test's operator had no working cluster-scoped RBAC, so instead of failing fast it timed out after ~30 minutes waiting for a DaemonSet that could never be created. createOrReplace now deletes and recreates a conflicting object so it's rebound to the current test's namespace, and InitializeClusterResources uses that behavior. The one other caller (global CRD setup in framework.go, run once per test binary) keeps the original skip behavior, since deleting a CRD would cascade-delete every custom resource of that type. Also widen runOCandGetOutput's retry window from 3 attempts/5s to the same pollInterval/pollTimeout used by getMetricResultsSupressWarning for the identical "metrics endpoint not ready yet" condition: the smaller window was not enough on a live run (TestMetricsHTTPVersion still failed after exhausting it). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Hi Michaël, The PR looks good. /lgtm |
|
/ok-to-test |
|
@parametalol: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3 participants
Summary
PRs #1003 and #1004 both failed
e2e-awsdespite neither touching e2e tests, RBAC, or the metrics endpoint. Both failed for the same underlying reason:tests/framework/resource.go:InitializeClusterResourceshard-fails if a cluster-scoped RBAC object (thefile-integrity-operatorClusterRole/ClusterRoleBinding) already exists. SinceE2E_SKIP_CLEANUP_ON_ERRORdefaults totrue, one failing test leaves these behind, and every subsequent test in the serial run then fails immediately at setup instead of just the one that actually broke.createFromYAMLalready supportsskipIfExistsfor exactly this; the only caller just never used it. This PR flips it totrue.tests/e2e/helpers.go:runOCandGetOutput(used to curl the metrics endpoint from an ephemeral pod) fails the test on the very first error. In practice curl exit 7 ("connection refused") happens routinely while the metrics service is still coming up — it shows up 11 times in a fully passing run I checked, just non-fatally, because that other call site already retries via await.Pollloop. This PR adds the same tolerance (retry a few times, 5s apart) directly inrunOCandGetOutputso the ~10 tests that go through it aren't one-shot.Together these mean a single transient hiccup stays isolated to (at most) one test instead of taking down the rest of the suite.
Test plan
go build ./tests/...go vet ./tests/...e2e-awsProw job passes (can't run locally — requires a live OpenShift cluster)🤖 Generated with Claude Code