src-bundle build flake: no node selector lets buildah pod land on arm64 node, fails pulling amd64-only pipeline:src manifest list

[kaovilai]

Note

Responses generated with Claude

Symptom

Operator bundle jobs across many repos intermittently fail the src-bundle step (seen e.g. on pull-ci-openshift-oadp-operator-oadp-dev-4.22-e2e-test-cli-aws / -e2e-test-aws for openshift/oadp-operator#2313, across several retrigger cycles):

Trying to pull image-registry.openshift-image-registry.svc:5000/<ci-op-namespace>/pipeline@sha256:<digest>...
error: error creating buildah builder: choosing an image from manifest list docker://image-registry.openshift-image-registry.svc:5000/<ci-op-namespace>/pipeline@sha256:<digest>: no image found in manifest list for architecture "arm64", variant "v8", OS "linux"
ERRO Some steps failed:
  * could not run steps: step src-bundle failed: could not get build src-bundle: builds.build.openshift.io "src-bundle" not found
INFO Reporting job state 'failed' with reason 'executing_graph:step_failed:building_bundle_source'

A /test <job> retry usually passes — it never reproduces twice in a row on the same PR, consistent with a scheduling race rather than a permanent breakage.

Mechanism

Three things combine:

1. pipeline:src is pushed as a manifest list; for repos without additional_architectures it contains only an amd64 entry.

2. Regular pipeline builds are arch-pinned: constructMultiArchBuilds (pkg/steps/source.go) sets a kubernetes.io/arch nodeSelector per build.

3. Bundle-source builds are not. bundleSourceStep.run (pkg/steps/bundle_source.go) deliberately creates the build with no node selector:

   // Bundle images are not multi-arch by design. Here we build it without creating a manifest-listed image.
   // Note that we are not configuring a node selector here, so the build will be scheduled on any available
   // node no matter the architecture.
   return handleBuild(ctx, s.client, s.podClient, *build)

On heterogeneous build-farm clusters (amd64 + arm64 nodes) the buildah pod sometimes lands on an arm64 node, which then cannot resolve the amd64-only pipeline:src manifest list → the error above. The retry lands on an amd64 node and succeeds.

Frequency / impact

search.ci (no image found in manifest list for architecture, build-logs, maxAge=24h) matches ~94 job-runs in 24 hours across at least: openshift/oadp-operator, kubevirt/hyperconverged-cluster-operator, openshift-kni/cluster-group-upgrades-operator, openshift-kni/numaresources-operator, medik8s/*, openshift/lvm-operator, openshift/custom-metrics-autoscaler-operator, openshift/assisted-service, openshift/csi-operator — i.e. essentially bundle/index-building jobs (ci-index, ci-bundle, OLM e2e) farm-wide.

Related prior work fixed manifest-list push races (#5004, #5072; #5016 explored the ImageStreamTag sync angle and was closed), but the bundle-build scheduling gap remains — the code comment explicitly documents it.

Proposed fix

Pin bundle-source (and index) builds to the architecture of the src image they consume (amd64 unless configured otherwise) via a kubernetes.io/arch nodeSelector, matching what constructMultiArchBuilds already does for pipeline builds.

/cc @droslean