pkg/steps: pin bundle builds to amd64 nodes#5329
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
🔗 Linked repositories identifiedCodeRabbit considers these linked repositories for cross-repo context during reviews:
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughWalkthroughBundle source and bundle-image builds now pin their build pods to AMD64 while preserving existing node selector entries. Tests validate the helper against single- and multi-architecture pipeline build construction. ChangesBundle build architecture
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 16 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (16 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull request overview
This PR addresses intermittent src-bundle / bundle image build failures on heterogeneous (amd64 + arm64) build clusters by ensuring single-arch bundle-related OpenShift Builds are scheduled onto amd64 nodes, avoiding “no image found in manifest list for architecture arm64” failures when inputs only contain amd64 images.
Changes:
- Pin
bundleSourceStep.runbuilds to amd64 viakubernetes.io/archnode selector. - Pin bundle image builds in
projectDirectoryImageBuildStep.run(theIsBundleImage()path) to amd64 via the same selector. - Update comments to document the manifest-list / scheduling failure mode that this resolves.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pkg/steps/project_image.go | Pins bundle image builds to amd64 nodes to avoid arm64 scheduling failures when consuming amd64-only manifest entries. |
| pkg/steps/bundle_source.go | Pins src-bundle (bundle-source) builds to amd64 nodes to prevent buildah failures caused by landing on arm64 nodes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kaovilai The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
6c97adc to
f97451d
Compare
f97451d to
416c24f
Compare
Bundle-source and bundle image builds are single-arch by design and are created without a node selector, so on heterogeneous build clusters the build pod can be scheduled onto an arm64 node. The build inputs (e.g. pipeline:src) are manifest lists that only contain an amd64 entry for single-arch repositories, so buildah then fails with: error creating buildah builder: choosing an image from manifest list docker://.../pipeline@sha256:...: no image found in manifest list for architecture "arm64", variant "v8", OS "linux" Pin these builds to amd64 nodes with a kubernetes.io/arch node selector, matching what constructMultiArchBuilds does for pipeline builds. Fixes openshift#5328 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tiger Kaovilai <passawit.kaovilai@gmail.com>
416c24f to
0d57bf8
Compare
|
Note Responses generated with Claude Retriggering infra flakes (test pods Pending >1h, 0/52 nodes available) and the unrelated /test checkconfig |
|
Scheduling tests matching the |
|
Note Responses generated with Claude CI status: The remaining failures — This is repo-wide, not from this PR: PR #5278's e2e runs today fail identically ( |
|
Note Responses generated with Claude Root cause of the |
|
Note Responses generated with Claude #5331 merged and the fixed ci-operator image promoted — retesting the jobs broken by the credential validation regression: /test breaking-changes |
|
@kaovilai: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Note Responses generated with Claude Post-#5331 retest results:
— the e2e-oo fixture references the long-EOL |
|
Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage. |
1 similar comment
|
Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage. |
| // builds to one architecture is only safe while every pipeline manifest list is guaranteed to | ||
| // contain that architecture. If any assertion below fails, that guarantee changed — update | ||
| // pinBuildToSingleArchNode together with the change that broke it. | ||
| func TestSingleArchBuildPinMatchesPipelineArchGuarantees(t *testing.T) { |
There was a problem hiding this comment.
It seems that the first 3 subtests don't exercises the new code at all. Is there any reason we want to do this?
Note
Responses generated with Claude
Summary
Bundle-source (
src-bundle) and bundle image builds are single-arch by design and were created without a node selector, so on heterogeneous build clusters the build pod can land on an arm64 node. The build inputs (e.g.pipeline:src) are manifest lists that contain only an amd64 entry for single-arch repositories, so buildah then fails with:This PR pins both bundle build sites (
bundleSourceStep.runinpkg/steps/bundle_source.goand theIsBundleImage()branch inpkg/steps/project_image.go) to amd64 nodes via a sharedpinBuildToSingleArchNodehelper that merges akubernetes.io/archentry into the build's node selector (merge rather than replace, so a selector set earlier on the build would survive), matching whatconstructMultiArchBuildsalready does for pipeline builds. A tripwire unit test (TestSingleArchBuildPinMatchesPipelineArchGuarantees) locks the pin to the pipeline-side guarantees it depends on.Fixes #5328
Why the previous "no node selector" behavior is wrong
The removed comment said the build "will be scheduled on any available node no matter the architecture" — that was written when it was harmless, but since pipeline outputs became manifest lists (#3587, reworked in #5004), the content buildah pulls is resolved by the architecture of the node the pod happens to land on. For a single-arch repo the manifest list has no arm64 entry (amd64 is the only architecture built unless more are configured), so the build's correctness depends on scheduler placement: amd64 node → works, arm64 node → hard failure. All 5 in-run retry attempts can land on arm64 too (
FetchImageContentFailed×5). That is a flake by construction, not a valid "any node" invariant.Evidence (fail → retry-pass on the same PR/job)
From openshift/oadp-operator#2313:
pull-ci-...oadp-dev-4.22-e2e-test-cli-aws: FAILURE 2026-07-23 00:48 UTC (exact error above in build-log) → retry SUCCESS 02:34 UTCpull-ci-...oadp-dev-4.23-ci-index: FAILURE 2026-07-22 22:40 UTC (same error, thensrc-bundle failed after 5 attempts ... FetchImageContentFailed) → retry SUCCESS 22:58 UTCci-indexpair the next day: FAILURE 13:19 UTC → retry SUCCESS 13:33 UTCFarm-wide: search.ci,
no image found in manifest list for architecture, build-logs, 24h → 93 job-runs in 24 hours, e.g. kubevirt/hyperconverged-cluster-operator#4421, assisted-service periodic, csi-operator periodic, plus openshift-kni/, medik8s/, lvm-operator.Why pinning to amd64 cannot break anyone (including
additional_architecturesrepos)additional_architecturesis strictly additive —pkg/api/types.go: "AdditionalArchitectures is a list of additional architectures to build for. AMD64 architecture is included by default." There is no knob to remove amd64, andprojectDirectoryImageBuildStep.ResolveMultiArch()insertsNodeArchitectureAMD64unconditionally before adding configured capabilities. So a repo withadditional_architectures: [arm64]produces apipeline:srcmanifest list containing both amd64 and arm64 — the amd64-pinned bundle build resolves its amd64 entry and works exactly as for single-arch repos. The failing direction (node arch missing from the list) cannot occur in reverse, because no configuration produces a manifest list without an amd64 entry.srcalready requires amd64 capacity:constructMultiArchBuildspinssrc-amd64(and every other pipeline build) tokubernetes.io/arch: amd64nodes. If a cluster could not schedule the pinned bundle build, it could not have builtsrcin the first place.index_generator.go: "At the moment, we support only amd64"), and bundle image content is architecture-independent metadata/YAML — the node arch never affects the produced bundle, only whether its inputs can be pulled.TestSingleArchBuildPinMatchesPipelineArchGuarantees(inbundle_source_test.go) asserts the invariants above against the real functions: the defaultconstructMultiArchBuildsoutput targets the pinned arch,ResolveMultiArch()contains it even when only other architectures are configured, multi-arch pipelines still produce a build for it, and pinning merges into (not replaces) an existing node selector. If a future change breaks any link in that chain, this test fails and points back atpinBuildToSingleArchNode.Test plan
go build ./pkg/steps/...gofmt -lcleango test ./pkg/steps/...passes, including newTestSingleArchBuildPinMatchesPipelineArchGuarantees/cc @droslean
Summary
src-bundleand the corresponding bundle images so they are scheduled only on amd64 nodes (kubernetes.io/archstable/amd64), even when the pipeline inputs are provided via manifest lists.src-bundlefailures caused by buildah pods landing on arm64 while consumingpipeline:srcmanifest lists that may only include amd64 images.pinBuildToSingleArchNodehelper (that merges with any existingNodeSelector), covering both:bundleSourceStep.runinpkg/steps/bundle_source.goIsBundleImage()branch inpkg/steps/project_image.goTestSingleArchBuildPinMatchesPipelineArchGuarantees) to enforce that the pinned architecture stays aligned with the pipeline multi-arch manifest-list guarantees and that the selector pinning logic merges rather than replaces.