[PLT-4265] Harden upgrade-provisioner.py: ENIConfig auto-fix, --restore, activate-capa-machinepool-features.py - #942
Open
iamjanr wants to merge 8 commits into
Open
[PLT-4265] Harden upgrade-provisioner.py: ENIConfig auto-fix, --restore, activate-capa-machinepool-features.py#942iamjanr wants to merge 8 commits into
iamjanr wants to merge 8 commits into
Conversation
…ot just warn apply_chart_crds() caught any exception from the whole pull+extract+apply sequence and only ever printed a WARN before continuing. A missing chart version in the configured helm repository (e.g. an outdated universe still pointing at an old K8s line) silently skipped the CRD update, leaving the upgraded cluster-operator running against a stale CRD schema. Split the failure modes: a failed helm pull (chart/version not found, unreachable repo) now raises and aborts the upgrade, since the CRDs are unavailable outright. Applying an already-downloaded CRD file remains best-effort, since a given CRD may have no real schema change in this version.
Patch while the controller is stopped, in the same critical section as the CAPI providers upgrade, with the controller started once at the very end — the 2026-08-13 verified-working design. The real fix for the underlying race lives in cluster-operator (branch PLT-4265-01). Also: never skip an already-at-target chart while --ecr-pull-through is enabled, since image path rewriting only happens on real chart upgrades.
…pa-machinepool-features.py Validates cluster-operator/CAPA minimum versions and CAPA feature gates required for MachinePool support. Read-only: it does not upgrade anything or patch CAPA directly, both are handled by upgrade-provisioner.py.
Restores the pre-upgrade state (CRDs, cluster-operator ConfigMap, Helm repository, ecr_pull_through_cache_enabled) from the automatic backup the script already creates before applying any change.
pods_cidr clusters created before the PLT-4509 fix carry an ENIConfig with the VPC's literal "default" security group instead of the real cluster one. MachineDeployment nodes self-heal via CAPA, MachinePool does not, so a pool added after the upgrade inherits the stale SG.
No "Phase 2" exists anywhere in the script, so the label only implied a multi-step flow that isn't there.
iamjanr
added
ai-assisted
wip
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PLT-4265 Harden upgrade-provisioner.py: ENIConfig auto-fix, --restore, activate-capa-machinepool-features.py
Description
Continuation of the PLT-4265 upgrade-path hardening work:
k8s_versioninside the same critical section as the CAPI/CAPA upgrade, instead of as a separate later step.migrate-workers-to-machinepool.pywithactivate-capa-machinepool-features.py— a simpler, read-only script that only validates prerequisites (cluster-operator/CAPA versions, CAPA feature gates) instead of attempting a guided migration that never worked reliably (check_mp_ready()'s node-readiness heuristic was structurally broken for MachinePool).--restoreflag toecr_pull_through.pyto roll back to the pre-upgrade backup on failure.ENIConfigsecurity group inupgrade-provisioner.py's pre-flight checks: clusters created before the PLT-4509 fix (< 0.9.0-m.5) withpods_cidrconfigured carry anENIConfigpointing at the VPC's literal "default" security group instead of the real cluster one.MachineDeploymentnodes self-heal via CAPA'sAWSMachinereconciler, butAWSManagedMachinePoolhas no equivalent, so aMachinePooladded after the upgrade would inherit the stale SG and lose connectivity. The check compares theENIConfig's security group againstaws eks describe-cluster's real one and patches it if they differ — read-only until then, no live ENI is ever touched (no cluster upgrading from before 0.9.0 can have aMachinePoolyet).AWS_LOAD_BALANCER_CONTROLLER_CHARTconstant.ecr_pull_through.py's output — no "Phase 2" exists anywhere in the script.Type of Change
Related Pull Requests
upgrade.adocto reflect this same flow, including themp_role_namerequirement this branch'sMachinePooltesting depends on.How Has This Been Tested?
activate-capa-machinepool-features.py: run for real (not dry-run, it's read-only) against two test EKS clusters, bothRESULT: OK.ENIConfigobjects.cloud-provisioner-upgradecontainer against a test cluster (created on 0.7.5, upgraded to 0.9.0/k8s 1.35 end-to-end in ~33 minutes) — the check fired in the pre-flight step and fixed the ENIConfig without blocking the rest of the upgrade.MachinePoolto two upgraded test clusters after their ENIConfig fix — both came up with correct security groups on all ENIs from birth, no retroactive fix needed. Moved all critical controllers (capi/capa/cluster-operator/coredns/etc.) onto theMachinePoolnodes on both clusters and confirmed real connectivity (ncto the ClusterIP, and the controllers themselves running healthy).Checklist
ai-assistedat minimum.Additional Notes
Unrelated, pre-existing finding surfaced during this testing (not fixed here, tracked separately):
keoscluster-controller-managerintermittently crashes with no logs on BottlerocketMachinePoolnodes — reproduced once on one test cluster, did not reproduce on another under the same conditions. Also unrelated: a stale0.0.0.0/0blackhole route (NAT gateway deleted well before this session, unrelated to this branch) was found and fixed on a shared test VPC.