Epic: Automatic CRD Storage Migration Before Service Upgrades

[ikhandamirov]

Understand the Epic

Service upgrades on all MCP variants should succeed without manual operator or customer intervention when a CRD storage version promotion is involved. When automatic migration is not possible, the platform must detect this in advance and surface a clear, actionable error rather than letting a Helm upgrade silently fail mid-flight.

Context / Background

The Kubernetes CRD Storage Version Problem

When a Kubernetes Custom Resource Definition (CRD) promotes a storage version (e.g. v1beta2 → v1), Kubernetes records the old version in CRD.status.storedVersions. Until a storage migration is explicitly performed — re-persisting all existing CRs in the new storage version and removing the old version from status.storedVersions — Kubernetes enforces that the old version name must remain in spec.versions. Any Helm chart upgrade or rollback that attempts to apply a CRD spec without that old version will be rejected by the API server with:

cannot patch "<crd-name>" with kind CustomResourceDefinition:
  status.storedVersions[N]: Invalid value: "<old-version>": missing from spec.versions;
  <old-version> was previously a storage version, and must remain in spec.versions until
  a storage migration ensures no data remains persisted in <old-version> and removes
  <old-version> from status.storedVersions

Production Incidents

This has already caused two separate incidents in production:

1. Flux rollback failure — After upgrading past the v1beta2→v1 boundary for the image automation CRDs (ImageUpdateAutomation, ImagePolicy, ImageRepository), rolling back to a prior flux2 chart version was blocked. Resolution: manually run flux migrate.

2. Flux upgrade failure (Task: Add available versions to ProviderConfig of all ServiceProviders #585) — Upgrading flux2 from 2.17.2 to 2.18.2 on a V1 MCP failed because ocirepositories.source.toolkit.fluxcd.io still had v1beta2 in status.storedVersions while the new chart dropped that version. Resolution: manually annotate all affected CRs and patch status.storedVersions.

Why This Is Systemic

The problem is not Flux-specific. Every service managed by a service provider that ships CRDs and promotes storage versions is exposed to this failure mode. The current list of affected candidates includes:

Service	CRD sets that version-promote over time
Flux (flux2)	source, image-automation, notification, kustomize, helm toolkits
Crossplane (core)	compositeresourcedefinitions, compositions, functionpipelines, …
Crossplane providers	provider-btp, provider-cloudfoundry, provider-kubernetes, provider-helm, …
External Secrets Operator	externalsecrets, secretstores, clustersecretstores, …
Kyverno	policies, clusterpolicies, admissionreports, …
Landscaper	installations, executions, deployitems, …
Velero	backups, restores, schedules, …
kro	resourcegraphdefinitions, …

Every time the openMCP release channel bumps a component version that crosses a storage-version boundary, operators must either proactively run a migration or face a blocked upgrade (or rollback). Currently that is entirely manual and relies on engineers knowing both the Kubernetes internals and the per-component migration tooling.

V1 vs V2 Scope

• V2 MCPs are managed by the service-provider-* controllers (one per service: service-provider-flux, service-provider-crossplane, etc.). These controllers reconcile Flux, Crossplane, ExternalSecretsOperator, etc. resources on the onboarding cluster and deploy the actual workloads onto MCP shoot clusters via FluxCD / Helm. The flux2 chart already has a built-in opt-in mechanism (crds.migration.enabled: true → Helm pre-upgrade hook that runs flux migrate), but it is not enabled in V2 service provider configurations today.
• V1 MCPs deploy services through the co-core cluster (FluxCD HelmRelease chain). There is no equivalent pre-upgrade hook infrastructure, so V1 is fully dependent on manual remediation today.

Proposed Approaches

Option A — Enable Per-Chart Migration Hooks Where Available (Quick Win, V2)

Several upstream charts already ship a built-in migration mechanism. The following have been verified:

Chart	Values key	Hook type	Default	Introduced
flux2 (fluxcd-community/helm-charts)	crds.migration.enabled	pre-upgrade Job running flux migrate	false	v2.8.x cycle (PR #280)
kyverno	crds.migration.enabled	post-upgrade Job running kyverno-cli migrate	true	v1.12.0 (PR #9481, Jan 2024)
velero	upgradeCRDs	pre-install/pre-upgrade/pre-rollback Job running velero install --crds-only --apply	true	v1.10.0 / Chart 12.0.1 (Dec 2022)

The following charts were also audited and do not ship a built-in migration hook:

Chart	Notes
crossplane	Runtime migrations exist in the application code but are not exposed as Helm hooks.
external-secrets	Only installCRDs: true and a conversion webhook flag (crds.conversion.enabled); no pre/post-upgrade migration job.
kro	No migration infrastructure; chart is still early-stage (v0.9.1).
Crossplane providers (provider-btp, etc.)	Individual provider charts do not ship migration hooks; they rely on Crossplane's core runtime.

For the three charts that do ship a migration mechanism, the service provider controllers should ensure the relevant value is enabled:

• service-provider-flux: set crds.migration.enabled: true in HelmRelease values (currently disabled).
• service-provider-kyverno: verify crds.migration.enabled is not overridden to false (default is already true upstream).
• service-provider-velero: verify upgradeCRDs is not overridden to false (default is already true upstream).

Action: Audit all service provider HelmRelease value overrides to confirm the upstream migration toggles are not suppressed, and explicitly enable crds.migration.enabled: true for Flux.

Option B — Generic Pre-Upgrade CRD Migration Job (Preferred, Both V1 and V2)

For services whose charts do not have a built-in migration hook (Crossplane core, all Crossplane providers, ESO, kro, OCM, and any future additions), build a generic CRD storage migration job that:

1. Detects migration need: Before the service provider reconciler applies a new HelmRelease version, compare the incoming chart's spec.versions for each CRD against the live status.storedVersions on the target MCP cluster. If any storage version listed in status.storedVersions is absent from the incoming spec.versions, a migration is required.

2. Executes migration: For each affected CRD:
   a. Enumerate all CRs in all namespaces (plus cluster-scoped CRs).
   b. Perform a no-op patch (e.g. kubectl annotate --overwrite) to force a re-persist at the current storage version.
   c. Patch CRD.status.storedVersions (subresource) to contain only the new storage version(s).

3. Gates the upgrade: Only proceed with the Helm upgrade (or FluxCD HelmRelease reconciliation) once migration succeeds. If migration fails, surface the error in the relevant service CR's .status.conditions and halt — do not proceed to a half-upgraded state.

This logic should live in the service provider controller layer, executed as a pre-reconciliation step whenever a version bump is detected.

Option C — Cluster-Level Admission / Pre-Flight Check

As a complementary safety net (not a replacement), add a pre-flight check to the upgrade pipeline that refuses to schedule a version bump if a CRD storage migration is pending. This converts a mid-upgrade failure into an early, actionable error, buying time to run the migration separately. This is especially relevant for V1 MCPs where the FluxCD HelmRelease chain is the driver.

Key Design Considerations

Where does the migration job run?

The migration job must run on the MCP shoot cluster (the customer's control plane), since that is where the CRDs and CRs live. The service provider controller (running on the platform cluster) must therefore use its existing mechanism for deploying workloads to shoot clusters (likely a short-lived Job via the connected kubeconfig or a dedicated HelmRelease with helm.sh/hook: pre-upgrade).

RBAC implications

Migration requires patch on the CRD subresource (status) and list/patch/annotate on all instances of the affected custom resources — which may be cluster-scoped or namespace-scoped. The migration job's ServiceAccount must be granted these permissions without over-granting. Consider scoping the ClusterRole tightly to the specific resource groups being migrated.

Large clusters

On MCP clusters with many managed resources (e.g. thousands of provider-btp managed resources), the "touch all CRs" step can be slow and generate significant API server load. The migration job should:

• Use LIST + paginated PATCH (not kubectl annotate --all, which loads everything into memory).
• Be rate-limited.
• Support a timeout with a configurable deadline.

Version skew / concurrent upgrades

If two services share a CRD (unlikely but possible in the Crossplane ecosystem), the migration logic must be idempotent and safe to run concurrently. Coordinate via ownership labels or a migration lease.

Upstream crds.migration.enabled adoption

For services where upstream charts gain this capability over time, prefer enabling the upstream hook over maintaining custom migration logic. Track which charts have adopted the feature and switch to the upstream mechanism when available. This should inform prioritization of Phase 3.

Notification to customers

Even when migration is fully automated, customers should be informed. Add a .status.conditions entry on the service CR (e.g. StorageMigrationInProgress) that is visible via the Onboarding API and shows clearly what is happening and why the upgrade is briefly delayed.

Out of Scope

• Migrating CRDs that are installed directly by customers (not managed by a service provider). Customers are responsible for CRDs they install themselves.
• CRD version migration during the V1→V2 MCP migration process itself (tracked separately).
• Removing deprecated CRD versions from the API server once all data is migrated (this is an upstream chart responsibility, not the platform's).

References

• Flux rollback incident: knowledge base known-issue-flux-crd-storedversions-rollback.md
• fluxcd-community/helm-charts PR Visualize relation of Crossplane and its providers in ManagedControlPlane wizard #280 (adds crds.migration.enabled): chore: upgrade charts to Flux v2.8.5 release (adds flux migrate pre-upgrade hook) fluxcd-community/helm-charts#280
• Kubernetes docs — CRD versioning & storage migration: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#upgrade-existing-objects-to-a-new-stored-version
• flux migrate CLI reference: https://fluxcd.io/flux/cmd/flux_migrate/

User Stories or tasks

No response

What is required to accept the Epic as finished.

• Full service provider coverage: Every service provider that ships CRDs (Flux, Crossplane core, all Crossplane providers, External Secrets Operator, Kyverno, Velero, Landscaper, kro, OCM) has storage-migration protection in place — either via an enabled upstream chart hook or via the generic migration job.
• Upgrade safety on V2: A version upgrade that crosses a CRD storage-version boundary on a V2 MCP completes without any manual operator or customer action, for all covered service providers.
• Upgrade safety on V1: A version upgrade that crosses a CRD storage-version boundary on a V1 MCP either:
  a. Completes automatically, or
  b. Is blocked with a clear, actionable error condition before the Helm upgrade begins — for all covered service providers.
• Rollback safety: A rollback that would fail due to status.storedVersions incompatibility is blocked with a clear error before it begins, for all covered service providers.
• Observability: Each automatic migration emits a platform metric and surfaces a condition on the service CR that is visible to both operators and customers via the Onboarding API.
• Idempotency: The migration mechanism is safe to re-run — executing it twice on a fully migrated cluster produces no errors and no resource modifications.

Dependencies of this Epic

No response

Risks of this Epic

No response

Known Stakeholders of this Epic

No response

Milestone Definitions for this Epic.

Milestone 1 — Immediate Mitigation (V2, Flux only)

• Enable crds.migration.enabled: true in service-provider-flux's default HelmRelease values for Flux, for all landscapes.
• Document the manual flux migrate procedure in the internal runbook so support engineers know what to do when a similar issue is reported on V1 MCPs.

Milestone 2 — Detection and Blocking (Both V1 and V2)

• Implement a CRDStorageVersionChecker utility that, given a set of CRD manifests (from an incoming Helm chart), computes the set of CRDs requiring storage migration on a target cluster.
• Integrate the checker into the service provider reconcile loop: if migration is needed, set a condition (StorageMigrationRequired: True) on the service CR and do not apply the new HelmRelease version.
• For V1, integrate the same check into the co-core FluxCD HelmRelease pipeline (or add a pre-upgrade Job resource via the HelmRelease preUpgrade hook).

Milestone 3 — Automated Migration (Both V1 and V2)

• Implement a CRDStorageMigrationJob that the service provider can trigger on the target MCP shoot cluster. It should:
  • Run as a Job in a dedicated namespace (e.g. kube-system or openmcp-system) with the minimum RBAC needed to list and annotate CRs for the specific CRDs being migrated.
  • Be idempotent — safe to re-run.
  • Report completion/failure back to the service provider controller via a well-known annotation or status condition on the Job.
• On successful migration, automatically clear the blocking condition and re-trigger the upgrade.
• Add observability: emit a metric/event for each migration performed, allowing platform operators to monitor via Dynatrace.

Milestone 4 — Rollback Safety

• Extend the checker to also validate rollback targets: before a rollback is allowed, verify that status.storedVersions for all affected CRDs is compatible with the rollback chart's spec.versions. Block the rollback and surface the same StorageMigrationRequired condition if not.

Metrics Definition for this Epic.

1. Zero manual CRD migration interventions — Number of support tickets or operator interventions required to unblock a service upgrade due to status.storedVersions conflicts drops to zero after full rollout.
2. Upgrade success rate — Percentage of service version upgrades (across all service providers and landscapes) that complete without error on first attempt, measured before and after rollout.
3. Time-to-unblock — For upgrades requiring migration, the elapsed time from version bump detection to successful upgrade completion (entirely automated; target: under 10 minutes).
4. Migration job idempotency — Re-running the migration job on an already-migrated cluster produces exit code 0 and no resource modifications (verified by automated test).
5. Condition visibility — 100% of in-progress or blocked migrations have a corresponding .status.conditions entry on the service CR that is readable via the Onboarding API at the time the block occurs.
6. Service coverage — Number of service providers with storage-migration protection enabled, tracked as a ratio against the total number of service providers that ship CRDs (target: 100% by end of Milestone 3).

Any further valuable resources.

No response