fix(operator,chart): adopt pre-existing aif-ui-config ConfigMap into Helm ownership

[leomiraanda]

Summary

Fixes SUSEAI-1039: helm install/upgrade of the aif-ui-server release can fail with:

ConfigMap "aif-ui-config" in namespace "cattle-ui-plugin-system" exists and cannot be
imported into the current release: invalid ownership metadata

The operator's syncUIConfigMap self-heal recreates aif-ui-config if it's missing, on every successful InstallAIExtension reconcile — and it creates it unowned by Helm. Once that happens, the next Helm install/upgrade of aif-ui-server permanently fails and the InstallAIExtension CR gets stuck in Failed, retrying the same doomed install every reconcile.

This PR bundles two complementary fixes for that problem.

Fix 1 — adopt any pre-existing ConfigMap via Helm's native TakeOwnership

An earlier version of this fix hand-rolled the ownership stamping (mirroring the label/annotations the chart's configmap.yaml sets). Code review on this PR found three real gaps in that approach — the UI's hardcoded release name vs. the operator's chart-URL-derived one only agreeing on the default path, the UI's create path stamping ownership even on the git-sourced branch its own comment named, and a chart-URL change silently transferring ownership instead of failing like a mismatched-name bare install would.

Replaced all of that with Helm's own TakeOwnership (vendored v3.21.1's install.go/upgrade.go), which this codebase already uses for the identical problem via Fleet's equivalent flag (blueprint.go, NVIDIA NIM charts templating their own pre-delivered ngc-secret). Set on all three of the operator's Helm action sites in operator/internal/infra/helm/action.go — install, upgrade, and the dry-run render EnsureRelease uses to decide whether an upgrade is even needed (Helm's ownership check runs before the DryRun branch, so it would otherwise block the real upgrade from ever being attempted).

adoptUIConfigMap and the ownership-stamping half of syncUIConfigMap are gone — Helm itself now adopts the ConfigMap regardless of who created it or what it currently says, using the correct current release identity every time. syncUIConfigMap keeps only its Data self-heal.

Fix 2 — make aif-ui-config survive helm uninstall

syncUIConfigMap's own doc comment already claimed the ConfigMap "is intentionally not deleted when the CR is removed," but nothing enforced that — configmap.yaml had no helm.sh/resource-policy: keep, so a plain uninstall (including the operator's own finalizer path) deleted it along with the rest of the release, silently losing custom operator-connection/catalog settings on reinstall.

Added helm.sh/resource-policy: keep. Helm's uninstall now leaves the object behind with its ownership labels/annotations intact, so a same-identity reinstall adopts it with no extra help needed — and Fix 1's TakeOwnership now covers every other case too (identity change, or an object created unowned entirely), not just this one.

Dropped after merging main

This branch originally carried a third fix: saveOperatorConfig's PUT was a full-object replace that never carried forward the existing object's labels/annotations, so the next Settings save after the ConfigMap became Helm-owned stripped that ownership straight back off.

main has since removed the UI's Settings write path entirely (getOperatorConfig, saveOperatorConfig, invalidateOperatorConfig and the extension-check helpers are all gone from ui/pkg/aif-ui/utils/operator-config.ts). There is no longer a function to fix, so that change and its Vitest suite were dropped in the merge rather than carried forward. The UI is no longer a writer of aif-ui-config at all, which is why the summary above names only one.

Testing

• ui_configmap_ownership_test.go (renamed to reflect scope: syncUIConfigMap no longer touches ownership) now pins that the self-heal only ever touches Data, regardless of pre-existing labels/annotations — including recreating an unowned ConfigMap without adding any ownership metadata.
• Verified against the live cluster two ways:
  • A standalone program using the same vendored Helm SDK directly: install fails on a pre-existing unowned ConfigMap without TakeOwnership; with it, Helm patches rather than replaces the object, correctly stamping the real release's identity and merging data instead of clobbering it.
  • The actual operator binary end to end (locally-built image loaded into the cluster node): unowned ConfigMap recreated by hand, next reconcile logs Helm release not found, installing → installed successfully with the same object uid throughout and correct ownership stamped.
  • Plain uninstall → reinstall regression check (no leftover, clean self-heal).
  • helm uninstall → Helm explicitly reports These resources were kept due to the resource policy: [ConfigMap] aif-ui-config; a same-name reinstall then adopts with the exact same uid/resourceVersion throughout — never touched, not deleted+recreated.

Known limitations (not fixed by this PR)

Standalone helm CLI runs. A bare helm install/helm upgrade aif-ui-server with no operator involved still isn't covered automatically — TakeOwnership is set on the operator's own Helm SDK calls, not on the chart itself. This can't be made automatic from inside the chart: Helm's ownership check runs before any pre-install/pre-upgrade hook could execute (confirmed against the vendored v3.21.1 source — the ownership check computes its adoption set before performInstall, which is what fires HookPreInstall). Since TakeOwnership is the same flag Helm exposes as --take-ownership on helm install/upgrade (v3.15+), documented this as a manual workaround instead — see the new "Standalone install/upgrade" section in charts/aif-ui/README.md.

useStaticCatalog is not preserved by the self-heal. main added a third key to the chart's configmap.yaml, but syncUIConfigMap writes only operatorNamespace and operatorService. On update that's harmless (CreateOrUpdate merges, so the key survives), but if the ConfigMap is deleted outright, the self-heal recreates it without useStaticCatalog and the UI falls back to its true default. That matches the chart default, so it's invisible in the common case — but an admin who installed with appCatalog.useStaticCatalog=false gets silently reverted to the static catalog until the next helm upgrade. Out of scope here: the self-heal has no source for that value today, so fixing it is design work. Tracking it with SUSEAI-910, which covers the same "objects in cattle-ui-plugin-system survive in a degraded state" territory.

Trade-off

TakeOwnership applies to every resource in a chart's manifest, not just this one ConfigMap, and adopting an object owned by another Helm release transfers deletion rights to this one. In practice we only ever point InstallAIExtension at the aif-ui chart, whose objects are release-scoped apart from aif-ui-config — but that's a deployment convention, not an invariant: the chart comes from the CR (spec.source.helm.chartURL, or spec.extension.name on the ClusterRepo path). Accepted deliberately, since the narrower alternative is the hand-rolled stamping this replaced. Worth revisiting if we ever install more than the one chart.

Test plan

• go build ./..., go vet, gofmt clean on touched files
• helm lint / helm template clean
• Full operator test suite passes (make test), including updated specs
• UI test suite (vitest run) and tsc typecheck pass
• Live cluster verification (see Testing above)
• main merged into the branch; conflicts resolved, obsolete UI fix dropped (see above)
• Reviewer sign-off on scope (operator-only fix + same-identity-reinstall + documented standalone-install workaround)

[leomiraanda]

Test coverage matrix

Two axes matter: who's installing (operator-orchestrated vs. standalone helm) and what caused the leftover unowned ConfigMap.

#	Scenario	Works?	Why	Evidence
1	Operator-orchestrated install — leftover unowned CM exists before the first Helm install (from a self-heal, a git→Helm switch, or a UI race)	✅ Works	adoptUIConfigMap pre-stamps ownership right before EnsureRelease runs, inside the operator's own reconcile loop	Live-tested (exact ticket repro)
2	Operator-orchestrated — CM deleted after a successful install, operator self-heals it	✅ Works	syncUIConfigMap re-stamps ownership on every successful reconcile	Live-tested
3	Operator-orchestrated — plain uninstall → reinstall, CM deleted along with the release, no leftover	✅ Works	Nothing to adopt — chart creates it fresh; confirms no regression	Live-tested
4	Operator-orchestrated — extension is git-sourced (no Helm release at all)	➖ N/A	Correctly left unowned by design — there's no release to adopt into	Unit-tested
5	Operator-orchestrated — extension organically switches git-sourced → Helm-sourced	✅ Should work	Same adoptUIConfigMap call fires before every Helm install regardless of how the extension got there	Not independently driven end-to-end; same code path as #1
6	Operator-orchestrated — UI save and operator's adopt/install race concurrently	✅ Works	Kubernetes optimistic concurrency; worst case is a transient conflict, not corruption	Automated concurrency test
7	UI Settings page save, extension is operator-managed (CR exists)	➖ N/A	UI deliberately skips writing the ConfigMap at all (Settings.vue:443) — defers entirely to the operator	Confirmed live
8	UI Settings page save, extension is unmanaged (no CR exists)	✅ Works	saveOperatorConfig's POST-create path stamps ownership itself	Live-tested (real browser via serve-pkgs)
9	Standalone helm install aif-ui-server, leftover unowned CM, no operator involved at all	❌ Does not work	Adoption logic lives only in operator Go code; a chart-level pre-install hook is architecturally impossible — Helm's ownership check (install.go:356) runs before any hook (:450)	Live-tested, reproduces the ticket's exact error; confirmed via Helm source + zero hook Jobs ever created
10	Standalone helm upgrade aif-ui-server, CM's ownership stripped out-of-band	❌ Does not work	Same architectural gate — upgrade.go runs the identical ownership check (:350) before HookPreUpgrade (:421)	Verified via Helm source; not live-tested, but same mechanism as #9
11	Multiple InstallAIExtension CRs mislabeling the CM with a different release's identity	⚠️ Theoretical risk	syncUIConfigMap/adoptUIConfigMap aren't scoped to a specific extension name — a second, unrelated CR reconciling successfully would stamp the wrong release name	Not tested; current tooling only ever creates one CR in practice

Bottom line: every path through the aif-operator (rows 1-8) is fixed and verified. The one confirmed gap is any aif-ui-server install/upgrade run by hand, outside the operator (rows 9-10) — permanent unless something external to the chart runs before the helm command, since Helm's own ownership check rejects before any chart hook gets a chance to run.

[leomiraanda]

Test coverage matrix — revised after the TakeOwnership rewrite + main merge

The matrix above still stands as a record of what was verified at the time, but it describes two mechanisms that no longer exist:

• adoptUIConfigMap — removed when this PR replaced hand-rolled ownership stamping with Helm's native TakeOwnership.
• saveOperatorConfig — removed on main by 48191334 (refactor(ui): remove advanced settings section), along with the rest of the UI's ConfigMap write path.

Six of the eleven rows are affected. Leaving the original in place; here's the delta.

#	Then	Now	What changed
1	✅ via adoptUIConfigMap	✅	Conclusion holds, mechanism doesn't — TakeOwnership on install does the adoption. Re-verified live against the current revision (see the PR description's Testing section).
2	✅ syncUIConfigMap re-stamps ownership on every reconcile	⚠️ rationale no longer true	Detail below.
3	✅ CM deleted with the release, nothing left over	✅	Premise inverted by resource-policy: keep: uninstall now keeps the object. Reinstall adopts it — verified same uid/resourceVersion throughout. Outcome unchanged, but the row no longer tests what it says.
4	➖ N/A — left unowned by design	➖	Still accurate, and now on a firmer basis: the self-heal never writes ownership metadata at all.
5	✅ same code path as #1	✅	Rationale stale (adoptUIConfigMap → TakeOwnership), conclusion holds.
6	✅ UI-save / operator-install race	➖ cannot occur	There is no UI writer.
7	➖ N/A — UI skips the write when a CR exists	➖	Moot: the UI never writes, CR or no CR.
8	✅ saveOperatorConfig POST-create stamps ownership	➖ scenario removed	Detail below.
9	❌ standalone helm install	❌	Unchanged. Now has a documented --take-ownership workaround (charts/aif-ui/README.md).
10	❌ standalone helm upgrade	❌	Unchanged, same workaround.
11	⚠️ theoretical: wrong-release stamping	⚠️ different shape	Detail below.
12	(new)	⚠️	useStaticCatalog is not preserved across a self-heal. Detail below.

Row 2 — the self-heal no longer re-stamps ownership

syncUIConfigMap now deliberately touches only Data. So if the ConfigMap is deleted, the self-heal recreates it unowned, and nothing re-adopts it on its own: decideRelease selects its action "from storage state alone, before any chart is pulled" (operator/internal/infra/helm/action.go:369) and never inspects live cluster objects. A deleted-and-recreated ConfigMap therefore produces no upgrade, and the object can stay unowned indefinitely.

Benign on every path I can trace — the UI reads Data and doesn't care about ownership, the next genuine upgrade adopts via TakeOwnership, and uninstall doesn't need ownership thanks to resource-policy: keep. But it is a real behavioural change from "re-stamped every 60s" to "unowned until something else triggers an upgrade," and the original row asserts the opposite.

Row 8 — not broken, deleted

main removed the advanced settings section, so Settings.vue only reads now (loadOperatorConfig, getOperatorNamespace). Consequence worth being explicit about: an unmanaged install can no longer correct wrong operator coordinates from the UI — that now requires chart values and a helm upgrade. Upstream product decision, not a change this PR makes, but the capability existed when the matrix was written.

Row 11 — risk changed shape rather than going away

The wrong-release-stamping code is gone, but TakeOwnership makes the multi-CR case concrete rather than theoretical: two InstallAIExtension CRs means two releases, and each install/upgrade adopts aif-ui-config away from the other, ping-ponging ownership. Now called out in the PR description's Trade-off section.

Row 12 (new) — useStaticCatalog lost on self-heal

main added a third key to the chart's configmap.yaml, but syncUIConfigMap writes only operatorNamespace and operatorService. On update this is harmless — CreateOrUpdate merges, so the key survives. If the ConfigMap is deleted outright, the self-heal recreates it without useStaticCatalog and the UI falls back to its true default. That matches the chart default, so it's invisible in the common case, but an admin who installed with appCatalog.useStaticCatalog=false is silently reverted to the static catalog until the next helm upgrade.

Same trigger as row 2, but the only one of the two with user-visible impact. Out of scope here — the self-heal has no source for that value today, so fixing it is design work. Tracking with SUSEAI-910, which covers the same "objects in cattle-ui-plugin-system surviving in a degraded state" territory.

Bottom line

Unchanged from the original: every operator-orchestrated path is fixed and verified, and the one confirmed gap is a by-hand helm install/upgrade outside the operator (rows 9-10), which Helm's own ownership check makes impossible to fix from inside the chart. What's new is that rows 6-8 can no longer happen at all, and rows 2 and 12 are latent-but-benign consequences of the self-heal no longer owning anything.