From d18d3af489689c0d68806c643df4dc6513f2bf60 Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 03:25:51 +0800 Subject: [PATCH 01/14] docs: add split system agent auth for Global DR --- docs/en/global/disaster_recovery.mdx | 244 +++++++++++++++++++++++---- docs/en/global/install.mdx | 151 +++++++++++++++-- 2 files changed, 344 insertions(+), 51 deletions(-) diff --git a/docs/en/global/disaster_recovery.mdx b/docs/en/global/disaster_recovery.mdx index 72440038..f59d2176 100644 --- a/docs/en/global/disaster_recovery.mdx +++ b/docs/en/global/disaster_recovery.mdx @@ -27,7 +27,7 @@ That deployment procedure is the authoritative source for the installation-time - Bare Metal deployments reference the shared encryption provider Secret from `BaremetalCluster.spec.encryptionProviderConfigRef` and use the same Kubernetes ServiceAccount signing key on both sides. - Huawei DCS, VMware vSphere, and Huawei Cloud Stack create the `dcs-import-extra-resources` ConfigMap before installer import so the installation can import referenced credential Secrets and, where applicable, provider-specific infrastructure resources. The name keeps the `dcs` prefix for historical installer compatibility. - Bare Metal creates the `dcs-import-extra-resources` ConfigMap before installer import so the installation can preserve bare-metal and elemental resources required by handoff. -- The primary cluster installs `global-etcd-sync` with the standby cluster connection values after both installations succeed. +- The standby cluster installs `global-etcd-sync` with connection values that point to the active primary after both installations succeed. ## Operational Scope @@ -45,26 +45,34 @@ A disaster recovery switchover and a DR-aware `global` cluster upgrade both unin ## Bare Metal DR Model -Bare Metal DR adds host-level state that does not exist in VM-based providers. The `global` cluster does not create physical machines; it manages long-lived `MachineInventory` objects and writes plan Secrets that are consumed by `elemental-system-agent` on each host. Because of that, a standby `global` cluster must be able to accept the existing system-agent token and must receive the workload cluster inventory and plan state without overwriting its own `global` inventory. +Bare Metal DR adds host-level state that does not exist in VM-based providers. The `global` cluster does not create physical machines; it manages long-lived `MachineInventory` objects and writes plan Secrets that are consumed by `elemental-system-agent` on each host. Global-cluster machines therefore use a cluster-local identity, while non-`global` workload machines use a shared identity that follows the synchronized workload state. A standby `global` cluster must preserve its local Global identity while accepting the shared workload token from primary. ### Endpoint Rules Use different system-agent endpoints for `global` hosts and workload-cluster hosts: -| Machine type | Endpoint | -|--------------|----------| -| Machines that belong to the active `global` cluster | `https:///kubernetes/global` | -| Machines that belong to non-`global` workload clusters | `https:///kubernetes/global` | +| Machine type | Endpoint | Identity | Synchronized | +|--------------|----------|----------|--------------| +| Machines that belong to the local `global` cluster | `https://:6443` | `baremetal-global-system-agent` | No | +| Machines that belong to non-`global` workload clusters | `https:///kubernetes/global` | `baremetal-system-agent` | Yes, primary to standby | -By default both endpoints are platform ingress paths, not direct kube-apiserver `:6443` endpoints. For the bootstrap KIND phase, `global` host registration uses the bootstrap host endpoint, normally `https://:12443`. Handoff changes only the installed `global` hosts to the final VIP endpoint. +Global hosts always use their own side's direct kube-apiserver VIP; they do not follow the platform domain when active and standby switch roles. Workload hosts always use the platform domain so that their endpoint follows the active Global cluster. -For a DR `global` cluster that must keep managing its own hosts without depending on its own ingress, opt in to direct kube-apiserver access by setting `handoffHook.directAPIServer: true` on the Bare Metal provider AppRelease (this requires `handoffHook.controlPlaneVIP`). Handoff then points `global`-cluster machines at `https://:6443` directly, with no `/kubernetes/global` ingress path, and the agent trusts the apiserver with the in-cluster apiserver CA instead of the platform-ingress certificate. The parameter defaults to `false` (the ingress endpoint above); it applies only to `global`-cluster machines, and workload-cluster machines always keep the ingress path. +During the bootstrap KIND phase, a Global host registration still uses the bootstrap endpoint, normally `https://:12443`. The bootstrap `MachineRegistration` must set the Global auth scope, but it must not set `baremetal.cluster.io/system-agent-server-url`; handoff has not happened yet. + +```yaml +metadata: + annotations: + baremetal.cluster.io/system-agent-auth-scope: global +``` + +Set `handoffHook.controlPlaneVIP` to the current side's VIP and set `handoffHook.directAPIServer: true` on both Bare Metal provider AppRelease objects. Handoff then rewrites only the installed Global hosts to `https://:6443`, with no `/kubernetes/global` path. The agent uses the kube-apiserver CA from the local ServiceAccount token Secret. Workload hosts keep the platform ingress path. A `SeedImage` generated in the bootstrap KIND environment points at the bootstrap registration endpoint. After the `global` cluster is installed and handed off, create new `MachineRegistration` and `SeedImage` resources on the active `global` cluster for any new `global` hosts. A pre-failover workload ISO can be reused after failover only when its registration URL uses the platform domain and the matching non-`global` `MachineRegistration` was synchronized to standby. -For a post-handoff `global` host registration, set the system-agent server URL annotation to the current active `global` control-plane VIP. The elemental operator appends the `/kubernetes/global` path when it renders the system-agent endpoint. Use the primary VIP before failover and the standby VIP after failover. +For every post-handoff Global host registration, set the Global auth scope, the local side's direct API URL, and direct endpoint mode. A primary Global registration always uses the primary VIP, and a standby Global registration always uses the standby VIP; do not change existing Global host endpoints during failover. ```yaml apiVersion: elemental.cattle.io/v1beta1 @@ -73,32 +81,101 @@ metadata: name: global- namespace: cpaas-system annotations: - baremetal.cluster.io/system-agent-server-url: https:// + baremetal.cluster.io/system-agent-auth-scope: global + baremetal.cluster.io/system-agent-server-url: https://:6443 + baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver ``` -When the `global` cluster was handed off with `handoffHook.directAPIServer: true`, keep new `global` registrations on the same direct-apiserver endpoint instead: set `baremetal.cluster.io/system-agent-server-url: https://:6443` and add `baremetal.cluster.io/system-agent-direct: "true"`. The elemental operator then serves that URL verbatim and does not append `/kubernetes/global`. +`baremetal.cluster.io/system-agent-direct: "true"` remains an upgrade-compatibility alias for `direct-apiserver`. Do not use it in new registrations. If both annotations are present, the explicit `baremetal.cluster.io/system-agent-endpoint-mode` value takes precedence. + +### System-Agent Identity Requirements + +Bare Metal DR uses two identity bundles: -### Shared Token Requirements +- The local Global bundle consists of `ServiceAccount`, token Secret, Role, and RoleBinding named `baremetal-global-system-agent` (the Secret name has the `-token` suffix). Each side generates and retains its own bundle. Its Role contains only that side's Global plan Secrets. +- The workload shared bundle consists of the four `baremetal-system-agent` objects. Primary is authoritative before failover, and `etcd-sync` copies the bundle exactly to standby. Its Role contains only non-`global` workload plan Secrets. -Existing hosts keep the `baremetal-system-agent` token that they received before failover. For that token to work on the standby cluster: + + Do not customize `baremetal-global-system-agent`, `baremetal-global-system-agent-token`, `baremetal-system-agent`, or `baremetal-system-agent-token`. The default `etcd-sync` exact-sync and ignore rules match these names literally. If a deployment must use different names, update and validate all corresponding `etcd-sync` rules before starting synchronization. + + +Primary and standby must still use the same `/etc/kubernetes/pki/sa.key`, `/etc/kubernetes/pki/sa.pub`, `service-account-issuer`, and `api-audiences` values so that the shared workload token is valid on both API servers. The two local Global tokens are intentionally different and must not be synchronized. + +Set `elemental.systemAgent.splitAuthEnabled: true` on both provider AppRelease objects. This is the single source of truth used by both `elemental-operator` and handoff; do not configure a separate handoff auth-enable switch. With the split enabled, the active Global must set `elemental.systemAgent.sharedAuthReadOnly: false` so that `elemental-operator` reconciles the shared workload Role. The inactive Global must set it to `true` so that `etcd-sync`, not the local operator, owns the synchronized shared bundle. Both operators continue to reconcile their own Global-local bundles. + +### Migrating Existing Handoff State -- Primary and standby must use the same `/etc/kubernetes/pki/sa.key` and `/etc/kubernetes/pki/sa.pub`. -- Primary and standby must use the same `service-account-issuer` and `api-audiences` values, normally `https://kubernetes.default.svc.cluster.local`. -- The `ServiceAccount`, token Secret, Role, and RoleBinding for `baremetal-system-agent` must be synchronized exactly to standby. -- `Role/baremetal-system-agent.rules[].resourceNames` must contain only plan Secret names. Do not grant namespace-wide Secret access and do not add registry, bootstrap, kubeconfig, or platform credential Secrets. +`ConfigMap/cpaas-system/baremetal-system-agent-handoff` is a versioned completion record. The direct-API split-auth configuration requires both `data.ready: "true"` and `data.profile: split-local-direct-v1`. A ConfigMap written by an older release with only `ready: "true"`, or with `legacy-shared-erebus-v1` or `legacy-shared-direct-v1`, must not cause the new handoff to skip. -Verify the permission boundary on standby before failover. +The upgrade migrates the already installed Global hosts in place; it does not require the original bootstrap plan Secret. Keep this safety order for every current Global plan Secret: + +1. Create and validate the local Global ServiceAccount, token Secret, Role, and RoleBinding, while preserving the old shared identity's access to the same current Global plan Secret. +2. Write the handoff plan on that current Global plan Secret so the host switches to the local token and local direct API endpoint. +3. Wait until the handoff checksum is Applied, then write and observe a Global-side probe with the new local identity. +4. Only after the probe succeeds, remove the old shared identity's temporary access to the Global plan and reconcile the shared Role back to workload-only plan Secrets. +5. Treat migration as complete only when all hosts have passed the probe, the permission scopes are narrowed, and the completion ConfigMap reports the exact new profile. + +This temporary overlap is deliberate: narrowing the shared permission before the old host has consumed its token-switch plan can strand the host with `403 Forbidden`. Do not grant namespace-wide Secret access, do not use a `ClusterRoleBinding`, and do not manually patch the completion profile to bypass the migration. ```shell -TOKEN=$(kubectl -n cpaas-system get secret baremetal-system-agent-token -o jsonpath='{.data.token}' | base64 -d) +kubectl --kubeconfig \ + -n cpaas-system get configmap baremetal-system-agent-handoff \ + -o jsonpath='{.data.ready}{" "}{.data.profile}{"\n"}' +``` + +Expected output: `true split-local-direct-v1`. -kubectl --token="${TOKEN}" -n cpaas-system auth can-i get secret -kubectl --token="${TOKEN}" -n cpaas-system auth can-i patch secret -kubectl --token="${TOKEN}" -n cpaas-system auth can-i get secret global-registry-auth -kubectl --token="${TOKEN}" -n cpaas-system auth can-i list secrets +Verify both permission boundaries before failover. Repeat the checks on primary and standby with an administrator kubeconfig that is allowed to impersonate ServiceAccounts. Set the two example Secret names from that same cluster. + +```shell +export LOCAL_GLOBAL_KUBECONFIG="" +export LOCAL_GLOBAL_PLAN_SECRET="" +export WORKLOAD_PLAN_SECRET="" + +# Expected: yes +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i get "secret/${LOCAL_GLOBAL_PLAN_SECRET}" +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i patch "secret/${LOCAL_GLOBAL_PLAN_SECRET}" +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ + -n cpaas-system auth can-i get "secret/${WORKLOAD_PLAN_SECRET}" +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ + -n cpaas-system auth can-i patch "secret/${WORKLOAD_PLAN_SECRET}" + +# Expected: no (cross-scope access) +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i get "secret/${WORKLOAD_PLAN_SECRET}" +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i patch "secret/${WORKLOAD_PLAN_SECRET}" +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ + -n cpaas-system auth can-i get "secret/${LOCAL_GLOBAL_PLAN_SECRET}" +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ + -n cpaas-system auth can-i patch "secret/${LOCAL_GLOBAL_PLAN_SECRET}" + +# Expected: no (namespace-wide or non-plan Secret access) +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i list secrets +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ + -n cpaas-system auth can-i list secrets +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i get secret/global-registry-auth +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ + -n cpaas-system auth can-i get secret/global-registry-auth ``` -Expected result: `get` and `patch` are allowed only for plan Secrets; `global-registry-auth` and namespace-wide `list secrets` are denied. +Expected result: `get` and `patch` are allowed only within the matching permission scope. Every cross-scope check, both namespace-wide `list secrets` checks, and both `global-registry-auth` checks return `no`. Also compare the token hash in `/var/lib/elemental/agent/elemental_connection.json` on every Global host with the local `baremetal-global-system-agent-token` Secret; checking only the Kubernetes Secret does not prove that an existing host uses the expected token. ### etcd-sync Scope @@ -124,8 +201,53 @@ Do not synchronize: - Standby `Cluster/cpaas-system/global`. - Standby `BaremetalCluster/cpaas-system/global`. - Standby `BaremetalMachine`, `MachineInventoryPool`, `MachineInventory`, and `MachineRegistration` objects that belong to the standby `global` cluster. +- `ServiceAccount/cpaas-system/baremetal-global-system-agent`, `Secret/cpaas-system/baremetal-global-system-agent-token`, `Role/cpaas-system/baremetal-global-system-agent`, and `RoleBinding/cpaas-system/baremetal-global-system-agent` from either side. +- Plan Secrets that belong to either side's local `Cluster/global`. Add their exact etcd keys to `etcd_sync.additional_ignore_equal` before starting synchronization. - Secrets from the primary cluster that are not required by workload cluster reconciliation. +Before installing the ModuleInfo, follow the executable two-kubeconfig procedure in [Install Primary and Standby Clusters](./install.mdx#install-primary-and-standby-clusters) to collect `Role/baremetal-global-system-agent.rules[].resourceNames` from both sides, merge and de-duplicate the names, and populate `spec.config.additional_ignore_equal` with exact `/registry/secrets/cpaas-system/` keys. Repeat that update before synchronization continues whenever either local Global Role gains another plan Secret. + +The mirror and monitor read their filter files at process startup. After every dynamic `additional_ignore_equal` update, first verify that the rendered `ConfigMap/cpaas-system/etcd-sync-ignore-text` contains every exact key, then trigger and wait for rollouts of both `Deployment/etcd-sync` and `Deployment/etcd-sync-monitor`. Updating ModuleInfo or waiting for the ConfigMap volume to refresh is not sufficient. + +```shell +export STANDBY_GLOBAL_KUBECONFIG="" + +IGNORE_EQUAL_TEXT="$( + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get configmap etcd-sync-ignore-text \ + -o jsonpath='{.data.ignore-equal\.txt}' +)" + +printf '%s\n' "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" | jq -r '.[]' | \ + while IFS= read -r key; do + if ! printf '%s\n' "${IGNORE_EQUAL_TEXT}" | grep -Fqx -- "${key}"; then + echo "missing etcd-sync ignore key: ${key}" >&2 + exit 1 + fi + done + +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system rollout restart deployment/etcd-sync +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system rollout restart deployment/etcd-sync-monitor +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system rollout status deployment/etcd-sync --timeout=5m +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system rollout status deployment/etcd-sync-monitor --timeout=5m +``` + +### Adding a Global Host While Synchronization Is Configured \{#adding-a-global-host-while-synchronization-is-configured} + +Never create or allocate a new primary or standby Global host while the mirror is actively synchronizing and the new plan Secret is absent from `additional_ignore_equal`. Use this sequence: + +1. Stop or suspend the `etcd-sync` mirror with the supported plugin lifecycle operation, and verify that `Deployment/cpaas-system/etcd-sync` has no Ready mirror Pods. If a direct scale-to-zero is immediately reverted by ModuleInfo or AppRelease reconciliation, use the owning platform operation instead of racing the controller. +2. Create the local Global `MachineRegistration` and `SeedImage`, register the host, and wait until the local `Role/baremetal-global-system-agent` contains the new plan Secret. Keep synchronization stopped. +3. Re-run the two-kubeconfig collection, update `global-etcd-sync` ModuleInfo with the complete merged exact-key list, and verify the rendered `etcd-sync-ignore-text` ConfigMap. +4. Restart the monitor and restore or roll out the mirror only after the new exact key is present. Wait for both Deployments to become available and confirm that the mirror remains healthy. +5. Re-run the Global-local and shared RBAC allow/deny checks for the new plan Secret before considering the host operational. + +Do not guess a plan Secret name from the inventory name. Read it from the local Global Role or `MachineInventory.status.plan.secretRef.name`, and do not resume synchronization if the two sources disagree. + Do not raw-sync every key under `/registry/secrets` when the primary and standby apiservers do not share the same Kubernetes encryption-provider configuration. The standby apiserver cannot decrypt ciphertext written with a different key. Sync only the required Secrets through a safe path that lets the standby apiserver store them with the standby encryption key, or use an approved exact-sync mechanism for the fixed auth bundle. @@ -136,20 +258,67 @@ Use this sequence for a planned or declared failover: 1. Fence the primary `global` write path. Stop or isolate the controllers and platform entry points that can continue writing workload cluster resources. Do this at the owning CR or platform-resource layer where possible; scaling a Deployment alone can be reverted by its owner. 2. Wait until primary-to-standby `etcd-sync` is caught up, then freeze or stop it. Do not allow old primary data to overwrite new standby writes after failover. -3. If your provider uses a sync gate, clear the gate after the sync has been stopped so standby controllers do not keep requeueing because they think synchronization is still in progress. -4. Switch the platform domain to the standby platform entrance. -5. Verify DNS and ingress: +3. On the standby that is becoming active, update the Bare Metal provider AppRelease to `elemental.systemAgent.sharedAuthReadOnly: false` and wait for `Deployment/cpaas-system/elemental-operator` to roll out. Verify that its arguments no longer contain `--system-agent-shared-auth-read-only`, and verify that `Role/cpaas-system/baremetal-system-agent.resourceNames` matches the workload-scoped `MachineInventory.status.plan.secretRef.name` set. This confirms that the new active operator has taken ownership of shared RBAC. + + ```shell + export NEW_ACTIVE_GLOBAL_KUBECONFIG="" + + kubectl --kubeconfig "${NEW_ACTIVE_GLOBAL_KUBECONFIG}" \ + -n cpaas-system rollout status deployment/elemental-operator + + if kubectl --kubeconfig "${NEW_ACTIVE_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get deployment elemental-operator -o json | \ + jq -e '[.spec.template.spec.containers[].args[]?] + | index("--system-agent-shared-auth-read-only") != null' >/dev/null; then + echo "elemental-operator is still in shared auth read-only mode" >&2 + exit 1 + fi + + WORKLOAD_PLAN_SECRET_NAMES="$( + kubectl --kubeconfig "${NEW_ACTIVE_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get machineinventories.elemental.cattle.io -o json | \ + jq -c '[ + .items[] + | select((.metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] // "shared") == "shared") + | .status.plan.secretRef.name? // empty + ] | unique | sort' + )" + SHARED_ROLE_SECRET_NAMES="$( + kubectl --kubeconfig "${NEW_ACTIVE_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get role baremetal-system-agent -o json | \ + jq -c '[ + .rules[]? + | select(any(.apiGroups[]?; . == "")) + | select(any(.resources[]?; . == "secrets")) + | .resourceNames[]? + ] | unique | sort' + )" + + if [ "${WORKLOAD_PLAN_SECRET_NAMES}" != "${SHARED_ROLE_SECRET_NAMES}" ]; then + echo "shared Role does not match workload plan Secrets" >&2 + exit 1 + fi + ``` + +4. Before the former primary is returned to service as the inactive peer or used as the target of reverse synchronization, set its AppRelease to `elemental.systemAgent.sharedAuthReadOnly: true`. It must not reconcile the shared bundle while it is the new standby. +5. If your provider uses a sync gate, clear the gate after the sync has been stopped so standby controllers do not keep requeueing because they think synchronization is still in progress. +6. Switch the platform domain to the standby platform entrance. +7. Verify the workload ingress and the new active Global's direct API endpoint: ```shell dig +short - curl -kI https:///kubernetes/global/version - curl -kI https:///kubernetes/global/version + curl -ksS -w '\nHTTP %{http_code}\n' \ + https:///kubernetes/global/version + curl -ksS -w '\nHTTP %{http_code}\n' \ + https://:6443/version ``` - A `401` response is acceptable; it proves the request reached the platform ingress. + A Kubernetes version response with HTTP `200`, or an authentication/authorization response with HTTP `401` or `403`, proves that the request reached the intended API endpoint. + +8. Confirm the standby cluster has the provider components needed to manage Bare Metal workload clusters: CAPI core, kubeadm provider, Bare Metal provider, elemental-operator, cluster-transformer, and alive-related delivery components. +9. Confirm existing workload hosts reconnect through the platform domain and can watch or patch their plan Secrets. -6. Confirm the standby cluster has the provider components needed to manage Bare Metal workload clusters: CAPI core, kubeadm provider, Bare Metal provider, elemental-operator, cluster-transformer, and alive-related delivery components. -7. Confirm existing workload hosts reconnect through the platform domain and can watch or patch their plan Secrets. +Failback uses the same ownership sequence in reverse. The currently active side remains `sharedAuthReadOnly: false`; the inactive failback target must be `true` while reverse `etcd-sync` is running. After reverse synchronization is caught up and stopped, change the side that is becoming active to `false`, verify that its operator has taken ownership, change the side becoming inactive to `true`, and only then switch the platform domain back. ### Post-Failover Validation @@ -161,7 +330,7 @@ After standby is active, validate in increasing risk order: - Worker scale-out allocates an available inventory and updates `Role/baremetal-system-agent.resourceNames` with the new plan Secret. - Worker scale-in runs a clean plan and releases inventory as designed. - Control-plane replacement restores the desired KCP replica count and etcd membership. -- A new `MachineRegistration` and `SeedImage` created on standby can register a new host. +- After following the stop-sync and exact-ignore update procedure, a new `MachineRegistration` and `SeedImage` created on standby can register a new host. - A pre-failover workload ISO is usable only if its registration URL uses the platform domain. ## Bare Metal Troubleshooting @@ -169,8 +338,8 @@ After standby is active, validate in increasing risk order: | Symptom | Likely cause | First check | |---------|--------------|-------------| | Handoff target list is empty | `dcs-import-extra-resources` was missing or incomplete before the installer API call | `kubectl -n cpaas-system get cm dcs-import-extra-resources -o yaml`; check `baremetal-system-agent-handoff` Job logs | -| Existing hosts get `401` or `Forbidden` after failover | ServiceAccount signing key, issuer, audience, token Secret, or Role `resourceNames` does not match | `sha256sum /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub`; `kubectl auth can-i` with the token | -| Existing hosts fail TLS verification after DNS switch | Platform certificate chain or SAN does not cover the platform domain and both VIPs | Check `cpaas-system/dex.tls`, certificate SANs, and `curl -kI https:///kubernetes/global/version` | +| Existing hosts get `401` or `Forbidden` after failover | ServiceAccount signing key, issuer, audience, token Secret, auth scope, or Role `resourceNames` does not match | `sha256sum /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub`; repeat the scoped `kubectl auth can-i` checks above | +| Existing workload hosts fail TLS verification after the domain switch, or Global hosts fail against their local VIP | The platform certificate does not cover the platform domain, or the local kube-apiserver certificate/CA does not cover and trust the direct VIP | Check `cpaas-system/dex.tls`, the kube-apiserver serving certificate SANs, the agent connection CA, `curl -ksS https:///kubernetes/global/version`, and `curl -ksS https://:6443/version` | | Standby `global` resources are replaced by primary resources | `etcd-sync` ignore rules did not exclude `global` resources | Inspect the `etcd-sync` ignore ConfigMap and stop sync before restoring standby resources | | New host cannot register with an old ISO | ISO registration URL points to bootstrap or primary-only IP, or `MachineRegistration` was not synchronized | Inspect the `MachineRegistration` status and rebuild the ISO on active standby when needed | @@ -203,9 +372,10 @@ Bare Metal requires all of the following to be aligned: - Same ServiceAccount signing key files and same issuer/audience values on both sides. - Same trusted platform certificate chain, with SANs that cover the platform domain, primary control-plane VIP, and standby control-plane VIP. - `dcs-import-extra-resources` created before the installer API call on both sides. -- Bare Metal provider AppRelease values set with `handoffHook.controlPlaneVIP` for the local side and `elemental.systemAgent.authMode: shared`. Add `handoffHook.directAPIServer: true` only when the `global`-cluster machines must reach the kube-apiserver directly at `https://:6443` instead of the ingress path (see Endpoint Rules); it defaults to `false`. +- Bare Metal provider AppRelease values set with the local `handoffHook.controlPlaneVIP`, `handoffHook.directAPIServer: true`, `elemental.systemAgent.authMode: shared`, `elemental.systemAgent.splitAuthEnabled: true`, and `elemental.systemAgent.globalServiceAccountName: baremetal-global-system-agent`. The active side sets `elemental.systemAgent.sharedAuthReadOnly: false`; the inactive side sets it to `true`. +- The deployed `elemental-operator` image and the `elemental-register` binary in `baremetal-base-image` and `baremetal-base-image-iso` are built from the same pinned elemental commit. Rebuild the base image and ISO whenever that commit changes; an operator-only upgrade does not update the host registration binary. -Do not sync `SeedImage` from primary to standby. Sync non-`global` `MachineRegistration` resources when workload ISOs need to remain valid across failover. Do not sync primary `global` `MachineRegistration` resources to standby; create them locally on the active side and set `baremetal.cluster.io/system-agent-server-url` to that side's control-plane VIP. +Do not sync `SeedImage` from primary to standby. Sync non-`global` `MachineRegistration` resources when workload ISOs need to remain valid across failover. Do not sync primary Global `MachineRegistration` resources to standby. Create Global registrations locally with `baremetal.cluster.io/system-agent-auth-scope: global`, `baremetal.cluster.io/system-agent-server-url: https://:6443`, and `baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver`. After failover, keep the primary fenced until a separate failback plan is executed. Do not run primary and standby Bare Metal providers against the same workload inventories at the same time. diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index 142659a0..c5e91057 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -532,10 +532,14 @@ Upload the packages. -r "${LOCAL_REGISTRY_ADDRESS}" -c "${KUBEADM_PROVIDER_PACK}" ``` + + The Bare Metal provider package deploys `elemental-operator`, while the Bare Metal base image and ISO contain the `elemental-register` binary used by each host. Build both artifacts from the same pinned `elemental-operator` commit. Rebuilding only the provider package is not sufficient when registration, endpoint, token, or CA handling changes. Rebuild `baremetal-base-image` and `baremetal-base-image-iso`, record the elemental commit and image digests, and verify that they match the deployed operator before booting the ISO. Do not rely on a mutable branch name or a cached base-image layer as proof of source consistency. + + Create and apply the AppRelease resources for the Kubeadm provider and the Bare Metal provider. - During bootstrap, the `global` cluster has not handed control to the final VIP yet. Keep the bare-metal registration path on the bootstrap host: `global.platformUrl` points to the bootstrap host, and `elemental.server.url` points to `https://:12443`. Do not set `baremetal.cluster.io/system-agent-server-url` on the `MachineRegistration` used for the `global` machines during this phase. The handoff job changes the `global` machines to `https:///kubernetes/global` after the platform installation completes. + During bootstrap, the `global` cluster has not handed control to the final VIP yet. Keep the bare-metal registration path on the bootstrap host: `global.platformUrl` points to the bootstrap host, and `elemental.server.url` points to `https://:12443`. Do not set `baremetal.cluster.io/system-agent-server-url` on the `MachineRegistration` used for the `global` machines during this phase. For Bare Metal DR, add `baremetal.cluster.io/system-agent-auth-scope: global` to that bootstrap `MachineRegistration`; this is the only DR-specific system-agent annotation required before handoff. The DR handoff job later moves those machines to the local direct API endpoint at `https://:6443`. Prepare the bootstrap HTTPS certificate used by `global-alb2` before you install the Bare Metal provider. The `elemental-operator` mounts the CA from `cpaas-system/dex.tls`, and `elemental-system-agent` uses that CA when `elemental.tls.agentTLSMode` is `strict`. The Secret must contain `tls.crt`, `tls.key`, and the CA bundle key that you configure in the AppRelease, shown below as `ca.crt`. The serving certificate must include `${HOST_IP}` as an IP SAN because the bootstrap endpoint is `https://${HOST_IP}:12443`. @@ -1150,11 +1154,11 @@ Use [Creating Clusters on Bare Metal](../create-cluster/bare-metal.mdx), [Managi - Put `${CONTROL_PLANE_VIP}` and `${PLATFORM_HOST}` in `KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.serverCertSANs`. - Set `BaremetalCluster.spec.controlPlaneLoadBalancer.host` to `${CONTROL_PLANE_VIP}`, `port` to `6443`, and use a `vrid` that is unique in the control-plane Layer-2 domain. - For a normal non-DR deployment, `BaremetalCluster.spec.encryptionProviderConfigRef` can be omitted. For DR, set it as described in [Optional Disaster Recovery Deployment](#optional-disaster-recovery-deployment); do not deliver `/etc/kubernetes/encryption-provider.conf` by adding it to `KubeadmControlPlane.spec.kubeadmConfigSpec.files`. -- Do not set `baremetal.cluster.io/system-agent-server-url` on the `MachineRegistration` used for the bootstrap `global` hosts. The bootstrap ISO must register through the bootstrap host; the handoff job later moves the `global` machines to the VIP. +- For Bare Metal DR, add `baremetal.cluster.io/system-agent-auth-scope: global` to the `MachineRegistration` used for the bootstrap `global` hosts. Do not set `baremetal.cluster.io/system-agent-server-url` during bootstrap. The bootstrap ISO must register through the bootstrap host; the handoff job later moves the `global` machines to the local direct API endpoint. - If a `global` VM or physical host does not have DHCP during the live-ISO boot, configure the NIC manually from the host console before waiting for `MachineInventory` registration. Use the same NetworkManager procedure described in [Creating Clusters on Bare Metal](../create-cluster/bare-metal.mdx#network-connectivity), replacing the example address, gateway, DNS, and connection name with the values for that host. - Do not depend on OS hostname side effects. The bare-metal provider normalizes kubeadm node names and provider IDs from the CAPI and inventory objects. - The `SeedImage` created in the bootstrap cluster is a bootstrap artifact. After handoff, create any new `MachineRegistration` or `SeedImage` on the active `global` cluster. -- When adding new machines to an already installed `global` cluster, explicitly set the `baremetal.cluster.io/system-agent-server-url` annotation on that `MachineRegistration` to the current active `global` control-plane VIP. New `global` machines do not pass through the bootstrap handoff job, so this annotation is what makes their system-agent watch plan Secrets through `https:///kubernetes/global`. Non-`global` workload-cluster machines should continue to use the platform domain path. +- Before adding a machine to an already installed DR `global` cluster, stop `etcd-sync` and follow [Adding a Global Host While Synchronization Is Configured](./disaster_recovery.mdx#adding-a-global-host-while-synchronization-is-configured) so the new plan Secret is added to the exact ignore list before synchronization resumes. Explicitly set all three annotations shown below. New `global` machines do not pass through the bootstrap handoff job, so they must use the local Global auth scope and the local direct kube-apiserver endpoint. Non-`global` workload-cluster machines continue to use the platform domain path at `https:///kubernetes/global`. ```yaml apiVersion: elemental.cattle.io/v1beta1 @@ -1163,9 +1167,13 @@ Use [Creating Clusters on Bare Metal](../create-cluster/bare-metal.mdx), [Managi name: global- namespace: cpaas-system annotations: - baremetal.cluster.io/system-agent-server-url: https:// + baremetal.cluster.io/system-agent-auth-scope: global + baremetal.cluster.io/system-agent-server-url: https://:6443 + baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver ``` + `baremetal.cluster.io/system-agent-direct: "true"` remains accepted only as an upgrade-compatibility alias for `direct-apiserver`. Do not add it to new manifests. If both annotations are present, the explicit `baremetal.cluster.io/system-agent-endpoint-mode` value takes precedence. + The following YAML is a differential fragment, not a complete manifest that you can apply directly. Merge these `global`-specific changes into the manifest that you prepare from the Bare Metal create-cluster references, then apply the complete manifest file. @@ -1842,7 +1850,7 @@ Issues that are not listed here usually point to environment-specific causes. Ca Use this section when you deploy primary and standby `global` clusters for disaster recovery. Complete these additions before you apply the provider-specific manifest for each `global` cluster. -Primary and standby clusters must use the same encryption provider configuration. For Bare Metal, primary and standby must also use the same Kubernetes ServiceAccount signing key so that the fixed `baremetal-system-agent` token created on the primary cluster is accepted by the standby API server after failover. For DCS and Bare Metal, the provider-specific cluster resource references a Secret that contains `encryption-provider.conf`; for HCS, normal non-DR deployments do not add `/etc/kubernetes/encryption-provider.conf` to `KubeadmControlPlane.spec.kubeadmConfigSpec.files`. VMware vSphere keeps the release manifest's `/etc/kubernetes/encryption-provider.conf` file entry. +Primary and standby clusters must use the same encryption provider configuration. For Bare Metal, primary and standby must also use the same Kubernetes ServiceAccount signing key so that the workload-cluster `baremetal-system-agent` token created on the primary cluster is accepted by the standby API server after failover. The machines that form each `global` cluster use a separate cluster-local identity and do not share their token with the peer `global` cluster. For DCS and Bare Metal, the provider-specific cluster resource references a Secret that contains `encryption-provider.conf`; for HCS, normal non-DR deployments do not add `/etc/kubernetes/encryption-provider.conf` to `KubeadmControlPlane.spec.kubeadmConfigSpec.files`. VMware vSphere keeps the release manifest's `/etc/kubernetes/encryption-provider.conf` file entry. ### Prepare Shared DR Variables @@ -2058,34 +2066,61 @@ Also keep the shared ServiceAccount signing key configuration from [Prepare Shar Create the Bare Metal `dcs-import-extra-resources` ConfigMap from Step 7 on both installation environments. The ConfigMap must import the bare-metal and elemental resources required by handoff, including the encryption-provider Secret referenced by `BaremetalCluster.spec.encryptionProviderConfigRef`, and must not import `SeedImage`. -The Bare Metal provider AppRelease on both sides must enable shared system-agent auth and handoff delivery: +The Bare Metal provider AppRelease on both sides must enable the Global-local identity, shared workload identity, and direct kube-apiserver handoff for Global machines: ```yaml handoffHook: controlPlaneVIP: + directAPIServer: true delivery: enabled: true mode: always - # directAPIServer: true # DR opt-in only. Points global-cluster machines at - # https://:6443 directly instead of the platform ingress - # path. Defaults to false. See "Bare Metal DR Model" in Disaster Recovery. elemental: systemAgent: authMode: shared + splitAuthEnabled: true serviceAccountName: baremetal-system-agent + globalServiceAccountName: baremetal-global-system-agent tls: agentTLSMode: strict caCertSecretName: dex.tls caCertSecretKey: ca.crt ``` -The generated `Role/cpaas-system/baremetal-system-agent` must restrict `secrets` access with `resourceNames` that contain only plan Secret names. Do not grant namespace-wide Secret access and do not include registry, bootstrap, or platform credential Secrets. +`elemental.systemAgent.splitAuthEnabled` is the single switch for both `elemental-operator` and the handoff job. Do not configure a separate handoff auth-enable switch. Keeping one source of truth prevents a Global host from retaining a shared token after the operator has already narrowed the shared Role to workload plans. + +Set the shared bundle ownership separately on the two AppRelease objects. The active primary reconciles workload permissions, while the inactive standby leaves the synchronized shared bundle read-only: + +```yaml title="Primary active Global" +elemental: + systemAgent: + sharedAuthReadOnly: false +``` + +```yaml title="Standby inactive Global" +elemental: + systemAgent: + sharedAuthReadOnly: true +``` + +The installation uses two non-overlapping system-agent permission sets: + +- `Role/cpaas-system/baremetal-global-system-agent` contains only plan Secret names for machines that belong to the local `Cluster/global`. Its ServiceAccount and token Secret are local to that `global` cluster and must not be synchronized to the peer. +- `Role/cpaas-system/baremetal-system-agent` contains only non-`global` workload plan Secret names. Its ServiceAccount, token Secret, Role, and RoleBinding are synchronized from primary to standby. + +Use namespaced `Role` and `RoleBinding` objects, not `ClusterRoleBinding`. Do not grant namespace-wide Secret access and do not include registry, bootstrap, cluster access credential, or platform credential Secrets. + + + Keep the Global-local objects named `baremetal-global-system-agent` and `baremetal-global-system-agent-token`, and keep the workload shared objects named `baremetal-system-agent` and `baremetal-system-agent-token`. The `etcd-sync` exact-sync and ignore rules use these literal names. Do not customize them unless you also update and validate every corresponding `etcd-sync` rule before synchronization starts. + + +The handoff completion signal is versioned. For this DR configuration, each final Global cluster must report `data.ready: "true"` and `data.profile: split-local-direct-v1` in `ConfigMap/cpaas-system/baremetal-system-agent-handoff`. A legacy ConfigMap that has only `ready: "true"`, or has a `legacy-shared-*` profile, does not satisfy the split-auth cleanup gate. Upgrading the provider reruns handoff against the current Global plan Secrets, migrates the installed Global hosts in place, and writes the new profile only after the new connection has been probed and the temporary shared access has been removed. Do not patch the profile manually. -### Install Primary and Standby Clusters +### Install Primary and Standby Clusters \{#install-primary-and-standby-clusters} Run Steps 1 through 9 for both the primary and standby `global` clusters. @@ -2097,13 +2132,98 @@ Use the provider-specific installer configuration differences for both sides: |----------|----------------------|----------------------| | Huawei DCS | Set `console.host` and `cluster.features.ha.vip` to the primary HA VIP. Create the DCS `dcs-import-extra-resources` ConfigMap from Step 7 and keep `PROVIDER_SECRET_NAME` aligned with `DCSCluster.spec.credentialSecretRef.name`. | Set `console.host` and `cluster.features.ha.vip` to the standby HA VIP. Create the DCS `dcs-import-extra-resources` ConfigMap from Step 7 and keep `PROVIDER_SECRET_NAME` aligned with `DCSCluster.spec.credentialSecretRef.name`. | | VMware vSphere | Set `VSphereCluster.spec.controlPlaneEndpoint.host` to the primary HA VIP used by the primary manifest. Create the VMware vSphere `dcs-import-extra-resources` ConfigMap from Step 7 and keep `global-vsphere-credentials` aligned with `VSphereCluster.spec.identityRef.name`. | Set `VSphereCluster.spec.controlPlaneEndpoint.host` to the standby HA VIP used by the standby manifest. Create the VMware vSphere `dcs-import-extra-resources` ConfigMap from Step 7 and keep `global-vsphere-credentials` aligned with `VSphereCluster.spec.identityRef.name`. | -| Huawei Cloud Stack | Keep `console.host: []`; the primary VIP is managed by the HCS ELB. Create the HCS `dcs-import-extra-resources` ConfigMap from Step 7 and keep `PROVIDER_SECRET_NAME` aligned with `HCSCluster.spec.identityRef.name`. | Keep `console.host: []`; the standby VIP is managed by the HCS ELB. Create the HCS `dcs-import-extra-resources` ConfigMap from Step 7 and keep `PROVIDER_SECRET_NAME` aligned with `HCSCluster.spec.identityRef.name`. | -| Bare Metal | Set `console.host`, `cluster.features.ha.vip`, `BaremetalCluster.spec.controlPlaneLoadBalancer.host`, and `handoffHook.controlPlaneVIP` to the primary control-plane VIP. Create the Bare Metal `dcs-import-extra-resources` ConfigMap from Step 7. | Set `console.host`, `cluster.features.ha.vip`, `BaremetalCluster.spec.controlPlaneLoadBalancer.host`, and `handoffHook.controlPlaneVIP` to the standby control-plane VIP. Create the Bare Metal `dcs-import-extra-resources` ConfigMap from Step 7. | +| Huawei Cloud Stack | Keep `console.host: []`; the primary VIP is managed by the HCS ELB. Create the HCS `dcs-import-extra-resources` ConfigMap from Step 7 and keep `HCS_SECRET_NAME` aligned with `HCSCluster.spec.identityRef.name`. | Keep `console.host: []`; the standby VIP is managed by the HCS ELB. Create the HCS `dcs-import-extra-resources` ConfigMap from Step 7 and keep `HCS_SECRET_NAME` aligned with `HCSCluster.spec.identityRef.name`. | +| Bare Metal | Set `console.host`, `cluster.features.ha.vip`, `BaremetalCluster.spec.controlPlaneLoadBalancer.host`, and `handoffHook.controlPlaneVIP` to the primary control-plane VIP. Set `handoffHook.directAPIServer: true`, `elemental.systemAgent.splitAuthEnabled: true`, and `elemental.systemAgent.sharedAuthReadOnly: false`. Create the Bare Metal `dcs-import-extra-resources` ConfigMap from Step 7. | Set `console.host`, `cluster.features.ha.vip`, `BaremetalCluster.spec.controlPlaneLoadBalancer.host`, and `handoffHook.controlPlaneVIP` to the standby control-plane VIP. Set `handoffHook.directAPIServer: true`, `elemental.systemAgent.splitAuthEnabled: true`, and `elemental.systemAgent.sharedAuthReadOnly: true`. Create the Bare Metal `dcs-import-extra-resources` ConfigMap from Step 7. | For the primary cluster, make sure the platform domain resolves to the primary HA VIP. In Step 8, set `hostIP` to the primary bootstrap host IP. For DCS, set `console.host` and `cluster.features.ha.vip` to the primary HA VIP. For VMware vSphere, set the control plane endpoint in the primary manifest to the primary HA VIP. For HCS, keep `console.host: []` because the VIP is owned by the HCS ELB. For Bare Metal, set both the manifest VIP and the installer VIP fields to the primary control-plane VIP. After the primary cluster installation succeeds, switch the platform domain to the standby HA VIP as required by the DR procedure. Then install the standby cluster. This DNS switch before the standby installation is required because several platform resources are rendered with the platform domain and must resolve to the standby entrance while the standby installer runs. In Step 8 on the standby bootstrap host, set `hostIP` to the standby bootstrap host IP. For DCS, set `console.host` and `cluster.features.ha.vip` to the standby HA VIP. For VMware vSphere, set the control plane endpoint in the standby manifest to the standby HA VIP. For HCS, keep `console.host: []`. For Bare Metal, set both the manifest VIP and the installer VIP fields to the standby control-plane VIP, and keep `REGISTRY_DOMAIN` as `${PLATFORM_HOST}:11443` on both sides. Get `INSTALLER_IP` from the `cpaas-installer` Pod on the standby bootstrap host; do not reuse the primary bootstrap host value. +For Bare Metal, verify the versioned handoff completion on both sides before decommissioning either bootstrap KIND cluster: + +```shell +kubectl --kubeconfig \ + -n cpaas-system get configmap baremetal-system-agent-handoff \ + -o jsonpath='{.data.ready}{" "}{.data.profile}{"\n"}' +kubectl --kubeconfig \ + -n cpaas-system get configmap baremetal-system-agent-handoff \ + -o jsonpath='{.data.ready}{" "}{.data.profile}{"\n"}' +``` + +Both commands must print `true split-local-direct-v1`. A bare `true`, a legacy profile, or a failed Global-side probe means the handoff is not complete. + +After the standby installation succeeds, switch the platform domain back to the primary entrance before installing `global-etcd-sync`. Primary remains active during initial synchronization, and workload system-agent endpoints use the platform domain. Verify that DNS and `https:///kubernetes/global/version` reach primary before continuing. + +Before you create the `global-etcd-sync` ModuleInfo, prepare the runtime exact-key ignore list. For providers other than Bare Metal, keep the list empty: + +```shell +export ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON='[]' +``` + +For Bare Metal, collect the local Global plan Secret names from both Global clusters and convert them to exact etcd keys. Run these commands from a host that has `jq` and two kubeconfigs. Each kubeconfig must address its own Global API server; do not rely on the platform domain, because that domain points to only one side at a time. If one kubeconfig contains both clusters, use two distinct contexts in the equivalent `kubectl --context` commands. + +```shell +export PRIMARY_GLOBAL_KUBECONFIG="" +export STANDBY_GLOBAL_KUBECONFIG="" +export PRIMARY_GLOBAL_CONTEXT="$(kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" config current-context)" +export STANDBY_GLOBAL_CONTEXT="$(kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" config current-context)" + +PRIMARY_GLOBAL_SERVER="$(kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" \ + --context "${PRIMARY_GLOBAL_CONTEXT}" config view --minify \ + -o jsonpath='{.clusters[0].cluster.server}')" +STANDBY_GLOBAL_SERVER="$(kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + --context "${STANDBY_GLOBAL_CONTEXT}" config view --minify \ + -o jsonpath='{.clusters[0].cluster.server}')" + +if [ "${PRIMARY_GLOBAL_SERVER}" = "${STANDBY_GLOBAL_SERVER}" ]; then + echo "primary and standby kubeconfigs resolve to the same API server" >&2 + exit 1 +fi + +global_plan_secret_names() { + kubeconfig=$1 + context=$2 + kubectl --kubeconfig "${kubeconfig}" --context "${context}" \ + -n cpaas-system get role baremetal-global-system-agent -o json | \ + jq -c '[ + .rules[]? + | select(any(.apiGroups[]?; . == "")) + | select(any(.resources[]?; . == "secrets")) + | .resourceNames[]? + ] | unique | sort' +} + +PRIMARY_GLOBAL_PLAN_SECRET_NAMES="$(global_plan_secret_names \ + "${PRIMARY_GLOBAL_KUBECONFIG}" "${PRIMARY_GLOBAL_CONTEXT}")" +STANDBY_GLOBAL_PLAN_SECRET_NAMES="$(global_plan_secret_names \ + "${STANDBY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_CONTEXT}")" + +printf '%s\n' "${PRIMARY_GLOBAL_PLAN_SECRET_NAMES}" | \ + jq -e 'length > 0' >/dev/null || { + echo "primary Role/baremetal-global-system-agent has no plan resourceNames" >&2 + exit 1 + } +printf '%s\n' "${STANDBY_GLOBAL_PLAN_SECRET_NAMES}" | \ + jq -e 'length > 0' >/dev/null || { + echo "standby Role/baremetal-global-system-agent has no plan resourceNames" >&2 + exit 1 + } + +export ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON="$( + jq -cn \ + --argjson primary "${PRIMARY_GLOBAL_PLAN_SECRET_NAMES}" \ + --argjson standby "${STANDBY_GLOBAL_PLAN_SECRET_NAMES}" \ + '($primary + $standby) + | unique + | sort + | map("/registry/secrets/cpaas-system/" + .)' +)" + +printf '%s\n' "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" | jq . +``` + +Do not install `etcd-sync` if either local Global Role has no plan `resourceNames`, or if the generated array omits a plan Secret from either side. The list is merged and de-duplicated because the broad Secret sync prefix would otherwise overwrite a local Global plan Secret whose name happens to exist in the source or target. Repeat the collection and update the ModuleInfo configuration before synchronization continues whenever either local Global Role gains a new plan Secret. + After both clusters are installed, get the primary `k8sadmin` token on a primary control plane node. `etcd-sync` is installed only on the standby cluster, and its `active_cluster_*` values point to the primary cluster. Keep this value in its original base64 Secret form for `active_cluster_token`. ```shell @@ -2143,7 +2263,8 @@ cat > "${ETCD_SYNC_MODULEINFO}" < Date: Tue, 14 Jul 2026 04:19:02 +0800 Subject: [PATCH 02/14] docs: harden baremetal Global split auth handoff --- docs/en/global/disaster_recovery.mdx | 26 +++++++++----- docs/en/global/install.mdx | 53 +++++++++++++++++++++++++--- 2 files changed, 65 insertions(+), 14 deletions(-) diff --git a/docs/en/global/disaster_recovery.mdx b/docs/en/global/disaster_recovery.mdx index f59d2176..3f5bb451 100644 --- a/docs/en/global/disaster_recovery.mdx +++ b/docs/en/global/disaster_recovery.mdx @@ -105,25 +105,25 @@ Set `elemental.systemAgent.splitAuthEnabled: true` on both provider AppRelease o ### Migrating Existing Handoff State -`ConfigMap/cpaas-system/baremetal-system-agent-handoff` is a versioned completion record. The direct-API split-auth configuration requires both `data.ready: "true"` and `data.profile: split-local-direct-v1`. A ConfigMap written by an older release with only `ready: "true"`, or with `legacy-shared-erebus-v1` or `legacy-shared-direct-v1`, must not cause the new handoff to skip. +`ConfigMap/cpaas-system/baremetal-system-agent-handoff` is a versioned completion record. The direct-API split-auth configuration requires `data.ready: "true"`, `data.profile: split-local-direct-v1`, and a `data.config` value that is exactly 64 lowercase hexadecimal characters. The digest covers the effective endpoint, token/CA, identity names, and each Global target's names and object UIDs. A ConfigMap written by an older release with only `ready: "true"`, with a `legacy-shared-*` profile, or with a stale, missing, or malformed config digest must not cause the new handoff to skip. The upgrade migrates the already installed Global hosts in place; it does not require the original bootstrap plan Secret. Keep this safety order for every current Global plan Secret: -1. Create and validate the local Global ServiceAccount, token Secret, Role, and RoleBinding, while preserving the old shared identity's access to the same current Global plan Secret. +1. Create and validate the local Global ServiceAccount, token Secret, Role, and canonical RoleBinding. Create the independent temporary `RoleBinding/baremetal-global-system-agent-handoff-bridge`, which references the Global-local Role and binds only the old shared ServiceAccount. Do not modify the canonical RoleBinding or the synchronized shared Role. 2. Write the handoff plan on that current Global plan Secret so the host switches to the local token and local direct API endpoint. 3. Wait until the handoff checksum is Applied, then write and observe a Global-side probe with the new local identity. -4. Only after the probe succeeds, remove the old shared identity's temporary access to the Global plan and reconcile the shared Role back to workload-only plan Secrets. -5. Treat migration as complete only when all hosts have passed the probe, the permission scopes are narrowed, and the completion ConfigMap reports the exact new profile. +4. Only after the probe succeeds, delete the temporary bridge. Workload targets and the shared Role are not rewritten by this migration. +5. Treat migration as complete only when all Global hosts have passed the probe, the bridge is absent, and the completion ConfigMap reports the exact new profile and config digest. -This temporary overlap is deliberate: narrowing the shared permission before the old host has consumed its token-switch plan can strand the host with `403 Forbidden`. Do not grant namespace-wide Secret access, do not use a `ClusterRoleBinding`, and do not manually patch the completion profile to bypass the migration. +This temporary overlap is deliberate: removing the bridge before the old host has consumed its token-switch plan can strand the host with `403 Forbidden`. Do not grant namespace-wide Secret access, do not use a `ClusterRoleBinding`, and do not manually patch completion fields to bypass the migration. ```shell kubectl --kubeconfig \ -n cpaas-system get configmap baremetal-system-agent-handoff \ - -o jsonpath='{.data.ready}{" "}{.data.profile}{"\n"}' + -o jsonpath='{.data.ready}{" "}{.data.profile}{" "}{.data.config}{"\n"}' ``` -Expected output: `true split-local-direct-v1`. +Expected output: `true split-local-direct-v1 <64-lowercase-hex-character-SHA-256>`. Verify both permission boundaries before failover. Repeat the checks on primary and standby with an administrator kubeconfig that is allowed to impersonate ServiceAccounts. Set the two example Secret names from that same cluster. @@ -201,7 +201,7 @@ Do not synchronize: - Standby `Cluster/cpaas-system/global`. - Standby `BaremetalCluster/cpaas-system/global`. - Standby `BaremetalMachine`, `MachineInventoryPool`, `MachineInventory`, and `MachineRegistration` objects that belong to the standby `global` cluster. -- `ServiceAccount/cpaas-system/baremetal-global-system-agent`, `Secret/cpaas-system/baremetal-global-system-agent-token`, `Role/cpaas-system/baremetal-global-system-agent`, and `RoleBinding/cpaas-system/baremetal-global-system-agent` from either side. +- `ServiceAccount/cpaas-system/baremetal-global-system-agent`, `Secret/cpaas-system/baremetal-global-system-agent-token`, `Role/cpaas-system/baremetal-global-system-agent`, `RoleBinding/cpaas-system/baremetal-global-system-agent`, and the temporary `RoleBinding/cpaas-system/baremetal-global-system-agent-handoff-bridge` from either side. - Plan Secrets that belong to either side's local `Cluster/global`. Add their exact etcd keys to `etcd_sync.additional_ignore_equal` before starting synchronization. - Secrets from the primary cluster that are not required by workload cluster reconciliation. @@ -279,7 +279,15 @@ Use this sequence for a planned or declared failover: -n cpaas-system get machineinventories.elemental.cattle.io -o json | \ jq -c '[ .items[] - | select((.metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] // "shared") == "shared") + | . as $inventory + | ( + $inventory.metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] // + (if (($inventory.metadata.annotations["baremetal.alauda.io/owner-cluster"] // "") | ascii_downcase) == "global" + then "global" + else "shared" + end) + ) as $scope + | select(($scope | ascii_downcase) == "shared") | .status.plan.secretRef.name? // empty ] | unique | sort' )" diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index c5e91057..e7a6f8f0 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -757,6 +757,9 @@ spec: replicas: 1 scheme: https handoffHook: + # Normal non-DR bootstrap baseline. Bare Metal DR must patch this to true + # before calling the installer API, as shown in Optional Disaster Recovery Deployment. + directAPIServer: false controlPlaneVIP: ${CONTROL_PLANE_VIP} delivery: enabled: true @@ -766,7 +769,12 @@ spec: url: https://${HOST_IP}:12443 systemAgent: authMode: shared + # Normal non-DR bootstrap baseline. Bare Metal DR must enable the split + # and set sharedAuthReadOnly according to this side's active/standby role. + splitAuthEnabled: false serviceAccountName: baremetal-system-agent + globalServiceAccountName: baremetal-global-system-agent + sharedAuthReadOnly: false tls: agentTLSMode: strict caCertSecretName: dex.tls @@ -788,6 +796,8 @@ until kubectl get crd machineinventories.elemental.cattle.io --ignore-not-found done ``` +The complete AppRelease above intentionally shows the normal, non-DR baseline. For a Bare Metal DR installation, do not call the installer API with those three baseline values unchanged. Apply the role-specific DR patch in [Optional Disaster Recovery Deployment](#optional-disaster-recovery-deployment) first, and verify the effective AppRelease values on that bootstrap cluster. + After the provider starts, verify that the chart values were accepted. A CrashLoopBackOff with an unknown flag such as `--system-agent-auth-mode` means the AppRelease chart and the elemental-operator image do not match; install a chart and image from the same release payload before continuing. ```shell @@ -2066,7 +2076,40 @@ Also keep the shared ServiceAccount signing key configuration from [Prepare Shar Create the Bare Metal `dcs-import-extra-resources` ConfigMap from Step 7 on both installation environments. The ConfigMap must import the bare-metal and elemental resources required by handoff, including the encryption-provider Secret referenced by `BaremetalCluster.spec.encryptionProviderConfigRef`, and must not import `SeedImage`. -The Bare Metal provider AppRelease on both sides must enable the Global-local identity, shared workload identity, and direct kube-apiserver handoff for Global machines: +The Bare Metal provider AppRelease on both sides must enable the Global-local identity, shared workload identity, and direct kube-apiserver handoff for Global machines. Apply one of the following patches on each side's bootstrap cluster after creating the AppRelease and before calling the installer API. + +On the primary bootstrap cluster: + +```shell +kubectl -n cpaas-system patch apprelease cluster-api-provider-baremetal \ + --type=merge \ + -p '{"spec":{"values":{"handoffHook":{"directAPIServer":true},"elemental":{"systemAgent":{"authMode":"shared","splitAuthEnabled":true,"serviceAccountName":"baremetal-system-agent","globalServiceAccountName":"baremetal-global-system-agent","sharedAuthReadOnly":false}}}}}' +``` + +On the standby bootstrap cluster: + +```shell +kubectl -n cpaas-system patch apprelease cluster-api-provider-baremetal \ + --type=merge \ + -p '{"spec":{"values":{"handoffHook":{"directAPIServer":true},"elemental":{"systemAgent":{"authMode":"shared","splitAuthEnabled":true,"serviceAccountName":"baremetal-system-agent","globalServiceAccountName":"baremetal-global-system-agent","sharedAuthReadOnly":true}}}}}' +``` + +Verify the effective values on each bootstrap cluster: + +```shell +export EXPECTED_SHARED_AUTH_READ_ONLY=false # Use true on the standby bootstrap cluster. + +kubectl -n cpaas-system get apprelease cluster-api-provider-baremetal -o json | \ + jq -e --argjson expected_read_only "${EXPECTED_SHARED_AUTH_READ_ONLY}" ' + .spec.values.handoffHook.directAPIServer == true and + .spec.values.elemental.systemAgent.splitAuthEnabled == true and + .spec.values.elemental.systemAgent.sharedAuthReadOnly == $expected_read_only + ' +``` + +Do not continue until this command succeeds. The primary must use `sharedAuthReadOnly: false`; the standby must use `sharedAuthReadOnly: true` while primary-to-standby synchronization is active. + +The resulting DR values are: ```yaml handoffHook: @@ -2115,7 +2158,7 @@ Use namespaced `Role` and `RoleBinding` objects, not `ClusterRoleBinding`. Do no Keep the Global-local objects named `baremetal-global-system-agent` and `baremetal-global-system-agent-token`, and keep the workload shared objects named `baremetal-system-agent` and `baremetal-system-agent-token`. The `etcd-sync` exact-sync and ignore rules use these literal names. Do not customize them unless you also update and validate every corresponding `etcd-sync` rule before synchronization starts. -The handoff completion signal is versioned. For this DR configuration, each final Global cluster must report `data.ready: "true"` and `data.profile: split-local-direct-v1` in `ConfigMap/cpaas-system/baremetal-system-agent-handoff`. A legacy ConfigMap that has only `ready: "true"`, or has a `legacy-shared-*` profile, does not satisfy the split-auth cleanup gate. Upgrading the provider reruns handoff against the current Global plan Secrets, migrates the installed Global hosts in place, and writes the new profile only after the new connection has been probed and the temporary shared access has been removed. Do not patch the profile manually. +The handoff completion signal is versioned. For this DR configuration, each final Global cluster must report `data.ready: "true"`, `data.profile: split-local-direct-v1`, and a `data.config` value that is exactly 64 lowercase hexadecimal characters in `ConfigMap/cpaas-system/baremetal-system-agent-handoff`. The digest covers the effective endpoint, token/CA, identity names, and each Global target's names and object UIDs. A ready-only ConfigMap, a `legacy-shared-*` profile, or a stale, missing, or malformed config digest does not satisfy the cleanup gate. Upgrading the provider migrates only installed Global hosts against their current Global plan Secrets. It uses the independent temporary `baremetal-global-system-agent-handoff-bridge` RoleBinding and writes the new profile/config only after the new connection has been probed and the bridge has been deleted. Do not patch completion fields manually. @@ -2144,13 +2187,13 @@ For Bare Metal, verify the versioned handoff completion on both sides before dec ```shell kubectl --kubeconfig \ -n cpaas-system get configmap baremetal-system-agent-handoff \ - -o jsonpath='{.data.ready}{" "}{.data.profile}{"\n"}' + -o jsonpath='{.data.ready}{" "}{.data.profile}{" "}{.data.config}{"\n"}' kubectl --kubeconfig \ -n cpaas-system get configmap baremetal-system-agent-handoff \ - -o jsonpath='{.data.ready}{" "}{.data.profile}{"\n"}' + -o jsonpath='{.data.ready}{" "}{.data.profile}{" "}{.data.config}{"\n"}' ``` -Both commands must print `true split-local-direct-v1`. A bare `true`, a legacy profile, or a failed Global-side probe means the handoff is not complete. +Both commands must print `true split-local-direct-v1 <64-lowercase-hex-character-SHA-256>`. A bare `true`, a legacy profile, an empty or malformed config digest, a remaining migration bridge, or a failed Global-side probe means the handoff is not complete. After the standby installation succeeds, switch the platform domain back to the primary entrance before installing `global-etcd-sync`. Primary remains active during initial synchronization, and workload system-agent endpoints use the platform domain. Verify that DNS and `https:///kubernetes/global/version` reach primary before continuing. From 8e014bf190942816cfdc47116db21f1230166d06 Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 09:39:36 +0800 Subject: [PATCH 03/14] docs: complete baremetal Global split auth DR procedure --- docs/en/global/disaster_recovery.mdx | 511 +++++++++++++++++- docs/en/global/install.mdx | 757 ++++++++++++++++++++++++++- 2 files changed, 1228 insertions(+), 40 deletions(-) diff --git a/docs/en/global/disaster_recovery.mdx b/docs/en/global/disaster_recovery.mdx index 3f5bb451..99d728a3 100644 --- a/docs/en/global/disaster_recovery.mdx +++ b/docs/en/global/disaster_recovery.mdx @@ -103,10 +103,160 @@ Primary and standby must still use the same `/etc/kubernetes/pki/sa.key`, `/etc/ Set `elemental.systemAgent.splitAuthEnabled: true` on both provider AppRelease objects. This is the single source of truth used by both `elemental-operator` and handoff; do not configure a separate handoff auth-enable switch. With the split enabled, the active Global must set `elemental.systemAgent.sharedAuthReadOnly: false` so that `elemental-operator` reconciles the shared workload Role. The inactive Global must set it to `true` so that `etcd-sync`, not the local operator, owns the synchronized shared bundle. Both operators continue to reconcile their own Global-local bundles. -### Migrating Existing Handoff State +### Migrating Existing Handoff State \{#migrating-existing-handoff-state} `ConfigMap/cpaas-system/baremetal-system-agent-handoff` is a versioned completion record. The direct-API split-auth configuration requires `data.ready: "true"`, `data.profile: split-local-direct-v1`, and a `data.config` value that is exactly 64 lowercase hexadecimal characters. The digest covers the effective endpoint, token/CA, identity names, and each Global target's names and object UIDs. A ConfigMap written by an older release with only `ready: "true"`, with a `legacy-shared-*` profile, or with a stale, missing, or malformed config digest must not cause the new handoff to skip. +#### Protect Existing Global Plan Secrets Before the Upgrade + +If `etcd-sync` is already installed, protect the current Global plan Secrets before upgrading either Bare Metal provider AppRelease or running the new handoff. An older environment does not yet have the authoritative `Role/baremetal-global-system-agent`, so derive this temporary exact-ignore set from the current `Cluster/global` `BaremetalMachine` references and cross-check every name against its `MachineInventory`. + +First stop or suspend the mirror with the supported plugin lifecycle operation, and verify that `Deployment/cpaas-system/etcd-sync` has no Ready mirror Pods. Then run the following from a secured host with both direct Global kubeconfigs: + +```shell +set -euo pipefail + +export PRIMARY_GLOBAL_KUBECONFIG="" +export STANDBY_GLOBAL_KUBECONFIG="" +export ETCD_SYNC_MODULEINFO_NAME="" + +existing_global_plan_secret_names() { + local kubeconfig=$1 targets names target inventory expected actual secret_type + + targets="$( + kubectl --kubeconfig "${kubeconfig}" \ + -n cpaas-system \ + get baremetalmachines.infrastructure.cluster.x-k8s.io -o json | \ + jq -c '[ + .items[] + | select(.metadata.labels["cluster.x-k8s.io/cluster-name"] == "global") + | { + inventory: (.status.machineInventoryRef.name // ""), + plan: (.status.planSecretRef.name // "") + } + ]' + )" + + printf '%s\n' "${targets}" | \ + jq -e 'length > 0 and all(.[]; .inventory != "" and .plan != "")' \ + >/dev/null || { + echo "Cluster/global has an incomplete inventory or plan reference" >&2 + return 1 + } + + names="$( + while IFS= read -r target; do + inventory="$(printf '%s\n' "${target}" | jq -r '.inventory')" + expected="$(printf '%s\n' "${target}" | jq -r '.plan')" + actual="$( + kubectl --kubeconfig "${kubeconfig}" \ + -n cpaas-system \ + get machineinventory.elemental.cattle.io "${inventory}" \ + -o jsonpath='{.status.plan.secretRef.name}' + )" + if [ "${actual}" != "${expected}" ]; then + echo "plan Secret mismatch for MachineInventory ${inventory}" >&2 + return 1 + fi + secret_type="$( + kubectl --kubeconfig "${kubeconfig}" \ + -n cpaas-system get secret "${actual}" -o jsonpath='{.type}' + )" + if [ "${secret_type}" != "elemental.cattle.io/plan" ]; then + echo "Secret ${actual} is not an elemental plan" >&2 + return 1 + fi + printf '%s\n' "${actual}" + done < <(printf '%s\n' "${targets}" | jq -c '.[]') + )" + + printf '%s\n' "${names}" | \ + jq -Rsc 'split("\n") | map(select(length > 0)) | unique | sort' +} + +PRIMARY_EXISTING_GLOBAL_PLAN_SECRET_NAMES="$( + existing_global_plan_secret_names "${PRIMARY_GLOBAL_KUBECONFIG}" +)" +STANDBY_EXISTING_GLOBAL_PLAN_SECRET_NAMES="$( + existing_global_plan_secret_names "${STANDBY_GLOBAL_KUBECONFIG}" +)" + +CURRENT_ADDITIONAL_IGNORE_EQUAL_JSON="$( + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + get moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" -o json | \ + jq -c '.spec.config.additional_ignore_equal // []' +)" + +export DISCOVERED_GLOBAL_PLAN_IGNORE_EQUAL_JSON="$( + jq -cn \ + --argjson primary "${PRIMARY_EXISTING_GLOBAL_PLAN_SECRET_NAMES}" \ + --argjson standby "${STANDBY_EXISTING_GLOBAL_PLAN_SECRET_NAMES}" ' + ($primary + $standby) + | unique + | sort + | map("/registry/secrets/cpaas-system/" + .) + ' +)" + +export TEMPORARY_GLOBAL_PLAN_IGNORE_EQUAL_JSON="$( + jq -cn \ + --argjson current "${CURRENT_ADDITIONAL_IGNORE_EQUAL_JSON}" \ + --argjson discovered "${DISCOVERED_GLOBAL_PLAN_IGNORE_EQUAL_JSON}" \ + '($current + $discovered) | unique | sort' +)" + +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + patch moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" \ + --type=merge \ + -p "$( + jq -cn --argjson keys "${TEMPORARY_GLOBAL_PLAN_IGNORE_EQUAL_JSON}" \ + '{spec:{config:{additional_ignore_equal:$keys}}}' + )" +``` + +Do not resume synchronization yet. Wait until `ConfigMap/cpaas-system/etcd-sync-ignore-text` contains every temporary exact key. Then restart the monitor and restore or roll out the mirror with the supported plugin lifecycle operation, and wait for both Deployments as described in [etcd-sync Scope](#etcd-sync-scope). No Ready mirror Pod may start before the rendered filter contains the complete temporary set. Keep this set in place throughout both AppRelease upgrades and handoff migrations. Without it, the broad Secret synchronization rule can delete, replace, or roll back the plan that a Global host must apply. + +#### Re-run the Handoff After the AppRelease Upgrade + +Upgrade the Bare Metal provider AppRelease on each side's retained bootstrap KIND cluster with `handoffHook.delivery.enabled: true` and `handoffHook.delivery.mode: always`, in addition to the split-auth values for that side. The chart's Helm `post-upgrade` Job delivers the current `acp-post-install` script and chart environment; it does not execute the installer post-install stage again. + +Also do not assume that the delivery Job changed the physical bootstrap host's `/cpaas`. In a nested KIND environment, its `hostPath: /cpaas` can refer to the KIND node filesystem. The reliable upgrade procedure is to read the official script and environment from the chart-rendered ConfigMap and explicitly run that script on the physical bootstrap host: + +```shell +set -euo pipefail + +export BOOTSTRAP_KUBECTL=/opt/kubectl +export BOOTSTRAP_KUBECONFIG=/cpaas/data/alauda.kubeconfig +export FINAL_GLOBAL_KUBECONFIG=/cpaas/data/admin.kubeconfig +export HANDOFF_RERUN_DIR=/root/baremetal-handoff-rerun + +sudo install -d -m 0700 "${HANDOFF_RERUN_DIR}" + +sudo "${BOOTSTRAP_KUBECTL}" \ + --kubeconfig "${BOOTSTRAP_KUBECONFIG}" \ + -n cpaas-system get configmap baremetal-acp-post-install \ + -o go-template='{{ index .data "acp-post-install" }}' | \ +sudo install -m 0700 /dev/stdin "${HANDOFF_RERUN_DIR}/acp-post-install" + +sudo "${BOOTSTRAP_KUBECTL}" \ + --kubeconfig "${BOOTSTRAP_KUBECONFIG}" \ + -n cpaas-system get configmap baremetal-acp-post-install \ + -o go-template='{{ index .data "baremetal-handoff.env" }}' | \ +sudo install -m 0600 /dev/stdin \ + "${HANDOFF_RERUN_DIR}/baremetal-handoff-chart.env" + +sudo env \ + KUBECTL="${BOOTSTRAP_KUBECTL}" \ + BOOTSTRAP_KUBECONFIG="${BOOTSTRAP_KUBECONFIG}" \ + GLOBAL_KUBECONFIG="${FINAL_GLOBAL_KUBECONFIG}" \ + BAREMETAL_HANDOFF_CHART_ENV_FILE="${HANDOFF_RERUN_DIR}/baremetal-handoff-chart.env" \ + "${HANDOFF_RERUN_DIR}/acp-post-install" +``` + +Run this separately on the primary and standby bootstrap hosts. Use the ConfigMap from that same side's upgraded AppRelease; do not copy a rendered environment from the peer. The script deliberately replaces a completed, failed, or still-running `baremetal-system-agent-handoff` Job with the current image and arguments. The handoff binary then compares the versioned profile and configuration digest, so a matching configuration exits without rewriting hosts while an old or drifted configuration performs the required in-place migration. Do not write a replacement Job or RBAC manifest by hand. + +This chart-delivered migration path requires the original side-specific bootstrap KIND cluster and its valid bootstrap and final-Global kubeconfigs. If that bootstrap environment has already been removed, stop and use an approved support procedure; installing the AppRelease only on the final Global cluster does not recreate the installer execution context. + The upgrade migrates the already installed Global hosts in place; it does not require the original bootstrap plan Secret. Keep this safety order for every current Global plan Secret: 1. Create and validate the local Global ServiceAccount, token Secret, Role, and canonical RoleBinding. Create the independent temporary `RoleBinding/baremetal-global-system-agent-handoff-bridge`, which references the Global-local Role and binds only the old shared ServiceAccount. Do not modify the canonical RoleBinding or the synchronized shared Role. @@ -125,6 +275,76 @@ kubectl --kubeconfig \ Expected output: `true split-local-direct-v1 <64-lowercase-hex-character-SHA-256>`. +After both sides report the expected completion and neither side has the temporary bridge, replace the temporary pre-upgrade list with the authoritative list generated from both local Global Roles: + +```shell +global_role_plan_secret_names() { + local kubeconfig=$1 + kubectl --kubeconfig "${kubeconfig}" \ + -n cpaas-system get role baremetal-global-system-agent -o json | \ + jq -c '[ + .rules[]? + | select(any(.apiGroups[]?; . == "")) + | select(any(.resources[]?; . == "secrets")) + | .resourceNames[]? + ] | unique | sort' +} + +PRIMARY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES="$( + global_role_plan_secret_names "${PRIMARY_GLOBAL_KUBECONFIG}" +)" +STANDBY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES="$( + global_role_plan_secret_names "${STANDBY_GLOBAL_KUBECONFIG}" +)" + +printf '%s\n' "${PRIMARY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES}" | \ + jq -e 'length > 0' >/dev/null +printf '%s\n' "${STANDBY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES}" | \ + jq -e 'length > 0' >/dev/null + +AUTHORITATIVE_GLOBAL_PLAN_IGNORE_EQUAL_JSON="$( + jq -cn \ + --argjson primary "${PRIMARY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES}" \ + --argjson standby "${STANDBY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES}" ' + ($primary + $standby) + | unique + | sort + | map("/registry/secrets/cpaas-system/" + .) + ' +)" + +CURRENT_ADDITIONAL_IGNORE_EQUAL_JSON="$( + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + get moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" -o json | \ + jq -c '.spec.config.additional_ignore_equal // []' +)" + +FINAL_ADDITIONAL_IGNORE_EQUAL_JSON="$( + jq -cn \ + --argjson current "${CURRENT_ADDITIONAL_IGNORE_EQUAL_JSON}" \ + --argjson temporary "${DISCOVERED_GLOBAL_PLAN_IGNORE_EQUAL_JSON}" \ + --argjson authoritative "${AUTHORITATIVE_GLOBAL_PLAN_IGNORE_EQUAL_JSON}" ' + (($current - $temporary) + $authoritative) | unique | sort + ' +)" + +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + patch moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" \ + --type=merge \ + -p "$( + jq -cn --argjson keys "${FINAL_ADDITIONAL_IGNORE_EQUAL_JSON}" \ + '{spec:{config:{additional_ignore_equal:$keys}}}' + )" +``` + +If the shell that performed the pre-upgrade collection is no longer available, re-run `existing_global_plan_secret_names` to reconstruct `DISCOVERED_GLOBAL_PLAN_IGNORE_EQUAL_JSON` before calculating the final list. Review any other custom entries separately. The final configuration must ensure that: + +- the Global plan entries are exactly the merged `resourceNames` from both `Role/baremetal-global-system-agent` objects; +- unrelated, explicitly approved additional ignore entries are preserved; and +- any temporary Global plan entry that is absent from both local Roles is removed. + +Verify the rendered ignore ConfigMap again and restart both `etcd-sync` Deployments before synchronization continues. From this point forward, the two local Roles, not the legacy BaremetalMachine list, are the authoritative source for Global plan exact-ignore entries. + Verify both permission boundaries before failover. Repeat the checks on primary and standby with an administrator kubeconfig that is allowed to impersonate ServiceAccounts. Set the two example Secret names from that same cluster. ```shell @@ -177,7 +397,7 @@ kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ Expected result: `get` and `patch` are allowed only within the matching permission scope. Every cross-scope check, both namespace-wide `list secrets` checks, and both `global-registry-auth` checks return `no`. Also compare the token hash in `/var/lib/elemental/agent/elemental_connection.json` on every Global host with the local `baremetal-global-system-agent-token` Secret; checking only the Kubernetes Secret does not prove that an existing host uses the expected token. -### etcd-sync Scope +### etcd-sync Scope \{#etcd-sync-scope} Deploy `etcd-sync` on the standby cluster. The sync direction is one-way before failover: @@ -256,9 +476,12 @@ Do not guess a plan Secret name from the inventory name. Read it from the local Use this sequence for a planned or declared failover: -1. Fence the primary `global` write path. Stop or isolate the controllers and platform entry points that can continue writing workload cluster resources. Do this at the owning CR or platform-resource layer where possible; scaling a Deployment alone can be reverted by its owner. -2. Wait until primary-to-standby `etcd-sync` is caught up, then freeze or stop it. Do not allow old primary data to overwrite new standby writes after failover. -3. On the standby that is becoming active, update the Bare Metal provider AppRelease to `elemental.systemAgent.sharedAuthReadOnly: false` and wait for `Deployment/cpaas-system/elemental-operator` to roll out. Verify that its arguments no longer contain `--system-agent-shared-auth-read-only`, and verify that `Role/cpaas-system/baremetal-system-agent.resourceNames` matches the workload-scoped `MachineInventory.status.plan.secretRef.name` set. This confirms that the new active operator has taken ownership of shared RBAC. +1. Stop new writes through the old active platform entrance, but keep the old active etcd endpoint available for the final synchronization check. Stop or isolate the owning platform resources that can restore an entry-point Deployment; scaling a Deployment alone is not a fence because its owner can recreate it. +2. On the old active Global cluster, update only the final Bare Metal provider AppRelease to `elemental.systemAgent.sharedAuthReadOnly: true`. Do not update the retained bootstrap AppRelease during the failover window. Wait for `Deployment/cpaas-system/elemental-operator` to roll out and verify that its arguments contain `--system-agent-shared-auth-read-only`. The new active side must still be read-only at this point, creating an intentional no-writer window for the shared bundle. +3. Run the final `etcd-sync-monitor /check` and the separate shared ServiceAccount/token/RBAC semantic checks. Require zero missed and surplus keys. Immediately after that check, hard-fence the old active control plane so it cannot create any later writes. A hard fence must have an external readback, such as all old active control-plane VMs being powered off with autostart disabled, or equivalent API and etcd network isolation. DNS changes and controller scale-to-zero operations are not hard fences. +4. Uninstall `etcd-sync` through its owning ModuleInfo. Do not delete only the AppRelease, scale the mirror, clear `ConfigMap/kube-public/etcd-sync`, or remove finalizers. Wait until the ModuleInfo, `AppRelease/cpaas-system/etcd-sync`, both `etcd-sync` Deployments, and `ConfigMap/kube-public/etcd-sync` are all absent. +5. Switch the platform domain to the new active Global entrance and wait for both the platform endpoint and `${PLATFORM_HOST}:11443` to resolve to and answer from that side. This step must be after the plugin uninstall because the plugin DNS checker must not resolve its source domain to the target cluster itself. It must be before changing the new active AppRelease when that AppRelease downloads its chart from the platform-domain registry; otherwise the ownership update cannot render after the old active registry is fenced. +6. On the Global cluster that is becoming active, update only the final Bare Metal provider AppRelease to `elemental.systemAgent.sharedAuthReadOnly: false` and wait for `Deployment/cpaas-system/elemental-operator` to roll out. Verify that its arguments no longer contain `--system-agent-shared-auth-read-only`, and verify that `Role/cpaas-system/baremetal-system-agent.resourceNames` matches the workload-scoped `MachineInventory.status.plan.secretRef.name` set. This confirms that the new active operator has taken ownership of shared RBAC. ```shell export NEW_ACTIVE_GLOBAL_KUBECONFIG="" @@ -308,25 +531,28 @@ Use this sequence for a planned or declared failover: fi ``` -4. Before the former primary is returned to service as the inactive peer or used as the target of reverse synchronization, set its AppRelease to `elemental.systemAgent.sharedAuthReadOnly: true`. It must not reconcile the shared bundle while it is the new standby. -5. If your provider uses a sync gate, clear the gate after the sync has been stopped so standby controllers do not keep requeueing because they think synchronization is still in progress. -6. Switch the platform domain to the standby platform entrance. -7. Verify the workload ingress and the new active Global's direct API endpoint: +7. Wait until at least 60 seconds after `ConfigMap/kube-public/etcd-sync` disappeared, then verify that the new active controllers are no longer gated. Do not delete the gate ConfigMap by hand. +8. Verify the workload ingress, registry, and the new active Global's direct API endpoint: ```shell - dig +short + export PLATFORM_DOMAIN="" + export NEW_ACTIVE_CONTROL_PLANE_VIP="" + + dig +short "${PLATFORM_DOMAIN}" + curl -ksS -w '\nHTTP %{http_code}\n' \ + "https://${PLATFORM_DOMAIN}/kubernetes/global/version" curl -ksS -w '\nHTTP %{http_code}\n' \ - https:///kubernetes/global/version + "https://${PLATFORM_DOMAIN}:11443/v2/" curl -ksS -w '\nHTTP %{http_code}\n' \ - https://:6443/version + "https://${NEW_ACTIVE_CONTROL_PLANE_VIP}:6443/version" ``` A Kubernetes version response with HTTP `200`, or an authentication/authorization response with HTTP `401` or `403`, proves that the request reached the intended API endpoint. -8. Confirm the standby cluster has the provider components needed to manage Bare Metal workload clusters: CAPI core, kubeadm provider, Bare Metal provider, elemental-operator, cluster-transformer, and alive-related delivery components. -9. Confirm existing workload hosts reconnect through the platform domain and can watch or patch their plan Secrets. +9. Confirm the standby cluster has the provider components needed to manage Bare Metal workload clusters: CAPI core, kubeadm provider, Bare Metal provider, elemental-operator, cluster-transformer, and alive-related delivery components. +10. Confirm existing workload hosts reconnect through the platform domain and can watch or patch their plan Secrets. Keep the former primary hard-fenced until the failback procedure has prepared it as the inactive peer. -Failback uses the same ownership sequence in reverse. The currently active side remains `sharedAuthReadOnly: false`; the inactive failback target must be `true` while reverse `etcd-sync` is running. After reverse synchronization is caught up and stopped, change the side that is becoming active to `false`, verify that its operator has taken ownership, change the side becoming inactive to `true`, and only then switch the platform domain back. +Failback uses the same ownership sequence in reverse. Start the original primary in isolation with `sharedAuthReadOnly: true`, install reverse `etcd-sync` on it, and synchronize from the currently active former standby. After reverse synchronization is caught up, first change the side becoming inactive to `true`, run the final checks, and hard-fence it. Uninstall the reverse mirror through its owning ModuleInfo and wait for the gate to disappear. Then switch the platform domain so the target AppRelease can use the target-side platform registry, change the side becoming active to `false`, verify ownership, and leave the inactive side fenced until forward synchronization has been re-established. Never reuse the old direction's token or exact-ignore array. ### Post-Failover Validation @@ -348,9 +574,262 @@ After standby is active, validate in increasing risk order: | Handoff target list is empty | `dcs-import-extra-resources` was missing or incomplete before the installer API call | `kubectl -n cpaas-system get cm dcs-import-extra-resources -o yaml`; check `baremetal-system-agent-handoff` Job logs | | Existing hosts get `401` or `Forbidden` after failover | ServiceAccount signing key, issuer, audience, token Secret, auth scope, or Role `resourceNames` does not match | `sha256sum /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub`; repeat the scoped `kubectl auth can-i` checks above | | Existing workload hosts fail TLS verification after the domain switch, or Global hosts fail against their local VIP | The platform certificate does not cover the platform domain, or the local kube-apiserver certificate/CA does not cover and trust the direct VIP | Check `cpaas-system/dex.tls`, the kube-apiserver serving certificate SANs, the agent connection CA, `curl -ksS https:///kubernetes/global/version`, and `curl -ksS https://:6443/version` | +| Recovery registration completes but an older host reports `x509: certificate signed by unknown authority` against the direct API VIP | The operator lacks the legacy direct-API CA bundle compatibility, or its configured ingress CA Secret/key does not match the mounted certificate chain | Verify the operator image contains Elemental PR #3, compare the provider AppRelease CA values with the `elemental-operator` volume and `--ca-cert-file`, and confirm the ServiceAccount token Secret contains the kube-apiserver `ca.crt` | | Standby `global` resources are replaced by primary resources | `etcd-sync` ignore rules did not exclude `global` resources | Inspect the `etcd-sync` ignore ConfigMap and stop sync before restoring standby resources | | New host cannot register with an old ISO | ISO registration URL points to bootstrap or primary-only IP, or `MachineRegistration` was not synchronized | Inspect the `MachineRegistration` status and rebuild the ISO on active standby when needed | +### Recover a Global Host That Cannot Authenticate During Migration \{#recover-a-global-host-that-cannot-authenticate-during-migration} + +Use this procedure only when an existing Global host cannot watch its plan because its legacy shared token no longer authenticates, for example after the shared ServiceAccount was replaced before that host consumed the split-auth migration plan. A temporary migration RoleBinding cannot repair an invalid JWT; it only preserves authorization for a token that the API server can still authenticate. + +This is an emergency connection repair for an already installed host. It reuses the existing registration state and MachineInventory. It must not reset, reinstall, or reprovision the host. + +Before starting, install the upgraded Bare Metal provider and `elemental-operator` on the affected final Global cluster. Wait for the local Global ServiceAccount, token Secret, Role, canonical RoleBinding, and the handoff-created migration bridge. Do not create the bridge manually. + +The operator image must include the legacy direct-kube-apiserver CA compatibility from Elemental PR #3 (`c8f7cf235cbfee0aa3973207f1091e5bc6c81cdd`, or an explicitly verified backport). For a direct-API registration update, that implementation returns the ingress and kube-apiserver CAs together in `Registration.CACert`, while keeping only the kube-apiserver CA in `SystemAgent.CACert`. Current register binaries use `SystemAgent.CACert`; older binaries, including the observed `fe5fd342` build that predates that field, ignore it and use the combined `Registration.CACert` when writing the direct-API kubeconfig. + +Confirm the deployed operator image digest belongs to a release containing that compatibility before creating a recovery capability. Record the host's existing `/usr/sbin/elemental-register --version`, but do not replace the installed binary. If the operator provenance is unknown, or it returns only the ingress CA in `Registration.CACert` for a direct-API registration, stop: the recovered agent will fail TLS verification against the kube-apiserver VIP. + +1. Create a dedicated recovery `MachineRegistration` on the affected final Global cluster. Copy only `spec` from the original bootstrap registration; do not copy `status`, UID, owner references, or a `SeedImage`. + + ```shell + set -euo pipefail + + export BOOTSTRAP_KUBECONFIG="" + export LOCAL_GLOBAL_KUBECONFIG="" + export BOOTSTRAP_MACHINE_REGISTRATION="" + export RECOVERY_MACHINE_REGISTRATION="global-system-agent-recovery" + export LOCAL_GLOBAL_CONTROL_PLANE_VIP="" + + kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" \ + -n cpaas-system \ + get machineregistration.elemental.cattle.io \ + "${BOOTSTRAP_MACHINE_REGISTRATION}" -o json | \ + jq --arg name "${RECOVERY_MACHINE_REGISTRATION}" \ + --arg vip "${LOCAL_GLOBAL_CONTROL_PLANE_VIP}" ' + { + apiVersion: .apiVersion, + kind: .kind, + metadata: { + name: $name, + namespace: "cpaas-system", + annotations: { + "baremetal.cluster.io/system-agent-auth-scope": "global", + "baremetal.cluster.io/system-agent-server-url": ("https://" + $vip + ":6443"), + "baremetal.cluster.io/system-agent-endpoint-mode": "direct-apiserver" + } + }, + spec: .spec + }' | \ + kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" apply -f - + + kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + -n cpaas-system wait --for=condition=Ready \ + "machineregistration.elemental.cattle.io/${RECOVERY_MACHINE_REGISTRATION}" \ + --timeout=180s + + test "$( + kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get machineregistration.elemental.cattle.io \ + "${RECOVERY_MACHINE_REGISTRATION}" \ + -o jsonpath='{.status.serviceAccountRef.name}' + )" = "baremetal-global-system-agent" + ``` + + Do not print or copy `status.registrationURL` into a ticket, terminal transcript, or shared log. It contains a registration capability token. + +2. Verify that the existing MachineInventory plan Secret has the name required by the current registration server. If the two names differ, stop; this repair would connect the agent to the wrong Secret. + + ```shell + export GLOBAL_MACHINE_INVENTORY="" + + PLAN_SECRET="$( + kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get machineinventory.elemental.cattle.io \ + "${GLOBAL_MACHINE_INVENTORY}" \ + -o jsonpath='{.status.plan.secretRef.name}' + )" + + test "${PLAN_SECRET}" = "${GLOBAL_MACHINE_INVENTORY}" + ``` + +3. Fetch the recovery registration configuration without switching global DNS and without exposing the capability URL in a process argument. Read the ingress CA Secret name and key from the affected cluster's actual Bare Metal provider AppRelease, and verify that they match the volume mounted into the running `elemental-operator`. Do not assume the key is `ca.crt`: a final Global `dex.tls` Secret can contain only `tls.crt` and `tls.key`, with the trusted certificate chain stored in `tls.crt`. + + ```shell + set -euo pipefail + umask 077 + + export PLATFORM_DOMAIN="" + export PROVIDER_APPRELEASE=cluster-api-provider-baremetal + W="$(mktemp -d)" + trap 'rm -rf "${W}"' EXIT + + PROVIDER_APPRELEASE_JSON="$( + kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get apprelease "${PROVIDER_APPRELEASE}" -o json + )" + + APPRELEASE_CA_SECRET_NAME="$( + printf '%s\n' "${PROVIDER_APPRELEASE_JSON}" | \ + jq -r '.spec.values.elemental.tls.caCertSecretName // empty' + )" + APPRELEASE_CA_SECRET_KEY="$( + printf '%s\n' "${PROVIDER_APPRELEASE_JSON}" | \ + jq -r '.spec.values.elemental.tls.caCertSecretKey // empty' + )" + + # If the AppRelease omitted either value, set the corresponding RECOVERY_* + # override only after inspecting the deployed operator volume and argument. + RECOVERY_CA_SECRET_NAME="${RECOVERY_CA_SECRET_NAME:-${APPRELEASE_CA_SECRET_NAME}}" + RECOVERY_CA_SECRET_KEY="${RECOVERY_CA_SECRET_KEY:-${APPRELEASE_CA_SECRET_KEY}}" + + if [ -z "${RECOVERY_CA_SECRET_NAME}" ] || [ -z "${RECOVERY_CA_SECRET_KEY}" ]; then + echo "the provider AppRelease does not identify the Elemental CA Secret and key" >&2 + exit 1 + fi + + ELEMENTAL_OPERATOR_DEPLOYMENT_JSON="$( + kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get deployment elemental-operator -o json + )" + DEPLOYED_CA_SECRET_NAME="$( + printf '%s\n' "${ELEMENTAL_OPERATOR_DEPLOYMENT_JSON}" | \ + jq -r '[.spec.template.spec.volumes[]? + | select(.name == "elemental-ca-cert") + | .secret.secretName][0] // empty' + )" + DEPLOYED_CA_SECRET_KEY="$( + printf '%s\n' "${ELEMENTAL_OPERATOR_DEPLOYMENT_JSON}" | \ + jq -r '[.spec.template.spec.volumes[]? + | select(.name == "elemental-ca-cert") + | .secret.items[]?.key][0] // empty' + )" + DEPLOYED_CA_FILE_ARG="$( + printf '%s\n' "${ELEMENTAL_OPERATOR_DEPLOYMENT_JSON}" | \ + jq -r '[.spec.template.spec.containers[].args[]? + | select(startswith("--ca-cert-file="))][0] // empty' + )" + DEPLOYED_CA_FILE_KEY="$(basename "${DEPLOYED_CA_FILE_ARG#*=}")" + + if [ "${RECOVERY_CA_SECRET_NAME}" != "${DEPLOYED_CA_SECRET_NAME}" ] || \ + [ "${RECOVERY_CA_SECRET_KEY}" != "${DEPLOYED_CA_SECRET_KEY}" ] || \ + [ "${RECOVERY_CA_SECRET_KEY}" != "${DEPLOYED_CA_FILE_KEY}" ]; then + echo "confirmed CA Secret/key does not match the running elemental-operator" >&2 + exit 1 + fi + + kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get machineregistration.elemental.cattle.io \ + "${RECOVERY_MACHINE_REGISTRATION}" \ + -o jsonpath='{.status.registrationURL}' > "${W}/registration-url" + + kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get secret "${RECOVERY_CA_SECRET_NAME}" -o json | \ + jq -er --arg key "${RECOVERY_CA_SECRET_KEY}" ' + .data[$key] | select(type == "string" and length > 0) + ' | base64 -d > "${W}/registration-ca.crt" + + grep -q -- '-----BEGIN CERTIFICATE-----' "${W}/registration-ca.crt" + openssl x509 -in "${W}/registration-ca.crt" -noout + + { + printf 'url = "' + tr -d '\r\n' < "${W}/registration-url" + printf '"\n' + printf 'cacert = "%s"\n' "${W}/registration-ca.crt" + printf 'resolve = "%s:443:%s"\n' \ + "${PLATFORM_DOMAIN}" "${LOCAL_GLOBAL_CONTROL_PLANE_VIP}" + printf 'fail\nsilent\nshow-error\n' + } > "${W}/curl.conf" + + chmod 0600 "${W}"/* + curl --config "${W}/curl.conf" \ + > "${W}/registration.yaml" \ + 2> "${W}/curl.log" + chmod 0600 "${W}/registration.yaml" + ``` + + This procedure fails closed if the AppRelease value, deployed Secret volume, and actual Secret key do not agree. If you must use a `RECOVERY_CA_SECRET_NAME` or `RECOVERY_CA_SECRET_KEY` override because an older AppRelease omitted explicit values, record the inspected Deployment volume and `--ca-cert-file` argument as evidence before setting it. Do not fall back from a missing `ca.crt` to `tls.crt` merely because that key exists. + + `registration-url`, `curl.conf`, `registration.yaml`, and the command log are sensitive. Keep them readable only by root and never attach them to support records. + +4. Copy `registration.yaml` to a root-only directory under `/run` on the affected host. + + ```shell + export GLOBAL_HOST_ADDRESS="" + + ssh "root@${GLOBAL_HOST_ADDRESS}" \ + 'umask 077 + install -d -m 0700 /run/elemental-reregister + install -m 0600 /dev/stdin /run/elemental-reregister/registration.yaml' \ + < "${W}/registration.yaml" + ``` + +5. On the affected host, verify and preserve the existing registration state, then run `elemental-register` in a temporary mount namespace. The temporary `/etc/hosts` bind mount makes only this command reach the local Global entrance; it does not change system DNS. + + ```shell + export PLATFORM_DOMAIN="" + export LOCAL_GLOBAL_CONTROL_PLANE_VIP="" + R=/run/elemental-reregister + + test -s /oem/registration/state.yaml + grep -q '^initialRegistration:' /oem/registration/state.yaml + + cp -a /etc/hosts "${R}/hosts" + printf '\n%s %s\n' \ + "${LOCAL_GLOBAL_CONTROL_PLANE_VIP}" "${PLATFORM_DOMAIN}" >> "${R}/hosts" + + cp -a /var/lib/elemental/agent/elemental_connection.json \ + "${R}/elemental_connection.before.json" + cp -a /oem/elemental-system-agent.yaml \ + "${R}/elemental-system-agent.before.yaml" + chmod 0600 "${R}"/* + + R="${R}" unshare --mount --propagation private sh -ceu ' + mount --bind "${R}/hosts" /etc/hosts + exec /usr/sbin/elemental-register \ + --config-path "${R}/registration.yaml" \ + --state-path /oem/registration/state.yaml \ + --install \ + --no-toolkit + ' > "${R}/register.log" 2>&1 + + systemctl restart elemental-system-agent.service + systemctl is-active --quiet elemental-system-agent.service + ``` + + Never add `--reset`: it discards the update state and turns this into a new registration or reprovision path. Do not add `--debug`: the returned system-agent configuration and token can be written to the log. Even normal logging includes the registration URL, so keep `register.log` root-only. + + The existing `/oem/registration/state.yaml` must retain the original TPM or emulated-TPM identity. If registration reports that the MachineInventory was not found and reprovisioning is needed, stop. Do not retry with an empty state or `--reset`. + + On an older register binary, the recovered runtime kubeconfig is expected to use the combined `Registration.CACert` bundle returned by the compatible operator. On a current binary, it uses `SystemAgent.CACert`. In both cases, the resulting direct-API kubeconfig must contain the kube-apiserver CA, not only the ingress CA. + +6. As soon as the runtime files have been written, the agent has restarted, and the handoff Job or plan feedback proves that the agent can authenticate and consume its current plan Secret, delete the recovery `MachineRegistration`. This revokes the registration capability; it does not remove the shared local Global ServiceAccount token already written to the host. Do not keep the capability until the final probe merely for convenience. + + ```shell + kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ + -n cpaas-system delete machineregistration.elemental.cattle.io \ + "${RECOVERY_MACHINE_REGISTRATION}" + ``` + + If the agent still reports `the server has asked for the client to provide credentials`, TLS verification failure, or no plan observation, do not treat registration as successful and do not proceed to the completion gate. Preserve the root-only diagnostic files while investigating. + +7. Let a still-running handoff Job continue. If the Job already reached its timeout or failed while the host was stranded, re-run the chart-delivered official post-install hook described above; it records the failed logs, deletes the failed Job, and recreates it with the current image and arguments. Do not handcraft a replacement Job. Wait for the affected host's migration handoff checksum and probe to become Applied, then verify all of the normal completion gates: + + - `/oem/elemental-system-agent.yaml` contains `https://:6443`; + - the runtime token matches the local `baremetal-global-system-agent-token` Secret when compared by hash; + - the temporary bridge is deleted by handoff, not manually; and + - the completion ConfigMap reports `true split-local-direct-v1 <64-lowercase-hex-character-SHA-256>`. + + The `--install --no-toolkit` command repairs the runtime system-agent connection files but does not write the persistent OEM handoff file or restart the agent by itself. The migration plan remains responsible for the persistent OEM update and final probe. + +8. Only after all completion gates pass, delete the root-only temporary files from the host: + + ```shell + ssh "root@${GLOBAL_HOST_ADDRESS}" \ + 'rm -rf /run/elemental-reregister' + ``` + ## Provider Notes @@ -381,7 +860,7 @@ Bare Metal requires all of the following to be aligned: - Same trusted platform certificate chain, with SANs that cover the platform domain, primary control-plane VIP, and standby control-plane VIP. - `dcs-import-extra-resources` created before the installer API call on both sides. - Bare Metal provider AppRelease values set with the local `handoffHook.controlPlaneVIP`, `handoffHook.directAPIServer: true`, `elemental.systemAgent.authMode: shared`, `elemental.systemAgent.splitAuthEnabled: true`, and `elemental.systemAgent.globalServiceAccountName: baremetal-global-system-agent`. The active side sets `elemental.systemAgent.sharedAuthReadOnly: false`; the inactive side sets it to `true`. -- The deployed `elemental-operator` image and the `elemental-register` binary in `baremetal-base-image` and `baremetal-base-image-iso` are built from the same pinned elemental commit. Rebuild the base image and ISO whenever that commit changes; an operator-only upgrade does not update the host registration binary. +- The deployed `elemental-operator` image and every newly produced `baremetal-base-image` and `baremetal-base-image-iso` are built from the same pinned elemental commit. Rebuild new installation media whenever that commit changes. Already installed hosts retain their original register binary and rely on operator-side backward compatibility; do not replace that binary during handoff or recovery. Do not sync `SeedImage` from primary to standby. Sync non-`global` `MachineRegistration` resources when workload ISOs need to remain valid across failover. Do not sync primary Global `MachineRegistration` resources to standby. Create Global registrations locally with `baremetal.cluster.io/system-agent-auth-scope: global`, `baremetal.cluster.io/system-agent-server-url: https://:6443`, and `baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver`. diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index e7a6f8f0..08164fd6 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -533,7 +533,9 @@ Upload the packages. ``` - The Bare Metal provider package deploys `elemental-operator`, while the Bare Metal base image and ISO contain the `elemental-register` binary used by each host. Build both artifacts from the same pinned `elemental-operator` commit. Rebuilding only the provider package is not sufficient when registration, endpoint, token, or CA handling changes. Rebuild `baremetal-base-image` and `baremetal-base-image-iso`, record the elemental commit and image digests, and verify that they match the deployed operator before booting the ISO. Do not rely on a mutable branch name or a cached base-image layer as proof of source consistency. + The Bare Metal provider package deploys `elemental-operator`, while the Bare Metal base image and ISO contain the `elemental-register` binary used by each host. For every newly produced installation medium, build the operator, `baremetal-base-image`, and `baremetal-base-image-iso` from the same pinned `elemental-operator` commit. When that commit changes, rebuild the base image and ISO before using them to install new hosts; record the elemental commit and image digests, and do not rely on a mutable branch name or cached layer as proof of source consistency. + + This release-build rule does not mean that an operator upgrade replaces `elemental-register` on an already installed host. Existing hosts retain the binary baked into their original image, so registration protocol and returned configuration changes must remain backward compatible. In particular, direct-kube-apiserver recovery of an older host requires an operator that includes the legacy CA-bundle compatibility described in [Recover a Global Host That Cannot Authenticate During Migration](./disaster_recovery.mdx#recover-a-global-host-that-cannot-authenticate-during-migration); do not copy a newer register binary onto an installed host as an upgrade workaround. Create and apply the AppRelease resources for the Kubeadm provider and the Bare Metal provider. @@ -1872,11 +1874,14 @@ export ENCRYPTION_PROVIDER_SECRET_B64="" export PRIMARY_CLUSTER_VIP="" export STANDBY_CLUSTER_VIP="" export BAREMETAL_ENCRYPTION_PROVIDER_SECRET="global-encryption-provider-config" -export ETCD_SYNC_VERSION="" +export ETCD_SYNC_VERSION="v4.3.16-hotfix.179.1.g746d4244-t" export ETCD_SYNC_MODULEINFO="/root/yamls/global-etcd-sync-moduleinfo.json" +export ETCD_SYNC_SELF_MONITOR_IGNORE_KEY="/registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor" export SERVICE_ACCOUNT_ISSUER="https://kubernetes.default.svc.cluster.local" ``` +`v4.3.16-hotfix.179.1.g746d4244-t` is the minimum build verified with this split-auth procedure. It contains `etcd-sync` commit `746d424`, which keeps the Global-local handoff objects out of primary-to-standby synchronization. A later build is acceptable only when its release provenance proves that it contains that commit or an explicitly verified equivalent backport. Do not infer compatibility from a higher-looking version string alone. + Create the encryption provider configuration file on both installation environments. ```shell @@ -2109,7 +2114,7 @@ kubectl -n cpaas-system get apprelease cluster-api-provider-baremetal -o json | Do not continue until this command succeeds. The primary must use `sharedAuthReadOnly: false`; the standby must use `sharedAuthReadOnly: true` while primary-to-standby synchronization is active. -The resulting DR values are: +The resulting bootstrap AppRelease values are: ```yaml handoffHook: @@ -2131,6 +2136,18 @@ elemental: caCertSecretKey: ca.crt ``` +Keep the bootstrap and final certificate inputs separate. The bootstrap `dex.tls` created in Step 3 has a dedicated `ca.crt` key and is used only for `https://:12443`. Do not import it into the final cluster. On each final Global cluster, the installer-managed `dex.tls` contains the platform full certificate chain in `tls.crt`; the final Bare Metal provider AppRelease must therefore use `elemental.tls.caCertSecretKey: tls.crt`: + +```yaml title="Final Global AppRelease" +elemental: + tls: + agentTLSMode: strict + caCertSecretName: dex.tls + caCertSecretKey: tls.crt +``` + +The final operator uses this chain for its registration endpoint. Direct-API handoff still obtains the kube-apiserver CA from `Secret/cpaas-system/baremetal-global-system-agent-token` key `ca.crt`; do not replace that CA with `dex.tls/tls.crt` in a Global host's direct `kubeconfig`. + `elemental.systemAgent.splitAuthEnabled` is the single switch for both `elemental-operator` and the handoff job. Do not configure a separate handoff auth-enable switch. Keeping one source of truth prevents a Global host from retaining a shared token after the operator has already narrowed the shared Role to workload plans. Set the shared bundle ownership separately on the two AppRelease objects. The active primary reconciles workload permissions, while the inactive standby leaves the synchronized shared bundle read-only: @@ -2160,6 +2177,225 @@ Use namespaced `Role` and `RoleBinding` objects, not `ClusterRoleBinding`. Do no The handoff completion signal is versioned. For this DR configuration, each final Global cluster must report `data.ready: "true"`, `data.profile: split-local-direct-v1`, and a `data.config` value that is exactly 64 lowercase hexadecimal characters in `ConfigMap/cpaas-system/baremetal-system-agent-handoff`. The digest covers the effective endpoint, token/CA, identity names, and each Global target's names and object UIDs. A ready-only ConfigMap, a `legacy-shared-*` profile, or a stale, missing, or malformed config digest does not satisfy the cleanup gate. Upgrading the provider migrates only installed Global hosts against their current Global plan Secrets. It uses the independent temporary `baremetal-global-system-agent-handoff-bridge` RoleBinding and writes the new profile/config only after the new connection has been probed and the bridge has been deleted. Do not patch completion fields manually. +#### Upgrade Both AppRelease Copies Before Migrating Existing Hosts + +A retained bootstrap AppRelease and the corresponding final Global AppRelease have different jobs during an upgrade. The final AppRelease upgrades the running provider and `elemental-operator`; the bootstrap AppRelease delivers the current handoff script and side-specific chart environment. Upgrading only one copy leaves either the running operator or the official handoff execution context stale. + +When upgrading an existing pair, stop `etcd-sync` and protect the current Global plan Secrets first as described in [Migrating Existing Handoff State](./disaster_recovery.mdx#migrating-existing-handoff-state). Then patch all four AppRelease objects: primary bootstrap, primary final Global, standby bootstrap, and standby final Global. The bootstrap copies use `ca.crt`; the final copies use `tls.crt`. + +```shell +set -euo pipefail + +export BAREMETAL_PROVIDER_VERSION="" +export PRIMARY_BOOTSTRAP_KUBECONFIG="" +export PRIMARY_GLOBAL_KUBECONFIG="" +export STANDBY_BOOTSTRAP_KUBECONFIG="" +export STANDBY_GLOBAL_KUBECONFIG="" + +patch_baremetal_apprelease() { + kubeconfig=$1 + control_plane_vip=$2 + shared_auth_read_only=$3 + ca_secret_key=$4 + + test "$( + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get apprelease cluster-api-provider-baremetal \ + -o jsonpath='{.spec.source.charts[0].name}' + )" = "ait/chart-cluster-api-provider-baremetal" + + patch="$({ + jq -cn \ + --arg version "${BAREMETAL_PROVIDER_VERSION}" \ + --arg vip "${control_plane_vip}" \ + --argjson read_only "${shared_auth_read_only}" \ + --arg ca_key "${ca_secret_key}" ' + { + spec: { + source: { + charts: [{ + name: "ait/chart-cluster-api-provider-baremetal", + releaseName: "cluster-api-provider-baremetal", + targetRevision: $version + }] + }, + values: { + handoffHook: { + controlPlaneVIP: $vip, + directAPIServer: true, + delivery: {enabled: true, mode: "always"} + }, + elemental: { + systemAgent: { + authMode: "shared", + splitAuthEnabled: true, + serviceAccountName: "baremetal-system-agent", + globalServiceAccountName: "baremetal-global-system-agent", + sharedAuthReadOnly: $read_only + }, + tls: { + agentTLSMode: "strict", + caCertSecretName: "dex.tls", + caCertSecretKey: $ca_key + } + } + } + } + }' + })" + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + patch apprelease cluster-api-provider-baremetal \ + --type=merge -p "${patch}" +} + +wait_baremetal_apprelease() { + kubeconfig=$1 + deadline=$((SECONDS + 600)) + + until kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get apprelease cluster-api-provider-baremetal -o json | \ + jq -e --arg version "${BAREMETAL_PROVIDER_VERSION}" ' + .spec.source.charts[0].targetRevision == $version and + ([.status.charts[]?] + | any(.installedRevision == $version and .phase == "Success")) + ' >/dev/null; do + if [ "${SECONDS}" -ge "${deadline}" ]; then + echo "Bare Metal provider AppRelease did not reach Success" >&2 + return 1 + fi + sleep 10 + done +} + +patch_baremetal_apprelease \ + "${PRIMARY_BOOTSTRAP_KUBECONFIG}" "${PRIMARY_CLUSTER_VIP}" false ca.crt +patch_baremetal_apprelease \ + "${PRIMARY_GLOBAL_KUBECONFIG}" "${PRIMARY_CLUSTER_VIP}" false tls.crt +patch_baremetal_apprelease \ + "${STANDBY_BOOTSTRAP_KUBECONFIG}" "${STANDBY_CLUSTER_VIP}" true ca.crt +patch_baremetal_apprelease \ + "${STANDBY_GLOBAL_KUBECONFIG}" "${STANDBY_CLUSTER_VIP}" true tls.crt + +wait_baremetal_apprelease "${PRIMARY_BOOTSTRAP_KUBECONFIG}" +wait_baremetal_apprelease "${PRIMARY_GLOBAL_KUBECONFIG}" +wait_baremetal_apprelease "${STANDBY_BOOTSTRAP_KUBECONFIG}" +wait_baremetal_apprelease "${STANDBY_GLOBAL_KUBECONFIG}" +``` + +Verify the final certificate key against both the Secret and the running operator on each final Global cluster: + +```shell +verify_final_elemental_tls() { + kubeconfig=$1 + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret dex.tls -o json | \ + jq -e ' + (.data["tls.crt"] | type == "string" and length > 0) and + (.data["tls.key"] | type == "string" and length > 0) + ' >/dev/null + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get apprelease cluster-api-provider-baremetal -o json | \ + jq -e ' + .spec.values.elemental.tls.caCertSecretName == "dex.tls" and + .spec.values.elemental.tls.caCertSecretKey == "tls.crt" + ' >/dev/null + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get deployment elemental-operator -o json | \ + jq -e ' + any(.spec.template.spec.volumes[]?; + .name == "elemental-ca-cert" and + .secret.secretName == "dex.tls" and + any(.secret.items[]?; .key == "tls.crt" and .path == "tls.crt")) and + any(.spec.template.spec.containers[].args[]?; + . == "--ca-cert-file=/etc/elemental/tls/tls.crt") + ' >/dev/null +} + +verify_final_elemental_tls "${PRIMARY_GLOBAL_KUBECONFIG}" +verify_final_elemental_tls "${STANDBY_GLOBAL_KUBECONFIG}" +``` + +After all four upgrades succeed, explicitly re-run the chart-delivered handoff script on each physical bootstrap host. An AppRelease or Helm `post-upgrade` alone delivers the script but does not execute the installer post-install stage. Use the commands in [Migrating Existing Handoff State](./disaster_recovery.mdx#migrating-existing-handoff-state), and use the ConfigMap from the same side's upgraded bootstrap AppRelease. + +#### Validate a Returned Recovery Configuration Before Use + +The emergency recovery procedure returns a configuration that contains a registration capability and the new system-agent token. Before copying it to a host or running `elemental-register`, validate it on the secured operator host with `yq` v4. Never print the file, enable shell tracing, or attach any derived command output that contains the token or registration URL. + +```shell +set -euo pipefail +set +x +umask 077 + +export RECOVERY_CONFIG="" +export LOCAL_GLOBAL_KUBECONFIG="" +export LOCAL_GLOBAL_CONTROL_PLANE_VIP="" +export PLATFORM_DOMAIN="" +export PLAN_SECRET="" +export EXPECTED_REGISTRATION_PREFIX="https://${PLATFORM_DOMAIN}/" +export EXPECTED_SYSTEM_AGENT_URL="https://${LOCAL_GLOBAL_CONTROL_PLANE_VIP}:6443" + +test "$(yq -r '.elemental."system-agent".url' "${RECOVERY_CONFIG}")" = \ + "${EXPECTED_SYSTEM_AGENT_URL}" +test "$(yq -r '.elemental."system-agent"."secret-name"' "${RECOVERY_CONFIG}")" = \ + "${PLAN_SECRET}" +test "$(yq -r '.elemental."system-agent"."secret-namespace"' "${RECOVERY_CONFIG}")" = \ + "cpaas-system" + +yq -e '.elemental."system-agent".strictTLSMode == true' \ + "${RECOVERY_CONFIG}" >/dev/null +yq -e '.elemental.registration."no-toolkit" == true' \ + "${RECOVERY_CONFIG}" >/dev/null +yq -e '.elemental."system-agent".token | tag == "!!str" and length > 0' \ + "${RECOVERY_CONFIG}" >/dev/null +yq -e ' + .elemental.registration.url | + startswith(strenv(EXPECTED_REGISTRATION_PREFIX)) +' "${RECOVERY_CONFIG}" >/dev/null + +W="$(mktemp -d)" +trap 'rm -rf "${W}"' EXIT + +yq -r '.elemental."system-agent"."ca-cert"' \ + "${RECOVERY_CONFIG}" > "${W}/returned-apiserver-ca.crt" +yq -r '.elemental.registration."ca-cert"' \ + "${RECOVERY_CONFIG}" > "${W}/returned-registration-ca.crt" + +kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" -n cpaas-system \ + get secret baremetal-global-system-agent-token \ + -o jsonpath='{.data.ca\.crt}' | base64 -d > "${W}/expected-apiserver-ca.crt" + +cmp -s "${W}/returned-apiserver-ca.crt" "${W}/expected-apiserver-ca.crt" +openssl verify -CAfile "${W}/returned-registration-ca.crt" \ + "${W}/expected-apiserver-ca.crt" >/dev/null + +EXPECTED_TOKEN_SHA="$({ + kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" -n cpaas-system \ + get secret baremetal-global-system-agent-token \ + -o jsonpath='{.data.token}' | base64 -d | sha256sum | awk '{print $1}' +})" +RETURNED_TOKEN_SHA="$({ + yq -r '.elemental."system-agent".token' "${RECOVERY_CONFIG}" | \ + tr -d '\r\n' | sha256sum | awk '{print $1}' +})" +test "${RETURNED_TOKEN_SHA}" = "${EXPECTED_TOKEN_SHA}" + +HTTP_CODE="$({ + curl --cacert "${W}/returned-apiserver-ca.crt" \ + -sS -o /dev/null -w '%{http_code}' \ + "${EXPECTED_SYSTEM_AGENT_URL}/version" +})" +case "${HTTP_CODE}" in + 200|401|403) ;; + *) echo "direct kube-apiserver TLS probe failed" >&2; exit 1 ;; +esac +``` + +This gate proves that the returned endpoint, plan Secret, namespace, token, and direct-API CA belong to the affected local Global cluster, and that the legacy registration CA bundle also trusts the kube-apiserver CA. If any check fails, delete the temporary recovery capability and stop. The `--install --no-toolkit` invocation in the recovery procedure is mandatory; removing `--no-toolkit` changes the operation from connection repair to an installation path. + @@ -2182,28 +2418,263 @@ For the primary cluster, make sure the platform domain resolves to the primary H After the primary cluster installation succeeds, switch the platform domain to the standby HA VIP as required by the DR procedure. Then install the standby cluster. This DNS switch before the standby installation is required because several platform resources are rendered with the platform domain and must resolve to the standby entrance while the standby installer runs. In Step 8 on the standby bootstrap host, set `hostIP` to the standby bootstrap host IP. For DCS, set `console.host` and `cluster.features.ha.vip` to the standby HA VIP. For VMware vSphere, set the control plane endpoint in the standby manifest to the standby HA VIP. For HCS, keep `console.host: []`. For Bare Metal, set both the manifest VIP and the installer VIP fields to the standby control-plane VIP, and keep `REGISTRY_DOMAIN` as `${PLATFORM_HOST}:11443` on both sides. Get `INSTALLER_IP` from the `cpaas-installer` Pod on the standby bootstrap host; do not reuse the primary bootstrap host value. -For Bare Metal, verify the versioned handoff completion on both sides before decommissioning either bootstrap KIND cluster: +For Bare Metal, run the complete handoff gate on both sides before decommissioning either bootstrap KIND cluster. `GLOBAL_HOSTS` is the space-separated management address list for that side's installed Global machines. Run this from a secured host that can use both Global kubeconfigs and SSH to the machines. Keep shell tracing disabled because the runtime connection file contains a bearer token. ```shell -kubectl --kubeconfig \ - -n cpaas-system get configmap baremetal-system-agent-handoff \ - -o jsonpath='{.data.ready}{" "}{.data.profile}{" "}{.data.config}{"\n"}' -kubectl --kubeconfig \ - -n cpaas-system get configmap baremetal-system-agent-handoff \ - -o jsonpath='{.data.ready}{" "}{.data.profile}{" "}{.data.config}{"\n"}' +set -euo pipefail +set +x + +export PRIMARY_GLOBAL_KUBECONFIG="" +export STANDBY_GLOBAL_KUBECONFIG="" +export PRIMARY_BOOTSTRAP_KUBECONFIG="" +export STANDBY_BOOTSTRAP_KUBECONFIG="" +export PRIMARY_GLOBAL_HOSTS=" " +export STANDBY_GLOBAL_HOSTS=" " + +verify_baremetal_handoff() { + bootstrap_kubeconfig=$1 + kubeconfig=$2 + control_plane_vip=$3 + global_hosts=$4 + expected_endpoint="https://${control_plane_vip}:6443" + + kubectl --kubeconfig "${bootstrap_kubeconfig}" -n cpaas-system \ + wait --for=condition=complete \ + job/baremetal-system-agent-handoff --timeout=30m + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get configmap baremetal-system-agent-handoff -o json | \ + jq -e ' + .data.ready == "true" and + .data.profile == "split-local-direct-v1" and + (.data.config | test("^[0-9a-f]{64}$")) + ' >/dev/null + + if kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get rolebinding baremetal-global-system-agent-handoff-bridge \ + >/dev/null 2>&1; then + echo "temporary handoff bridge still exists" >&2 + return 1 + fi + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system get \ + serviceaccount/baremetal-global-system-agent \ + secret/baremetal-global-system-agent-token \ + role/baremetal-global-system-agent \ + rolebinding/baremetal-global-system-agent >/dev/null + + global_plan_names="$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get role baremetal-global-system-agent -o json | \ + jq -c '[ + .rules[]? + | select(any(.apiGroups[]?; . == "")) + | select(any(.resources[]?; . == "secrets")) + | .resourceNames[]? + ] | unique | sort' + })" + printf '%s\n' "${global_plan_names}" | jq -e 'length > 0' >/dev/null + + while IFS= read -r plan_secret; do + test "$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret "${plan_secret}" -o jsonpath='{.type}' + })" = "elemental.cattle.io/plan" + done < <(printf '%s\n' "${global_plan_names}" | jq -r '.[]') + + expected_token_sha="$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret baremetal-global-system-agent-token \ + -o jsonpath='{.data.token}' | base64 -d | \ + sha256sum | awk '{print $1}' + })" + expected_ca_sha="$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret baremetal-global-system-agent-token \ + -o jsonpath='{.data.ca\.crt}' | base64 -d | \ + sha256sum | awk '{print $1}' + })" + + for host in ${global_hosts}; do + actual_endpoint="$({ + ssh "root@${host}" \ + 'cat /var/lib/elemental/agent/elemental_connection.json' | \ + jq -r '.kubeConfig' | \ + awk '$1 == "server:" {print $2; exit}' + })" + test "${actual_endpoint}" = "${expected_endpoint}" + + actual_namespace="$({ + ssh "root@${host}" \ + 'cat /var/lib/elemental/agent/elemental_connection.json' | \ + jq -r '.namespace' + })" + test "${actual_namespace}" = "cpaas-system" + + actual_plan_secret="$({ + ssh "root@${host}" \ + 'cat /var/lib/elemental/agent/elemental_connection.json' | \ + jq -r '.secretName' + })" + printf '%s\n' "${global_plan_names}" | \ + jq -e --arg name "${actual_plan_secret}" 'index($name) != null' \ + >/dev/null + + actual_token_sha="$({ + ssh "root@${host}" \ + 'cat /var/lib/elemental/agent/elemental_connection.json' | \ + jq -r '.kubeConfig' | \ + awk '$1 == "token:" {print $2; exit}' | \ + tr -d '\r\n' | sha256sum | awk '{print $1}' + })" + test "${actual_token_sha}" = "${expected_token_sha}" + + actual_ca_sha="$({ + ssh "root@${host}" \ + 'cat /var/lib/elemental/agent/elemental_connection.json' | \ + jq -r '.kubeConfig' | \ + awk '$1 == "certificate-authority-data:" {print $2; exit}' | \ + base64 -d | sha256sum | awk '{print $1}' + })" + test "${actual_ca_sha}" = "${expected_ca_sha}" + + ssh "root@${host}" \ + "grep -Fq -- '${expected_endpoint}' /oem/elemental-system-agent.yaml" + ssh "root@${host}" \ + 'test "$(stat -c %a /oem/elemental-system-agent.yaml)" = 600' + ssh "root@${host}" \ + 'grep -Fqx -- "CATTLE_AGENT_STRICT_VERIFY=\"true\"" /etc/rancher/elemental/agent/envs' + ssh "root@${host}" \ + 'systemctl is-active --quiet elemental-system-agent.service' + done +} + +verify_baremetal_handoff \ + "${PRIMARY_BOOTSTRAP_KUBECONFIG}" "${PRIMARY_GLOBAL_KUBECONFIG}" \ + "${PRIMARY_CLUSTER_VIP}" \ + "${PRIMARY_GLOBAL_HOSTS}" +verify_baremetal_handoff \ + "${STANDBY_BOOTSTRAP_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}" \ + "${STANDBY_CLUSTER_VIP}" \ + "${STANDBY_GLOBAL_HOSTS}" +``` + +Also verify the permission split. An empty shared `Role/baremetal-system-agent` is valid immediately after Global installation when no non-`global` workload plan exists. In that branch, keep the four shared objects present and synchronized, require all broad or cross-scope requests to be denied, and repeat the positive shared-plan checks after the first workload plan is created. + +```shell +verify_system_agent_rbac() { + kubeconfig=$1 + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system get \ + serviceaccount/baremetal-system-agent \ + secret/baremetal-system-agent-token \ + role/baremetal-system-agent \ + rolebinding/baremetal-system-agent >/dev/null + + local_plan_names="$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get role baremetal-global-system-agent -o json | \ + jq -c '[ + .rules[]? + | select(any(.apiGroups[]?; . == "")) + | select(any(.resources[]?; . == "secrets")) + | .resourceNames[]? + ] | unique | sort' + })" + shared_plan_names="$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get role baremetal-system-agent -o json | \ + jq -c '[ + .rules[]? + | select(any(.apiGroups[]?; . == "")) + | select(any(.resources[]?; . == "secrets")) + | .resourceNames[]? + ] | unique | sort' + })" + + local_plan="$(printf '%s\n' "${local_plan_names}" | jq -er '.[0]')" + + test "$({ + kubectl --kubeconfig "${kubeconfig}" \ + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i get "secret/${local_plan}" + })" = yes + test "$({ + kubectl --kubeconfig "${kubeconfig}" \ + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i patch "secret/${local_plan}" + })" = yes + + for service_account in baremetal-global-system-agent baremetal-system-agent; do + test "$({ + kubectl --kubeconfig "${kubeconfig}" \ + --as="system:serviceaccount:cpaas-system:${service_account}" \ + -n cpaas-system auth can-i list secrets + })" = no + test "$({ + kubectl --kubeconfig "${kubeconfig}" \ + --as="system:serviceaccount:cpaas-system:${service_account}" \ + -n cpaas-system auth can-i create secrets + })" = no + test "$({ + kubectl --kubeconfig "${kubeconfig}" \ + --as="system:serviceaccount:cpaas-system:${service_account}" \ + -n cpaas-system auth can-i delete secrets + })" = no + test "$({ + kubectl --kubeconfig "${kubeconfig}" \ + --as="system:serviceaccount:cpaas-system:${service_account}" \ + -n cpaas-system auth can-i get secret/global-registry-auth + })" = no + done + + test "$({ + kubectl --kubeconfig "${kubeconfig}" \ + --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ + -n cpaas-system auth can-i get "secret/${local_plan}" + })" = no + + if printf '%s\n' "${shared_plan_names}" | jq -e 'length == 0' >/dev/null; then + echo "shared workload Role is empty; positive workload checks are deferred" + return 0 + fi + + workload_plan="$(printf '%s\n' "${shared_plan_names}" | jq -er '.[0]')" + test "$({ + kubectl --kubeconfig "${kubeconfig}" \ + --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ + -n cpaas-system auth can-i get "secret/${workload_plan}" + })" = yes + test "$({ + kubectl --kubeconfig "${kubeconfig}" \ + --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ + -n cpaas-system auth can-i patch "secret/${workload_plan}" + })" = yes + test "$({ + kubectl --kubeconfig "${kubeconfig}" \ + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i get "secret/${workload_plan}" + })" = no +} + +verify_system_agent_rbac "${PRIMARY_GLOBAL_KUBECONFIG}" +verify_system_agent_rbac "${STANDBY_GLOBAL_KUBECONFIG}" ``` -Both commands must print `true split-local-direct-v1 <64-lowercase-hex-character-SHA-256>`. A bare `true`, a legacy profile, an empty or malformed config digest, a remaining migration bridge, or a failed Global-side probe means the handoff is not complete. +Do not reduce this to a ready-only ConfigMap check. A bare `true`, a legacy profile, an empty or malformed config digest, a remaining migration bridge, a runtime token or CA mismatch, a stale bootstrap endpoint in the persistent OEM file, or an incorrect permission boundary means handoff is incomplete. After the standby installation succeeds, switch the platform domain back to the primary entrance before installing `global-etcd-sync`. Primary remains active during initial synchronization, and workload system-agent endpoints use the platform domain. Verify that DNS and `https:///kubernetes/global/version` reach primary before continuing. -Before you create the `global-etcd-sync` ModuleInfo, prepare the runtime exact-key ignore list. For providers other than Bare Metal, keep the list empty: +Before you create the `global-etcd-sync` ModuleInfo, prepare the runtime exact-key ignore list. Keep the plugin's own `ServiceMonitor/etcd-sync-monitor` local on whichever Global cluster runs the plugin. This entry is required for every provider; without it, the mirror can treat its own target-side monitoring resource as source surplus and repeatedly replace or delete it. ```shell -export ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON='[]' +export ETCD_SYNC_SELF_MONITOR_IGNORE_KEY="/registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor" +export ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON="$({ + jq -cn --arg key "${ETCD_SYNC_SELF_MONITOR_IGNORE_KEY}" '[$key]' +})" ``` -For Bare Metal, collect the local Global plan Secret names from both Global clusters and convert them to exact etcd keys. Run these commands from a host that has `jq` and two kubeconfigs. Each kubeconfig must address its own Global API server; do not rely on the platform domain, because that domain points to only one side at a time. If one kubeconfig contains both clusters, use two distinct contexts in the equivalent `kubectl --context` commands. +For Bare Metal, add both sides' local Global plan Secret names to that baseline list and convert them to exact etcd keys. Run these commands from a host that has `jq` and two kubeconfigs. Each kubeconfig must address its own Global API server; do not rely on the platform domain, because that domain points to only one side at a time. If one kubeconfig contains both clusters, use two distinct contexts in the equivalent `kubectl --context` commands. ```shell export PRIMARY_GLOBAL_KUBECONFIG="" @@ -2254,18 +2725,22 @@ printf '%s\n' "${STANDBY_GLOBAL_PLAN_SECRET_NAMES}" | \ export ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON="$( jq -cn \ + --argjson baseline "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" \ --argjson primary "${PRIMARY_GLOBAL_PLAN_SECRET_NAMES}" \ --argjson standby "${STANDBY_GLOBAL_PLAN_SECRET_NAMES}" \ - '($primary + $standby) + '$baseline + + (($primary + $standby) + | unique + | sort + | map("/registry/secrets/cpaas-system/" + .)) | unique - | sort - | map("/registry/secrets/cpaas-system/" + .)' + | sort' )" printf '%s\n' "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" | jq . ``` -Do not install `etcd-sync` if either local Global Role has no plan `resourceNames`, or if the generated array omits a plan Secret from either side. The list is merged and de-duplicated because the broad Secret sync prefix would otherwise overwrite a local Global plan Secret whose name happens to exist in the source or target. Repeat the collection and update the ModuleInfo configuration before synchronization continues whenever either local Global Role gains a new plan Secret. +Do not install `etcd-sync` if either local Global Role has no plan `resourceNames`, if the generated array omits a plan Secret from either side, or if it omits `ETCD_SYNC_SELF_MONITOR_IGNORE_KEY`. The list is merged and de-duplicated because the broad Secret sync prefix would otherwise overwrite a local Global plan Secret whose name happens to exist in the source or target. Repeat the collection and update the ModuleInfo configuration before synchronization continues whenever either local Global Role gains a new plan Secret. After both clusters are installed, get the primary `k8sadmin` token on a primary control plane node. `etcd-sync` is installed only on the standby cluster, and its `active_cluster_*` values point to the primary cluster. Keep this value in its original base64 Secret form for `active_cluster_token`. @@ -2288,6 +2763,7 @@ export STANDBY_CLUSTER_BEARER_TOKEN="" Create the `global-etcd-sync` ModuleInfo payload for the standby cluster. The `active_cluster_vip` and `active_cluster_token` values must point to the primary cluster. ```shell +umask 077 cat > "${ETCD_SYNC_MODULEINFO}" < "${ETCD_SYNC_MODULEINFO}" <` name, so capture the actual object name from the response and use that name for every later patch or delete operation. The request and response both contain credentials; keep their files root-only and do not print them. ```shell -curl -sk -X POST "https://${STANDBY_CLUSTER_VIP}/apis/cluster.alauda.io/v1alpha1/moduleinfoes" \ - -H "Authorization: Bearer ${STANDBY_CLUSTER_BEARER_TOKEN}" \ - -H "Content-Type: application/json" \ - -d @"${ETCD_SYNC_MODULEINFO}" +set -euo pipefail +set +x +umask 077 + +ETCD_SYNC_HEADERS="$(mktemp)" +ETCD_SYNC_RESPONSE="$(mktemp)" +trap 'rm -f "${ETCD_SYNC_HEADERS}" "${ETCD_SYNC_RESPONSE}"' EXIT + +{ + printf 'Authorization: Bearer %s\n' "${STANDBY_CLUSTER_BEARER_TOKEN}" + printf 'Content-Type: application/json\n' +} > "${ETCD_SYNC_HEADERS}" + +HTTP_CODE="$({ + curl -sk -X POST \ + "https://${STANDBY_CLUSTER_VIP}/apis/cluster.alauda.io/v1alpha1/moduleinfoes" \ + -H @"${ETCD_SYNC_HEADERS}" \ + -d @"${ETCD_SYNC_MODULEINFO}" \ + -o "${ETCD_SYNC_RESPONSE}" \ + -w '%{http_code}' +})" + +case "${HTTP_CODE}" in + 200|201) ;; + *) echo "ModuleInfo API returned HTTP ${HTTP_CODE}" >&2; exit 1 ;; +esac + +export ETCD_SYNC_MODULEINFO_NAME="$({ + jq -er '.metadata.name | select(type == "string" and length > 0)' \ + "${ETCD_SYNC_RESPONSE}" +})" +export ETCD_SYNC_MODULEINFO_NAME_FILE="/root/yamls/global-etcd-sync-moduleinfo.name" +printf '%s\n' "${ETCD_SYNC_MODULEINFO_NAME}" > \ + "${ETCD_SYNC_MODULEINFO_NAME_FILE}" +chmod 0600 "${ETCD_SYNC_MODULEINFO_NAME_FILE}" + +rm -f "${ETCD_SYNC_HEADERS}" "${ETCD_SYNC_RESPONSE}" +trap - EXIT +rm -f "${ETCD_SYNC_MODULEINFO}" +unset PRIMARY_CLUSTER_TOKEN_B64 STANDBY_CLUSTER_BEARER_TOKEN +``` + +Wait for the normalized ModuleInfo and its managed `AppRelease/cpaas-system/etcd-sync`. Do not declare the plugin installed until the AppRelease phase is `Success`, its installed revision matches `ETCD_SYNC_VERSION`, and both Deployments use an image built from `etcd-sync` commit `746d424` or the explicitly verified equivalent backport. + +```shell +deadline=$((SECONDS + 600)) +until kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + get moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" -o json | \ + jq -e --arg version "${ETCD_SYNC_VERSION}" \ + '.spec.version == $version' >/dev/null && \ + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get apprelease etcd-sync -o json | \ + jq -e --arg version "${ETCD_SYNC_VERSION}" ' + [.status.charts[]?] + | any(.installedRevision == $version and .phase == "Success") + ' >/dev/null; do + if [ "${SECONDS}" -ge "${deadline}" ]; then + echo "etcd-sync ModuleInfo/AppRelease did not reach Success" >&2 + exit 1 + fi + sleep 10 +done + +export ETCD_SYNC_IMAGE_PROVENANCE_MARKER="746d424" +for deployment in etcd-sync etcd-sync-monitor; do + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get deployment "${deployment}" -o json | \ + jq -e --arg marker "${ETCD_SYNC_IMAGE_PROVENANCE_MARKER}" ' + all(.spec.template.spec.containers[]?; .image | contains($marker)) + ' >/dev/null +done ``` -Wait until `ConfigMap/cpaas-system/etcd-sync-ignore-text` contains every key from `ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON`, then wait for both `Deployment/etcd-sync` and `Deployment/etcd-sync-monitor` to become available. The mirror and monitor processes read their filter files at startup. Whenever you later change `additional_ignore_equal`, updating ModuleInfo or the ConfigMap alone is not enough: after the rendered ConfigMap is correct, explicitly restart and verify both Deployments before synchronization is allowed to continue. +The marker is `746d424` for the verified minimum build. If an approved equivalent backport uses a different image tag, set `ETCD_SYNC_IMAGE_PROVENANCE_MARKER` to that build's auditable commit or release marker only after recording the backport evidence; do not disable the provenance check. + +Wait until `ConfigMap/cpaas-system/etcd-sync-ignore-text` contains every key from `ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON`, including the plugin's own ServiceMonitor key. The mirror and monitor processes read their filter files at startup. After the rendered ConfigMap is correct, explicitly restart and verify both Deployments; updating ModuleInfo or waiting for the projected ConfigMap volume to refresh is not sufficient. + +```shell +IGNORE_EQUAL_TEXT="$({ + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get configmap etcd-sync-ignore-text \ + -o jsonpath='{.data.ignore-equal\.txt}' +})" + +printf '%s\n' "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" | jq -r '.[]' | \ + while IFS= read -r key; do + if ! printf '%s\n' "${IGNORE_EQUAL_TEXT}" | grep -Fqx -- "${key}"; then + echo "missing etcd-sync exact-ignore key: ${key}" >&2 + exit 1 + fi + done + +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system rollout restart deployment/etcd-sync +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system rollout restart deployment/etcd-sync-monitor +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system rollout status deployment/etcd-sync --timeout=5m +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system rollout status deployment/etcd-sync-monitor --timeout=5m +``` + +Trigger an on-demand monitor comparison and require zero missed and zero surplus keys. Also require the filtered local and remote target-key counts to match. The `/check` endpoint returns HTTP 425 when the preceding check ran less than one minute ago; retry without treating that response as success. + +```shell +set -euo pipefail +umask 077 + +PORT_FORWARD_LOG="$(mktemp)" +CHECK_RESPONSE="$(mktemp)" +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system port-forward service/etcd-sync-monitor 18080:80 \ + >"${PORT_FORWARD_LOG}" 2>&1 & +PORT_FORWARD_PID=$! +trap 'kill "${PORT_FORWARD_PID}" 2>/dev/null || true; rm -f "${PORT_FORWARD_LOG}" "${CHECK_RESPONSE}"' EXIT + +until curl -fsS http://127.0.0.1:18080/healthz >/dev/null; do + kill -0 "${PORT_FORWARD_PID}" + sleep 1 +done + +while true; do + HTTP_CODE="$({ + curl -sS -H 'Accept: application/json' \ + -o "${CHECK_RESPONSE}" -w '%{http_code}' \ + http://127.0.0.1:18080/check + })" + case "${HTTP_CODE}" in + 200) break ;; + 425) sleep 10 ;; + *) echo "etcd-sync monitor check returned HTTP ${HTTP_CODE}" >&2; exit 1 ;; + esac +done + +jq -e ' + (.localMissedKeys | length) == 0 and + (.localSurplusKeys | length) == 0 +' "${CHECK_RESPONSE}" >/dev/null + +METRICS="$(curl -fsS http://127.0.0.1:18080/metrics)" +LOCAL_TARGET_COUNT="$({ + printf '%s\n' "${METRICS}" | \ + awk '$1 ~ /^local_target_keys_count/ {value=$2} END {print value}' +})" +REMOTE_TARGET_COUNT="$({ + printf '%s\n' "${METRICS}" | \ + awk '$1 ~ /^remote_target_keys_count/ {value=$2} END {print value}' +})" +test -n "${LOCAL_TARGET_COUNT}" +test "${LOCAL_TARGET_COUNT}" = "${REMOTE_TARGET_COUNT}" + +kill "${PORT_FORWARD_PID}" +wait "${PORT_FORWARD_PID}" 2>/dev/null || true +rm -f "${PORT_FORWARD_LOG}" "${CHECK_RESPONSE}" +trap - EXIT +``` + +After the first successful synchronization, verify that the shared workload bundle is semantically identical on both sides. This check is valid even when the shared Role has no rules because no workload plan exists yet. The two Global-local token Secrets must remain different. + +```shell +shared_bundle_digest() { + kubeconfig=$1 + resource=$2 + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get "${resource}" -o json | \ + jq -S ' + del( + .metadata.creationTimestamp, + .metadata.generation, + .metadata.managedFields, + .metadata.resourceVersion, + .metadata.uid + ) + ' | sha256sum | awk '{print $1}' +} + +for resource in \ + serviceaccount/baremetal-system-agent \ + secret/baremetal-system-agent-token \ + role/baremetal-system-agent \ + rolebinding/baremetal-system-agent; do + PRIMARY_DIGEST="$(shared_bundle_digest \ + "${PRIMARY_GLOBAL_KUBECONFIG}" "${resource}")" + STANDBY_DIGEST="$(shared_bundle_digest \ + "${STANDBY_GLOBAL_KUBECONFIG}" "${resource}")" + test "${PRIMARY_DIGEST}" = "${STANDBY_DIGEST}" +done + +PRIMARY_LOCAL_TOKEN_SHA="$({ + kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" -n cpaas-system \ + get secret baremetal-global-system-agent-token \ + -o jsonpath='{.data.token}' | base64 -d | sha256sum | awk '{print $1}' +})" +STANDBY_LOCAL_TOKEN_SHA="$({ + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system \ + get secret baremetal-global-system-agent-token \ + -o jsonpath='{.data.token}' | base64 -d | sha256sum | awk '{print $1}' +})" +test "${PRIMARY_LOCAL_TOKEN_SHA}" != "${STANDBY_LOCAL_TOKEN_SHA}" +``` + +Every later configuration update must patch the actual generated ModuleInfo name, then repeat the rendered ConfigMap, rollout, and monitor gates: + +```shell +export ETCD_SYNC_MODULEINFO_NAME="$({ + cat /root/yamls/global-etcd-sync-moduleinfo.name +})" + +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + patch moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" \ + --type=merge \ + -p "$({ + jq -cn --argjson keys "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" \ + '{spec:{config:{additional_ignore_equal:$keys}}}' + })" +``` + +For a planned failover, uninstall the plugin through its owning ModuleInfo only after the consistency gate reports zero missed and surplus keys and the primary has a verified hard fence. Do not use Deployment scale as the lifecycle operation; AppRelease or ModuleInfo reconciliation can restore the mirror. Change `sharedAuthReadOnly` only on the two final Global AppRelease objects during the role switch. Do not patch either retained bootstrap AppRelease in the failover window: `handoffHook.delivery.mode: always` can deliver and re-run handoff state when that AppRelease changes. + +```shell +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + delete moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + wait --for=delete \ + "moduleinfoes.cluster.alauda.io/${ETCD_SYNC_MODULEINFO_NAME}" --timeout=10m +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system wait --for=delete apprelease/etcd-sync --timeout=10m +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system wait --for=delete deployment/etcd-sync --timeout=10m +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system wait --for=delete deployment/etcd-sync-monitor --timeout=10m +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n kube-public wait --for=delete configmap/etcd-sync --timeout=10m +``` + +Only after all of these owned resources disappear may the platform domain move to the newly active side. Keep both final Global AppRelease objects in `sharedAuthReadOnly: true` through the hard-fence, uninstall, and DNS transition. This deliberate no-writer window avoids split-brain ownership. Because both provider AppRelease objects use `${PLATFORM_HOST}:11443` as their chart repository and image registry, wait until the domain and registry answer from the newly active side before setting that side's final AppRelease to `sharedAuthReadOnly: false`; attempting that chart update while the domain still resolves to a fenced old registry fails with `ChartDownloadFailed`. Then wait for the new active `elemental-operator` rollout, verify that the read-only flag is absent, and verify the shared Role against the workload plan set. Never update either retained bootstrap AppRelease during this transition. + +For failback, synchronization runs in the opposite direction. Install a new ModuleInfo on the original primary, which is now the inactive target, and point `active_cluster_vip` and `active_cluster_token` at the currently active former standby. Before creating that reverse ModuleInfo, re-run the two-kubeconfig Role collection against the current state and build `additional_ignore_equal` from the union of both local `Role/baremetal-global-system-agent` plan names plus `ETCD_SYNC_SELF_MONITOR_IGNORE_KEY`. Do not reuse the old forward-direction array or start the reverse mirror before its rendered ConfigMap passes the same exact-key and zero-difference gates. At failback, set the currently active side to read-only before the final check and hard fence, uninstall the reverse ModuleInfo, switch the platform domain to the target registry, and only then set the target final AppRelease to writable. Restart the Pods that must reload DR and endpoint configuration. Run the same commands on a primary control plane node and on a standby control plane node. From f4ee08e15b1a4c9a2625409d021ddc5798a9b11f Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 10:55:08 +0800 Subject: [PATCH 04/14] docs: scope baremetal split auth to fresh installs --- docs/en/global/disaster_recovery.mdx | 695 +-------------------------- docs/en/global/install.mdx | 669 +++++++++++--------------- 2 files changed, 295 insertions(+), 1069 deletions(-) diff --git a/docs/en/global/disaster_recovery.mdx b/docs/en/global/disaster_recovery.mdx index 99d728a3..96a8b006 100644 --- a/docs/en/global/disaster_recovery.mdx +++ b/docs/en/global/disaster_recovery.mdx @@ -72,7 +72,7 @@ Set `handoffHook.controlPlaneVIP` to the current side's VIP and set `handoffHook A `SeedImage` generated in the bootstrap KIND environment points at the bootstrap registration endpoint. After the `global` cluster is installed and handed off, create new `MachineRegistration` and `SeedImage` resources on the active `global` cluster for any new `global` hosts. A pre-failover workload ISO can be reused after failover only when its registration URL uses the platform domain and the matching non-`global` `MachineRegistration` was synchronized to standby. -For every post-handoff Global host registration, set the Global auth scope, the local side's direct API URL, and direct endpoint mode. A primary Global registration always uses the primary VIP, and a standby Global registration always uses the standby VIP; do not change existing Global host endpoints during failover. +For every post-handoff Global host registration, set the Global auth scope, the local side's direct API URL, and direct endpoint mode. A primary Global registration uses the primary VIP, and a standby Global registration uses the standby VIP. ```yaml apiVersion: elemental.cattle.io/v1beta1 @@ -86,14 +86,14 @@ metadata: baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver ``` -`baremetal.cluster.io/system-agent-direct: "true"` remains an upgrade-compatibility alias for `direct-apiserver`. Do not use it in new registrations. If both annotations are present, the explicit `baremetal.cluster.io/system-agent-endpoint-mode` value takes precedence. +Use the explicit `baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver` annotation for every new Global registration. ### System-Agent Identity Requirements Bare Metal DR uses two identity bundles: - The local Global bundle consists of `ServiceAccount`, token Secret, Role, and RoleBinding named `baremetal-global-system-agent` (the Secret name has the `-token` suffix). Each side generates and retains its own bundle. Its Role contains only that side's Global plan Secrets. -- The workload shared bundle consists of the four `baremetal-system-agent` objects. Primary is authoritative before failover, and `etcd-sync` copies the bundle exactly to standby. Its Role contains only non-`global` workload plan Secrets. +- The workload shared bundle consists of the four `baremetal-system-agent` objects. It may be absent on both sides before the first shared-scope workload plan exists. Once primary creates it, `etcd-sync` copies the bundle exactly to standby. Its Role contains only explicitly shared-scope, non-`global` workload plan Secrets. Do not customize `baremetal-global-system-agent`, `baremetal-global-system-agent-token`, `baremetal-system-agent`, or `baremetal-system-agent-token`. The default `etcd-sync` exact-sync and ignore rules match these names literally. If a deployment must use different names, update and validate all corresponding `etcd-sync` rules before starting synchronization. @@ -103,300 +103,6 @@ Primary and standby must still use the same `/etc/kubernetes/pki/sa.key`, `/etc/ Set `elemental.systemAgent.splitAuthEnabled: true` on both provider AppRelease objects. This is the single source of truth used by both `elemental-operator` and handoff; do not configure a separate handoff auth-enable switch. With the split enabled, the active Global must set `elemental.systemAgent.sharedAuthReadOnly: false` so that `elemental-operator` reconciles the shared workload Role. The inactive Global must set it to `true` so that `etcd-sync`, not the local operator, owns the synchronized shared bundle. Both operators continue to reconcile their own Global-local bundles. -### Migrating Existing Handoff State \{#migrating-existing-handoff-state} - -`ConfigMap/cpaas-system/baremetal-system-agent-handoff` is a versioned completion record. The direct-API split-auth configuration requires `data.ready: "true"`, `data.profile: split-local-direct-v1`, and a `data.config` value that is exactly 64 lowercase hexadecimal characters. The digest covers the effective endpoint, token/CA, identity names, and each Global target's names and object UIDs. A ConfigMap written by an older release with only `ready: "true"`, with a `legacy-shared-*` profile, or with a stale, missing, or malformed config digest must not cause the new handoff to skip. - -#### Protect Existing Global Plan Secrets Before the Upgrade - -If `etcd-sync` is already installed, protect the current Global plan Secrets before upgrading either Bare Metal provider AppRelease or running the new handoff. An older environment does not yet have the authoritative `Role/baremetal-global-system-agent`, so derive this temporary exact-ignore set from the current `Cluster/global` `BaremetalMachine` references and cross-check every name against its `MachineInventory`. - -First stop or suspend the mirror with the supported plugin lifecycle operation, and verify that `Deployment/cpaas-system/etcd-sync` has no Ready mirror Pods. Then run the following from a secured host with both direct Global kubeconfigs: - -```shell -set -euo pipefail - -export PRIMARY_GLOBAL_KUBECONFIG="" -export STANDBY_GLOBAL_KUBECONFIG="" -export ETCD_SYNC_MODULEINFO_NAME="" - -existing_global_plan_secret_names() { - local kubeconfig=$1 targets names target inventory expected actual secret_type - - targets="$( - kubectl --kubeconfig "${kubeconfig}" \ - -n cpaas-system \ - get baremetalmachines.infrastructure.cluster.x-k8s.io -o json | \ - jq -c '[ - .items[] - | select(.metadata.labels["cluster.x-k8s.io/cluster-name"] == "global") - | { - inventory: (.status.machineInventoryRef.name // ""), - plan: (.status.planSecretRef.name // "") - } - ]' - )" - - printf '%s\n' "${targets}" | \ - jq -e 'length > 0 and all(.[]; .inventory != "" and .plan != "")' \ - >/dev/null || { - echo "Cluster/global has an incomplete inventory or plan reference" >&2 - return 1 - } - - names="$( - while IFS= read -r target; do - inventory="$(printf '%s\n' "${target}" | jq -r '.inventory')" - expected="$(printf '%s\n' "${target}" | jq -r '.plan')" - actual="$( - kubectl --kubeconfig "${kubeconfig}" \ - -n cpaas-system \ - get machineinventory.elemental.cattle.io "${inventory}" \ - -o jsonpath='{.status.plan.secretRef.name}' - )" - if [ "${actual}" != "${expected}" ]; then - echo "plan Secret mismatch for MachineInventory ${inventory}" >&2 - return 1 - fi - secret_type="$( - kubectl --kubeconfig "${kubeconfig}" \ - -n cpaas-system get secret "${actual}" -o jsonpath='{.type}' - )" - if [ "${secret_type}" != "elemental.cattle.io/plan" ]; then - echo "Secret ${actual} is not an elemental plan" >&2 - return 1 - fi - printf '%s\n' "${actual}" - done < <(printf '%s\n' "${targets}" | jq -c '.[]') - )" - - printf '%s\n' "${names}" | \ - jq -Rsc 'split("\n") | map(select(length > 0)) | unique | sort' -} - -PRIMARY_EXISTING_GLOBAL_PLAN_SECRET_NAMES="$( - existing_global_plan_secret_names "${PRIMARY_GLOBAL_KUBECONFIG}" -)" -STANDBY_EXISTING_GLOBAL_PLAN_SECRET_NAMES="$( - existing_global_plan_secret_names "${STANDBY_GLOBAL_KUBECONFIG}" -)" - -CURRENT_ADDITIONAL_IGNORE_EQUAL_JSON="$( - kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - get moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" -o json | \ - jq -c '.spec.config.additional_ignore_equal // []' -)" - -export DISCOVERED_GLOBAL_PLAN_IGNORE_EQUAL_JSON="$( - jq -cn \ - --argjson primary "${PRIMARY_EXISTING_GLOBAL_PLAN_SECRET_NAMES}" \ - --argjson standby "${STANDBY_EXISTING_GLOBAL_PLAN_SECRET_NAMES}" ' - ($primary + $standby) - | unique - | sort - | map("/registry/secrets/cpaas-system/" + .) - ' -)" - -export TEMPORARY_GLOBAL_PLAN_IGNORE_EQUAL_JSON="$( - jq -cn \ - --argjson current "${CURRENT_ADDITIONAL_IGNORE_EQUAL_JSON}" \ - --argjson discovered "${DISCOVERED_GLOBAL_PLAN_IGNORE_EQUAL_JSON}" \ - '($current + $discovered) | unique | sort' -)" - -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - patch moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" \ - --type=merge \ - -p "$( - jq -cn --argjson keys "${TEMPORARY_GLOBAL_PLAN_IGNORE_EQUAL_JSON}" \ - '{spec:{config:{additional_ignore_equal:$keys}}}' - )" -``` - -Do not resume synchronization yet. Wait until `ConfigMap/cpaas-system/etcd-sync-ignore-text` contains every temporary exact key. Then restart the monitor and restore or roll out the mirror with the supported plugin lifecycle operation, and wait for both Deployments as described in [etcd-sync Scope](#etcd-sync-scope). No Ready mirror Pod may start before the rendered filter contains the complete temporary set. Keep this set in place throughout both AppRelease upgrades and handoff migrations. Without it, the broad Secret synchronization rule can delete, replace, or roll back the plan that a Global host must apply. - -#### Re-run the Handoff After the AppRelease Upgrade - -Upgrade the Bare Metal provider AppRelease on each side's retained bootstrap KIND cluster with `handoffHook.delivery.enabled: true` and `handoffHook.delivery.mode: always`, in addition to the split-auth values for that side. The chart's Helm `post-upgrade` Job delivers the current `acp-post-install` script and chart environment; it does not execute the installer post-install stage again. - -Also do not assume that the delivery Job changed the physical bootstrap host's `/cpaas`. In a nested KIND environment, its `hostPath: /cpaas` can refer to the KIND node filesystem. The reliable upgrade procedure is to read the official script and environment from the chart-rendered ConfigMap and explicitly run that script on the physical bootstrap host: - -```shell -set -euo pipefail - -export BOOTSTRAP_KUBECTL=/opt/kubectl -export BOOTSTRAP_KUBECONFIG=/cpaas/data/alauda.kubeconfig -export FINAL_GLOBAL_KUBECONFIG=/cpaas/data/admin.kubeconfig -export HANDOFF_RERUN_DIR=/root/baremetal-handoff-rerun - -sudo install -d -m 0700 "${HANDOFF_RERUN_DIR}" - -sudo "${BOOTSTRAP_KUBECTL}" \ - --kubeconfig "${BOOTSTRAP_KUBECONFIG}" \ - -n cpaas-system get configmap baremetal-acp-post-install \ - -o go-template='{{ index .data "acp-post-install" }}' | \ -sudo install -m 0700 /dev/stdin "${HANDOFF_RERUN_DIR}/acp-post-install" - -sudo "${BOOTSTRAP_KUBECTL}" \ - --kubeconfig "${BOOTSTRAP_KUBECONFIG}" \ - -n cpaas-system get configmap baremetal-acp-post-install \ - -o go-template='{{ index .data "baremetal-handoff.env" }}' | \ -sudo install -m 0600 /dev/stdin \ - "${HANDOFF_RERUN_DIR}/baremetal-handoff-chart.env" - -sudo env \ - KUBECTL="${BOOTSTRAP_KUBECTL}" \ - BOOTSTRAP_KUBECONFIG="${BOOTSTRAP_KUBECONFIG}" \ - GLOBAL_KUBECONFIG="${FINAL_GLOBAL_KUBECONFIG}" \ - BAREMETAL_HANDOFF_CHART_ENV_FILE="${HANDOFF_RERUN_DIR}/baremetal-handoff-chart.env" \ - "${HANDOFF_RERUN_DIR}/acp-post-install" -``` - -Run this separately on the primary and standby bootstrap hosts. Use the ConfigMap from that same side's upgraded AppRelease; do not copy a rendered environment from the peer. The script deliberately replaces a completed, failed, or still-running `baremetal-system-agent-handoff` Job with the current image and arguments. The handoff binary then compares the versioned profile and configuration digest, so a matching configuration exits without rewriting hosts while an old or drifted configuration performs the required in-place migration. Do not write a replacement Job or RBAC manifest by hand. - -This chart-delivered migration path requires the original side-specific bootstrap KIND cluster and its valid bootstrap and final-Global kubeconfigs. If that bootstrap environment has already been removed, stop and use an approved support procedure; installing the AppRelease only on the final Global cluster does not recreate the installer execution context. - -The upgrade migrates the already installed Global hosts in place; it does not require the original bootstrap plan Secret. Keep this safety order for every current Global plan Secret: - -1. Create and validate the local Global ServiceAccount, token Secret, Role, and canonical RoleBinding. Create the independent temporary `RoleBinding/baremetal-global-system-agent-handoff-bridge`, which references the Global-local Role and binds only the old shared ServiceAccount. Do not modify the canonical RoleBinding or the synchronized shared Role. -2. Write the handoff plan on that current Global plan Secret so the host switches to the local token and local direct API endpoint. -3. Wait until the handoff checksum is Applied, then write and observe a Global-side probe with the new local identity. -4. Only after the probe succeeds, delete the temporary bridge. Workload targets and the shared Role are not rewritten by this migration. -5. Treat migration as complete only when all Global hosts have passed the probe, the bridge is absent, and the completion ConfigMap reports the exact new profile and config digest. - -This temporary overlap is deliberate: removing the bridge before the old host has consumed its token-switch plan can strand the host with `403 Forbidden`. Do not grant namespace-wide Secret access, do not use a `ClusterRoleBinding`, and do not manually patch completion fields to bypass the migration. - -```shell -kubectl --kubeconfig \ - -n cpaas-system get configmap baremetal-system-agent-handoff \ - -o jsonpath='{.data.ready}{" "}{.data.profile}{" "}{.data.config}{"\n"}' -``` - -Expected output: `true split-local-direct-v1 <64-lowercase-hex-character-SHA-256>`. - -After both sides report the expected completion and neither side has the temporary bridge, replace the temporary pre-upgrade list with the authoritative list generated from both local Global Roles: - -```shell -global_role_plan_secret_names() { - local kubeconfig=$1 - kubectl --kubeconfig "${kubeconfig}" \ - -n cpaas-system get role baremetal-global-system-agent -o json | \ - jq -c '[ - .rules[]? - | select(any(.apiGroups[]?; . == "")) - | select(any(.resources[]?; . == "secrets")) - | .resourceNames[]? - ] | unique | sort' -} - -PRIMARY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES="$( - global_role_plan_secret_names "${PRIMARY_GLOBAL_KUBECONFIG}" -)" -STANDBY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES="$( - global_role_plan_secret_names "${STANDBY_GLOBAL_KUBECONFIG}" -)" - -printf '%s\n' "${PRIMARY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES}" | \ - jq -e 'length > 0' >/dev/null -printf '%s\n' "${STANDBY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES}" | \ - jq -e 'length > 0' >/dev/null - -AUTHORITATIVE_GLOBAL_PLAN_IGNORE_EQUAL_JSON="$( - jq -cn \ - --argjson primary "${PRIMARY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES}" \ - --argjson standby "${STANDBY_AUTHORITATIVE_GLOBAL_PLAN_SECRET_NAMES}" ' - ($primary + $standby) - | unique - | sort - | map("/registry/secrets/cpaas-system/" + .) - ' -)" - -CURRENT_ADDITIONAL_IGNORE_EQUAL_JSON="$( - kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - get moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" -o json | \ - jq -c '.spec.config.additional_ignore_equal // []' -)" - -FINAL_ADDITIONAL_IGNORE_EQUAL_JSON="$( - jq -cn \ - --argjson current "${CURRENT_ADDITIONAL_IGNORE_EQUAL_JSON}" \ - --argjson temporary "${DISCOVERED_GLOBAL_PLAN_IGNORE_EQUAL_JSON}" \ - --argjson authoritative "${AUTHORITATIVE_GLOBAL_PLAN_IGNORE_EQUAL_JSON}" ' - (($current - $temporary) + $authoritative) | unique | sort - ' -)" - -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - patch moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" \ - --type=merge \ - -p "$( - jq -cn --argjson keys "${FINAL_ADDITIONAL_IGNORE_EQUAL_JSON}" \ - '{spec:{config:{additional_ignore_equal:$keys}}}' - )" -``` - -If the shell that performed the pre-upgrade collection is no longer available, re-run `existing_global_plan_secret_names` to reconstruct `DISCOVERED_GLOBAL_PLAN_IGNORE_EQUAL_JSON` before calculating the final list. Review any other custom entries separately. The final configuration must ensure that: - -- the Global plan entries are exactly the merged `resourceNames` from both `Role/baremetal-global-system-agent` objects; -- unrelated, explicitly approved additional ignore entries are preserved; and -- any temporary Global plan entry that is absent from both local Roles is removed. - -Verify the rendered ignore ConfigMap again and restart both `etcd-sync` Deployments before synchronization continues. From this point forward, the two local Roles, not the legacy BaremetalMachine list, are the authoritative source for Global plan exact-ignore entries. - -Verify both permission boundaries before failover. Repeat the checks on primary and standby with an administrator kubeconfig that is allowed to impersonate ServiceAccounts. Set the two example Secret names from that same cluster. - -```shell -export LOCAL_GLOBAL_KUBECONFIG="" -export LOCAL_GLOBAL_PLAN_SECRET="" -export WORKLOAD_PLAN_SECRET="" - -# Expected: yes -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ - -n cpaas-system auth can-i get "secret/${LOCAL_GLOBAL_PLAN_SECRET}" -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ - -n cpaas-system auth can-i patch "secret/${LOCAL_GLOBAL_PLAN_SECRET}" -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ - -n cpaas-system auth can-i get "secret/${WORKLOAD_PLAN_SECRET}" -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ - -n cpaas-system auth can-i patch "secret/${WORKLOAD_PLAN_SECRET}" - -# Expected: no (cross-scope access) -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ - -n cpaas-system auth can-i get "secret/${WORKLOAD_PLAN_SECRET}" -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ - -n cpaas-system auth can-i patch "secret/${WORKLOAD_PLAN_SECRET}" -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ - -n cpaas-system auth can-i get "secret/${LOCAL_GLOBAL_PLAN_SECRET}" -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ - -n cpaas-system auth can-i patch "secret/${LOCAL_GLOBAL_PLAN_SECRET}" - -# Expected: no (namespace-wide or non-plan Secret access) -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ - -n cpaas-system auth can-i list secrets -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ - -n cpaas-system auth can-i list secrets -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ - -n cpaas-system auth can-i get secret/global-registry-auth -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ - -n cpaas-system auth can-i get secret/global-registry-auth -``` - -Expected result: `get` and `patch` are allowed only within the matching permission scope. Every cross-scope check, both namespace-wide `list secrets` checks, and both `global-registry-auth` checks return `no`. Also compare the token hash in `/var/lib/elemental/agent/elemental_connection.json` on every Global host with the local `baremetal-global-system-agent-token` Secret; checking only the Kubernetes Secret does not prove that an existing host uses the expected token. - ### etcd-sync Scope \{#etcd-sync-scope} Deploy `etcd-sync` on the standby cluster. The sync direction is one-way before failover: @@ -413,7 +119,7 @@ Synchronize the workload-cluster resources that allow standby to continue reconc - Non-`global` elemental resources: `MachineInventory` and `MachineRegistration`. - Plan Secrets, bootstrap data Secrets, workload kubeconfig Secrets, and workload credential Secrets that are required by the synchronized resources. - Provider referenced Secrets that must exist on the target side, including the encryption-provider Secret referenced by `BaremetalCluster.spec.encryptionProviderConfigRef` when Bare Metal DR is enabled. -- Exact shared auth bundle: `ServiceAccount/cpaas-system/baremetal-system-agent`, `Secret/cpaas-system/baremetal-system-agent-token`, `Role/cpaas-system/baremetal-system-agent`, and `RoleBinding/cpaas-system/baremetal-system-agent`. +- Exact shared auth bundle, once it exists: `ServiceAccount/cpaas-system/baremetal-system-agent`, `Secret/cpaas-system/baremetal-system-agent-token`, `Role/cpaas-system/baremetal-system-agent`, and `RoleBinding/cpaas-system/baremetal-system-agent`. Do not synchronize: @@ -421,138 +127,40 @@ Do not synchronize: - Standby `Cluster/cpaas-system/global`. - Standby `BaremetalCluster/cpaas-system/global`. - Standby `BaremetalMachine`, `MachineInventoryPool`, `MachineInventory`, and `MachineRegistration` objects that belong to the standby `global` cluster. -- `ServiceAccount/cpaas-system/baremetal-global-system-agent`, `Secret/cpaas-system/baremetal-global-system-agent-token`, `Role/cpaas-system/baremetal-global-system-agent`, `RoleBinding/cpaas-system/baremetal-global-system-agent`, and the temporary `RoleBinding/cpaas-system/baremetal-global-system-agent-handoff-bridge` from either side. +- `ServiceAccount/cpaas-system/baremetal-global-system-agent`, `Secret/cpaas-system/baremetal-global-system-agent-token`, `Role/cpaas-system/baremetal-global-system-agent`, and `RoleBinding/cpaas-system/baremetal-global-system-agent` from either side. +- `ConfigMap/cpaas-system/baremetal-system-agent-handoff` from either side. +- The standby-local token Secret selected by `active_cluster_token_secret_ref` and `ServiceMonitor/cpaas-system/etcd-sync-monitor`; MR `!179` renders exact-ignore entries for both. - Plan Secrets that belong to either side's local `Cluster/global`. Add their exact etcd keys to `etcd_sync.additional_ignore_equal` before starting synchronization. - Secrets from the primary cluster that are not required by workload cluster reconciliation. -Before installing the ModuleInfo, follow the executable two-kubeconfig procedure in [Install Primary and Standby Clusters](./install.mdx#install-primary-and-standby-clusters) to collect `Role/baremetal-global-system-agent.rules[].resourceNames` from both sides, merge and de-duplicate the names, and populate `spec.config.additional_ignore_equal` with exact `/registry/secrets/cpaas-system/` keys. Repeat that update before synchronization continues whenever either local Global Role gains another plan Secret. - -The mirror and monitor read their filter files at process startup. After every dynamic `additional_ignore_equal` update, first verify that the rendered `ConfigMap/cpaas-system/etcd-sync-ignore-text` contains every exact key, then trigger and wait for rollouts of both `Deployment/etcd-sync` and `Deployment/etcd-sync-monitor`. Updating ModuleInfo or waiting for the ConfigMap volume to refresh is not sufficient. - -```shell -export STANDBY_GLOBAL_KUBECONFIG="" - -IGNORE_EQUAL_TEXT="$( - kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system get configmap etcd-sync-ignore-text \ - -o jsonpath='{.data.ignore-equal\.txt}' -)" - -printf '%s\n' "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" | jq -r '.[]' | \ - while IFS= read -r key; do - if ! printf '%s\n' "${IGNORE_EQUAL_TEXT}" | grep -Fqx -- "${key}"; then - echo "missing etcd-sync ignore key: ${key}" >&2 - exit 1 - fi - done - -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system rollout restart deployment/etcd-sync -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system rollout restart deployment/etcd-sync-monitor -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system rollout status deployment/etcd-sync --timeout=5m -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system rollout status deployment/etcd-sync-monitor --timeout=5m -``` - -### Adding a Global Host While Synchronization Is Configured \{#adding-a-global-host-while-synchronization-is-configured} - -Never create or allocate a new primary or standby Global host while the mirror is actively synchronizing and the new plan Secret is absent from `additional_ignore_equal`. Use this sequence: - -1. Stop or suspend the `etcd-sync` mirror with the supported plugin lifecycle operation, and verify that `Deployment/cpaas-system/etcd-sync` has no Ready mirror Pods. If a direct scale-to-zero is immediately reverted by ModuleInfo or AppRelease reconciliation, use the owning platform operation instead of racing the controller. -2. Create the local Global `MachineRegistration` and `SeedImage`, register the host, and wait until the local `Role/baremetal-global-system-agent` contains the new plan Secret. Keep synchronization stopped. -3. Re-run the two-kubeconfig collection, update `global-etcd-sync` ModuleInfo with the complete merged exact-key list, and verify the rendered `etcd-sync-ignore-text` ConfigMap. -4. Restart the monitor and restore or roll out the mirror only after the new exact key is present. Wait for both Deployments to become available and confirm that the mirror remains healthy. -5. Re-run the Global-local and shared RBAC allow/deny checks for the new plan Secret before considering the host operational. +Before installing the ModuleInfo, follow the executable two-kubeconfig procedure in [Install Primary and Standby Clusters](./install.mdx#install-primary-and-standby-clusters) to collect `Role/baremetal-global-system-agent.rules[].resourceNames` from both sides, merge and de-duplicate the names, and populate `spec.config.additional_ignore_equal` with exactly the corresponding `/registry/secrets/cpaas-system/` keys. Do not duplicate static ignore keys in this dynamic list. -Do not guess a plan Secret name from the inventory name. Read it from the local Global Role or `MachineInventory.status.plan.secretRef.name`, and do not resume synchronization if the two sources disagree. +MR `!179` adds a checksum of the rendered sync-key ConfigMap to both Deployment Pod templates. A fresh installation therefore starts with the final filter and does not need a manual rollout restart. Verify the dynamic and static entries, checksum annotations, Deployment readiness, and zero-difference monitor result with the installation guide's acceptance commands. - Do not raw-sync every key under `/registry/secrets` when the primary and standby apiservers do not share the same Kubernetes encryption-provider configuration. The standby apiserver cannot decrypt ciphertext written with a different key. Sync only the required Secrets through a safe path that lets the standby apiserver store them with the standby encryption key, or use an approved exact-sync mechanism for the fixed auth bundle. + Do not raw-sync every key under `/registry/secrets` when the primary and standby apiservers do not share the same Kubernetes encryption-provider configuration. The standby apiserver cannot decrypt ciphertext written with a different key. The fresh DR installation must use the same encryption provider configuration on both sides before synchronization starts. ### Failover Procedure Use this sequence for a planned or declared failover: -1. Stop new writes through the old active platform entrance, but keep the old active etcd endpoint available for the final synchronization check. Stop or isolate the owning platform resources that can restore an entry-point Deployment; scaling a Deployment alone is not a fence because its owner can recreate it. -2. On the old active Global cluster, update only the final Bare Metal provider AppRelease to `elemental.systemAgent.sharedAuthReadOnly: true`. Do not update the retained bootstrap AppRelease during the failover window. Wait for `Deployment/cpaas-system/elemental-operator` to roll out and verify that its arguments contain `--system-agent-shared-auth-read-only`. The new active side must still be read-only at this point, creating an intentional no-writer window for the shared bundle. -3. Run the final `etcd-sync-monitor /check` and the separate shared ServiceAccount/token/RBAC semantic checks. Require zero missed and surplus keys. Immediately after that check, hard-fence the old active control plane so it cannot create any later writes. A hard fence must have an external readback, such as all old active control-plane VMs being powered off with autostart disabled, or equivalent API and etcd network isolation. DNS changes and controller scale-to-zero operations are not hard fences. -4. Uninstall `etcd-sync` through its owning ModuleInfo. Do not delete only the AppRelease, scale the mirror, clear `ConfigMap/kube-public/etcd-sync`, or remove finalizers. Wait until the ModuleInfo, `AppRelease/cpaas-system/etcd-sync`, both `etcd-sync` Deployments, and `ConfigMap/kube-public/etcd-sync` are all absent. -5. Switch the platform domain to the new active Global entrance and wait for both the platform endpoint and `${PLATFORM_HOST}:11443` to resolve to and answer from that side. This step must be after the plugin uninstall because the plugin DNS checker must not resolve its source domain to the target cluster itself. It must be before changing the new active AppRelease when that AppRelease downloads its chart from the platform-domain registry; otherwise the ownership update cannot render after the old active registry is fenced. -6. On the Global cluster that is becoming active, update only the final Bare Metal provider AppRelease to `elemental.systemAgent.sharedAuthReadOnly: false` and wait for `Deployment/cpaas-system/elemental-operator` to roll out. Verify that its arguments no longer contain `--system-agent-shared-auth-read-only`, and verify that `Role/cpaas-system/baremetal-system-agent.resourceNames` matches the workload-scoped `MachineInventory.status.plan.secretRef.name` set. This confirms that the new active operator has taken ownership of shared RBAC. - - ```shell - export NEW_ACTIVE_GLOBAL_KUBECONFIG="" - - kubectl --kubeconfig "${NEW_ACTIVE_GLOBAL_KUBECONFIG}" \ - -n cpaas-system rollout status deployment/elemental-operator - - if kubectl --kubeconfig "${NEW_ACTIVE_GLOBAL_KUBECONFIG}" \ - -n cpaas-system get deployment elemental-operator -o json | \ - jq -e '[.spec.template.spec.containers[].args[]?] - | index("--system-agent-shared-auth-read-only") != null' >/dev/null; then - echo "elemental-operator is still in shared auth read-only mode" >&2 - exit 1 - fi - - WORKLOAD_PLAN_SECRET_NAMES="$( - kubectl --kubeconfig "${NEW_ACTIVE_GLOBAL_KUBECONFIG}" \ - -n cpaas-system get machineinventories.elemental.cattle.io -o json | \ - jq -c '[ - .items[] - | . as $inventory - | ( - $inventory.metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] // - (if (($inventory.metadata.annotations["baremetal.alauda.io/owner-cluster"] // "") | ascii_downcase) == "global" - then "global" - else "shared" - end) - ) as $scope - | select(($scope | ascii_downcase) == "shared") - | .status.plan.secretRef.name? // empty - ] | unique | sort' - )" - SHARED_ROLE_SECRET_NAMES="$( - kubectl --kubeconfig "${NEW_ACTIVE_GLOBAL_KUBECONFIG}" \ - -n cpaas-system get role baremetal-system-agent -o json | \ - jq -c '[ - .rules[]? - | select(any(.apiGroups[]?; . == "")) - | select(any(.resources[]?; . == "secrets")) - | .resourceNames[]? - ] | unique | sort' - )" - - if [ "${WORKLOAD_PLAN_SECRET_NAMES}" != "${SHARED_ROLE_SECRET_NAMES}" ]; then - echo "shared Role does not match workload plan Secrets" >&2 - exit 1 - fi - ``` - -7. Wait until at least 60 seconds after `ConfigMap/kube-public/etcd-sync` disappeared, then verify that the new active controllers are no longer gated. Do not delete the gate ConfigMap by hand. -8. Verify the workload ingress, registry, and the new active Global's direct API endpoint: +1. Fence the primary `global` write path. Stop or isolate the controllers and platform entry points that can continue writing workload cluster resources. Do this at the owning CR or platform-resource layer where possible; scaling a Deployment alone can be reverted by its owner. +2. Wait until primary-to-standby `etcd-sync` is caught up, then freeze or stop it. Do not allow old primary data to overwrite new standby writes after failover. +3. If your provider uses a sync gate, clear the gate after the sync has been stopped so standby controllers do not keep requeueing because they think synchronization is still in progress. +4. Switch the platform domain to the standby platform entrance. +5. Verify DNS and ingress: ```shell - export PLATFORM_DOMAIN="" - export NEW_ACTIVE_CONTROL_PLANE_VIP="" - - dig +short "${PLATFORM_DOMAIN}" - curl -ksS -w '\nHTTP %{http_code}\n' \ - "https://${PLATFORM_DOMAIN}/kubernetes/global/version" - curl -ksS -w '\nHTTP %{http_code}\n' \ - "https://${PLATFORM_DOMAIN}:11443/v2/" - curl -ksS -w '\nHTTP %{http_code}\n' \ - "https://${NEW_ACTIVE_CONTROL_PLANE_VIP}:6443/version" + dig +short + curl -kI https:///kubernetes/global/version + curl -kI https:///kubernetes/global/version ``` - A Kubernetes version response with HTTP `200`, or an authentication/authorization response with HTTP `401` or `403`, proves that the request reached the intended API endpoint. - -9. Confirm the standby cluster has the provider components needed to manage Bare Metal workload clusters: CAPI core, kubeadm provider, Bare Metal provider, elemental-operator, cluster-transformer, and alive-related delivery components. -10. Confirm existing workload hosts reconnect through the platform domain and can watch or patch their plan Secrets. Keep the former primary hard-fenced until the failback procedure has prepared it as the inactive peer. + A `401` response is acceptable; it proves the request reached the platform ingress. -Failback uses the same ownership sequence in reverse. Start the original primary in isolation with `sharedAuthReadOnly: true`, install reverse `etcd-sync` on it, and synchronize from the currently active former standby. After reverse synchronization is caught up, first change the side becoming inactive to `true`, run the final checks, and hard-fence it. Uninstall the reverse mirror through its owning ModuleInfo and wait for the gate to disappear. Then switch the platform domain so the target AppRelease can use the target-side platform registry, change the side becoming active to `false`, verify ownership, and leave the inactive side fenced until forward synchronization has been re-established. Never reuse the old direction's token or exact-ignore array. +6. Confirm the standby cluster has the provider components needed to manage Bare Metal workload clusters: CAPI core, kubeadm provider, Bare Metal provider, elemental-operator, cluster-transformer, and alive-related delivery components. +7. Confirm existing workload hosts reconnect through the platform domain and can watch or patch their plan Secrets. ### Post-Failover Validation @@ -564,7 +172,7 @@ After standby is active, validate in increasing risk order: - Worker scale-out allocates an available inventory and updates `Role/baremetal-system-agent.resourceNames` with the new plan Secret. - Worker scale-in runs a clean plan and releases inventory as designed. - Control-plane replacement restores the desired KCP replica count and etcd membership. -- After following the stop-sync and exact-ignore update procedure, a new `MachineRegistration` and `SeedImage` created on standby can register a new host. +- A new `MachineRegistration` and `SeedImage` created on standby can register a new host. - A pre-failover workload ISO is usable only if its registration URL uses the platform domain. ## Bare Metal Troubleshooting @@ -572,264 +180,11 @@ After standby is active, validate in increasing risk order: | Symptom | Likely cause | First check | |---------|--------------|-------------| | Handoff target list is empty | `dcs-import-extra-resources` was missing or incomplete before the installer API call | `kubectl -n cpaas-system get cm dcs-import-extra-resources -o yaml`; check `baremetal-system-agent-handoff` Job logs | -| Existing hosts get `401` or `Forbidden` after failover | ServiceAccount signing key, issuer, audience, token Secret, auth scope, or Role `resourceNames` does not match | `sha256sum /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub`; repeat the scoped `kubectl auth can-i` checks above | -| Existing workload hosts fail TLS verification after the domain switch, or Global hosts fail against their local VIP | The platform certificate does not cover the platform domain, or the local kube-apiserver certificate/CA does not cover and trust the direct VIP | Check `cpaas-system/dex.tls`, the kube-apiserver serving certificate SANs, the agent connection CA, `curl -ksS https:///kubernetes/global/version`, and `curl -ksS https://:6443/version` | -| Recovery registration completes but an older host reports `x509: certificate signed by unknown authority` against the direct API VIP | The operator lacks the legacy direct-API CA bundle compatibility, or its configured ingress CA Secret/key does not match the mounted certificate chain | Verify the operator image contains Elemental PR #3, compare the provider AppRelease CA values with the `elemental-operator` volume and `--ca-cert-file`, and confirm the ServiceAccount token Secret contains the kube-apiserver `ca.crt` | +| Existing hosts get `401` or `Forbidden` after failover | ServiceAccount signing key, issuer, audience, token Secret, or Role `resourceNames` does not match | `sha256sum /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub`; `kubectl auth can-i` with the token | +| Existing hosts fail TLS verification after DNS switch | Platform certificate chain or SAN does not cover the platform domain and both VIPs | Check `cpaas-system/dex.tls`, certificate SANs, and `curl -kI https:///kubernetes/global/version` | | Standby `global` resources are replaced by primary resources | `etcd-sync` ignore rules did not exclude `global` resources | Inspect the `etcd-sync` ignore ConfigMap and stop sync before restoring standby resources | | New host cannot register with an old ISO | ISO registration URL points to bootstrap or primary-only IP, or `MachineRegistration` was not synchronized | Inspect the `MachineRegistration` status and rebuild the ISO on active standby when needed | -### Recover a Global Host That Cannot Authenticate During Migration \{#recover-a-global-host-that-cannot-authenticate-during-migration} - -Use this procedure only when an existing Global host cannot watch its plan because its legacy shared token no longer authenticates, for example after the shared ServiceAccount was replaced before that host consumed the split-auth migration plan. A temporary migration RoleBinding cannot repair an invalid JWT; it only preserves authorization for a token that the API server can still authenticate. - -This is an emergency connection repair for an already installed host. It reuses the existing registration state and MachineInventory. It must not reset, reinstall, or reprovision the host. - -Before starting, install the upgraded Bare Metal provider and `elemental-operator` on the affected final Global cluster. Wait for the local Global ServiceAccount, token Secret, Role, canonical RoleBinding, and the handoff-created migration bridge. Do not create the bridge manually. - -The operator image must include the legacy direct-kube-apiserver CA compatibility from Elemental PR #3 (`c8f7cf235cbfee0aa3973207f1091e5bc6c81cdd`, or an explicitly verified backport). For a direct-API registration update, that implementation returns the ingress and kube-apiserver CAs together in `Registration.CACert`, while keeping only the kube-apiserver CA in `SystemAgent.CACert`. Current register binaries use `SystemAgent.CACert`; older binaries, including the observed `fe5fd342` build that predates that field, ignore it and use the combined `Registration.CACert` when writing the direct-API kubeconfig. - -Confirm the deployed operator image digest belongs to a release containing that compatibility before creating a recovery capability. Record the host's existing `/usr/sbin/elemental-register --version`, but do not replace the installed binary. If the operator provenance is unknown, or it returns only the ingress CA in `Registration.CACert` for a direct-API registration, stop: the recovered agent will fail TLS verification against the kube-apiserver VIP. - -1. Create a dedicated recovery `MachineRegistration` on the affected final Global cluster. Copy only `spec` from the original bootstrap registration; do not copy `status`, UID, owner references, or a `SeedImage`. - - ```shell - set -euo pipefail - - export BOOTSTRAP_KUBECONFIG="" - export LOCAL_GLOBAL_KUBECONFIG="" - export BOOTSTRAP_MACHINE_REGISTRATION="" - export RECOVERY_MACHINE_REGISTRATION="global-system-agent-recovery" - export LOCAL_GLOBAL_CONTROL_PLANE_VIP="" - - kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" \ - -n cpaas-system \ - get machineregistration.elemental.cattle.io \ - "${BOOTSTRAP_MACHINE_REGISTRATION}" -o json | \ - jq --arg name "${RECOVERY_MACHINE_REGISTRATION}" \ - --arg vip "${LOCAL_GLOBAL_CONTROL_PLANE_VIP}" ' - { - apiVersion: .apiVersion, - kind: .kind, - metadata: { - name: $name, - namespace: "cpaas-system", - annotations: { - "baremetal.cluster.io/system-agent-auth-scope": "global", - "baremetal.cluster.io/system-agent-server-url": ("https://" + $vip + ":6443"), - "baremetal.cluster.io/system-agent-endpoint-mode": "direct-apiserver" - } - }, - spec: .spec - }' | \ - kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" apply -f - - - kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - -n cpaas-system wait --for=condition=Ready \ - "machineregistration.elemental.cattle.io/${RECOVERY_MACHINE_REGISTRATION}" \ - --timeout=180s - - test "$( - kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - -n cpaas-system get machineregistration.elemental.cattle.io \ - "${RECOVERY_MACHINE_REGISTRATION}" \ - -o jsonpath='{.status.serviceAccountRef.name}' - )" = "baremetal-global-system-agent" - ``` - - Do not print or copy `status.registrationURL` into a ticket, terminal transcript, or shared log. It contains a registration capability token. - -2. Verify that the existing MachineInventory plan Secret has the name required by the current registration server. If the two names differ, stop; this repair would connect the agent to the wrong Secret. - - ```shell - export GLOBAL_MACHINE_INVENTORY="" - - PLAN_SECRET="$( - kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - -n cpaas-system get machineinventory.elemental.cattle.io \ - "${GLOBAL_MACHINE_INVENTORY}" \ - -o jsonpath='{.status.plan.secretRef.name}' - )" - - test "${PLAN_SECRET}" = "${GLOBAL_MACHINE_INVENTORY}" - ``` - -3. Fetch the recovery registration configuration without switching global DNS and without exposing the capability URL in a process argument. Read the ingress CA Secret name and key from the affected cluster's actual Bare Metal provider AppRelease, and verify that they match the volume mounted into the running `elemental-operator`. Do not assume the key is `ca.crt`: a final Global `dex.tls` Secret can contain only `tls.crt` and `tls.key`, with the trusted certificate chain stored in `tls.crt`. - - ```shell - set -euo pipefail - umask 077 - - export PLATFORM_DOMAIN="" - export PROVIDER_APPRELEASE=cluster-api-provider-baremetal - W="$(mktemp -d)" - trap 'rm -rf "${W}"' EXIT - - PROVIDER_APPRELEASE_JSON="$( - kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - -n cpaas-system get apprelease "${PROVIDER_APPRELEASE}" -o json - )" - - APPRELEASE_CA_SECRET_NAME="$( - printf '%s\n' "${PROVIDER_APPRELEASE_JSON}" | \ - jq -r '.spec.values.elemental.tls.caCertSecretName // empty' - )" - APPRELEASE_CA_SECRET_KEY="$( - printf '%s\n' "${PROVIDER_APPRELEASE_JSON}" | \ - jq -r '.spec.values.elemental.tls.caCertSecretKey // empty' - )" - - # If the AppRelease omitted either value, set the corresponding RECOVERY_* - # override only after inspecting the deployed operator volume and argument. - RECOVERY_CA_SECRET_NAME="${RECOVERY_CA_SECRET_NAME:-${APPRELEASE_CA_SECRET_NAME}}" - RECOVERY_CA_SECRET_KEY="${RECOVERY_CA_SECRET_KEY:-${APPRELEASE_CA_SECRET_KEY}}" - - if [ -z "${RECOVERY_CA_SECRET_NAME}" ] || [ -z "${RECOVERY_CA_SECRET_KEY}" ]; then - echo "the provider AppRelease does not identify the Elemental CA Secret and key" >&2 - exit 1 - fi - - ELEMENTAL_OPERATOR_DEPLOYMENT_JSON="$( - kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - -n cpaas-system get deployment elemental-operator -o json - )" - DEPLOYED_CA_SECRET_NAME="$( - printf '%s\n' "${ELEMENTAL_OPERATOR_DEPLOYMENT_JSON}" | \ - jq -r '[.spec.template.spec.volumes[]? - | select(.name == "elemental-ca-cert") - | .secret.secretName][0] // empty' - )" - DEPLOYED_CA_SECRET_KEY="$( - printf '%s\n' "${ELEMENTAL_OPERATOR_DEPLOYMENT_JSON}" | \ - jq -r '[.spec.template.spec.volumes[]? - | select(.name == "elemental-ca-cert") - | .secret.items[]?.key][0] // empty' - )" - DEPLOYED_CA_FILE_ARG="$( - printf '%s\n' "${ELEMENTAL_OPERATOR_DEPLOYMENT_JSON}" | \ - jq -r '[.spec.template.spec.containers[].args[]? - | select(startswith("--ca-cert-file="))][0] // empty' - )" - DEPLOYED_CA_FILE_KEY="$(basename "${DEPLOYED_CA_FILE_ARG#*=}")" - - if [ "${RECOVERY_CA_SECRET_NAME}" != "${DEPLOYED_CA_SECRET_NAME}" ] || \ - [ "${RECOVERY_CA_SECRET_KEY}" != "${DEPLOYED_CA_SECRET_KEY}" ] || \ - [ "${RECOVERY_CA_SECRET_KEY}" != "${DEPLOYED_CA_FILE_KEY}" ]; then - echo "confirmed CA Secret/key does not match the running elemental-operator" >&2 - exit 1 - fi - - kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - -n cpaas-system get machineregistration.elemental.cattle.io \ - "${RECOVERY_MACHINE_REGISTRATION}" \ - -o jsonpath='{.status.registrationURL}' > "${W}/registration-url" - - kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - -n cpaas-system get secret "${RECOVERY_CA_SECRET_NAME}" -o json | \ - jq -er --arg key "${RECOVERY_CA_SECRET_KEY}" ' - .data[$key] | select(type == "string" and length > 0) - ' | base64 -d > "${W}/registration-ca.crt" - - grep -q -- '-----BEGIN CERTIFICATE-----' "${W}/registration-ca.crt" - openssl x509 -in "${W}/registration-ca.crt" -noout - - { - printf 'url = "' - tr -d '\r\n' < "${W}/registration-url" - printf '"\n' - printf 'cacert = "%s"\n' "${W}/registration-ca.crt" - printf 'resolve = "%s:443:%s"\n' \ - "${PLATFORM_DOMAIN}" "${LOCAL_GLOBAL_CONTROL_PLANE_VIP}" - printf 'fail\nsilent\nshow-error\n' - } > "${W}/curl.conf" - - chmod 0600 "${W}"/* - curl --config "${W}/curl.conf" \ - > "${W}/registration.yaml" \ - 2> "${W}/curl.log" - chmod 0600 "${W}/registration.yaml" - ``` - - This procedure fails closed if the AppRelease value, deployed Secret volume, and actual Secret key do not agree. If you must use a `RECOVERY_CA_SECRET_NAME` or `RECOVERY_CA_SECRET_KEY` override because an older AppRelease omitted explicit values, record the inspected Deployment volume and `--ca-cert-file` argument as evidence before setting it. Do not fall back from a missing `ca.crt` to `tls.crt` merely because that key exists. - - `registration-url`, `curl.conf`, `registration.yaml`, and the command log are sensitive. Keep them readable only by root and never attach them to support records. - -4. Copy `registration.yaml` to a root-only directory under `/run` on the affected host. - - ```shell - export GLOBAL_HOST_ADDRESS="" - - ssh "root@${GLOBAL_HOST_ADDRESS}" \ - 'umask 077 - install -d -m 0700 /run/elemental-reregister - install -m 0600 /dev/stdin /run/elemental-reregister/registration.yaml' \ - < "${W}/registration.yaml" - ``` - -5. On the affected host, verify and preserve the existing registration state, then run `elemental-register` in a temporary mount namespace. The temporary `/etc/hosts` bind mount makes only this command reach the local Global entrance; it does not change system DNS. - - ```shell - export PLATFORM_DOMAIN="" - export LOCAL_GLOBAL_CONTROL_PLANE_VIP="" - R=/run/elemental-reregister - - test -s /oem/registration/state.yaml - grep -q '^initialRegistration:' /oem/registration/state.yaml - - cp -a /etc/hosts "${R}/hosts" - printf '\n%s %s\n' \ - "${LOCAL_GLOBAL_CONTROL_PLANE_VIP}" "${PLATFORM_DOMAIN}" >> "${R}/hosts" - - cp -a /var/lib/elemental/agent/elemental_connection.json \ - "${R}/elemental_connection.before.json" - cp -a /oem/elemental-system-agent.yaml \ - "${R}/elemental-system-agent.before.yaml" - chmod 0600 "${R}"/* - - R="${R}" unshare --mount --propagation private sh -ceu ' - mount --bind "${R}/hosts" /etc/hosts - exec /usr/sbin/elemental-register \ - --config-path "${R}/registration.yaml" \ - --state-path /oem/registration/state.yaml \ - --install \ - --no-toolkit - ' > "${R}/register.log" 2>&1 - - systemctl restart elemental-system-agent.service - systemctl is-active --quiet elemental-system-agent.service - ``` - - Never add `--reset`: it discards the update state and turns this into a new registration or reprovision path. Do not add `--debug`: the returned system-agent configuration and token can be written to the log. Even normal logging includes the registration URL, so keep `register.log` root-only. - - The existing `/oem/registration/state.yaml` must retain the original TPM or emulated-TPM identity. If registration reports that the MachineInventory was not found and reprovisioning is needed, stop. Do not retry with an empty state or `--reset`. - - On an older register binary, the recovered runtime kubeconfig is expected to use the combined `Registration.CACert` bundle returned by the compatible operator. On a current binary, it uses `SystemAgent.CACert`. In both cases, the resulting direct-API kubeconfig must contain the kube-apiserver CA, not only the ingress CA. - -6. As soon as the runtime files have been written, the agent has restarted, and the handoff Job or plan feedback proves that the agent can authenticate and consume its current plan Secret, delete the recovery `MachineRegistration`. This revokes the registration capability; it does not remove the shared local Global ServiceAccount token already written to the host. Do not keep the capability until the final probe merely for convenience. - - ```shell - kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" \ - -n cpaas-system delete machineregistration.elemental.cattle.io \ - "${RECOVERY_MACHINE_REGISTRATION}" - ``` - - If the agent still reports `the server has asked for the client to provide credentials`, TLS verification failure, or no plan observation, do not treat registration as successful and do not proceed to the completion gate. Preserve the root-only diagnostic files while investigating. - -7. Let a still-running handoff Job continue. If the Job already reached its timeout or failed while the host was stranded, re-run the chart-delivered official post-install hook described above; it records the failed logs, deletes the failed Job, and recreates it with the current image and arguments. Do not handcraft a replacement Job. Wait for the affected host's migration handoff checksum and probe to become Applied, then verify all of the normal completion gates: - - - `/oem/elemental-system-agent.yaml` contains `https://:6443`; - - the runtime token matches the local `baremetal-global-system-agent-token` Secret when compared by hash; - - the temporary bridge is deleted by handoff, not manually; and - - the completion ConfigMap reports `true split-local-direct-v1 <64-lowercase-hex-character-SHA-256>`. - - The `--install --no-toolkit` command repairs the runtime system-agent connection files but does not write the persistent OEM handoff file or restart the agent by itself. The migration plan remains responsible for the persistent OEM update and final probe. - -8. Only after all completion gates pass, delete the root-only temporary files from the host: - - ```shell - ssh "root@${GLOBAL_HOST_ADDRESS}" \ - 'rm -rf /run/elemental-reregister' - ``` - ## Provider Notes @@ -860,7 +215,7 @@ Bare Metal requires all of the following to be aligned: - Same trusted platform certificate chain, with SANs that cover the platform domain, primary control-plane VIP, and standby control-plane VIP. - `dcs-import-extra-resources` created before the installer API call on both sides. - Bare Metal provider AppRelease values set with the local `handoffHook.controlPlaneVIP`, `handoffHook.directAPIServer: true`, `elemental.systemAgent.authMode: shared`, `elemental.systemAgent.splitAuthEnabled: true`, and `elemental.systemAgent.globalServiceAccountName: baremetal-global-system-agent`. The active side sets `elemental.systemAgent.sharedAuthReadOnly: false`; the inactive side sets it to `true`. -- The deployed `elemental-operator` image and every newly produced `baremetal-base-image` and `baremetal-base-image-iso` are built from the same pinned elemental commit. Rebuild new installation media whenever that commit changes. Already installed hosts retain their original register binary and rely on operator-side backward compatibility; do not replace that binary during handoff or recovery. +- The deployed `elemental-operator` image and every newly produced `baremetal-base-image` and `baremetal-base-image-iso` are built from the same pinned elemental commit. Rebuild the installation media whenever that commit changes. Do not sync `SeedImage` from primary to standby. Sync non-`global` `MachineRegistration` resources when workload ISOs need to remain valid across failover. Do not sync primary Global `MachineRegistration` resources to standby. Create Global registrations locally with `baremetal.cluster.io/system-agent-auth-scope: global`, `baremetal.cluster.io/system-agent-server-url: https://:6443`, and `baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver`. diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index 08164fd6..3d06cb0f 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -535,7 +535,6 @@ Upload the packages. The Bare Metal provider package deploys `elemental-operator`, while the Bare Metal base image and ISO contain the `elemental-register` binary used by each host. For every newly produced installation medium, build the operator, `baremetal-base-image`, and `baremetal-base-image-iso` from the same pinned `elemental-operator` commit. When that commit changes, rebuild the base image and ISO before using them to install new hosts; record the elemental commit and image digests, and do not rely on a mutable branch name or cached layer as proof of source consistency. - This release-build rule does not mean that an operator upgrade replaces `elemental-register` on an already installed host. Existing hosts retain the binary baked into their original image, so registration protocol and returned configuration changes must remain backward compatible. In particular, direct-kube-apiserver recovery of an older host requires an operator that includes the legacy CA-bundle compatibility described in [Recover a Global Host That Cannot Authenticate During Migration](./disaster_recovery.mdx#recover-a-global-host-that-cannot-authenticate-during-migration); do not copy a newer register binary onto an installed host as an upgrade workaround. Create and apply the AppRelease resources for the Kubeadm provider and the Bare Metal provider. @@ -1170,21 +1169,6 @@ Use [Creating Clusters on Bare Metal](../create-cluster/bare-metal.mdx), [Managi - If a `global` VM or physical host does not have DHCP during the live-ISO boot, configure the NIC manually from the host console before waiting for `MachineInventory` registration. Use the same NetworkManager procedure described in [Creating Clusters on Bare Metal](../create-cluster/bare-metal.mdx#network-connectivity), replacing the example address, gateway, DNS, and connection name with the values for that host. - Do not depend on OS hostname side effects. The bare-metal provider normalizes kubeadm node names and provider IDs from the CAPI and inventory objects. - The `SeedImage` created in the bootstrap cluster is a bootstrap artifact. After handoff, create any new `MachineRegistration` or `SeedImage` on the active `global` cluster. -- Before adding a machine to an already installed DR `global` cluster, stop `etcd-sync` and follow [Adding a Global Host While Synchronization Is Configured](./disaster_recovery.mdx#adding-a-global-host-while-synchronization-is-configured) so the new plan Secret is added to the exact ignore list before synchronization resumes. Explicitly set all three annotations shown below. New `global` machines do not pass through the bootstrap handoff job, so they must use the local Global auth scope and the local direct kube-apiserver endpoint. Non-`global` workload-cluster machines continue to use the platform domain path at `https:///kubernetes/global`. - - ```yaml - apiVersion: elemental.cattle.io/v1beta1 - kind: MachineRegistration - metadata: - name: global- - namespace: cpaas-system - annotations: - baremetal.cluster.io/system-agent-auth-scope: global - baremetal.cluster.io/system-agent-server-url: https://:6443 - baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver - ``` - - `baremetal.cluster.io/system-agent-direct: "true"` remains accepted only as an upgrade-compatibility alias for `direct-apiserver`. Do not add it to new manifests. If both annotations are present, the explicit `baremetal.cluster.io/system-agent-endpoint-mode` value takes precedence. The following YAML is a differential fragment, not a complete manifest that you can apply directly. Merge these `global`-specific changes into the manifest that you prepare from the Bare Metal create-cluster references, then apply the complete manifest file. @@ -1874,13 +1858,12 @@ export ENCRYPTION_PROVIDER_SECRET_B64="" export PRIMARY_CLUSTER_VIP="" export STANDBY_CLUSTER_VIP="" export BAREMETAL_ENCRYPTION_PROVIDER_SECRET="global-encryption-provider-config" -export ETCD_SYNC_VERSION="v4.3.16-hotfix.179.1.g746d4244-t" +export ETCD_SYNC_VERSION="" export ETCD_SYNC_MODULEINFO="/root/yamls/global-etcd-sync-moduleinfo.json" -export ETCD_SYNC_SELF_MONITOR_IGNORE_KEY="/registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor" export SERVICE_ACCOUNT_ISSUER="https://kubernetes.default.svc.cluster.local" ``` -`v4.3.16-hotfix.179.1.g746d4244-t` is the minimum build verified with this split-auth procedure. It contains `etcd-sync` commit `746d424`, which keeps the Global-local handoff objects out of primary-to-standby synchronization. A later build is acceptable only when its release provenance proves that it contains that commit or an explicitly verified equivalent backport. Do not infer compatibility from a higher-looking version string alone. +Use the tested artifact produced from `etcd-sync` MR `!179` head commit `555c699`. That revision contains the Global-local identity, handoff completion, and plugin ServiceMonitor exact-ignore rules, plus the dynamic Global plan-key input. Replace the placeholder with the published artifact version whose provenance proves it contains `555c699` or an explicitly verified equivalent merge commit. Do not infer compatibility from a higher-looking version string alone. Create the encryption provider configuration file on both installation environments. @@ -2175,226 +2158,7 @@ Use namespaced `Role` and `RoleBinding` objects, not `ClusterRoleBinding`. Do no Keep the Global-local objects named `baremetal-global-system-agent` and `baremetal-global-system-agent-token`, and keep the workload shared objects named `baremetal-system-agent` and `baremetal-system-agent-token`. The `etcd-sync` exact-sync and ignore rules use these literal names. Do not customize them unless you also update and validate every corresponding `etcd-sync` rule before synchronization starts. -The handoff completion signal is versioned. For this DR configuration, each final Global cluster must report `data.ready: "true"`, `data.profile: split-local-direct-v1`, and a `data.config` value that is exactly 64 lowercase hexadecimal characters in `ConfigMap/cpaas-system/baremetal-system-agent-handoff`. The digest covers the effective endpoint, token/CA, identity names, and each Global target's names and object UIDs. A ready-only ConfigMap, a `legacy-shared-*` profile, or a stale, missing, or malformed config digest does not satisfy the cleanup gate. Upgrading the provider migrates only installed Global hosts against their current Global plan Secrets. It uses the independent temporary `baremetal-global-system-agent-handoff-bridge` RoleBinding and writes the new profile/config only after the new connection has been probed and the bridge has been deleted. Do not patch completion fields manually. - -#### Upgrade Both AppRelease Copies Before Migrating Existing Hosts - -A retained bootstrap AppRelease and the corresponding final Global AppRelease have different jobs during an upgrade. The final AppRelease upgrades the running provider and `elemental-operator`; the bootstrap AppRelease delivers the current handoff script and side-specific chart environment. Upgrading only one copy leaves either the running operator or the official handoff execution context stale. - -When upgrading an existing pair, stop `etcd-sync` and protect the current Global plan Secrets first as described in [Migrating Existing Handoff State](./disaster_recovery.mdx#migrating-existing-handoff-state). Then patch all four AppRelease objects: primary bootstrap, primary final Global, standby bootstrap, and standby final Global. The bootstrap copies use `ca.crt`; the final copies use `tls.crt`. - -```shell -set -euo pipefail - -export BAREMETAL_PROVIDER_VERSION="" -export PRIMARY_BOOTSTRAP_KUBECONFIG="" -export PRIMARY_GLOBAL_KUBECONFIG="" -export STANDBY_BOOTSTRAP_KUBECONFIG="" -export STANDBY_GLOBAL_KUBECONFIG="" - -patch_baremetal_apprelease() { - kubeconfig=$1 - control_plane_vip=$2 - shared_auth_read_only=$3 - ca_secret_key=$4 - - test "$( - kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ - get apprelease cluster-api-provider-baremetal \ - -o jsonpath='{.spec.source.charts[0].name}' - )" = "ait/chart-cluster-api-provider-baremetal" - - patch="$({ - jq -cn \ - --arg version "${BAREMETAL_PROVIDER_VERSION}" \ - --arg vip "${control_plane_vip}" \ - --argjson read_only "${shared_auth_read_only}" \ - --arg ca_key "${ca_secret_key}" ' - { - spec: { - source: { - charts: [{ - name: "ait/chart-cluster-api-provider-baremetal", - releaseName: "cluster-api-provider-baremetal", - targetRevision: $version - }] - }, - values: { - handoffHook: { - controlPlaneVIP: $vip, - directAPIServer: true, - delivery: {enabled: true, mode: "always"} - }, - elemental: { - systemAgent: { - authMode: "shared", - splitAuthEnabled: true, - serviceAccountName: "baremetal-system-agent", - globalServiceAccountName: "baremetal-global-system-agent", - sharedAuthReadOnly: $read_only - }, - tls: { - agentTLSMode: "strict", - caCertSecretName: "dex.tls", - caCertSecretKey: $ca_key - } - } - } - } - }' - })" - - kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ - patch apprelease cluster-api-provider-baremetal \ - --type=merge -p "${patch}" -} - -wait_baremetal_apprelease() { - kubeconfig=$1 - deadline=$((SECONDS + 600)) - - until kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ - get apprelease cluster-api-provider-baremetal -o json | \ - jq -e --arg version "${BAREMETAL_PROVIDER_VERSION}" ' - .spec.source.charts[0].targetRevision == $version and - ([.status.charts[]?] - | any(.installedRevision == $version and .phase == "Success")) - ' >/dev/null; do - if [ "${SECONDS}" -ge "${deadline}" ]; then - echo "Bare Metal provider AppRelease did not reach Success" >&2 - return 1 - fi - sleep 10 - done -} - -patch_baremetal_apprelease \ - "${PRIMARY_BOOTSTRAP_KUBECONFIG}" "${PRIMARY_CLUSTER_VIP}" false ca.crt -patch_baremetal_apprelease \ - "${PRIMARY_GLOBAL_KUBECONFIG}" "${PRIMARY_CLUSTER_VIP}" false tls.crt -patch_baremetal_apprelease \ - "${STANDBY_BOOTSTRAP_KUBECONFIG}" "${STANDBY_CLUSTER_VIP}" true ca.crt -patch_baremetal_apprelease \ - "${STANDBY_GLOBAL_KUBECONFIG}" "${STANDBY_CLUSTER_VIP}" true tls.crt - -wait_baremetal_apprelease "${PRIMARY_BOOTSTRAP_KUBECONFIG}" -wait_baremetal_apprelease "${PRIMARY_GLOBAL_KUBECONFIG}" -wait_baremetal_apprelease "${STANDBY_BOOTSTRAP_KUBECONFIG}" -wait_baremetal_apprelease "${STANDBY_GLOBAL_KUBECONFIG}" -``` - -Verify the final certificate key against both the Secret and the running operator on each final Global cluster: - -```shell -verify_final_elemental_tls() { - kubeconfig=$1 - - kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ - get secret dex.tls -o json | \ - jq -e ' - (.data["tls.crt"] | type == "string" and length > 0) and - (.data["tls.key"] | type == "string" and length > 0) - ' >/dev/null - - kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ - get apprelease cluster-api-provider-baremetal -o json | \ - jq -e ' - .spec.values.elemental.tls.caCertSecretName == "dex.tls" and - .spec.values.elemental.tls.caCertSecretKey == "tls.crt" - ' >/dev/null - - kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ - get deployment elemental-operator -o json | \ - jq -e ' - any(.spec.template.spec.volumes[]?; - .name == "elemental-ca-cert" and - .secret.secretName == "dex.tls" and - any(.secret.items[]?; .key == "tls.crt" and .path == "tls.crt")) and - any(.spec.template.spec.containers[].args[]?; - . == "--ca-cert-file=/etc/elemental/tls/tls.crt") - ' >/dev/null -} - -verify_final_elemental_tls "${PRIMARY_GLOBAL_KUBECONFIG}" -verify_final_elemental_tls "${STANDBY_GLOBAL_KUBECONFIG}" -``` - -After all four upgrades succeed, explicitly re-run the chart-delivered handoff script on each physical bootstrap host. An AppRelease or Helm `post-upgrade` alone delivers the script but does not execute the installer post-install stage. Use the commands in [Migrating Existing Handoff State](./disaster_recovery.mdx#migrating-existing-handoff-state), and use the ConfigMap from the same side's upgraded bootstrap AppRelease. - -#### Validate a Returned Recovery Configuration Before Use - -The emergency recovery procedure returns a configuration that contains a registration capability and the new system-agent token. Before copying it to a host or running `elemental-register`, validate it on the secured operator host with `yq` v4. Never print the file, enable shell tracing, or attach any derived command output that contains the token or registration URL. - -```shell -set -euo pipefail -set +x -umask 077 - -export RECOVERY_CONFIG="" -export LOCAL_GLOBAL_KUBECONFIG="" -export LOCAL_GLOBAL_CONTROL_PLANE_VIP="" -export PLATFORM_DOMAIN="" -export PLAN_SECRET="" -export EXPECTED_REGISTRATION_PREFIX="https://${PLATFORM_DOMAIN}/" -export EXPECTED_SYSTEM_AGENT_URL="https://${LOCAL_GLOBAL_CONTROL_PLANE_VIP}:6443" - -test "$(yq -r '.elemental."system-agent".url' "${RECOVERY_CONFIG}")" = \ - "${EXPECTED_SYSTEM_AGENT_URL}" -test "$(yq -r '.elemental."system-agent"."secret-name"' "${RECOVERY_CONFIG}")" = \ - "${PLAN_SECRET}" -test "$(yq -r '.elemental."system-agent"."secret-namespace"' "${RECOVERY_CONFIG}")" = \ - "cpaas-system" - -yq -e '.elemental."system-agent".strictTLSMode == true' \ - "${RECOVERY_CONFIG}" >/dev/null -yq -e '.elemental.registration."no-toolkit" == true' \ - "${RECOVERY_CONFIG}" >/dev/null -yq -e '.elemental."system-agent".token | tag == "!!str" and length > 0' \ - "${RECOVERY_CONFIG}" >/dev/null -yq -e ' - .elemental.registration.url | - startswith(strenv(EXPECTED_REGISTRATION_PREFIX)) -' "${RECOVERY_CONFIG}" >/dev/null - -W="$(mktemp -d)" -trap 'rm -rf "${W}"' EXIT - -yq -r '.elemental."system-agent"."ca-cert"' \ - "${RECOVERY_CONFIG}" > "${W}/returned-apiserver-ca.crt" -yq -r '.elemental.registration."ca-cert"' \ - "${RECOVERY_CONFIG}" > "${W}/returned-registration-ca.crt" - -kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" -n cpaas-system \ - get secret baremetal-global-system-agent-token \ - -o jsonpath='{.data.ca\.crt}' | base64 -d > "${W}/expected-apiserver-ca.crt" - -cmp -s "${W}/returned-apiserver-ca.crt" "${W}/expected-apiserver-ca.crt" -openssl verify -CAfile "${W}/returned-registration-ca.crt" \ - "${W}/expected-apiserver-ca.crt" >/dev/null - -EXPECTED_TOKEN_SHA="$({ - kubectl --kubeconfig "${LOCAL_GLOBAL_KUBECONFIG}" -n cpaas-system \ - get secret baremetal-global-system-agent-token \ - -o jsonpath='{.data.token}' | base64 -d | sha256sum | awk '{print $1}' -})" -RETURNED_TOKEN_SHA="$({ - yq -r '.elemental."system-agent".token' "${RECOVERY_CONFIG}" | \ - tr -d '\r\n' | sha256sum | awk '{print $1}' -})" -test "${RETURNED_TOKEN_SHA}" = "${EXPECTED_TOKEN_SHA}" - -HTTP_CODE="$({ - curl --cacert "${W}/returned-apiserver-ca.crt" \ - -sS -o /dev/null -w '%{http_code}' \ - "${EXPECTED_SYSTEM_AGENT_URL}/version" -})" -case "${HTTP_CODE}" in - 200|401|403) ;; - *) echo "direct kube-apiserver TLS probe failed" >&2; exit 1 ;; -esac -``` - -This gate proves that the returned endpoint, plan Secret, namespace, token, and direct-API CA belong to the affected local Global cluster, and that the legacy registration CA bundle also trusts the kube-apiserver CA. If any check fails, delete the temporary recovery capability and stop. The `--install --no-toolkit` invocation in the recovery procedure is mandatory; removing `--no-toolkit` changes the operation from connection repair to an installation path. +For this fresh DR installation, each final Global cluster must report `data.ready: "true"` in `ConfigMap/cpaas-system/baremetal-system-agent-handoff`. This is the bootstrap cleanup signal, not the complete acceptance result: also verify the runtime endpoint, token, CA, plan feedback, and RBAC boundaries below. Fresh installation does not use `RoleBinding/cpaas-system/baremetal-global-system-agent-handoff-bridge`; it must be absent. Do not patch the completion signal manually. @@ -2444,11 +2208,7 @@ verify_baremetal_handoff() { kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ get configmap baremetal-system-agent-handoff -o json | \ - jq -e ' - .data.ready == "true" and - .data.profile == "split-local-direct-v1" and - (.data.config | test("^[0-9a-f]{64}$")) - ' >/dev/null + jq -e '.data.ready == "true"' >/dev/null if kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ get rolebinding baremetal-global-system-agent-handoff-bridge \ @@ -2559,17 +2319,17 @@ verify_baremetal_handoff \ "${STANDBY_GLOBAL_HOSTS}" ``` -Also verify the permission split. An empty shared `Role/baremetal-system-agent` is valid immediately after Global installation when no non-`global` workload plan exists. In that branch, keep the four shared objects present and synchronized, require all broad or cross-scope requests to be denied, and repeat the positive shared-plan checks after the first workload plan is created. +Also verify the Global-local permission boundary on both sides. Before the first non-`global` workload plan exists, the shared `baremetal-system-agent` ServiceAccount, token Secret, Role, and RoleBinding may be absent on both clusters. Do not create or copy that bundle by hand merely to satisfy the Global installation gate; the active operator creates it when the first shared-scope registration is reconciled, and `etcd-sync` then copies it to standby. ```shell -verify_system_agent_rbac() { +verify_global_local_system_agent_rbac() { kubeconfig=$1 kubectl --kubeconfig "${kubeconfig}" -n cpaas-system get \ - serviceaccount/baremetal-system-agent \ - secret/baremetal-system-agent-token \ - role/baremetal-system-agent \ - rolebinding/baremetal-system-agent >/dev/null + serviceaccount/baremetal-global-system-agent \ + secret/baremetal-global-system-agent-token \ + role/baremetal-global-system-agent \ + rolebinding/baremetal-global-system-agent >/dev/null local_plan_names="$({ kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ @@ -2581,100 +2341,55 @@ verify_system_agent_rbac() { | .resourceNames[]? ] | unique | sort' })" - shared_plan_names="$({ - kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ - get role baremetal-system-agent -o json | \ - jq -c '[ - .rules[]? - | select(any(.apiGroups[]?; . == "")) - | select(any(.resources[]?; . == "secrets")) - | .resourceNames[]? - ] | unique | sort' - })" - - local_plan="$(printf '%s\n' "${local_plan_names}" | jq -er '.[0]')" + printf '%s\n' "${local_plan_names}" | jq -e 'length > 0' >/dev/null - test "$({ - kubectl --kubeconfig "${kubeconfig}" \ - --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ - -n cpaas-system auth can-i get "secret/${local_plan}" - })" = yes - test "$({ - kubectl --kubeconfig "${kubeconfig}" \ - --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ - -n cpaas-system auth can-i patch "secret/${local_plan}" - })" = yes + while IFS= read -r plan_secret; do + test "$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret "${plan_secret}" -o jsonpath='{.type}' + })" = "elemental.cattle.io/plan" - for service_account in baremetal-global-system-agent baremetal-system-agent; do test "$({ kubectl --kubeconfig "${kubeconfig}" \ - --as="system:serviceaccount:cpaas-system:${service_account}" \ - -n cpaas-system auth can-i list secrets - })" = no + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i get "secret/${plan_secret}" + })" = yes test "$({ kubectl --kubeconfig "${kubeconfig}" \ - --as="system:serviceaccount:cpaas-system:${service_account}" \ - -n cpaas-system auth can-i create secrets - })" = no + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i patch "secret/${plan_secret}" + })" = yes test "$({ kubectl --kubeconfig "${kubeconfig}" \ - --as="system:serviceaccount:cpaas-system:${service_account}" \ - -n cpaas-system auth can-i delete secrets + --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ + -n cpaas-system auth can-i get "secret/${plan_secret}" })" = no + done < <(printf '%s\n' "${local_plan_names}" | jq -r '.[]') + + for verb in list create delete; do test "$({ kubectl --kubeconfig "${kubeconfig}" \ - --as="system:serviceaccount:cpaas-system:${service_account}" \ - -n cpaas-system auth can-i get secret/global-registry-auth + --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ + -n cpaas-system auth can-i "${verb}" secrets })" = no done - test "$({ - kubectl --kubeconfig "${kubeconfig}" \ - --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ - -n cpaas-system auth can-i get "secret/${local_plan}" - })" = no - - if printf '%s\n' "${shared_plan_names}" | jq -e 'length == 0' >/dev/null; then - echo "shared workload Role is empty; positive workload checks are deferred" - return 0 - fi - - workload_plan="$(printf '%s\n' "${shared_plan_names}" | jq -er '.[0]')" - test "$({ - kubectl --kubeconfig "${kubeconfig}" \ - --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ - -n cpaas-system auth can-i get "secret/${workload_plan}" - })" = yes - test "$({ - kubectl --kubeconfig "${kubeconfig}" \ - --as=system:serviceaccount:cpaas-system:baremetal-system-agent \ - -n cpaas-system auth can-i patch "secret/${workload_plan}" - })" = yes test "$({ kubectl --kubeconfig "${kubeconfig}" \ --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \ - -n cpaas-system auth can-i get "secret/${workload_plan}" + -n cpaas-system auth can-i get secret/global-registry-auth })" = no } -verify_system_agent_rbac "${PRIMARY_GLOBAL_KUBECONFIG}" -verify_system_agent_rbac "${STANDBY_GLOBAL_KUBECONFIG}" +verify_global_local_system_agent_rbac "${PRIMARY_GLOBAL_KUBECONFIG}" +verify_global_local_system_agent_rbac "${STANDBY_GLOBAL_KUBECONFIG}" ``` -Do not reduce this to a ready-only ConfigMap check. A bare `true`, a legacy profile, an empty or malformed config digest, a remaining migration bridge, a runtime token or CA mismatch, a stale bootstrap endpoint in the persistent OEM file, or an incorrect permission boundary means handoff is incomplete. +Do not reduce acceptance to the `ready` signal alone. A runtime token or CA mismatch, a stale bootstrap endpoint in the persistent OEM file, a failed plan probe, or an incorrect permission boundary means fresh handoff is incomplete even when `data.ready` is `true`. After the standby installation succeeds, switch the platform domain back to the primary entrance before installing `global-etcd-sync`. Primary remains active during initial synchronization, and workload system-agent endpoints use the platform domain. Verify that DNS and `https:///kubernetes/global/version` reach primary before continuing. -Before you create the `global-etcd-sync` ModuleInfo, prepare the runtime exact-key ignore list. Keep the plugin's own `ServiceMonitor/etcd-sync-monitor` local on whichever Global cluster runs the plugin. This entry is required for every provider; without it, the mirror can treat its own target-side monitoring resource as source surplus and repeatedly replace or delete it. - -```shell -export ETCD_SYNC_SELF_MONITOR_IGNORE_KEY="/registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor" -export ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON="$({ - jq -cn --arg key "${ETCD_SYNC_SELF_MONITOR_IGNORE_KEY}" '[$key]' -})" -``` - -For Bare Metal, add both sides' local Global plan Secret names to that baseline list and convert them to exact etcd keys. Run these commands from a host that has `jq` and two kubeconfigs. Each kubeconfig must address its own Global API server; do not rely on the platform domain, because that domain points to only one side at a time. If one kubeconfig contains both clusters, use two distinct contexts in the equivalent `kubectl --context` commands. +Before you create the `global-etcd-sync` ModuleInfo, build `additional_ignore_equal` exclusively from both sides' local Global plan Secret names. MR `!179` already keeps the plugin's own `ServiceMonitor/etcd-sync-monitor` and the fixed Global-local objects local, so do not duplicate those static keys in the dynamic list. Run these commands from a host that has `jq` and two kubeconfigs. Each kubeconfig must address its own Global API server; do not rely on the platform domain, because that domain points to only one side at a time. If one kubeconfig contains both clusters, use two distinct contexts in the equivalent `kubectl --context` commands. ```shell export PRIMARY_GLOBAL_KUBECONFIG="" @@ -2725,11 +2440,9 @@ printf '%s\n' "${STANDBY_GLOBAL_PLAN_SECRET_NAMES}" | \ export ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON="$( jq -cn \ - --argjson baseline "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" \ --argjson primary "${PRIMARY_GLOBAL_PLAN_SECRET_NAMES}" \ --argjson standby "${STANDBY_GLOBAL_PLAN_SECRET_NAMES}" \ - '$baseline + - (($primary + $standby) + '(($primary + $standby) | unique | sort | map("/registry/secrets/cpaas-system/" + .)) @@ -2740,27 +2453,58 @@ export ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON="$( printf '%s\n' "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" | jq . ``` -Do not install `etcd-sync` if either local Global Role has no plan `resourceNames`, if the generated array omits a plan Secret from either side, or if it omits `ETCD_SYNC_SELF_MONITOR_IGNORE_KEY`. The list is merged and de-duplicated because the broad Secret sync prefix would otherwise overwrite a local Global plan Secret whose name happens to exist in the source or target. Repeat the collection and update the ModuleInfo configuration before synchronization continues whenever either local Global Role gains a new plan Secret. +Do not install `etcd-sync` if either local Global Role has no plan `resourceNames` or if the generated array is not exactly the de-duplicated union of both Roles converted to `/registry/secrets/cpaas-system/`. The broad Secret sync prefix would otherwise overwrite a local Global plan Secret whose name happens to exist in the source or target. -After both clusters are installed, get the primary `k8sadmin` token on a primary control plane node. `etcd-sync` is installed only on the standby cluster, and its `active_cluster_*` values point to the primary cluster. Keep this value in its original base64 Secret form for `active_cluster_token`. +Create a standby-local Secret that contains the primary cluster bearer token, then reference only its name from the plugin configuration. Do not put a token value in `active_cluster_token` for a fresh installation. The Secret must exist before the ModuleInfo is created because the chart's pre-install bootstrap Job mounts it. ```shell -export PRIMARY_CLUSTER_TOKEN_B64="$(sudo kubectl get secret -n cpaas-system k8sadmin -o jsonpath='{.data.token}')" -``` +set -euo pipefail +set +x +umask 077 -Get the standby `k8sadmin` token on a standby control plane node. Use this decoded bearer token to call the standby cluster ModuleInfo API. +export ETCD_SYNC_TOKEN_SECRET_NAME="etcd-sync-active-cluster-token" -```shell -export STANDBY_CLUSTER_BEARER_TOKEN="$(sudo kubectl get secret -n cpaas-system k8sadmin -o jsonpath='{.data.token}' | base64 -d)" +PRIMARY_CLUSTER_TOKEN_B64="$({ + kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get secret k8sadmin -o jsonpath='{.data.token}' +})" +test -n "${PRIMARY_CLUSTER_TOKEN_B64}" + +jq -cn \ + --arg name "${ETCD_SYNC_TOKEN_SECRET_NAME}" \ + --arg token "${PRIMARY_CLUSTER_TOKEN_B64}" ' + { + apiVersion: "v1", + kind: "Secret", + metadata: {name: $name, namespace: "cpaas-system"}, + type: "Opaque", + data: {token: $token} + } +' | kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" apply -f - +unset PRIMARY_CLUSTER_TOKEN_B64 + +TOKEN_CHECK_FILE="$(mktemp)" +trap 'rm -f "${TOKEN_CHECK_FILE}"' EXIT +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get secret "${ETCD_SYNC_TOKEN_SECRET_NAME}" \ + -o jsonpath='{.data.token}' | base64 -d > "${TOKEN_CHECK_FILE}" +grep -Eq '^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$' \ + "${TOKEN_CHECK_FILE}" +rm -f "${TOKEN_CHECK_FILE}" +trap - EXIT ``` -If you create the `global-etcd-sync` ModuleInfo payload from a different host, securely transfer the decoded value from the standby control plane node and export it there. +Get the standby `k8sadmin` token on a standby control-plane node. Use this decoded bearer token only to call the standby ModuleInfo API. ```shell -export STANDBY_CLUSTER_BEARER_TOKEN="" +export STANDBY_CLUSTER_BEARER_TOKEN="$({ + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get secret k8sadmin -o jsonpath='{.data.token}' | \ + base64 -d +})" ``` -Create the `global-etcd-sync` ModuleInfo payload for the standby cluster. The `active_cluster_vip` and `active_cluster_token` values must point to the primary cluster. +Create the `global-etcd-sync` ModuleInfo payload for the standby cluster. `active_cluster_vip` points to primary, while `active_cluster_token_secret_ref` names the standby-local Secret prepared above. ```shell umask 077 @@ -2782,7 +2526,7 @@ cat > "${ETCD_SYNC_MODULEINFO}" </dev/null + +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get apprelease etcd-sync -o json | \ +jq -e \ + --arg secret "${ETCD_SYNC_TOKEN_SECRET_NAME}" \ + --argjson expected_ignore "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" ' + .spec.values.etcd_sync.active_cluster_token_secret_ref == $secret and + ((.spec.values.etcd_sync.active_cluster_token // "") == "") and + ((.spec.values.etcd_sync.additional_ignore_equal // [] | sort) == + ($expected_ignore | sort)) +' >/dev/null + +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get deployment etcd-sync -o json | \ +jq -e --arg secret "${ETCD_SYNC_TOKEN_SECRET_NAME}" ' + any(.spec.template.spec.volumes[]?; + .name == "etcd-sync-token" and .secret.secretName == $secret) +' >/dev/null + +if kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get secret etcd-sync-token >/dev/null 2>&1; then + echo "legacy Secret/etcd-sync-token must not be rendered for a fresh install" >&2 + exit 1 +fi +if kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get secret "${ETCD_SYNC_TOKEN_SECRET_NAME}" \ + >/dev/null 2>&1; then + echo "the etcd-sync active-cluster token Secret must remain standby-local" >&2 + exit 1 +fi + IGNORE_EQUAL_TEXT="$({ kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ -n cpaas-system get configmap etcd-sync-ignore-text \ -o jsonpath='{.data.ignore-equal\.txt}' })" +SYNC_EQUAL_TEXT="$({ + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get configmap etcd-sync-ignore-text \ + -o jsonpath='{.data.sync-equal-file\.txt}' +})" printf '%s\n' "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" | jq -r '.[]' | \ while IFS= read -r key; do - if ! printf '%s\n' "${IGNORE_EQUAL_TEXT}" | grep -Fqx -- "${key}"; then - echo "missing etcd-sync exact-ignore key: ${key}" >&2 - exit 1 - fi + printf '%s\n' "${IGNORE_EQUAL_TEXT}" | grep -Fqx -- "${key}" done -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system rollout restart deployment/etcd-sync -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system rollout restart deployment/etcd-sync-monitor -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system rollout status deployment/etcd-sync --timeout=5m -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system rollout status deployment/etcd-sync-monitor --timeout=5m +for key in \ + "/registry/secrets/cpaas-system/${ETCD_SYNC_TOKEN_SECRET_NAME}" \ + /registry/serviceaccounts/cpaas-system/baremetal-global-system-agent \ + /registry/secrets/cpaas-system/baremetal-global-system-agent-token \ + /registry/roles/cpaas-system/baremetal-global-system-agent \ + /registry/rolebindings/cpaas-system/baremetal-global-system-agent \ + /registry/configmaps/cpaas-system/baremetal-system-agent-handoff \ + /registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor; do + printf '%s\n' "${IGNORE_EQUAL_TEXT}" | grep -Fqx -- "${key}" +done + +for key in \ + /registry/serviceaccounts/cpaas-system/baremetal-system-agent \ + /registry/secrets/cpaas-system/baremetal-system-agent-token \ + /registry/roles/cpaas-system/baremetal-system-agent \ + /registry/rolebindings/cpaas-system/baremetal-system-agent; do + printf '%s\n' "${SYNC_EQUAL_TEXT}" | grep -Fqx -- "${key}" +done + +for deployment in etcd-sync etcd-sync-monitor; do + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system get deployment "${deployment}" -o json | \ + jq -e '.spec.template.metadata.annotations["checksum/sync-key-config"] | length > 0' \ + >/dev/null + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ + -n cpaas-system rollout status "deployment/${deployment}" --timeout=5m +done ``` +The first installation renders the complete filter before the Pods start, so no manual rollout restart is required. The checksum annotation added by MR `!179` triggers both Deployment rollouts if the rendered filter changes later. + Trigger an on-demand monitor comparison and require zero missed and zero surplus keys. Also require the filtered local and remote target-key counts to match. The `/check` endpoint returns HTTP 425 when the preceding check ran less than one minute ago; retry without treating that response as success. ```shell @@ -2951,9 +2759,25 @@ rm -f "${PORT_FORWARD_LOG}" "${CHECK_RESPONSE}" trap - EXIT ``` -After the first successful synchronization, verify that the shared workload bundle is semantically identical on both sides. This check is valid even when the shared Role has no rules because no workload plan exists yet. The two Global-local token Secrets must remain different. +After the first successful synchronization, evaluate the shared workload bundle as one atomic set. Before the first shared-scope workload plan exists, all four objects may be absent on both sides. Partial presence, or presence on only one side, is a failure. Once the bundle exists, validate the synchronized ServiceAccount UID and the token Secret's ServiceAccount UID annotation explicitly; a content comparison that discards the ServiceAccount UID is not sufficient to prove that the token authenticates on standby. ```shell +shared_bundle_count() { + kubeconfig=$1 + count=0 + for resource in \ + serviceaccount/baremetal-system-agent \ + secret/baremetal-system-agent-token \ + role/baremetal-system-agent \ + rolebinding/baremetal-system-agent; do + if kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get "${resource}" >/dev/null 2>&1; then + count=$((count + 1)) + fi + done + printf '%s\n' "${count}" +} + shared_bundle_digest() { kubeconfig=$1 resource=$2 @@ -2971,17 +2795,100 @@ shared_bundle_digest() { ' | sha256sum | awk '{print $1}' } -for resource in \ - serviceaccount/baremetal-system-agent \ - secret/baremetal-system-agent-token \ - role/baremetal-system-agent \ - rolebinding/baremetal-system-agent; do - PRIMARY_DIGEST="$(shared_bundle_digest \ - "${PRIMARY_GLOBAL_KUBECONFIG}" "${resource}")" - STANDBY_DIGEST="$(shared_bundle_digest \ - "${STANDBY_GLOBAL_KUBECONFIG}" "${resource}")" - test "${PRIMARY_DIGEST}" = "${STANDBY_DIGEST}" -done +validate_shared_bundle_cluster() { + kubeconfig=$1 + + sa_uid="$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get serviceaccount baremetal-system-agent -o jsonpath='{.metadata.uid}' + })" + test -n "${sa_uid}" + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret baremetal-system-agent-token -o json | \ + jq -e --arg uid "${sa_uid}" ' + .type == "kubernetes.io/service-account-token" and + .metadata.annotations["kubernetes.io/service-account.name"] == + "baremetal-system-agent" and + .metadata.annotations["kubernetes.io/service-account.uid"] == $uid and + (.data.token | type == "string" and length > 0) + ' >/dev/null + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get rolebinding baremetal-system-agent -o json | \ + jq -e ' + .roleRef == { + apiGroup: "rbac.authorization.k8s.io", + kind: "Role", + name: "baremetal-system-agent" + } and + (.subjects | length) == 1 and + .subjects[0] == { + kind: "ServiceAccount", + name: "baremetal-system-agent", + namespace: "cpaas-system" + } + ' >/dev/null + + expected_shared_plans="$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get machineinventories.elemental.cattle.io -o json | \ + jq -c '[ + .items[] + | select( + (.metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] // "") == + "shared" + ) + | .status.plan.secretRef.name? // empty + ] | unique | sort' + })" + actual_shared_plans="$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get role baremetal-system-agent -o json | \ + jq -c '[ + .rules[]? + | select(any(.apiGroups[]?; . == "")) + | select(any(.resources[]?; . == "secrets")) + | .resourceNames[]? + ] | unique | sort' + })" + test "${actual_shared_plans}" = "${expected_shared_plans}" +} + +PRIMARY_SHARED_COUNT="$(shared_bundle_count "${PRIMARY_GLOBAL_KUBECONFIG}")" +STANDBY_SHARED_COUNT="$(shared_bundle_count "${STANDBY_GLOBAL_KUBECONFIG}")" + +if [ "${PRIMARY_SHARED_COUNT}" = 0 ] && [ "${STANDBY_SHARED_COUNT}" = 0 ]; then + echo "shared workload bundle is deferred until the first shared-scope workload plan" +elif [ "${PRIMARY_SHARED_COUNT}" = 4 ] && [ "${STANDBY_SHARED_COUNT}" = 4 ]; then + validate_shared_bundle_cluster "${PRIMARY_GLOBAL_KUBECONFIG}" + validate_shared_bundle_cluster "${STANDBY_GLOBAL_KUBECONFIG}" + + PRIMARY_SHARED_SA_UID="$({ + kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" -n cpaas-system \ + get serviceaccount baremetal-system-agent -o jsonpath='{.metadata.uid}' + })" + STANDBY_SHARED_SA_UID="$({ + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system \ + get serviceaccount baremetal-system-agent -o jsonpath='{.metadata.uid}' + })" + test "${PRIMARY_SHARED_SA_UID}" = "${STANDBY_SHARED_SA_UID}" + + for resource in \ + serviceaccount/baremetal-system-agent \ + secret/baremetal-system-agent-token \ + role/baremetal-system-agent \ + rolebinding/baremetal-system-agent; do + PRIMARY_DIGEST="$(shared_bundle_digest \ + "${PRIMARY_GLOBAL_KUBECONFIG}" "${resource}")" + STANDBY_DIGEST="$(shared_bundle_digest \ + "${STANDBY_GLOBAL_KUBECONFIG}" "${resource}")" + test "${PRIMARY_DIGEST}" = "${STANDBY_DIGEST}" + done +else + echo "shared workload bundle is only partially synchronized" >&2 + exit 1 +fi PRIMARY_LOCAL_TOKEN_SHA="$({ kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" -n cpaas-system \ @@ -2996,43 +2903,7 @@ STANDBY_LOCAL_TOKEN_SHA="$({ test "${PRIMARY_LOCAL_TOKEN_SHA}" != "${STANDBY_LOCAL_TOKEN_SHA}" ``` -Every later configuration update must patch the actual generated ModuleInfo name, then repeat the rendered ConfigMap, rollout, and monitor gates: - -```shell -export ETCD_SYNC_MODULEINFO_NAME="$({ - cat /root/yamls/global-etcd-sync-moduleinfo.name -})" - -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - patch moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" \ - --type=merge \ - -p "$({ - jq -cn --argjson keys "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" \ - '{spec:{config:{additional_ignore_equal:$keys}}}' - })" -``` - -For a planned failover, uninstall the plugin through its owning ModuleInfo only after the consistency gate reports zero missed and surplus keys and the primary has a verified hard fence. Do not use Deployment scale as the lifecycle operation; AppRelease or ModuleInfo reconciliation can restore the mirror. Change `sharedAuthReadOnly` only on the two final Global AppRelease objects during the role switch. Do not patch either retained bootstrap AppRelease in the failover window: `handoffHook.delivery.mode: always` can deliver and re-run handoff state when that AppRelease changes. - -```shell -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - delete moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - wait --for=delete \ - "moduleinfoes.cluster.alauda.io/${ETCD_SYNC_MODULEINFO_NAME}" --timeout=10m -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system wait --for=delete apprelease/etcd-sync --timeout=10m -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system wait --for=delete deployment/etcd-sync --timeout=10m -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n cpaas-system wait --for=delete deployment/etcd-sync-monitor --timeout=10m -kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ - -n kube-public wait --for=delete configmap/etcd-sync --timeout=10m -``` - -Only after all of these owned resources disappear may the platform domain move to the newly active side. Keep both final Global AppRelease objects in `sharedAuthReadOnly: true` through the hard-fence, uninstall, and DNS transition. This deliberate no-writer window avoids split-brain ownership. Because both provider AppRelease objects use `${PLATFORM_HOST}:11443` as their chart repository and image registry, wait until the domain and registry answer from the newly active side before setting that side's final AppRelease to `sharedAuthReadOnly: false`; attempting that chart update while the domain still resolves to a fenced old registry fails with `ChartDownloadFailed`. Then wait for the new active `elemental-operator` rollout, verify that the read-only flag is absent, and verify the shared Role against the workload plan set. Never update either retained bootstrap AppRelease during this transition. - -For failback, synchronization runs in the opposite direction. Install a new ModuleInfo on the original primary, which is now the inactive target, and point `active_cluster_vip` and `active_cluster_token` at the currently active former standby. Before creating that reverse ModuleInfo, re-run the two-kubeconfig Role collection against the current state and build `additional_ignore_equal` from the union of both local `Role/baremetal-global-system-agent` plan names plus `ETCD_SYNC_SELF_MONITOR_IGNORE_KEY`. Do not reuse the old forward-direction array or start the reverse mirror before its rendered ConfigMap passes the same exact-key and zero-difference gates. At failback, set the currently active side to read-only before the final check and hard fence, uninstall the reverse ModuleInfo, switch the platform domain to the target registry, and only then set the target final AppRelease to writable. +If the shared bundle was absent, repeat this entire block after creating the first non-`global` workload plan. Fresh-install acceptance is complete only after the deferred check passes when shared workload provisioning is first exercised. Restart the Pods that must reload DR and endpoint configuration. Run the same commands on a primary control plane node and on a standby control plane node. From cc1f6ad882b7f09642b57e089f2a8510b974e65b Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 11:06:10 +0800 Subject: [PATCH 05/14] docs: align etcd-sync provenance with MR head --- docs/en/global/install.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index 3d06cb0f..3dd7ea06 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -1858,12 +1858,12 @@ export ENCRYPTION_PROVIDER_SECRET_B64="" export PRIMARY_CLUSTER_VIP="" export STANDBY_CLUSTER_VIP="" export BAREMETAL_ENCRYPTION_PROVIDER_SECRET="global-encryption-provider-config" -export ETCD_SYNC_VERSION="" +export ETCD_SYNC_VERSION="" export ETCD_SYNC_MODULEINFO="/root/yamls/global-etcd-sync-moduleinfo.json" export SERVICE_ACCOUNT_ISSUER="https://kubernetes.default.svc.cluster.local" ``` -Use the tested artifact produced from `etcd-sync` MR `!179` head commit `555c699`. That revision contains the Global-local identity, handoff completion, and plugin ServiceMonitor exact-ignore rules, plus the dynamic Global plan-key input. Replace the placeholder with the published artifact version whose provenance proves it contains `555c699` or an explicitly verified equivalent merge commit. Do not infer compatibility from a higher-looking version string alone. +Use the tested artifact produced from `etcd-sync` MR `!179` head commit `214083173102507d6c7971d3d589fd8b7dd00efb` (`2140831`). That revision contains the Global-local identity, handoff completion, plugin ServiceMonitor exact-ignore rules, dynamic Global plan-key input, and the fresh-install removal of the migration-bridge ignore. Replace the placeholder with the published artifact version whose provenance proves it contains `2140831` or an explicitly verified equivalent merge commit. Do not infer compatibility from a higher-looking version string alone. Create the encryption provider configuration file on both installation environments. @@ -2580,7 +2580,7 @@ rm -f "${ETCD_SYNC_MODULEINFO}" unset STANDBY_CLUSTER_BEARER_TOKEN ``` -Wait for the normalized ModuleInfo and its managed `AppRelease/cpaas-system/etcd-sync`. Do not declare the plugin installed until the AppRelease phase is `Success`, its installed revision matches `ETCD_SYNC_VERSION`, and both Deployments use an image built from `etcd-sync` MR `!179` head commit `555c699` or the explicitly verified equivalent merge commit. +Wait for the normalized ModuleInfo and its managed `AppRelease/cpaas-system/etcd-sync`. Do not declare the plugin installed until the AppRelease phase is `Success`, its installed revision matches `ETCD_SYNC_VERSION`, and both Deployments use an image built from `etcd-sync` MR `!179` head commit `2140831` or the explicitly verified equivalent merge commit. ```shell deadline=$((SECONDS + 600)) @@ -2601,7 +2601,7 @@ until kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ sleep 10 done -export ETCD_SYNC_IMAGE_PROVENANCE_MARKER="555c699" +export ETCD_SYNC_IMAGE_PROVENANCE_MARKER="2140831" for deployment in etcd-sync etcd-sync-monitor; do kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ -n cpaas-system get deployment "${deployment}" -o json | \ @@ -2611,7 +2611,7 @@ for deployment in etcd-sync etcd-sync-monitor; do done ``` -The marker is `555c699` for the reviewed MR head. If the published image uses a merge-commit marker instead, set `ETCD_SYNC_IMAGE_PROVENANCE_MARKER` to that auditable marker only after recording that the merge contains `555c699`; do not disable the provenance check. +The marker is `2140831` for the reviewed MR head. If the published image uses a merge-commit marker instead, set `ETCD_SYNC_IMAGE_PROVENANCE_MARKER` to that auditable marker only after recording that the merge contains `214083173102507d6c7971d3d589fd8b7dd00efb`; do not disable the provenance check. Verify the Secret reference and the complete rendered filter before accepting the first synchronization. The dynamic list must contain only the two local Global Role plan-key union. The token Secret, Global-local bundle, handoff completion ConfigMap, and plugin ServiceMonitor are static exact-ignore entries in MR `!179` and must also be present. The shared workload bundle must remain in the exact-sync list. From 8082d8a7fffd5a5e8cfcbe7037f3405d785bb1d0 Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 11:13:38 +0800 Subject: [PATCH 06/14] docs: align split auth with final etcd-sync head --- docs/en/global/install.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index 3dd7ea06..c34cf92f 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -1858,12 +1858,12 @@ export ENCRYPTION_PROVIDER_SECRET_B64="" export PRIMARY_CLUSTER_VIP="" export STANDBY_CLUSTER_VIP="" export BAREMETAL_ENCRYPTION_PROVIDER_SECRET="global-encryption-provider-config" -export ETCD_SYNC_VERSION="" +export ETCD_SYNC_VERSION="" export ETCD_SYNC_MODULEINFO="/root/yamls/global-etcd-sync-moduleinfo.json" export SERVICE_ACCOUNT_ISSUER="https://kubernetes.default.svc.cluster.local" ``` -Use the tested artifact produced from `etcd-sync` MR `!179` head commit `214083173102507d6c7971d3d589fd8b7dd00efb` (`2140831`). That revision contains the Global-local identity, handoff completion, plugin ServiceMonitor exact-ignore rules, dynamic Global plan-key input, and the fresh-install removal of the migration-bridge ignore. Replace the placeholder with the published artifact version whose provenance proves it contains `2140831` or an explicitly verified equivalent merge commit. Do not infer compatibility from a higher-looking version string alone. +Use the tested artifact produced from `etcd-sync` MR `!179` head commit `4ff3b5ee52ad0d9c7febf7a969bb99b4ca95e8e0` (`4ff3b5e`). That revision contains the Global-local identity, handoff completion, plugin ServiceMonitor exact-ignore rules, and dynamic Global plan-key input. It also removes the newly introduced legacy owner/cluster fallback: fresh-install plan classification accepts only the explicit `baremetal.cluster.io/system-agent-auth-scope` annotation, and an unannotated inventory is not assigned to either synchronization domain. Replace the placeholder with the published artifact version whose provenance proves it contains `4ff3b5e` or an explicitly verified equivalent merge commit. Do not infer compatibility from a higher-looking version string alone. Create the encryption provider configuration file on both installation environments. @@ -2580,7 +2580,7 @@ rm -f "${ETCD_SYNC_MODULEINFO}" unset STANDBY_CLUSTER_BEARER_TOKEN ``` -Wait for the normalized ModuleInfo and its managed `AppRelease/cpaas-system/etcd-sync`. Do not declare the plugin installed until the AppRelease phase is `Success`, its installed revision matches `ETCD_SYNC_VERSION`, and both Deployments use an image built from `etcd-sync` MR `!179` head commit `2140831` or the explicitly verified equivalent merge commit. +Wait for the normalized ModuleInfo and its managed `AppRelease/cpaas-system/etcd-sync`. Do not declare the plugin installed until the AppRelease phase is `Success`, its installed revision matches `ETCD_SYNC_VERSION`, and both Deployments use an image built from `etcd-sync` MR `!179` head commit `4ff3b5e` or the explicitly verified equivalent merge commit. ```shell deadline=$((SECONDS + 600)) @@ -2601,7 +2601,7 @@ until kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ sleep 10 done -export ETCD_SYNC_IMAGE_PROVENANCE_MARKER="2140831" +export ETCD_SYNC_IMAGE_PROVENANCE_MARKER="4ff3b5e" for deployment in etcd-sync etcd-sync-monitor; do kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ -n cpaas-system get deployment "${deployment}" -o json | \ @@ -2611,7 +2611,7 @@ for deployment in etcd-sync etcd-sync-monitor; do done ``` -The marker is `2140831` for the reviewed MR head. If the published image uses a merge-commit marker instead, set `ETCD_SYNC_IMAGE_PROVENANCE_MARKER` to that auditable marker only after recording that the merge contains `214083173102507d6c7971d3d589fd8b7dd00efb`; do not disable the provenance check. +The marker is `4ff3b5e` for the reviewed MR head. If the published image uses a merge-commit marker instead, set `ETCD_SYNC_IMAGE_PROVENANCE_MARKER` to that auditable marker only after recording that the merge contains `4ff3b5ee52ad0d9c7febf7a969bb99b4ca95e8e0`; do not disable the provenance check. Verify the Secret reference and the complete rendered filter before accepting the first synchronization. The dynamic list must contain only the two local Global Role plan-key union. The token Secret, Global-local bundle, handoff completion ConfigMap, and plugin ServiceMonitor are static exact-ignore entries in MR `!179` and must also be present. The shared workload bundle must remain in the exact-sync list. From ba5e464eb464784b3dc3fbcf47c1208fb2630614 Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 11:21:41 +0800 Subject: [PATCH 07/14] docs: require explicit Global auth scope --- docs/en/global/install.mdx | 112 +++++++++++++++++++++++++++++++++++-- 1 file changed, 108 insertions(+), 4 deletions(-) diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index c34cf92f..c72ef56b 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -1863,7 +1863,7 @@ export ETCD_SYNC_MODULEINFO="/root/yamls/global-etcd-sync-moduleinfo.json" export SERVICE_ACCOUNT_ISSUER="https://kubernetes.default.svc.cluster.local" ``` -Use the tested artifact produced from `etcd-sync` MR `!179` head commit `4ff3b5ee52ad0d9c7febf7a969bb99b4ca95e8e0` (`4ff3b5e`). That revision contains the Global-local identity, handoff completion, plugin ServiceMonitor exact-ignore rules, and dynamic Global plan-key input. It also removes the newly introduced legacy owner/cluster fallback: fresh-install plan classification accepts only the explicit `baremetal.cluster.io/system-agent-auth-scope` annotation, and an unannotated inventory is not assigned to either synchronization domain. Replace the placeholder with the published artifact version whose provenance proves it contains `4ff3b5e` or an explicitly verified equivalent merge commit. Do not infer compatibility from a higher-looking version string alone. +Use the tested artifact produced from `etcd-sync` MR `!179` head commit `4ff3b5ee52ad0d9c7febf7a969bb99b4ca95e8e0` (`4ff3b5e`). That revision contains the Global-local identity, handoff completion, plugin ServiceMonitor exact-ignore rules, and dynamic Global plan-key input. It also removes the newly introduced legacy owner/cluster fallback: Global-local classification requires the explicit `baremetal.cluster.io/system-agent-auth-scope: global` annotation. An unannotated Elemental object is not protected as Global-local, remains eligible for the generic synchronization rules, and therefore fails fresh-install Global validation. Replace the placeholder with the published artifact version whose provenance proves it contains `4ff3b5e` or an explicitly verified equivalent merge commit. Do not infer compatibility from a higher-looking version string alone. Create the encryption provider configuration file on both installation environments. @@ -2194,12 +2194,50 @@ export PRIMARY_BOOTSTRAP_KUBECONFIG="" export STANDBY_BOOTSTRAP_KUBECONFIG="" export PRIMARY_GLOBAL_HOSTS=" " export STANDBY_GLOBAL_HOSTS=" " +export PRIMARY_GLOBAL_REGISTRATION_NAME="" +export STANDBY_GLOBAL_REGISTRATION_NAME="" + +verify_global_auth_scope() { + kubeconfig=$1 + registration_name=$2 + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get machineregistration.elemental.cattle.io "${registration_name}" -o json | \ + jq -e ' + .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] == + "global" + ' >/dev/null + + global_inventory_names="$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get baremetalmachines.infrastructure.cluster.x-k8s.io -o json | \ + jq -c '[ + .items[] + | select( + .metadata.labels["cluster.x-k8s.io/cluster-name"] == "global" + ) + | .status.machineInventoryRef.name? // empty + ] | unique | sort' + })" + printf '%s\n' "${global_inventory_names}" | \ + jq -e 'length > 0' >/dev/null + + while IFS= read -r inventory_name; do + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get machineinventory.elemental.cattle.io "${inventory_name}" -o json | \ + jq -e ' + .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] == + "global" + ' >/dev/null + done < <(printf '%s\n' "${global_inventory_names}" | jq -r '.[]') +} verify_baremetal_handoff() { bootstrap_kubeconfig=$1 kubeconfig=$2 control_plane_vip=$3 global_hosts=$4 + registration_name=$5 expected_endpoint="https://${control_plane_vip}:6443" kubectl --kubeconfig "${bootstrap_kubeconfig}" -n cpaas-system \ @@ -2210,6 +2248,8 @@ verify_baremetal_handoff() { get configmap baremetal-system-agent-handoff -o json | \ jq -e '.data.ready == "true"' >/dev/null + verify_global_auth_scope "${kubeconfig}" "${registration_name}" + if kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ get rolebinding baremetal-global-system-agent-handoff-bridge \ >/dev/null 2>&1; then @@ -2312,11 +2352,11 @@ verify_baremetal_handoff() { verify_baremetal_handoff \ "${PRIMARY_BOOTSTRAP_KUBECONFIG}" "${PRIMARY_GLOBAL_KUBECONFIG}" \ "${PRIMARY_CLUSTER_VIP}" \ - "${PRIMARY_GLOBAL_HOSTS}" + "${PRIMARY_GLOBAL_HOSTS}" "${PRIMARY_GLOBAL_REGISTRATION_NAME}" verify_baremetal_handoff \ "${STANDBY_BOOTSTRAP_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}" \ "${STANDBY_CLUSTER_VIP}" \ - "${STANDBY_GLOBAL_HOSTS}" + "${STANDBY_GLOBAL_HOSTS}" "${STANDBY_GLOBAL_REGISTRATION_NAME}" ``` Also verify the Global-local permission boundary on both sides. Before the first non-`global` workload plan exists, the shared `baremetal-system-agent` ServiceAccount, token Secret, Role, and RoleBinding may be absent on both clusters. Do not create or copy that bundle by hand merely to satisfy the Global installation gate; the active operator creates it when the first shared-scope registration is reconciled, and `etcd-sync` then copies it to standby. @@ -2343,6 +2383,20 @@ verify_global_local_system_agent_rbac() { })" printf '%s\n' "${local_plan_names}" | jq -e 'length > 0' >/dev/null + expected_local_plan_names="$({ + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get machineinventories.elemental.cattle.io -o json | \ + jq -c '[ + .items[] + | select( + .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] == + "global" + ) + | .status.plan.secretRef.name? // empty + ] | unique | sort' + })" + test "${local_plan_names}" = "${expected_local_plan_names}" + while IFS= read -r plan_secret; do test "$({ kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ @@ -2392,8 +2446,12 @@ After the standby installation succeeds, switch the platform domain back to the Before you create the `global-etcd-sync` ModuleInfo, build `additional_ignore_equal` exclusively from both sides' local Global plan Secret names. MR `!179` already keeps the plugin's own `ServiceMonitor/etcd-sync-monitor` and the fixed Global-local objects local, so do not duplicate those static keys in the dynamic list. Run these commands from a host that has `jq` and two kubeconfigs. Each kubeconfig must address its own Global API server; do not rely on the platform domain, because that domain points to only one side at a time. If one kubeconfig contains both clusters, use two distinct contexts in the equivalent `kubectl --context` commands. ```shell +set -euo pipefail + export PRIMARY_GLOBAL_KUBECONFIG="" export STANDBY_GLOBAL_KUBECONFIG="" +export PRIMARY_GLOBAL_REGISTRATION_NAME="" +export STANDBY_GLOBAL_REGISTRATION_NAME="" export PRIMARY_GLOBAL_CONTEXT="$(kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" config current-context)" export STANDBY_GLOBAL_CONTEXT="$(kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" config current-context)" @@ -2409,6 +2467,52 @@ if [ "${PRIMARY_GLOBAL_SERVER}" = "${STANDBY_GLOBAL_SERVER}" ]; then exit 1 fi +validate_global_auth_scope() { + kubeconfig=$1 + context=$2 + registration_name=$3 + + kubectl --kubeconfig "${kubeconfig}" --context "${context}" \ + -n cpaas-system get machineregistration.elemental.cattle.io \ + "${registration_name}" -o json | \ + jq -e ' + .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] == + "global" + ' >/dev/null + + global_inventory_names="$({ + kubectl --kubeconfig "${kubeconfig}" --context "${context}" \ + -n cpaas-system \ + get baremetalmachines.infrastructure.cluster.x-k8s.io -o json | \ + jq -c '[ + .items[] + | select( + .metadata.labels["cluster.x-k8s.io/cluster-name"] == "global" + ) + | .status.machineInventoryRef.name? // empty + ] | unique | sort' + })" + printf '%s\n' "${global_inventory_names}" | \ + jq -e 'length > 0' >/dev/null + + while IFS= read -r inventory_name; do + kubectl --kubeconfig "${kubeconfig}" --context "${context}" \ + -n cpaas-system get machineinventory.elemental.cattle.io \ + "${inventory_name}" -o json | \ + jq -e ' + .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] == + "global" + ' >/dev/null + done < <(printf '%s\n' "${global_inventory_names}" | jq -r '.[]') +} + +validate_global_auth_scope \ + "${PRIMARY_GLOBAL_KUBECONFIG}" "${PRIMARY_GLOBAL_CONTEXT}" \ + "${PRIMARY_GLOBAL_REGISTRATION_NAME}" +validate_global_auth_scope \ + "${STANDBY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_CONTEXT}" \ + "${STANDBY_GLOBAL_REGISTRATION_NAME}" + global_plan_secret_names() { kubeconfig=$1 context=$2 @@ -2453,7 +2557,7 @@ export ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON="$( printf '%s\n' "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" | jq . ``` -Do not install `etcd-sync` if either local Global Role has no plan `resourceNames` or if the generated array is not exactly the de-duplicated union of both Roles converted to `/registry/secrets/cpaas-system/`. The broad Secret sync prefix would otherwise overwrite a local Global plan Secret whose name happens to exist in the source or target. +Do not install `etcd-sync` if either Global `MachineRegistration` or any `Cluster/global` `MachineInventory` lacks the explicit `global` scope, if either local Global Role has no plan `resourceNames`, or if the generated array is not exactly the de-duplicated union of both Roles converted to `/registry/secrets/cpaas-system/`. An unannotated Elemental object remains eligible for generic synchronization, and the broad Secret sync prefix could overwrite a local Global plan Secret whose name happens to exist in the source or target. Create a standby-local Secret that contains the primary cluster bearer token, then reference only its name from the plugin configuration. Do not put a token value in `active_cluster_token` for a fresh installation. The Secret must exist before the ModuleInfo is created because the chart's pre-install bootstrap Job mounts it. From fee456baab46323d118f7807e46d20a6887e24a5 Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 13:58:45 +0800 Subject: [PATCH 08/14] docs: run bootstrap setup with bash --- docs/en/global/install.mdx | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index c72ef56b..87ec82ea 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -85,15 +85,24 @@ Use `LOCAL_REGISTRY_ADDRESS` when pushing packages from the bootstrap host. Use ### Step 2 — Create the Bootstrap Cluster -Run the bootstrap script provided by the Core Package. This brings up a temporary KIND-based bootstrap cluster named `minialauda` on the bootstrap host — the temporary Cluster API management cluster used only to provision the `global` cluster. +Run the bootstrap script provided by the Core Package with Bash. This brings up a temporary KIND-based bootstrap cluster named `minialauda` on the bootstrap host — the temporary Cluster API management cluster used only to provision the `global` cluster. After it completes, make the matching `kubectl` client from the bootstrap control-plane container available on the bootstrap host and configure the exported kubeconfig. ```shell mkdir -p /root/cpaas-install tar -xvf -C /root/cpaas-install cd /root/cpaas-install/installer -sh setup.sh -mkdir -p ~/.kube +bash setup.sh +mkdir -p "${HOME}/.local/bin" ~/.kube + +if ! command -v kubectl >/dev/null 2>&1; then + nerdctl cp minialauda-control-plane:/usr/bin/kubectl \ + "${HOME}/.local/bin/kubectl" + chmod 0755 "${HOME}/.local/bin/kubectl" + export PATH="${HOME}/.local/bin:${PATH}" +fi + cp /var/cpaas/data/alauda.kubeconfig ~/.kube/config +kubectl get nodes ``` The bootstrap script provisions an embedded registry, the Cluster API control plane, and the installer components that drive the `global` cluster installation. From 5d84b24ea43ac68a28ac7818ef13b1c65ac4a92a Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 14:42:09 +0800 Subject: [PATCH 09/14] docs: align baremetal manifest with current CRDs --- docs/en/global/install.mdx | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index 87ec82ea..97d6a673 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -1168,7 +1168,7 @@ Use [Creating Clusters on Bare Metal](../create-cluster/bare-metal.mdx), [Managi - Add `Cluster.metadata.labels.cluster-type: ProviderBaremetal`. - Add `Cluster.metadata.annotations["cpaas.io/registry-address"]` with `${NODE_REGISTRY_ADDRESS}`. - Add `Cluster.metadata.annotations["cpaas.io/kube-ovn-join-cidr"]`, `Cluster.metadata.annotations["cpaas.io/sentry-deploy-type"]: Baremetal`, and `Cluster.metadata.annotations["cpaas.io/alb-address-type"]: ClusterAddress`. -- Set `KubeadmControlPlane.spec.kubeadmConfigSpec.format: cloud-init`, or leave it unset because `cloud-init` is the provider path used by Bare Metal. +- Set `KubeadmControlPlane.spec.kubeadmConfigSpec.format: cloud-config`, or leave it unset because `cloud-config` is the provider path used by Bare Metal. - Set `KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge: 0`. Bare-metal pools cannot over-provision physical hosts. - Keep `controlplane.cluster.x-k8s.io/skip-kube-proxy: ""` on the `KubeadmControlPlane` when the release manifest uses kube-ovn. - Put `${CONTROL_PLANE_VIP}` and `${PLATFORM_HOST}` in `KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.serverCertSANs`. @@ -1258,7 +1258,6 @@ spec: spec: machineInventoryPoolRef: name: global-control-plane-pool - allocationPolicy: Ordered --- apiVersion: controlplane.cluster.x-k8s.io/v1beta1 kind: KubeadmControlPlane @@ -1282,7 +1281,7 @@ spec: kind: BaremetalMachineTemplate name: global-control-plane-template kubeadmConfigSpec: - format: cloud-init + format: cloud-config clusterConfiguration: etcd: local: From 8d2a154f336b300dd7dcbeea7ec495d92020d661 Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 15:11:27 +0800 Subject: [PATCH 10/14] docs: gate fresh baremetal DR installation --- docs/en/create-cluster/bare-metal.mdx | 33 +++++++++++++++++++-------- docs/en/global/disaster_recovery.mdx | 4 ++++ docs/en/global/install.mdx | 4 ++++ 3 files changed, 32 insertions(+), 9 deletions(-) diff --git a/docs/en/create-cluster/bare-metal.mdx b/docs/en/create-cluster/bare-metal.mdx index d08cbb58..6a0f8b06 100644 --- a/docs/en/create-cluster/bare-metal.mdx +++ b/docs/en/create-cluster/bare-metal.mdx @@ -196,17 +196,32 @@ Apply the manifest and wait for the SeedImage build to finish: ```bash kubectl apply -f 01-machineregistration-seedimage.yaml -kubectl -n cpaas-system get seedimage -registration-iso -w +kubectl -n cpaas-system wait \ + --for=condition=SeedImageReady=True \ + seedimage/-registration-iso \ + --timeout=30m + +SEEDIMAGE_REASON="$(kubectl -n cpaas-system \ + get seedimage -registration-iso \ + -o jsonpath='{.status.conditions[?(@.type=="SeedImageReady")].reason}')" +SEEDIMAGE_DOWNLOAD_URL="$(kubectl -n cpaas-system \ + get seedimage -registration-iso \ + -o jsonpath='{.status.downloadURL}')" +SEEDIMAGE_CHECKSUM_URL="$(kubectl -n cpaas-system \ + get seedimage -registration-iso \ + -o jsonpath='{.status.checksumURL}')" + +if [ "${SEEDIMAGE_REASON}" != "SeedImageBuildSuccess" ] || \ + [ -z "${SEEDIMAGE_DOWNLOAD_URL}" ] || \ + [ -z "${SEEDIMAGE_CHECKSUM_URL}" ]; then + echo "SeedImage did not produce a downloadable ISO and checksum" >&2 + exit 1 +fi + +printf '%s\n%s\n' "${SEEDIMAGE_DOWNLOAD_URL}" "${SEEDIMAGE_CHECKSUM_URL}" ``` -When `status.state` reaches `Completed`, fetch the download URL and ISO checksum: - -```bash -kubectl -n cpaas-system get seedimage -registration-iso \ - -o jsonpath='{.status.downloadURL}{"\n"}' -kubectl -n cpaas-system get seedimage -registration-iso \ - -o jsonpath='{.status.checksumURL}{"\n"}' -``` +Do not use `status.state` as the success gate; the current controller does not populate it. Continue only when `SeedImageReady=True` has reason `SeedImageBuildSuccess` and both `status.downloadURL` and `status.checksumURL` are non-empty. A SeedImage whose download lifetime has expired can still report `SeedImageReady=True`, but its reason changes and its URLs are cleared. Boot every target host from this ISO. `elemental-register` runs first (creates the `MachineInventory` and uploads `observedNetwork`), then `elemental install` writes the on-disk OS. After install completes, the host stays available for plan execution. If the live ISO environment has no DHCP address, configure NetworkManager manually on the host console as described in [Network Connectivity](#network-connectivity) before waiting for the `MachineInventory`. diff --git a/docs/en/global/disaster_recovery.mdx b/docs/en/global/disaster_recovery.mdx index 96a8b006..b2a326cd 100644 --- a/docs/en/global/disaster_recovery.mdx +++ b/docs/en/global/disaster_recovery.mdx @@ -45,6 +45,10 @@ A disaster recovery switchover and a DR-aware `global` cluster upgrade both unin ## Bare Metal DR Model + + The split-auth model below describes only the target state created by a fresh primary and standby `global` installation. It is not an upgrade or migration procedure for an existing Bare Metal DR pair and does not cover the repair or recovery of a failed environment. Use a separately validated operations runbook for those lifecycle tasks. + + Bare Metal DR adds host-level state that does not exist in VM-based providers. The `global` cluster does not create physical machines; it manages long-lived `MachineInventory` objects and writes plan Secrets that are consumed by `elemental-system-agent` on each host. Global-cluster machines therefore use a cluster-local identity, while non-`global` workload machines use a shared identity that follows the synchronized workload state. A standby `global` cluster must preserve its local Global identity while accepting the shared workload token from primary. ### Endpoint Rules diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index 97d6a673..f1aa5acd 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -1854,6 +1854,10 @@ Issues that are not listed here usually point to environment-specific causes. Ca Use this section when you deploy primary and standby `global` clusters for disaster recovery. Complete these additions before you apply the provider-specific manifest for each `global` cluster. + + The Bare Metal split-auth procedure in this section covers only a fresh installation of the primary and standby `global` clusters. It does not define or validate an in-place upgrade or migration of an existing Bare Metal DR pair, or the repair or recovery of a failed environment. Use a separately validated operations runbook for those lifecycle tasks. + + Primary and standby clusters must use the same encryption provider configuration. For Bare Metal, primary and standby must also use the same Kubernetes ServiceAccount signing key so that the workload-cluster `baremetal-system-agent` token created on the primary cluster is accepted by the standby API server after failover. The machines that form each `global` cluster use a separate cluster-local identity and do not share their token with the peer `global` cluster. For DCS and Bare Metal, the provider-specific cluster resource references a Secret that contains `encryption-provider.conf`; for HCS, normal non-DR deployments do not add `/etc/kubernetes/encryption-provider.conf` to `KubeadmControlPlane.spec.kubeadmConfigSpec.files`. VMware vSphere keeps the release manifest's `/etc/kubernetes/encryption-provider.conf` file entry. ### Prepare Shared DR Variables From 589c3015e0d4bc7f126f12000fe0ce7294bfe2d2 Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 15:58:25 +0800 Subject: [PATCH 11/14] docs: scope platform certificate SANs --- docs/en/global/disaster_recovery.mdx | 6 +++--- docs/en/global/install.mdx | 10 +++++----- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/en/global/disaster_recovery.mdx b/docs/en/global/disaster_recovery.mdx index b2a326cd..dfe586bb 100644 --- a/docs/en/global/disaster_recovery.mdx +++ b/docs/en/global/disaster_recovery.mdx @@ -20,7 +20,7 @@ Use [Optional Disaster Recovery Deployment](./install.mdx#optional-disaster-reco That deployment procedure is the authoritative source for the installation-time DR configuration, including: - Primary and standby clusters use the same Kubernetes API server encryption provider configuration. -- The etcd server certificate SAN list includes both the primary and standby control plane VIPs and the Platform Access Address. +- The `KubeadmControlPlane` etcd server certificate SAN list includes both the primary and standby control plane VIPs and the Platform Access Address. This list is separate from the platform `console.cert` SAN list. - Huawei DCS deployments reference the shared encryption provider Secret from `DCSCluster.spec.encryptionProviderConfigRef`. - VMware vSphere deployments write the same `/etc/kubernetes/encryption-provider.conf` file through `KubeadmControlPlane.spec.kubeadmConfigSpec.files`. - Huawei Cloud Stack deployments write the same `/etc/kubernetes/encryption-provider.conf` file through `KubeadmControlPlane.spec.kubeadmConfigSpec.files`. @@ -185,7 +185,7 @@ After standby is active, validate in increasing risk order: |---------|--------------|-------------| | Handoff target list is empty | `dcs-import-extra-resources` was missing or incomplete before the installer API call | `kubectl -n cpaas-system get cm dcs-import-extra-resources -o yaml`; check `baremetal-system-agent-handoff` Job logs | | Existing hosts get `401` or `Forbidden` after failover | ServiceAccount signing key, issuer, audience, token Secret, or Role `resourceNames` does not match | `sha256sum /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub`; `kubectl auth can-i` with the token | -| Existing hosts fail TLS verification after DNS switch | Platform certificate chain or SAN does not cover the platform domain and both VIPs | Check `cpaas-system/dex.tls`, certificate SANs, and `curl -kI https:///kubernetes/global/version` | +| Existing workload hosts fail TLS verification after DNS switch | The standby serves an untrusted platform certificate, or the certificate does not cover the stable platform domain | Check `cpaas-system/dex.tls`, verify the `${PLATFORM_HOST}` SAN, and run `curl -kI https:///kubernetes/global/version`. Check a VIP SAN only when platform HTTPS or registry access intentionally uses that VIP. | | Standby `global` resources are replaced by primary resources | `etcd-sync` ignore rules did not exclude `global` resources | Inspect the `etcd-sync` ignore ConfigMap and stop sync before restoring standby resources | | New host cannot register with an old ISO | ISO registration URL points to bootstrap or primary-only IP, or `MachineRegistration` was not synchronized | Inspect the `MachineRegistration` status and rebuild the ISO on active standby when needed | @@ -216,7 +216,7 @@ Bare Metal requires all of the following to be aligned: - Same Kubernetes encryption provider Secret content on both sides, referenced by `BaremetalCluster.spec.encryptionProviderConfigRef`. - The encryption provider Secret is imported into each final `global` cluster through `dcs-import-extra-resources`; it must not exist only in the bootstrap KIND cluster. - Same ServiceAccount signing key files and same issuer/audience values on both sides. -- Same trusted platform certificate chain, with SANs that cover the platform domain, primary control-plane VIP, and standby control-plane VIP. +- Same trusted platform certificate on both sides, with the stable platform domain in its SAN list. Add a control-plane VIP SAN only when platform HTTPS or registry access intentionally uses that VIP; do not add internal Service names to `console.cert`. - `dcs-import-extra-resources` created before the installer API call on both sides. - Bare Metal provider AppRelease values set with the local `handoffHook.controlPlaneVIP`, `handoffHook.directAPIServer: true`, `elemental.systemAgent.authMode: shared`, `elemental.systemAgent.splitAuthEnabled: true`, and `elemental.systemAgent.globalServiceAccountName: baremetal-global-system-agent`. The active side sets `elemental.systemAgent.sharedAuthReadOnly: false`; the inactive side sets it to `true`. - The deployed `elemental-operator` image and every newly produced `baremetal-base-image` and `baremetal-base-image-iso` are built from the same pinned elemental commit. Rebuild the installation media whenever that commit changes. diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index f1aa5acd..0183e799 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -665,7 +665,7 @@ kubectl -n cpaas-system rollout restart deploy/global-alb2 kubectl -n cpaas-system rollout status deploy/global-alb2 ``` -Do not include this bootstrap `dex.tls` in `dcs-import-extra-resources`. It is only for the temporary bootstrap endpoint. The final `global` cluster's `dex.tls` is created or maintained by the installer platform certificate flow. For DR, use the `thirdParty` console certificate guidance in Step 8 so the final platform certificate chain covers the platform domain, the primary VIP, and the standby VIP. +Do not include this bootstrap `dex.tls` in `dcs-import-extra-resources`. It is only for the temporary bootstrap endpoint. The final `global` cluster's `dex.tls` is created or maintained by the installer platform certificate flow. For DR, use the `thirdParty` console certificate guidance in Step 8 so both sides serve a certificate for the stable platform domain. A control-plane VIP needs to be in that platform certificate only when clients intentionally access platform HTTPS or the platform registry directly through that VIP. ```shell mkdir -p /root/yamls @@ -1791,11 +1791,11 @@ Set `console.host` and `cluster.features.ha.vip` to the local Bare Metal `global - The examples use a self-signed console certificate. If the environment requires a third-party certificate, replace `console.cert` with a `thirdParty` block that contains the base64 full certificate chain, private key, and optional PKCS#12 values before you submit the installer request. + The examples use a self-signed console certificate. If the environment requires a third-party certificate, replace `console.cert` with a `thirdParty` block that contains the base64 full certificate chain, private key, and optional PKCS#12 values before you submit the installer request. This certificate is for platform HTTPS and registry ingress; it does not configure the kube-apiserver or etcd certificates generated from `KubeadmControlPlane`. - For a primary/standby Bare Metal `global` DR deployment, do not let each side generate an unrelated self-signed certificate. Use a `thirdParty` certificate chain trusted by both sides. The certificate SAN list must cover `${PLATFORM_HOST}`, the primary control-plane VIP, the standby control-plane VIP, and any required platform ingress service names. Otherwise existing system-agents can fail TLS verification after DNS is switched to the standby cluster. + For a primary/standby Bare Metal `global` DR deployment, do not let each side generate an unrelated self-signed certificate. Configure the same trusted `thirdParty` platform certificate on both sides. Its required SAN is the stable `${PLATFORM_HOST}` domain. Do not add the primary or standby control-plane VIPs or internal Service names by default. Add a VIP SAN only when clients intentionally access platform HTTPS or the platform registry directly through that VIP. Global hosts handed off to `https://:6443` validate the separate kube-apiserver certificate and CA, not `console.cert` or `dex.tls`. ### Step 9 — Monitor the Installation @@ -1952,9 +1952,9 @@ grep -E 'service-account-private-key-file' \ /etc/kubernetes/manifests/kube-controller-manager.yaml ``` -### Add DR Certificate SANs to KubeadmControlPlane +### Add DR etcd Server Certificate SANs to KubeadmControlPlane -In the manifest generated in Step 4, include both the primary and standby control plane VIPs, the platform access address, and `etcd.kube-system` in `KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.serverCertSANs`. Use the same SAN list on both the primary and standby installation environments. +In the manifest generated in Step 4, include both the primary and standby control plane VIPs, the platform access address, and `etcd.kube-system` in `KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.serverCertSANs`. Use the same SAN list on both the primary and standby installation environments. These values configure the etcd server certificate generated by kubeadm; they are independent of the platform `console.cert` and must not be copied into its `thirdParty` SAN list. ```yaml serverCertSANs: From 79cd4145f2f757a6a5c74c312416239036bb8cac Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 16:25:07 +0800 Subject: [PATCH 12/14] docs: separate registry and console certificates --- docs/en/global/disaster_recovery.mdx | 4 ++-- docs/en/global/install.mdx | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/en/global/disaster_recovery.mdx b/docs/en/global/disaster_recovery.mdx index dfe586bb..e28e4343 100644 --- a/docs/en/global/disaster_recovery.mdx +++ b/docs/en/global/disaster_recovery.mdx @@ -185,7 +185,7 @@ After standby is active, validate in increasing risk order: |---------|--------------|-------------| | Handoff target list is empty | `dcs-import-extra-resources` was missing or incomplete before the installer API call | `kubectl -n cpaas-system get cm dcs-import-extra-resources -o yaml`; check `baremetal-system-agent-handoff` Job logs | | Existing hosts get `401` or `Forbidden` after failover | ServiceAccount signing key, issuer, audience, token Secret, or Role `resourceNames` does not match | `sha256sum /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub`; `kubectl auth can-i` with the token | -| Existing workload hosts fail TLS verification after DNS switch | The standby serves an untrusted platform certificate, or the certificate does not cover the stable platform domain | Check `cpaas-system/dex.tls`, verify the `${PLATFORM_HOST}` SAN, and run `curl -kI https:///kubernetes/global/version`. Check a VIP SAN only when platform HTTPS or registry access intentionally uses that VIP. | +| Existing workload hosts fail TLS verification after DNS switch | The standby serves an untrusted platform certificate, or the certificate does not cover the stable platform domain | Check `cpaas-system/dex.tls`, verify the `${PLATFORM_HOST}` SAN, and run `curl -kI https:///kubernetes/global/version`. Check a VIP SAN only when platform HTTPS intentionally uses that VIP. The registry on port `11443` uses its separate `global-registry-server` certificate chain. | | Standby `global` resources are replaced by primary resources | `etcd-sync` ignore rules did not exclude `global` resources | Inspect the `etcd-sync` ignore ConfigMap and stop sync before restoring standby resources | | New host cannot register with an old ISO | ISO registration URL points to bootstrap or primary-only IP, or `MachineRegistration` was not synchronized | Inspect the `MachineRegistration` status and rebuild the ISO on active standby when needed | @@ -216,7 +216,7 @@ Bare Metal requires all of the following to be aligned: - Same Kubernetes encryption provider Secret content on both sides, referenced by `BaremetalCluster.spec.encryptionProviderConfigRef`. - The encryption provider Secret is imported into each final `global` cluster through `dcs-import-extra-resources`; it must not exist only in the bootstrap KIND cluster. - Same ServiceAccount signing key files and same issuer/audience values on both sides. -- Same trusted platform certificate on both sides, with the stable platform domain in its SAN list. Add a control-plane VIP SAN only when platform HTTPS or registry access intentionally uses that VIP; do not add internal Service names to `console.cert`. +- Same trusted platform HTTPS certificate on both sides, with the stable platform domain in its SAN list. Add a control-plane VIP SAN only when platform HTTPS intentionally uses that VIP; do not add internal Service names to `console.cert`. The registry on port `11443` uses an independently managed `global-registry-server` certificate chain. - `dcs-import-extra-resources` created before the installer API call on both sides. - Bare Metal provider AppRelease values set with the local `handoffHook.controlPlaneVIP`, `handoffHook.directAPIServer: true`, `elemental.systemAgent.authMode: shared`, `elemental.systemAgent.splitAuthEnabled: true`, and `elemental.systemAgent.globalServiceAccountName: baremetal-global-system-agent`. The active side sets `elemental.systemAgent.sharedAuthReadOnly: false`; the inactive side sets it to `true`. - The deployed `elemental-operator` image and every newly produced `baremetal-base-image` and `baremetal-base-image-iso` are built from the same pinned elemental commit. Rebuild the installation media whenever that commit changes. diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index 0183e799..87f18bf9 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -665,7 +665,7 @@ kubectl -n cpaas-system rollout restart deploy/global-alb2 kubectl -n cpaas-system rollout status deploy/global-alb2 ``` -Do not include this bootstrap `dex.tls` in `dcs-import-extra-resources`. It is only for the temporary bootstrap endpoint. The final `global` cluster's `dex.tls` is created or maintained by the installer platform certificate flow. For DR, use the `thirdParty` console certificate guidance in Step 8 so both sides serve a certificate for the stable platform domain. A control-plane VIP needs to be in that platform certificate only when clients intentionally access platform HTTPS or the platform registry directly through that VIP. +Do not include this bootstrap `dex.tls` in `dcs-import-extra-resources`. It is only for the temporary bootstrap endpoint. The final `global` cluster's `dex.tls` is created or maintained by the installer platform certificate flow. For DR, use the `thirdParty` console certificate guidance in Step 8 so both sides serve a certificate for the stable platform domain. A control-plane VIP needs to be in that platform certificate only when clients intentionally access platform HTTPS directly through that VIP. The Global registry on port `11443` presents its own `global-registry-server` certificate chain and is not configured by `console.cert` or `dex.tls`. ```shell mkdir -p /root/yamls @@ -1791,11 +1791,11 @@ Set `console.host` and `cluster.features.ha.vip` to the local Bare Metal `global - The examples use a self-signed console certificate. If the environment requires a third-party certificate, replace `console.cert` with a `thirdParty` block that contains the base64 full certificate chain, private key, and optional PKCS#12 values before you submit the installer request. This certificate is for platform HTTPS and registry ingress; it does not configure the kube-apiserver or etcd certificates generated from `KubeadmControlPlane`. + The examples use a self-signed console certificate. If the environment requires a third-party certificate, replace `console.cert` with a `thirdParty` block that contains the base64 full certificate chain, private key, and optional PKCS#12 values before you submit the installer request. This certificate is only for platform HTTPS ingress; it does not configure the kube-apiserver or etcd certificates generated from `KubeadmControlPlane`. The Global registry on port `11443` uses an independently managed `global-registry-server` certificate chain, not `console.cert` or `dex.tls`. - For a primary/standby Bare Metal `global` DR deployment, do not let each side generate an unrelated self-signed certificate. Configure the same trusted `thirdParty` platform certificate on both sides. Its required SAN is the stable `${PLATFORM_HOST}` domain. Do not add the primary or standby control-plane VIPs or internal Service names by default. Add a VIP SAN only when clients intentionally access platform HTTPS or the platform registry directly through that VIP. Global hosts handed off to `https://:6443` validate the separate kube-apiserver certificate and CA, not `console.cert` or `dex.tls`. + For a primary/standby Bare Metal `global` DR deployment, do not let each side generate an unrelated self-signed certificate. Configure the same trusted `thirdParty` platform certificate on both sides. Its required SAN is the stable `${PLATFORM_HOST}` domain. Do not add the primary or standby control-plane VIPs or internal Service names by default. Add a VIP SAN only when clients intentionally access platform HTTPS directly through that VIP. Registry access through port `11443` does not use this certificate; registry certificate management is independent and outside this certificate step. Global hosts handed off to `https://:6443` validate the separate kube-apiserver certificate and CA, not `console.cert` or `dex.tls`. ### Step 9 — Monitor the Installation From 544abd823c0c48b8f039dfd7729774bea6d57404 Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 17:04:09 +0800 Subject: [PATCH 13/14] docs: rehydrate fresh handoff plan secrets --- docs/en/global/disaster_recovery.mdx | 6 ++-- docs/en/global/install.mdx | 48 +++++++++++++++------------- 2 files changed, 29 insertions(+), 25 deletions(-) diff --git a/docs/en/global/disaster_recovery.mdx b/docs/en/global/disaster_recovery.mdx index e28e4343..059e6de2 100644 --- a/docs/en/global/disaster_recovery.mdx +++ b/docs/en/global/disaster_recovery.mdx @@ -26,7 +26,7 @@ That deployment procedure is the authoritative source for the installation-time - Huawei Cloud Stack deployments write the same `/etc/kubernetes/encryption-provider.conf` file through `KubeadmControlPlane.spec.kubeadmConfigSpec.files`. - Bare Metal deployments reference the shared encryption provider Secret from `BaremetalCluster.spec.encryptionProviderConfigRef` and use the same Kubernetes ServiceAccount signing key on both sides. - Huawei DCS, VMware vSphere, and Huawei Cloud Stack create the `dcs-import-extra-resources` ConfigMap before installer import so the installation can import referenced credential Secrets and, where applicable, provider-specific infrastructure resources. The name keeps the `dcs` prefix for historical installer compatibility. -- Bare Metal creates the `dcs-import-extra-resources` ConfigMap before installer import so the installation can preserve bare-metal and elemental resources required by handoff. +- Bare Metal creates the `dcs-import-extra-resources` ConfigMap before installer import so the installation can preserve the durable bare-metal and elemental owner resources required by handoff. It does not import Global plan Secrets or kubeadm bootstrap data Secrets; fresh handoff creates the initial Global plan Secrets after all target Global `MachineInventory` objects exist. - The standby cluster installs `global-etcd-sync` with connection values that point to the active primary after both installations succeed. ## Operational Scope @@ -125,6 +125,8 @@ Synchronize the workload-cluster resources that allow standby to continue reconc - Provider referenced Secrets that must exist on the target side, including the encryption-provider Secret referenced by `BaremetalCluster.spec.encryptionProviderConfigRef` when Bare Metal DR is enabled. - Exact shared auth bundle, once it exists: `ServiceAccount/cpaas-system/baremetal-system-agent`, `Secret/cpaas-system/baremetal-system-agent-token`, `Role/cpaas-system/baremetal-system-agent`, and `RoleBinding/cpaas-system/baremetal-system-agent`. +Do not use the synchronization list in this section to populate Step 7 `dcs-import-extra-resources`; the initial Global handoff excludes Global plan Secrets, `KubeadmConfig` objects, and kubeadm bootstrap data Secrets. + Do not synchronize: - `SeedImage`. @@ -217,7 +219,7 @@ Bare Metal requires all of the following to be aligned: - The encryption provider Secret is imported into each final `global` cluster through `dcs-import-extra-resources`; it must not exist only in the bootstrap KIND cluster. - Same ServiceAccount signing key files and same issuer/audience values on both sides. - Same trusted platform HTTPS certificate on both sides, with the stable platform domain in its SAN list. Add a control-plane VIP SAN only when platform HTTPS intentionally uses that VIP; do not add internal Service names to `console.cert`. The registry on port `11443` uses an independently managed `global-registry-server` certificate chain. -- `dcs-import-extra-resources` created before the installer API call on both sides. +- `dcs-import-extra-resources` created before the installer API call on both sides, containing the durable Global Bare Metal and Elemental owner resources plus the referenced encryption-provider Secret. It must exclude Global plan Secrets, `KubeadmConfig` objects, and kubeadm bootstrap data Secrets; fresh handoff creates the initial Global plan Secrets with the live Global `MachineInventory` UID. - Bare Metal provider AppRelease values set with the local `handoffHook.controlPlaneVIP`, `handoffHook.directAPIServer: true`, `elemental.systemAgent.authMode: shared`, `elemental.systemAgent.splitAuthEnabled: true`, and `elemental.systemAgent.globalServiceAccountName: baremetal-global-system-agent`. The active side sets `elemental.systemAgent.sharedAuthReadOnly: false`; the inactive side sets it to `true`. - The deployed `elemental-operator` image and every newly produced `baremetal-base-image` and `baremetal-base-image-iso` are built from the same pinned elemental commit. Rebuild the installation media whenever that commit changes. diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index 87f18bf9..089834b0 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -1458,24 +1458,24 @@ kubectl apply -f /root/yamls/dcs-import-extra-resources.yaml -Create and apply the Bare Metal import ConfigMap before you trigger the installer. This ConfigMap is required because the installer must import the bare-metal and elemental resources that were created in the bootstrap cluster before the `global` cluster exists. +Create and apply the Bare Metal import ConfigMap before you trigger the installer. This ConfigMap imports the durable bare-metal and elemental owner resources that the fresh-install handoff uses to enumerate the installed Global machines. It must not import Elemental plan Secrets or kubeadm bootstrap data Secrets. - Create `dcs-import-extra-resources` before calling `POST /cpaas-installer/api/config/dcs`. If it is missing, the handoff job can run with an empty target list because the new `global` cluster does not contain the `BaremetalMachine`, `MachineInventory`, `MachineRegistration`, or plan Secret objects that describe the bootstrap `global` machines. + Create `dcs-import-extra-resources` before calling `POST /cpaas-installer/api/config/dcs`. If it is missing, the handoff job can run with an empty target list because the new `global` cluster does not contain the `BaremetalMachine`, `MachineInventory`, or `MachineRegistration` objects that describe the bootstrap `global` machines. Do not compensate by adding their plan Secrets or kubeadm bootstrap data Secrets to the ConfigMap. -Collect the plan Secret names and kubeadm bootstrap data Secret names after the `global` Cluster API resources have reconciled. These names do not exist before CAPI creates the Machines. For DR, also include the Secret referenced by `BaremetalCluster.spec.encryptionProviderConfigRef` so the final `global` cluster contains the same encryption-provider configuration. Do not import arbitrary platform credential Secrets, and do not import the MachineRegistration token Secret. +Collect the exact Global `MachineInventory` names from the reconciled `BaremetalMachine` objects and identify the bootstrap `MachineRegistration` used by those hosts. For DR, also collect the Secret name referenced by `BaremetalCluster.spec.encryptionProviderConfigRef` so the final `global` cluster contains the same encryption-provider configuration. Do not import arbitrary platform credential Secrets or the MachineRegistration token Secret. ```shell -kubectl -n cpaas-system get machineinventory.elemental.cattle.io \ - -o jsonpath='{range .items[*]}{.status.plan.secretRef.name}{"\n"}{end}' kubectl -n cpaas-system get baremetalmachine \ - -o jsonpath='{range .items[*]}{.status.planSecretRef.name}{"\n"}{end}' -kubectl -n cpaas-system get machine -l cluster.x-k8s.io/cluster-name=global \ - -o jsonpath='{range .items[*]}{.spec.bootstrap.dataSecretName}{"\n"}{end}' + -l cluster.x-k8s.io/cluster-name=global \ + -o custom-columns='NAME:.metadata.name,INVENTORY:.status.machineInventoryRef.name' +kubectl -n cpaas-system get machineregistration.elemental.cattle.io +kubectl -n cpaas-system get baremetalcluster global \ + -o jsonpath='{.spec.encryptionProviderConfigRef.name}{"\n"}' ``` -Create the ConfigMap. Replace the placeholder inventory, plan Secret, and bootstrap data Secret names with the values from the commands above. +Create the ConfigMap. Replace the inventory and MachineRegistration placeholders with the values from the commands above. For DR, also replace the encryption-provider Secret placeholder; for a non-DR installation that does not set `BaremetalCluster.spec.encryptionProviderConfigRef`, remove that Secret entry. ```shell mkdir -p /root/yamls @@ -1522,18 +1522,6 @@ data: names: [""] etcdKeyBase: "/registry/elemental.cattle.io/machineregistrations/cpaas-system/" method: etcdctl - - resource: "secrets" - names: - - "" - - "" - - "" - method: kubectl - - resource: "secrets" - names: - - "" - - "" - - "" - method: kubectl # Required for DR when BaremetalCluster.spec.encryptionProviderConfigRef is set. # The name must match the Secret referenced by BaremetalCluster. - resource: "secrets" @@ -1546,7 +1534,21 @@ kubectl apply -f /root/yamls/dcs-import-extra-resources.yaml kubectl -n cpaas-system get cm dcs-import-extra-resources -o yaml ``` -Do not add `SeedImage` objects or MachineRegistration token Secrets to this import list. A bootstrap `SeedImage` produces an ISO that points at the bootstrap environment and is no longer the correct lifecycle object after the `global` cluster has been handed off. The `seedimages.elemental.cattle.io` CRD is imported only so the new `global` cluster understands the API type. + + The installer applies `method: kubectl` extras before its batched etcd import has finished creating their owner objects. An Elemental plan Secret imported at that point still references the bootstrap `MachineInventory` UID; that owner UID does not identify the later imported Global `MachineInventory`, so Kubernetes garbage collection can delete the Secret. Do not add Elemental plan Secrets, `KubeadmConfig` objects, or `Machine.spec.bootstrap.dataSecretName` Secrets to this ConfigMap. + + +The first-version fresh-install handoff handles this ordering explicitly. It verifies that all target Global `MachineInventory` objects exist, reads each source plan Secret from the still-running bootstrap cluster, and creates the Global plan Secret with the live Global `MachineInventory` UID as its controller owner. + +`KubeadmConfig` objects and kubeadm bootstrap data Secrets are not used by the first-version fresh-install handoff and must not be added to this ConfigMap. + +Before continuing to Step 8, check the rendered ConfigMap: + +- It includes the Bare Metal CRDs, `BaremetalCluster/global`, Bare Metal machine resources, the exact Global `MachineInventory` objects, and the Global `MachineRegistration`. +- It does not include `SeedImage` objects, MachineRegistration token Secrets, Elemental plan Secrets, `KubeadmConfig` objects, or kubeadm bootstrap data Secrets. +- For DR, it still includes the encryption-provider Secret whose name exactly matches `BaremetalCluster.spec.encryptionProviderConfigRef.name`. + +A bootstrap `SeedImage` produces an ISO that points at the bootstrap environment and is no longer the correct lifecycle object after the `global` cluster has been handed off. The `seedimages.elemental.cattle.io` CRD is imported only so the new `global` cluster understands the API type. For DR, verify after installation that the final `global` cluster has the imported encryption-provider Secret. @@ -2074,7 +2076,7 @@ Keep the same DR `serverCertSANs` list on both the primary and standby installat Also keep the shared ServiceAccount signing key configuration from [Prepare Shared ServiceAccount Signing Key](#prepare-shared-serviceaccount-signing-key). Without that key, the standby API server cannot validate the `baremetal-system-agent` token that existing hosts received before failover. -Create the Bare Metal `dcs-import-extra-resources` ConfigMap from Step 7 on both installation environments. The ConfigMap must import the bare-metal and elemental resources required by handoff, including the encryption-provider Secret referenced by `BaremetalCluster.spec.encryptionProviderConfigRef`, and must not import `SeedImage`. +Create the Bare Metal `dcs-import-extra-resources` ConfigMap from Step 7 on both installation environments. The ConfigMap must import the durable bare-metal and elemental owner resources required by handoff, including the encryption-provider Secret referenced by `BaremetalCluster.spec.encryptionProviderConfigRef`. It must not import `SeedImage`, Elemental plan Secrets, `KubeadmConfig` objects, or kubeadm bootstrap data Secrets; the fresh-install handoff creates the initial Global plan Secrets only after all target Global `MachineInventory` objects are present. The Bare Metal provider AppRelease on both sides must enable the Global-local identity, shared workload identity, and direct kube-apiserver handoff for Global machines. Apply one of the following patches on each side's bootstrap cluster after creating the AppRelease and before calling the installer API. From d168c6e51100ae41a769aca90046582e09391662 Mon Sep 17 00:00:00 2001 From: Gang Wang Date: Tue, 14 Jul 2026 21:33:54 +0800 Subject: [PATCH 14/14] docs: keep DR registries local to each Global --- docs/en/global/install.mdx | 64 ++++++++++++++++++++++++++++++++------ 1 file changed, 54 insertions(+), 10 deletions(-) diff --git a/docs/en/global/install.mdx b/docs/en/global/install.mdx index 089834b0..4932efe7 100644 --- a/docs/en/global/install.mdx +++ b/docs/en/global/install.mdx @@ -1734,7 +1734,7 @@ curl -k -X POST "http://${INSTALLER_IP}:8080/cpaas-installer/api/config/dcs" \ -The Bare Metal installer request includes the control-plane VIP because the `global` cluster uses the VIP exposed by `alive` after handoff. Set `REGISTRY_DOMAIN` to the platform registry address that the final `global` cluster should use. For DR, use `${PLATFORM_HOST}:11443` on both primary and standby so the registry follows the platform-domain switch. For a non-DR deployment, `${CONTROL_PLANE_VIP}:11443` is also valid. Do not use the bootstrap registry address in this field. +The Bare Metal installer request includes the control-plane VIP because the `global` cluster uses the VIP exposed by `alive` after handoff. Set `REGISTRY_DOMAIN` to `${CONTROL_PLANE_VIP}:11443` for that side. In a DR pair, primary and standby must therefore use different Registry addresses: each Global keeps its own Registry on its own control-plane VIP, and Registry traffic does not follow the platform-domain DNS switch. Keep `registry.externalAddress` unset so the installer deploys and populates the local platform Registry. Do not use the bootstrap Registry address or `${PLATFORM_HOST}:11443` in this field. ```shell mkdir -p /root/yamls @@ -1872,12 +1872,12 @@ export ENCRYPTION_PROVIDER_SECRET_B64="" export PRIMARY_CLUSTER_VIP="" export STANDBY_CLUSTER_VIP="" export BAREMETAL_ENCRYPTION_PROVIDER_SECRET="global-encryption-provider-config" -export ETCD_SYNC_VERSION="" +export ETCD_SYNC_VERSION="" export ETCD_SYNC_MODULEINFO="/root/yamls/global-etcd-sync-moduleinfo.json" export SERVICE_ACCOUNT_ISSUER="https://kubernetes.default.svc.cluster.local" ``` -Use the tested artifact produced from `etcd-sync` MR `!179` head commit `4ff3b5ee52ad0d9c7febf7a969bb99b4ca95e8e0` (`4ff3b5e`). That revision contains the Global-local identity, handoff completion, plugin ServiceMonitor exact-ignore rules, and dynamic Global plan-key input. It also removes the newly introduced legacy owner/cluster fallback: Global-local classification requires the explicit `baremetal.cluster.io/system-agent-auth-scope: global` annotation. An unannotated Elemental object is not protected as Global-local, remains eligible for the generic synchronization rules, and therefore fails fresh-install Global validation. Replace the placeholder with the published artifact version whose provenance proves it contains `4ff3b5e` or an explicitly verified equivalent merge commit. Do not infer compatibility from a higher-looking version string alone. +Use the tested artifact produced from `etcd-sync` MR `!179` head commit `86c2ce500c059ff5226dd2f5ebb5b6e908faf06d` (`86c2ce5`). That revision contains the Global-local identity rules, plugin ServiceMonitor exact-ignore rule, dynamic Global plan-key input, and the local `public-registry-credential` exact-ignore rule required when each Global uses its own VIP Registry. It also removes the newly introduced legacy owner/cluster fallback: Global-local classification requires the explicit `baremetal.cluster.io/system-agent-auth-scope: global` annotation. An unannotated Elemental object is not protected as Global-local, remains eligible for the generic synchronization rules, and therefore fails fresh-install Global validation. Replace the placeholder with the published artifact version whose provenance proves it contains `86c2ce5` or an explicitly verified equivalent merge commit. Do not infer compatibility from a higher-looking version string alone. Create the encryption provider configuration file on both installation environments. @@ -2194,7 +2194,51 @@ Use the provider-specific installer configuration differences for both sides: For the primary cluster, make sure the platform domain resolves to the primary HA VIP. In Step 8, set `hostIP` to the primary bootstrap host IP. For DCS, set `console.host` and `cluster.features.ha.vip` to the primary HA VIP. For VMware vSphere, set the control plane endpoint in the primary manifest to the primary HA VIP. For HCS, keep `console.host: []` because the VIP is owned by the HCS ELB. For Bare Metal, set both the manifest VIP and the installer VIP fields to the primary control-plane VIP. -After the primary cluster installation succeeds, switch the platform domain to the standby HA VIP as required by the DR procedure. Then install the standby cluster. This DNS switch before the standby installation is required because several platform resources are rendered with the platform domain and must resolve to the standby entrance while the standby installer runs. In Step 8 on the standby bootstrap host, set `hostIP` to the standby bootstrap host IP. For DCS, set `console.host` and `cluster.features.ha.vip` to the standby HA VIP. For VMware vSphere, set the control plane endpoint in the standby manifest to the standby HA VIP. For HCS, keep `console.host: []`. For Bare Metal, set both the manifest VIP and the installer VIP fields to the standby control-plane VIP, and keep `REGISTRY_DOMAIN` as `${PLATFORM_HOST}:11443` on both sides. Get `INSTALLER_IP` from the `cpaas-installer` Pod on the standby bootstrap host; do not reuse the primary bootstrap host value. +After the primary cluster installation succeeds, switch the platform domain to the standby HA VIP as required by the DR procedure. Then install the standby cluster. This DNS switch before the standby installation is required because several platform resources are rendered with the platform domain and must resolve to the standby entrance while the standby installer runs. In Step 8 on the standby bootstrap host, set `hostIP` to the standby bootstrap host IP. For DCS, set `console.host` and `cluster.features.ha.vip` to the standby HA VIP. For VMware vSphere, set the control plane endpoint in the standby manifest to the standby HA VIP. For HCS, keep `console.host: []`. For Bare Metal, set both the manifest VIP and the installer VIP fields to the standby control-plane VIP, and set `REGISTRY_DOMAIN` to the standby `${CONTROL_PLANE_VIP}:11443`. The primary installation must likewise use its own control-plane VIP. Do not change either Registry address when the platform domain switches. Get `INSTALLER_IP` from the `cpaas-installer` Pod on the standby bootstrap host; do not reuse the primary bootstrap host value. + +For Bare Metal, verify the permanent Registry propagation on each final Global before installing `etcd-sync`. Keep `ProductBase.spec.registry.preferPlatformURL` at the installer-generated value; the Global-local Registry address is carried by `spec.registry.address` and the two Cluster annotations. + +```shell +verify_baremetal_global_registry() { + kubeconfig=$1 + expected_registry=$2 + + test "$(kubectl --kubeconfig "${kubeconfig}" \ + get productbase.product.alauda.io base \ + -o jsonpath='{.spec.registry.address}')" = "${expected_registry}" + test "$(kubectl --kubeconfig "${kubeconfig}" \ + get cluster.platform.tkestack.io global \ + -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}')" = \ + "${expected_registry}" + test "$(kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get cluster.cluster.x-k8s.io global \ + -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}')" = \ + "${expected_registry}" + test "$(kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret public-registry-credential -o jsonpath='{.data.registry}' | \ + base64 -d)" = "${expected_registry}" + + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret global-registry-auth -o jsonpath='{.data.\.dockerconfigjson}' | \ + base64 -d | jq -e --arg registry "${expected_registry}" \ + '.auths[$registry] != null' >/dev/null + + kubectl --kubeconfig "${kubeconfig}" get apprelease -A -o json | \ + jq -e --arg registry "${expected_registry}" ' + all(.items[]; + .spec.source.repoURL == $registry and + .spec.values.global.registry.address == $registry) + ' >/dev/null +} + +export PRIMARY_GLOBAL_REGISTRY_ADDRESS=":11443" +export STANDBY_GLOBAL_REGISTRY_ADDRESS=":11443" + +verify_baremetal_global_registry \ + "${PRIMARY_GLOBAL_KUBECONFIG}" "${PRIMARY_GLOBAL_REGISTRY_ADDRESS}" +verify_baremetal_global_registry \ + "${STANDBY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_REGISTRY_ADDRESS}" +``` For Bare Metal, run the complete handoff gate on both sides before decommissioning either bootstrap KIND cluster. `GLOBAL_HOSTS` is the space-separated management address list for that side's installed Global machines. Run this from a secured host that can use both Global kubeconfigs and SSH to the machines. Keep shell tracing disabled because the runtime connection file contains a bearer token. @@ -2457,7 +2501,7 @@ Do not reduce acceptance to the `ready` signal alone. A runtime token or CA mism After the standby installation succeeds, switch the platform domain back to the primary entrance before installing `global-etcd-sync`. Primary remains active during initial synchronization, and workload system-agent endpoints use the platform domain. Verify that DNS and `https:///kubernetes/global/version` reach primary before continuing. -Before you create the `global-etcd-sync` ModuleInfo, build `additional_ignore_equal` exclusively from both sides' local Global plan Secret names. MR `!179` already keeps the plugin's own `ServiceMonitor/etcd-sync-monitor` and the fixed Global-local objects local, so do not duplicate those static keys in the dynamic list. Run these commands from a host that has `jq` and two kubeconfigs. Each kubeconfig must address its own Global API server; do not rely on the platform domain, because that domain points to only one side at a time. If one kubeconfig contains both clusters, use two distinct contexts in the equivalent `kubectl --context` commands. +Before you create the `global-etcd-sync` ModuleInfo, build `additional_ignore_equal` exclusively from both sides' local Global plan Secret names. MR `!179` already keeps the plugin's own `ServiceMonitor/etcd-sync-monitor`, the fixed Global-local objects, and `public-registry-credential` local, so do not duplicate those static keys in the dynamic list. Run these commands from a host that has `jq` and two kubeconfigs. Each kubeconfig must address its own Global API server; do not rely on the platform domain, because that domain points to only one side at a time. If one kubeconfig contains both clusters, use two distinct contexts in the equivalent `kubectl --context` commands. ```shell set -euo pipefail @@ -2698,7 +2742,7 @@ rm -f "${ETCD_SYNC_MODULEINFO}" unset STANDBY_CLUSTER_BEARER_TOKEN ``` -Wait for the normalized ModuleInfo and its managed `AppRelease/cpaas-system/etcd-sync`. Do not declare the plugin installed until the AppRelease phase is `Success`, its installed revision matches `ETCD_SYNC_VERSION`, and both Deployments use an image built from `etcd-sync` MR `!179` head commit `4ff3b5e` or the explicitly verified equivalent merge commit. +Wait for the normalized ModuleInfo and its managed `AppRelease/cpaas-system/etcd-sync`. Do not declare the plugin installed until the AppRelease phase is `Success`, its installed revision matches `ETCD_SYNC_VERSION`, and both Deployments use an image built from `etcd-sync` MR `!179` head commit `86c2ce5` or the explicitly verified equivalent merge commit. ```shell deadline=$((SECONDS + 600)) @@ -2719,7 +2763,7 @@ until kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ sleep 10 done -export ETCD_SYNC_IMAGE_PROVENANCE_MARKER="4ff3b5e" +export ETCD_SYNC_IMAGE_PROVENANCE_MARKER="86c2ce5" for deployment in etcd-sync etcd-sync-monitor; do kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ -n cpaas-system get deployment "${deployment}" -o json | \ @@ -2729,9 +2773,9 @@ for deployment in etcd-sync etcd-sync-monitor; do done ``` -The marker is `4ff3b5e` for the reviewed MR head. If the published image uses a merge-commit marker instead, set `ETCD_SYNC_IMAGE_PROVENANCE_MARKER` to that auditable marker only after recording that the merge contains `4ff3b5ee52ad0d9c7febf7a969bb99b4ca95e8e0`; do not disable the provenance check. +The marker is `86c2ce5` for the reviewed MR head. If the published image uses a merge-commit marker instead, set `ETCD_SYNC_IMAGE_PROVENANCE_MARKER` to that auditable marker only after recording that the merge contains `86c2ce500c059ff5226dd2f5ebb5b6e908faf06d`; do not disable the provenance check. -Verify the Secret reference and the complete rendered filter before accepting the first synchronization. The dynamic list must contain only the two local Global Role plan-key union. The token Secret, Global-local bundle, handoff completion ConfigMap, and plugin ServiceMonitor are static exact-ignore entries in MR `!179` and must also be present. The shared workload bundle must remain in the exact-sync list. +Verify the Secret reference and the complete rendered filter before accepting the first synchronization. The dynamic list must contain only the two local Global Role plan-key union. The token Secret, Global-local bundle, local `public-registry-credential`, and plugin ServiceMonitor are static exact-ignore entries in MR `!179` and must also be present. `ConfigMap/baremetal-system-agent-handoff` is already outside the configured synchronization ranges, so it remains local without an exact-ignore entry. The shared workload bundle must remain in the exact-sync list. ```shell kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \ @@ -2797,7 +2841,7 @@ for key in \ /registry/secrets/cpaas-system/baremetal-global-system-agent-token \ /registry/roles/cpaas-system/baremetal-global-system-agent \ /registry/rolebindings/cpaas-system/baremetal-global-system-agent \ - /registry/configmaps/cpaas-system/baremetal-system-agent-handoff \ + /registry/secrets/cpaas-system/public-registry-credential \ /registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor; do printf '%s\n' "${IGNORE_EQUAL_TEXT}" | grep -Fqx -- "${key}" done