diff --git a/CLAUDE.md b/CLAUDE.md index af28aa0..51e9122 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,5 +25,5 @@ CP-INV-013 -- CiliumPending on TalosCluster is not a degraded state. It is the e ### Session protocol additions Step 4a -- Read platform-design.md in this repository. -Step 4b -- Determine which category the target CRD belongs to before implementing any reconciler: CAPI-managed lifecycle (TalosCluster target path, SeamInfrastructureCluster, SeamInfrastructureMachine -- no RunnerConfig); operational runner Job CRDs (TalosBackup, TalosEtcdMaintenance, TalosPKIRotation, TalosRecovery, TalosHardeningApply, TalosNodePatch, TalosCredentialRotation, TalosClusterReset -- verify capability in conductor-schema.md first); tenant coordination (PlatformTenant, QueueProfile -- no runner capability, no CAPI objects). +Step 4b -- Determine which category the target CRD belongs to before implementing any reconciler: CAPI-managed lifecycle (TalosCluster target path, SeamInfrastructureCluster, SeamInfrastructureMachine -- no RunnerConfig); operational runner Job CRDs (TalosBackup, TalosEtcdMaintenance, TalosPKIRotation, TalosRecovery, TalosHardeningApply, TalosNodePatch, TalosCredentialRotation, TalosClusterReset -- verify capability in conductor-schema.md first). PlatformTenant is dropped: tenant coordination is handled by InfrastructureTalosCluster (mode=import or mode=bootstrap) plus the conductor role=tenant Deployment managed by the compiler enable bundle. Step 4c -- For any Seam Infrastructure Provider session: confirm talos goclient access is bounded to SeamInfrastructureClusterReconciler and SeamInfrastructureMachineReconciler only. Any other file importing talos goclient is a CP-INV-001 violation. diff --git a/CODEBASE.md b/CODEBASE.md deleted file mode 100644 index d104125..0000000 --- a/CODEBASE.md +++ /dev/null @@ -1,208 +0,0 @@ -# platform: Codebase Reference - -## 1. Purpose - -Platform is the cluster lifecycle authority for the ONT platform. It owns the complete creation, upgrade, and decommission lifecycle of Talos-based Kubernetes clusters via a custom CAPI infrastructure provider. It is the sole namespace creation authority for `seam-tenant-{clusterName}` (CP-INV-004). It submits Conductor execute-mode Jobs for all day-2 operations. Platform does NOT compile manifests (conductor/compiler), deliver packs (wrapper), or own RBAC governance (guardian). - -Two code paths: `reconcileDirectBootstrap()` for the management cluster (mode=bootstrap, capi.enabled=false), and `reconcileCAPIPath()` for target clusters (mode=bootstrap or mode=import, capi.enabled=true). - -Talos goclient is permitted ONLY in `SeamInfrastructureClusterReconciler` and `SeamInfrastructureMachineReconciler` (CP-INV-001). All other reconcilers are strictly prohibited. - ---- - -## 2. Key Files and Locations - -### API types (`api/v1alpha1/`) - -| File | Type | -|------|------| -| `taloscluster_types.go` | `TalosCluster` -- platform's CR, owned by seam-core schema. `spec.mode` (bootstrap/import), `spec.capi.enabled`, `spec.role` (management/tenant). | -| `etcdmaintenance_types.go` | `EtcdMaintenance` day-2 CR | -| `nodemaintenance_types.go` | `NodeMaintenance` day-2 CR | -| `pkirotation_types.go` | `PKIRotation` day-2 CR | -| `clusterreset_types.go` | `ClusterReset` day-2 CR (requires `ontai.dev/reset-approved=true` annotation, CP-INV-006) | -| `upgradepolicy_types.go` | `UpgradePolicy` -- dual-path: CAPI (modifies TalosControlPlane) or direct Job | -| `nodeoperation_types.go` | `NodeOperation` -- dual-path | -| `clustermaintenance_types.go` | `ClusterMaintenance` day-2 CR | -| `hardeningprofile_types.go` | `HardeningProfile` day-2 CR | -| `maintenancebundle_types.go` | `MaintenanceBundle` day-2 CR | - -### CAPI provider types (`api/infrastructure/v1alpha1/`) - -| File | Type | -|------|------| -| `seaminfrastructurecluster_types.go` | `SeamInfrastructureCluster` -- platform's CAPI InfrastructureCluster implementation | -| `seaminfrastructuremachine_types.go` | `SeamInfrastructureMachine` -- holds `spec.address` (pre-provisioned node IP); `status.ready=true` after machineconfig applied | - -### Controllers (`internal/controller/`) - -#### `taloscluster_controller.go` - -`TalosClusterReconciler` at L46. `Reconcile()` L83 with deferred status patch at L110. - -`machineApplyAttemptsHaltThreshold = 3` (L24) -- number of consecutive port-50000 ApplyConfiguration failures before TalosClusterReconciler raises `ControlPlaneUnreachable` condition. - -`reconcileDirectBootstrap()` L209 -- management cluster path. Submits bootstrap Job for new clusters, handles `mode=import` kubeconfig import, calls `ensureManagementOnboarding()` when complete. - -`reconcileCAPIPath()` L449 -- target cluster path. Creates SeamInfrastructureCluster, CAPI Cluster, TalosControlPlane (CACPPT), TalosConfigTemplate (CABPT), MachineDeployments, SeamInfrastructureMachineTemplates. - -`checkMachineReachability()` L662 -- lists SeamInfrastructureMachine nodes in `seam-tenant-{cluster}`, checks for port-50000 ApplyConfiguration failures. Halts control-plane reconcile after `machineApplyAttemptsHaltThreshold` consecutive failures (L696). - -`ensureConductorReadyAndTransition()` L590 -- waits for RunnerConfig capabilities non-empty + remote conductor bootstrap complete. - -`EnsureRemoteConductorBootstrap()` L732 -- creates conductor Deployment + RBAC on target cluster via direct kubeconfig. Calls `ensureRemoteNamespace()` L831, `ensureRemoteConductorServiceAccount()` L842, `EnsureRemoteConductorRBAC()` L860. - -`EnsureRemoteTalosClusterCopy()` L965 -- copies InfrastructureTalosCluster CR to `ont-system` on target cluster via dynamic client. Non-fatal on NotFound (seam-core enable bundle may not yet be applied). - -#### `taloscluster_helpers.go` - -`handleTalosClusterDeletion()` -- **Decision H deletion cascade (T-24)**: -- Step 0 (`finalizerDecisionHCascade`, role=tenant only): Decision H ordered teardown. Deletes all InfrastructurePackExecutions and InfrastructurePackInstances in `seam-tenant-{cluster}`, deletes `conductor-tenant` RBACProfile in `seam-tenant-{cluster}`, removes cluster from `seam-platform-rbac-policy.spec.allowedClusters`, removes cluster from `spec.targetClusters` on `rbac-wrapper`, `rbac-conductor`, `rbac-platform`, `rbac-seam-core` profiles in `seam-system`. mode=bootstrap: permanent decommission. mode=import: management severance only (cluster continues). Both share this cleanup order. -- Step 1 (`finalizerRunnerConfigCleanup`, annotation-gated): Deletes RunnerConfig in `ont-system` + kubeconfig/talosconfig Secrets in `seam-tenant-{cluster}`. -- Step 2 (`finalizerTenantNamespaceCleanup`, CAPI-only): Deletes tenant namespace `seam-tenant-{cluster}`. -- Step 3 (`finalizerWrapperRunnerCRBCleanup`, role=tenant only): Deletes cluster-scoped wrapper-runner ClusterRoleBinding. - -`finalizerDecisionHCascade = "platform.ontai.dev/decision-h-cascade"` -- added by `ensureDecisionHCascadeFinalizer()` for all role=tenant clusters. Added in Step C0 of Reconcile alongside the other finalizer-ensure calls. T-24. - -`packExecutionTenantGVK`, `packInstanceTenantGVK` -- GVKs for InfrastructurePackExecution/PackInstance under infrastructure.ontai.dev/v1alpha1. Used in Decision H cascade. - -`removeFromUnstructuredStringSlice()` -- mirror of `appendToUnstructuredStringSlice()` that removes a value from a string slice field via MergePatch. Returns nil on NotFound (non-fatal). Used in Decision H cascade for allowedClusters and targetClusters cleanup. T-24. - -`ensureTenantOnboarding()` -- called on new tenant cluster registration: -1. Append cluster to `seam-platform-rbac-policy` spec.allowedClusters via `appendToUnstructuredStringSlice()`. -2. Append cluster to targetClusters for profiles: `rbac-wrapper`, `rbac-conductor`, `rbac-platform`, `rbac-seam-core`. -3. Create LocalQueue `pack-deploy-queue` in tenant namespace for Kueue. -4. Call `ensureExecutorTalosconfig()` -- copies talosconfig Secret to `ont-system` and `seam-tenant-{cluster}`. -5. Call `ensureTenantExecutorResources()` -- creates executor SA/Role/RoleBinding for day-2 Jobs. -6. Call `ensureWrapperRunnerResources()` -- creates wrapper-runner SA/Role/RoleBinding/ClusterRoleBinding for pack-deploy Jobs. - -`ensureManagementOnboarding()` -- called for management cluster: appends "management" to rbac-policy allowedClusters, copies talosconfig, creates executor resources. - -`appendToUnstructuredStringSlice()` -- reads object via GVK/namespace/name, appends value to string slice field at fieldPath via MergePatch. Returns nil on NotFound (non-fatal for test environments). - -`ensureWrapperRunnerResources()` -- creates `wrapper-runner-{cluster}` SA + `wrapper-runner` Role + `wrapper-runner-{cluster}` RoleBinding + `wrapper-runner-{cluster}` ClusterRoleBinding. Cleanup by `finalizerWrapperRunnerCRBCleanup`. - -`ensureTenantExecutorResources()` -- creates `platform-executor` SA/Role/RoleBinding in `seam-tenant-{cluster}`. The `platform-executor` Role grants access to all day-2 operation CRD groups including `platform.ontai.dev` resources: `etcdmaintenances`, `hardeningprofiles`, `nodemaintenances`, `nodeoperations`, `pkirotations`, `upgradepolicies`. The `upgradepolicies` resource is required for the `talos-upgrade` capability to list UpgradePolicy CRs and read the target version. - -#### `driftsignal_reconciler.go` - -`DriftSignalReconciler` -- new reconciler (T-23). Watches `DriftSignal` objects (seam-core typed). Dispatches on `spec.state=pending` by `affectedCRRef.Kind`: - -**`InfrastructureRunnerConfig` case:** -1. Derives cluster name from namespace: `strings.TrimPrefix(req.Namespace, "seam-tenant-")`. -2. Finds TalosCluster by name in `seam-system`. -3. Annotates TalosCluster with `ontai.dev/runnerconfig-drift-requeue={timestamp}` to trigger reconciliation. -4. Advances DriftSignal `spec.state` to `queued` via MergePatch. - -**`InfrastructureTalosCluster` case (T-23 Talos version drift, added session/17):** -1. Parses `observedVersion` from `spec.driftReason` field `observedTalosVersion:{version}`. -2. Patches `TalosCluster.status.observedTalosVersion` to `observedVersion`. -3. Appends a synthetic out-of-band TCOR operation record: capability `talos-version-drift`, status Succeeded. -4. Calls `bumpTCORRevision()` with the observed version (creates a new revision epoch anchored at the observed version). -5. Calls `ensureCorrectiveUpgradePolicy()` -- creates `drift-version-{cluster}` UpgradePolicy in `seam-tenant-{cluster}` with `upgradeType=talos`, `targetTalosVersion=spec.talosVersion` (the declared desired state to restore). -6. Advances DriftSignal `spec.state` to `queued`. - -`ensureCorrectiveUpgradePolicy()` is idempotent: no-op if UpgradePolicy already exists with the same target version. Registered in `cmd/platform/main.go`. - -Other kinds and non-pending states are no-ops. If TalosCluster not found, advances state to queued to avoid retry storms. - -#### `seaminfrastructuremachine_reconciler.go` - -`SeamInfrastructureMachineReconciler` -- the ONLY reconciler permitted talos goclient access outside `SeamInfrastructureClusterReconciler` (CP-INV-001). Delivers machineconfig to Talos node on port 50000 via `ApplyConfiguration`. Sets `status.ready=true` after node exits maintenance mode. - -`port50000RetryBase = 10 * time.Second` (L36). `port50000RetryCap` (L39) -- max retry interval. - -#### `pki_cert_helpers.go` - -Certificate expiry detection and PKI rotation triggering. platform-schema.md §13. - -`ParsePEMCertExpiry(pemData []byte) (*time.Time, error)` -- iterates PEM blocks, returns earliest NotAfter across all CERTIFICATE blocks. Exported for unit tests. - -`ParseKubeconfigCertExpiry(kubeconfigYAML []byte) (*time.Time, error)` -- parses kubeconfig via client-go clientcmd, iterates AuthInfos, returns earliest expiry from ClientCertificateData entries. - -`ParseTalosConfigCertExpiry(talosConfigYAML []byte) (*time.Time, error)` -- parses talosconfig YAML, reads active context crt field (base64-encoded PEM), returns cert expiry. - -`detectClusterPKIExpiry(ctx, c, clusterName)` -- reads both Secrets via readSecretAndParseExpiry, returns earliest expiry across both. Tolerates NotFound gracefully. - -`syncPKIExpiry(ctx, c, tc) (bool, error)` -- calls detectClusterPKIExpiry, writes result to tc.Status.PkiExpiryDate, returns rotationNeeded when expiry is within spec.pkiRotationThresholdDays (default 30 days). - -`ensureAutoRotationPKI(ctx, c, scheme, tc) error` -- creates PKIRotation CR named `{cluster}-pki-auto-{ts}` with label `pki-trigger=auto`. Idempotent: skips if an in-progress PKIRotation already exists for the cluster. - -`ensureAnnotationRotationPKI(ctx, c, scheme, tc) error` -- creates PKIRotation CR named `{cluster}-pki-manual-{ts}` with label `pki-trigger=manual`. Caller removes the annotation. - -#### `operational_job_base.go` - -`operationalJobBackoffLimit = int32(0)` (L45) -- no retries on gate failures (INV-018). Applied at `jobSpec()` L61 via `BackoffLimit: &backoff` L76. - -`jobSpec()` L61 -- builds conductor execute-mode Job manifest for a named capability in a namespace. - -`jobSpecWithExclusions()` L234 -- same as `jobSpec()` but adds node affinity exclusions. - -`getClusterRunnerConfig()` L337 -- reads RunnerConfig for `clusterName` from `ont-system`. - -`hasCapability()` L344 -- checks if RunnerConfig `status.capabilities` contains named capability. - -`readOperationRecord()` L156 -- reads PackOperationResult (TCOR) for Job completion status. - -`ensureTCOR()` L180 -- creates TalosClusterOperationResult for a cluster. - -`bumpTCORRevision()` L209 -- increments TCOR revision on version upgrade. - -`resolveOperatorLeaderNode()` L265 -- identifies the node hosting the current operator leader Pod (for node exclusion in day-2 ops). - -`buildNodeExclusions()` L307 -- builds list of node names to exclude from Job scheduling. - -#### `s3_env_secret.go` - -Cross-namespace S3 credential projection for executor Jobs. Source secret lives in `seam-system`; executor Job runs in `seam-tenant-{cluster}`. Direct `envFrom` across namespaces is not possible in Kubernetes, so this file manages a projected copy. - -`ensureS3EnvSecret(ctx, c, scheme, sourceName, sourceNS string, em) (string, error)` -- reads source secret, normalizes keys via `NormalizeS3SecretData`, creates/updates `{em.Name}-s3-env` Secret in `em.Namespace` with an ownerReference to `em`. Returns the projected secret name. - -`NormalizeS3SecretData(data map[string][]byte) (map[string][]byte, error)` (exported) -- accepts both provider key conventions and outputs canonical AWS SDK env var names. See platform-schema.md §10 for the full key contract. - -`appendS3EnvFrom(job *batchv1.Job, envSecretName string)` -- appends an `envFrom` entry for `envSecretName` to the first container of the Job's pod template. No-op when `envSecretName` is empty (non-backup operations). - -`resolveS3CredentialsForRestore(ctx, c, em) (string, string, bool, error)` -- resolves S3 credentials for restore: first checks `spec.s3SnapshotPath.credentialsSecretRef`, then falls back to `seam-etcd-backup-config` in `seam-system`. - ---- - -## 3. Primary Data Flows - -**Management cluster bootstrap**: `reconcileDirectBootstrap()` L209 reads RunnerConfig, submits bootstrap Conductor Job if capabilities empty, polls Job completion via `readOperationRecord()` L156, calls `ensureConductorReadyAndTransition()` L590 when bootstrap complete. - -**CAPI cluster creation**: `reconcileCAPIPath()` L449 creates SeamInfrastructureCluster + CAPI Cluster + TalosControlPlane + TalosConfigTemplate + MachineDeployments. CABPT renders machineconfigs into bootstrap Secrets per Machine. `SeamInfrastructureMachineReconciler` picks up each Secret, delivers machineconfig to node port 50000, sets `status.ready=true`. - -**Mode=import path**: `reconcileDirectBootstrap()` for import path imports existing kubeconfig (no machineconfig delivery). Cluster is governed but not bootstrapped by platform. - -**Day-2 op path (direct)**: Human creates day-2 CR (e.g., EtcdMaintenance) --> reconciler calls `getClusterRunnerConfig()` + `hasCapability()` --> for backup/restore operations: `resolveEtcdBackupS3Secret()` or `resolveS3CredentialsForRestore()` resolves the source Secret, then `ensureS3EnvSecret()` projects a normalized copy into `em.Namespace` -- if no S3 secret is found, `EtcdBackupDestinationAbsent` condition is set and reconcile stops --> builds Job spec via `jobSpec()` or `jobSpecWithExclusions()` --> `appendS3EnvFrom()` mounts the projected secret via `envFrom` --> submits Conductor executor Job --> polls `readOperationRecord()` --> updates CR status. - -**PKI rotation path**: `Reconcile()` Step F fires only for stable-Ready clusters (clusters that were Ready before this reconcile pass). Two triggers: (1) annotation `platform.ontai.dev/rotate-pki=true` calls `ensureAnnotationRotationPKI()` which creates a manual PKIRotation CR; annotation is cleared via Patch. (2) `syncPKIExpiry()` reads kubeconfig and talosconfig Secrets, parses X.509 cert expiry, writes `tc.Status.PkiExpiryDate`, and when expiry is within threshold calls `ensureAutoRotationPKI()`. Stable-Ready clusters requeue every 24h for daily expiry monitoring. platform-schema.md §13. - ---- - -## 4. Invariants - -| ID | Rule | Location | -|----|------|----------| -| CP-INV-001 | Talos goclient restricted to SeamInfrastructureClusterReconciler and SeamInfrastructureMachineReconciler | `seaminfrastructuremachine_reconciler.go`, `seaminfrastructurecluster_reconciler.go` | -| CP-INV-004 | Platform is sole namespace creation authority for `seam-tenant-{cluster}` | `taloscluster_helpers.go:278` `ensureTenantNamespace()` | -| CP-INV-006 | TalosClusterReset requires `ontai.dev/reset-approved=true` before reconciliation | `clusterreset_reconciler.go` | -| CP-INV-007 | Leader election required; lease: `platform-leader` in `seam-system` | `cmd/main.go` | -| CP-INV-008 | TalosCluster owns all CAPI objects via ownerReference | `taloscluster_controller.go:449` | -| CP-INV-009 | Every TalosConfigTemplate includes `cluster.network.cni.name: none` and BPF kernel params | `taloscluster_helpers.go:420` `ensureTalosConfigTemplate()` | -| CP-INV-010 | Kueue not used in platform; operational runner Jobs submit directly | `operational_job_base.go:61` | -| CP-INV-013 | CiliumPending on TalosCluster is not a degraded state | `isCiliumPackInstanceReady()` L683 | -| INV-006 | No Jobs on delete path -- deletion triggers events only | `handleTalosClusterDeletion()` L1073 | - ---- - -## 6. Test Contract - -| Package | Coverage | -|---------|----------| -| `internal/controller` | `taloscluster_helpers_test.go`: Decision H cascade (T-24) -- `TestHandleTalosClusterDeletion_DecisionHCascade_DeletesPackExecutions`, `TestHandleTalosClusterDeletion_DecisionHCascade_RemovesFromAllowedClusters`, `TestHandleTalosClusterDeletion_DecisionHCascade_NotTenant`. `removeFromUnstructuredStringSlice` round-trip. `driftsignal_reconciler_test.go`: `TestDriftSignalReconciler_RunnerConfigKind_RequeuesTalosCluster`, `TestDriftSignalReconciler_NonPending_NoOp`, `TestDriftSignalReconciler_UnknownKind_NoOp`, `TestDriftSignalReconciler_NotFound_NoOp` (T-23 RunnerConfig case); `TestDriftSignalReconciler_TalosVersionDrift_FullFlow` verifies observedTalosVersion patch + out-of-band TCOR record + TCOR revision bump + UpgradePolicy creation + DriftSignal state=queued; `TestDriftSignalReconciler_TalosVersionDrift_AlreadyQueued` no-op guard (T-23 Talos version drift case). | -| `test/unit/controller` | TalosClusterReconciler (bootstrap, CAPI, import paths), handleTalosClusterDeletion, ensureTenantOnboarding, operational job base (jobSpec, hasCapability) | -| `test/unit/controller` (s3) | `NormalizeS3SecretData`: required-key validation, camelCase input, AWS SDK env var input, mixed keys, optional endpoint omission | -| `test/unit/controller` (pki) | `ParsePEMCertExpiry`: single cert, multiple certs (earliest wins), empty input, non-cert PEM. `ParseKubeconfigCertExpiry`: valid embedded cert data, no cert data. `ParseTalosConfigCertExpiry`: valid crt field, missing crt, no active context. | -| `test/integration/day2` | EtcdMaintenance reconciler (backup with S3, S3-absent condition, etcd defrag, restore path); verifies SSA status patch, Job creation with capability label, S3 projected secret creation via `ensureS3EnvSecret` | -| `test/e2e` | Stub files; all skip when `MGMT_KUBECONFIG` absent; skip reasons reference backlog item IDs. PKI rotation automation stubs (pki_rotation_automation_test.go): annotation-triggered rotation, synthetic expiry injection, idempotency guard -- all unconditionally skip pending DAY2-OPS-TENANT closure. HardeningProfile live specs (hardeningprofile_e2e_test.go, 6 specs): `MGMT-HP-PROFILE`, `MGMT-HP-CLUSTER`, `MGMT-HP-NODE`, `TENANT-HP-PROFILE`, `TENANT-HP-CLUSTER`, `TENANT-HP-NODE`. PKI rotation live specs (pkirotation_e2e_test.go, 2 specs): `TENANT-PKI-ROTATE` (PKIRotation CR reaches Ready=True; kubeconfig Secrets refreshed), `TENANT-PKI-CLUSTER-REACH` (post-rotation ClusterPack probe proves cluster reachable). Both sets require `MGMT_KUBECONFIG`. `TENANT-HP-NODE` also requires `TENANT_WORKER_NODE`. Uses safe idempotent machineconfig patches (net.ipv4 sysctls). | diff --git a/api/v1alpha1/machineconfigbackup_types.go b/api/v1alpha1/machineconfigbackup_types.go new file mode 100644 index 0000000..3d70284 --- /dev/null +++ b/api/v1alpha1/machineconfigbackup_types.go @@ -0,0 +1,113 @@ +package v1alpha1 + +import ( + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + "github.com/ontai-dev/seam-core/pkg/lineage" +) + +// Condition type and reason constants for TalosMachineConfigBackup. +const ( + // ConditionTypeMachineConfigBackupReady indicates the backup completed successfully. + ConditionTypeMachineConfigBackupReady = "Ready" + + // ConditionTypeMachineConfigBackupRunning indicates the Conductor Job is running. + ConditionTypeMachineConfigBackupRunning = "Running" + + // ConditionTypeMachineConfigBackupDegraded indicates the backup failed. + ConditionTypeMachineConfigBackupDegraded = "Degraded" + + // ReasonMachineConfigBackupJobSubmitted is set when the Conductor executor Job is submitted. + ReasonMachineConfigBackupJobSubmitted = "JobSubmitted" + + // ReasonMachineConfigBackupJobComplete is set when the Job completed successfully. + ReasonMachineConfigBackupJobComplete = "JobComplete" + + // ReasonMachineConfigBackupJobFailed is set when the Job failed. INV-018 applies. + ReasonMachineConfigBackupJobFailed = "JobFailed" + + // ReasonMachineConfigBackupS3Absent indicates no S3 backup destination is configured. + ReasonMachineConfigBackupS3Absent = "S3DestinationAbsent" + + // ConditionTypeMachineConfigBackupS3Absent is the condition type for absent S3 config. + ConditionTypeMachineConfigBackupS3Absent = "S3DestinationAbsent" +) + +// TalosMachineConfigBackupSpec defines the desired state of TalosMachineConfigBackup. +type TalosMachineConfigBackupSpec struct { + // ClusterRef references the TalosCluster whose node machine configs are to be backed up. + ClusterRef LocalObjectRef `json:"clusterRef"` + + // S3BackupSecretRef references a Secret containing S3 backup credentials for this + // operation. Takes precedence over the cluster-wide seam-etcd-backup-config Secret + // in seam-system. platform-schema.md §10. + // +optional + S3BackupSecretRef *corev1.SecretReference `json:"s3BackupSecretRef,omitempty"` + + // S3Destination is the S3 location to write node machine configs to. + // The bucket is required. The key prefix is auto-generated as: + // {cluster}/machineconfigs/{TIMESTAMP}/{hostname}.yaml + S3Destination S3Ref `json:"s3Destination"` + + // Lineage is the sealed causal chain record for this root declaration. + // Authored once at object creation time and immutable thereafter. + // seam-core-schema.md §5, CLAUDE.md §14 Decision 1. + // +optional + Lineage *lineage.SealedCausalChain `json:"lineage,omitempty"` +} + +// TalosMachineConfigBackupStatus defines the observed state of TalosMachineConfigBackup. +type TalosMachineConfigBackupStatus struct { + // ObservedGeneration is the generation of the spec last reconciled. + // +optional + ObservedGeneration int64 `json:"observedGeneration,omitempty"` + + // JobName is the name of the most recently submitted Conductor executor Job. + // +optional + JobName string `json:"jobName,omitempty"` + + // OperationResult is the message from the Conductor OperationResult ConfigMap. + // +optional + OperationResult string `json:"operationResult,omitempty"` + + // Conditions is the list of status conditions for this TalosMachineConfigBackup. + // Condition types: Ready, Running, Degraded, S3DestinationAbsent, LineageSynced. + // +optional + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` +} + +// TalosMachineConfigBackup triggers a machine config backup for all nodes of a target +// cluster. The Conductor executor reads each node's running config via GetMachineConfig +// and uploads it to S3 at {cluster}/machineconfigs/{TIMESTAMP}/{hostname}.yaml. +// Named Conductor capability: machineconfig-backup. platform-schema.md §11. +// +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Namespaced,shortName=mcb +// +kubebuilder:printcolumn:name="Cluster",type=string,JSONPath=".spec.clusterRef.name" +// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=".status.conditions[?(@.type==\"Ready\")].status" +// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=".metadata.creationTimestamp" +type TalosMachineConfigBackup struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec TalosMachineConfigBackupSpec `json:"spec,omitempty"` + Status TalosMachineConfigBackupStatus `json:"status,omitempty"` +} + +// TalosMachineConfigBackupList is the list type for TalosMachineConfigBackup. +// +// +kubebuilder:object:root=true +type TalosMachineConfigBackupList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []TalosMachineConfigBackup `json:"items"` +} + +func init() { + SchemeBuilder.Register(&TalosMachineConfigBackup{}, &TalosMachineConfigBackupList{}) +} diff --git a/api/v1alpha1/taloscluster_types.go b/api/v1alpha1/taloscluster_types.go index f8ac47e..e0271b4 100644 --- a/api/v1alpha1/taloscluster_types.go +++ b/api/v1alpha1/taloscluster_types.go @@ -70,6 +70,7 @@ const ( ConditionTypeKubeconfigUnavailable = conditions.ConditionTypeKubeconfigUnavailable ConditionTypeVersionUpgradePending = conditions.ConditionTypeVersionUpgradePending ConditionTypeVersionRegressionBlocked = conditions.ConditionTypeVersionRegressionBlocked + ConditionTypeHardeningApplied = conditions.ConditionTypeHardeningApplied ) // Reason constants for TalosCluster -- re-exported from seam-core/pkg/conditions. @@ -95,4 +96,7 @@ const ( ReasonVersionUpgradeSubmitted = conditions.ReasonVersionUpgradeSubmitted ReasonVersionUpgradeComplete = conditions.ReasonVersionUpgradeComplete ReasonVersionRegressionAttempted = conditions.ReasonVersionRegressionAttempted + ReasonHardeningApplied = conditions.ReasonHardeningApplied + ReasonHardeningPending = conditions.ReasonHardeningPending + ReasonHardeningProfileNotValid = conditions.ReasonHardeningProfileNotValid ) diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index 8be2008..0f2b41c 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -1043,3 +1043,111 @@ func (in *UpgradePolicyStatus) DeepCopy() *UpgradePolicyStatus { in.DeepCopyInto(out) return out } + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *TalosMachineConfigBackup) DeepCopyInto(out *TalosMachineConfigBackup) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new TalosMachineConfigBackup. +func (in *TalosMachineConfigBackup) DeepCopy() *TalosMachineConfigBackup { + if in == nil { + return nil + } + out := new(TalosMachineConfigBackup) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *TalosMachineConfigBackup) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *TalosMachineConfigBackupList) DeepCopyInto(out *TalosMachineConfigBackupList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]TalosMachineConfigBackup, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new TalosMachineConfigBackupList. +func (in *TalosMachineConfigBackupList) DeepCopy() *TalosMachineConfigBackupList { + if in == nil { + return nil + } + out := new(TalosMachineConfigBackupList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *TalosMachineConfigBackupList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *TalosMachineConfigBackupSpec) DeepCopyInto(out *TalosMachineConfigBackupSpec) { + *out = *in + out.ClusterRef = in.ClusterRef + if in.S3BackupSecretRef != nil { + in, out := &in.S3BackupSecretRef, &out.S3BackupSecretRef + *out = new(corev1.SecretReference) + **out = **in + } + out.S3Destination = in.S3Destination + if in.Lineage != nil { + in, out := &in.Lineage, &out.Lineage + *out = new(lineage.SealedCausalChain) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new TalosMachineConfigBackupSpec. +func (in *TalosMachineConfigBackupSpec) DeepCopy() *TalosMachineConfigBackupSpec { + if in == nil { + return nil + } + out := new(TalosMachineConfigBackupSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *TalosMachineConfigBackupStatus) DeepCopyInto(out *TalosMachineConfigBackupStatus) { + *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new TalosMachineConfigBackupStatus. +func (in *TalosMachineConfigBackupStatus) DeepCopy() *TalosMachineConfigBackupStatus { + if in == nil { + return nil + } + out := new(TalosMachineConfigBackupStatus) + in.DeepCopyInto(out) + return out +} diff --git a/cmd/platform/main.go b/cmd/platform/main.go index 1047612..86a4291 100644 --- a/cmd/platform/main.go +++ b/cmd/platform/main.go @@ -210,6 +210,16 @@ func main() { os.Exit(1) } + if err := (&controller.MachineConfigBackupReconciler{ + Client: mgr.GetClient(), + APIReader: mgr.GetAPIReader(), + Scheme: mgr.GetScheme(), + Recorder: mgr.GetEventRecorder("machineconfigbackup-controller"), + }).SetupWithManager(mgr); err != nil { + setupLog.Error(err, "unable to create controller", "controller", "MachineConfigBackup") + os.Exit(1) + } + if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil { setupLog.Error(err, "unable to set up health check") os.Exit(1) diff --git a/internal/controller/machineconfigbackup_reconciler.go b/internal/controller/machineconfigbackup_reconciler.go new file mode 100644 index 0000000..190f8b6 --- /dev/null +++ b/internal/controller/machineconfigbackup_reconciler.go @@ -0,0 +1,260 @@ +package controller + +// MachineConfigBackupReconciler reconciles TalosMachineConfigBackup CRs. +// +// Pattern: read the cluster RunnerConfig from ont-system, gate on capability +// availability, project S3 credentials into the job namespace, then submit a +// single batch/v1 Conductor executor Job. Watches the OperationResult ConfigMap +// for completion. conductor-schema.md §5 §17. +// +// Named Conductor capability: machineconfig-backup. +// platform-schema.md §11 TalosMachineConfigBackup. +// +// CP-INV-003: RunnerConfig is generated at runtime, never hand-coded. +// INV-018: gate failures are permanent -- backoffLimit=0, no retries. + +import ( + "context" + "fmt" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + clientevents "k8s.io/client-go/tools/events" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + "sigs.k8s.io/controller-runtime/pkg/log" + + platformv1alpha1 "github.com/ontai-dev/platform/api/v1alpha1" +) + +// machineconfig Conductor capability name per conductor-schema.md. +const capabilityMachineConfigBackup = "machineconfig-backup" + +// MachineConfigBackupReconciler reconciles TalosMachineConfigBackup objects. +type MachineConfigBackupReconciler struct { + Client client.Client + APIReader client.Reader + Scheme *runtime.Scheme + Recorder clientevents.EventRecorder +} + +// +kubebuilder:rbac:groups=platform.ontai.dev,resources=talosmachineconfigbackups,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups=platform.ontai.dev,resources=talosmachineconfigbackups/status,verbs=get;update;patch +// +kubebuilder:rbac:groups=platform.ontai.dev,resources=talosmachineconfigbackups/finalizers,verbs=update +// +kubebuilder:rbac:groups="",resources=events,verbs=create;patch +// +kubebuilder:rbac:groups=infrastructure.ontai.dev,resources=infrastructurerunnerconfigs,verbs=get;list;watch +// +kubebuilder:rbac:groups=infrastructure.ontai.dev,resources=infrastructuretalosclusteroperationresults,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups=batch,resources=jobs,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups="",resources=configmaps,verbs=get;list;watch +// +kubebuilder:rbac:groups=rbac.authorization.k8s.io,resources=roles;rolebindings,verbs=get;list;watch;create;update;patch +// +kubebuilder:rbac:groups="",resources=serviceaccounts,verbs=get;list;watch;create;update;patch +// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch;create;update;patch + +func (r *MachineConfigBackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + logger := log.FromContext(ctx) + + mcb := &platformv1alpha1.TalosMachineConfigBackup{} + if err := r.Client.Get(ctx, req.NamespacedName, mcb); err != nil { + if apierrors.IsNotFound(err) { + return ctrl.Result{}, nil + } + return ctrl.Result{}, fmt.Errorf("get TalosMachineConfigBackup %s: %w", req.NamespacedName, err) + } + + patchBase := client.MergeFrom(mcb.DeepCopy()) + defer func() { + if err := r.Client.Status().Patch(ctx, mcb, patchBase); err != nil { + if !apierrors.IsNotFound(err) { + logger.Error(err, "failed to patch TalosMachineConfigBackup status", + "name", mcb.Name, "namespace", mcb.Namespace) + } + } + }() + + mcb.Status.ObservedGeneration = mcb.Generation + + // Initialize LineageSynced on first observation -- one-time write. + if platformv1alpha1.FindCondition(mcb.Status.Conditions, platformv1alpha1.ConditionTypeLineageSynced) == nil { + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeLineageSynced, + metav1.ConditionFalse, + platformv1alpha1.ReasonLineageControllerAbsent, + "InfrastructureLineageController is not yet deployed.", + mcb.Generation, + ) + } + + // Already complete -- one-shot CR. + readyCond := platformv1alpha1.FindCondition(mcb.Status.Conditions, platformv1alpha1.ConditionTypeMachineConfigBackupReady) + if readyCond != nil && readyCond.Status == metav1.ConditionTrue { + return ctrl.Result{}, nil + } + + // Gate: S3 bucket must be non-empty. + if mcb.Spec.S3Destination.Bucket == "" { + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeMachineConfigBackupS3Absent, + metav1.ConditionTrue, + platformv1alpha1.ReasonMachineConfigBackupS3Absent, + "spec.s3Destination.bucket is required. platform-schema.md §11.", + mcb.Generation, + ) + return ctrl.Result{}, nil + } + + // Gate: read the cluster RunnerConfig from ont-system and verify capability. + clusterRC, err := getClusterRunnerConfig(ctx, r.Client, mcb.Spec.ClusterRef.Name) + if err != nil { + return ctrl.Result{}, fmt.Errorf("MachineConfigBackupReconciler: get cluster RunnerConfig: %w", err) + } + if clusterRC == nil { + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeCapabilityUnavailable, + metav1.ConditionTrue, + platformv1alpha1.ReasonRunnerConfigNotFound, + "Cluster RunnerConfig not yet present in ont-system. Waiting for Conductor agent.", + mcb.Generation, + ) + return ctrl.Result{RequeueAfter: capabilityUnavailableRetryInterval}, nil + } + if !hasCapability(clusterRC, capabilityMachineConfigBackup) { + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeCapabilityUnavailable, + metav1.ConditionTrue, + platformv1alpha1.ReasonCapabilityNotPublished, + fmt.Sprintf("Capability %q not yet published by Conductor agent.", capabilityMachineConfigBackup), + mcb.Generation, + ) + return ctrl.Result{RequeueAfter: capabilityUnavailableRetryInterval}, nil + } + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeCapabilityUnavailable, + metav1.ConditionFalse, + platformv1alpha1.ReasonCapabilityNotPublished, + "", + mcb.Generation, + ) + + jobName := operationalJobName(mcb.Name, capabilityMachineConfigBackup) + + // Check for an existing Job. + existingJob, err := getOperationalJob(ctx, r.Client, mcb.Namespace, jobName) + if err != nil { + return ctrl.Result{}, fmt.Errorf("MachineConfigBackupReconciler: check job: %w", err) + } + + if existingJob == nil { + // Resolve S3 credentials and project them into the job namespace. + s3Name, s3NS, found, sErr := resolveS3BackupSecretRef(ctx, r.Client, mcb.Spec.S3BackupSecretRef) + if sErr != nil { + return ctrl.Result{}, fmt.Errorf("MachineConfigBackupReconciler: resolve S3 secret: %w", sErr) + } + if !found { + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeMachineConfigBackupS3Absent, + metav1.ConditionTrue, + platformv1alpha1.ReasonMachineConfigBackupS3Absent, + "No S3 credentials configured: spec.s3BackupSecretRef is absent and seam-etcd-backup-config Secret not found in seam-system. platform-schema.md §10.", + mcb.Generation, + ) + r.Recorder.Eventf(mcb, nil, "Warning", "S3CredentialsAbsent", + "TalosMachineConfigBackup %s/%s: no S3 credentials configured", mcb.Namespace, mcb.Name) + return ctrl.Result{}, nil + } + + s3EnvSecretName := mcb.Name + s3EnvSecretSuffix + if err := ensureS3EnvSecretFor(ctx, r.Client, r.Scheme, s3Name, s3NS, mcb, s3EnvSecretName, mcb.Namespace); err != nil { + return ctrl.Result{}, fmt.Errorf("MachineConfigBackupReconciler: project S3 env secret: %w", err) + } + + job := jobSpec(jobName, mcb.Namespace, mcb.Spec.ClusterRef.Name, capabilityMachineConfigBackup, clusterRC.Spec.RunnerImage) + appendS3EnvFrom(job, s3EnvSecretName) + if err := controllerutil.SetControllerReference(mcb, job, r.Scheme); err != nil { + return ctrl.Result{}, fmt.Errorf("MachineConfigBackupReconciler: set owner reference: %w", err) + } + if err := r.Client.Create(ctx, job); err != nil { + return ctrl.Result{}, fmt.Errorf("MachineConfigBackupReconciler: create job: %w", err) + } + mcb.Status.JobName = jobName + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeMachineConfigBackupRunning, + metav1.ConditionTrue, + platformv1alpha1.ReasonMachineConfigBackupJobSubmitted, + fmt.Sprintf("Conductor executor Job %s submitted for %s.", jobName, capabilityMachineConfigBackup), + mcb.Generation, + ) + r.Recorder.Eventf(mcb, nil, "Normal", "JobSubmitted", "JobSubmitted", + "Submitted Conductor executor Job %s for machineconfig-backup", jobName) + logger.Info("submitted Conductor executor Job", + "name", mcb.Name, "jobName", jobName, "capability", capabilityMachineConfigBackup) + return ctrl.Result{RequeueAfter: operationalJobPollInterval}, nil + } + + // Job exists -- check OperationResult ConfigMap. + complete, failed, result := readOperationRecord(ctx, r.Client, mcb.Spec.ClusterRef.Name, jobName) + if failed { + mcb.Status.OperationResult = result + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeMachineConfigBackupDegraded, + metav1.ConditionTrue, + platformv1alpha1.ReasonMachineConfigBackupJobFailed, + fmt.Sprintf("Conductor executor Job %s failed: %s", jobName, result), + mcb.Generation, + ) + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeMachineConfigBackupRunning, + metav1.ConditionFalse, + platformv1alpha1.ReasonMachineConfigBackupJobFailed, + "Job failed.", + mcb.Generation, + ) + r.Recorder.Eventf(mcb, nil, "Warning", "JobFailed", "JobFailed", + "Conductor executor Job %s failed: %s", jobName, result) + return ctrl.Result{}, nil + } + if !complete { + return ctrl.Result{RequeueAfter: operationalJobPollInterval}, nil + } + + // Job complete. + mcb.Status.OperationResult = result + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeMachineConfigBackupRunning, + metav1.ConditionFalse, + platformv1alpha1.ReasonMachineConfigBackupJobComplete, + "Job completed.", + mcb.Generation, + ) + platformv1alpha1.SetCondition( + &mcb.Status.Conditions, + platformv1alpha1.ConditionTypeMachineConfigBackupReady, + metav1.ConditionTrue, + platformv1alpha1.ReasonMachineConfigBackupJobComplete, + fmt.Sprintf("Conductor executor Job %s completed successfully.", jobName), + mcb.Generation, + ) + r.Recorder.Eventf(mcb, nil, "Normal", "JobComplete", "JobComplete", + "Conductor executor Job %s completed successfully", jobName) + logger.Info("TalosMachineConfigBackup complete", + "name", mcb.Name, "capability", capabilityMachineConfigBackup) + return ctrl.Result{}, nil +} + +// SetupWithManager registers MachineConfigBackupReconciler with the manager. +func (r *MachineConfigBackupReconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + For(&platformv1alpha1.TalosMachineConfigBackup{}). + Complete(r) +} diff --git a/internal/controller/s3_env_secret.go b/internal/controller/s3_env_secret.go index f963ad5..33ad6cd 100644 --- a/internal/controller/s3_env_secret.go +++ b/internal/controller/s3_env_secret.go @@ -132,6 +132,67 @@ func appendS3EnvFrom(job *batchv1.Job, envSecretName string) { ) } +// resolveS3BackupSecretRef resolves S3 credentials from an optional per-operation +// SecretReference, falling back to the cluster-wide seam-etcd-backup-config Secret. +// Generic: used by any backup reconciler that follows the §10 S3 resolution hierarchy. +// Returns (secretName, secretNamespace, found, error). +func resolveS3BackupSecretRef(ctx context.Context, c client.Client, secretRef *corev1.SecretReference) (string, string, bool, error) { + if secretRef != nil && secretRef.Name != "" { + ns := secretRef.Namespace + if ns == "" { + ns = "seam-system" + } + secret := &corev1.Secret{} + if err := c.Get(ctx, types.NamespacedName{Name: secretRef.Name, Namespace: ns}, secret); err != nil { + if apierrors.IsNotFound(err) { + return "", "", false, nil + } + return "", "", false, fmt.Errorf("get S3 secret %s/%s: %w", ns, secretRef.Name, err) + } + return secretRef.Name, ns, true, nil + } + const defaultName = "seam-etcd-backup-config" + const defaultNS = "seam-system" + secret := &corev1.Secret{} + if err := c.Get(ctx, types.NamespacedName{Name: defaultName, Namespace: defaultNS}, secret); err != nil { + if apierrors.IsNotFound(err) { + return "", "", false, nil + } + return "", "", false, fmt.Errorf("get default S3 secret %s/%s: %w", defaultNS, defaultName, err) + } + return defaultName, defaultNS, true, nil +} + +// ensureS3EnvSecretFor projects S3 credentials from sourceName/sourceNS into a Secret +// owned by owner in projNS. The projected secret is named projName. +// Generic: works for any CR type as owner. +func ensureS3EnvSecretFor(ctx context.Context, c client.Client, scheme *runtime.Scheme, sourceName, sourceNS string, owner client.Object, projName, projNS string) error { + src := &corev1.Secret{} + if err := c.Get(ctx, types.NamespacedName{Name: sourceName, Namespace: sourceNS}, src); err != nil { + return fmt.Errorf("read S3 source secret %s/%s: %w", sourceNS, sourceName, err) + } + envData, err := NormalizeS3SecretData(src.Data) + if err != nil { + return fmt.Errorf("normalize S3 secret %s/%s: %w", sourceNS, sourceName, err) + } + proj := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: projName, + Namespace: projNS, + }, + } + if err := controllerutil.SetControllerReference(owner, proj, scheme); err != nil { + return fmt.Errorf("set owner reference on S3 env secret: %w", err) + } + if _, err := controllerutil.CreateOrUpdate(ctx, c, proj, func() error { + proj.Data = envData + return nil + }); err != nil { + return fmt.Errorf("upsert S3 env secret %s/%s: %w", projNS, projName, err) + } + return nil +} + // resolveS3CredentialsForRestore resolves S3 credentials for an etcd restore operation. // Resolution order (mirrors backup resolution, platform-schema.md §10): // 1. spec.s3SnapshotPath.credentialsSecretRef — per-operation override. diff --git a/internal/controller/taloscluster_bootstrap_hardening.go b/internal/controller/taloscluster_bootstrap_hardening.go new file mode 100644 index 0000000..e8ff836 --- /dev/null +++ b/internal/controller/taloscluster_bootstrap_hardening.go @@ -0,0 +1,134 @@ +package controller + +import ( + "context" + "fmt" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + platformv1alpha1 "github.com/ontai-dev/platform/api/v1alpha1" +) + +const hardeningBootstrapLabel = "ontai.dev/hardening-trigger" +const hardeningBootstrapLabelValue = "bootstrap" +const hardeningRequeueInterval = 30 * time.Second + +// ensureBootstrapHardening creates a NodeMaintenance with operation=hardening-apply and +// label ontai.dev/hardening-trigger=bootstrap in seam-tenant-{cluster} when +// spec.hardeningProfileRef is set and the cluster is Ready (ONT-native path only). +// Sets HardeningApplied on TalosCluster: +// - False/HardeningProfileNotValid: HardeningProfile Valid=True not yet set +// - False/HardeningPending: NodeMaintenance created, not yet Ready +// - True/HardeningApplied: NodeMaintenance reached Ready=True +// +// Returns a non-zero RequeueAfter when NodeMaintenance is pending. +func (r *TalosClusterReconciler) ensureBootstrapHardening( + ctx context.Context, + tc *platformv1alpha1.TalosCluster, +) (ctrl.Result, error) { + if tc.Spec.HardeningProfileRef == nil { + return ctrl.Result{}, nil + } + + hpNS := tc.Spec.HardeningProfileRef.Namespace + if hpNS == "" { + hpNS = tc.Namespace + } + + // Verify the HardeningProfile exists and has Valid=True before creating NodeMaintenance. + hp := &platformv1alpha1.HardeningProfile{} + if err := r.Client.Get(ctx, types.NamespacedName{ + Name: tc.Spec.HardeningProfileRef.Name, + Namespace: hpNS, + }, hp); err != nil { + return ctrl.Result{}, fmt.Errorf("ensureBootstrapHardening: get HardeningProfile: %w", err) + } + validCond := platformv1alpha1.FindCondition(hp.Status.Conditions, platformv1alpha1.ConditionTypeHardeningProfileValid) + if validCond == nil || validCond.Status != metav1.ConditionTrue { + platformv1alpha1.SetCondition( + &tc.Status.Conditions, + platformv1alpha1.ConditionTypeHardeningApplied, + metav1.ConditionFalse, + platformv1alpha1.ReasonHardeningProfileNotValid, + fmt.Sprintf("HardeningProfile %s/%s does not have Valid=True.", hpNS, hp.Name), + tc.Generation, + ) + return ctrl.Result{RequeueAfter: hardeningRequeueInterval}, nil + } + + tenantNS := "seam-tenant-" + tc.Name + + // Idempotency guard: check for an existing bootstrap NodeMaintenance. + nmList := &platformv1alpha1.NodeMaintenanceList{} + if err := r.Client.List(ctx, nmList, + client.InNamespace(tenantNS), + client.MatchingLabels{hardeningBootstrapLabel: hardeningBootstrapLabelValue}, + ); err != nil { + return ctrl.Result{}, fmt.Errorf("ensureBootstrapHardening: list NodeMaintenance: %w", err) + } + + if len(nmList.Items) > 0 { + nm := nmList.Items[0] + readyCond := platformv1alpha1.FindCondition(nm.Status.Conditions, platformv1alpha1.ConditionTypeNodeMaintenanceReady) + if readyCond != nil && readyCond.Status == metav1.ConditionTrue { + platformv1alpha1.SetCondition( + &tc.Status.Conditions, + platformv1alpha1.ConditionTypeHardeningApplied, + metav1.ConditionTrue, + platformv1alpha1.ReasonHardeningApplied, + fmt.Sprintf("Bootstrap NodeMaintenance %s reached Ready=True.", nm.Name), + tc.Generation, + ) + return ctrl.Result{}, nil + } + platformv1alpha1.SetCondition( + &tc.Status.Conditions, + platformv1alpha1.ConditionTypeHardeningApplied, + metav1.ConditionFalse, + platformv1alpha1.ReasonHardeningPending, + fmt.Sprintf("Bootstrap NodeMaintenance %s pending.", nm.Name), + tc.Generation, + ) + return ctrl.Result{RequeueAfter: hardeningRequeueInterval}, nil + } + + // Create the bootstrap NodeMaintenance. + nm := &platformv1alpha1.NodeMaintenance{ + ObjectMeta: metav1.ObjectMeta{ + GenerateName: tc.Name + "-bootstrap-hardening-", + Namespace: tenantNS, + Labels: map[string]string{ + hardeningBootstrapLabel: hardeningBootstrapLabelValue, + }, + }, + Spec: platformv1alpha1.NodeMaintenanceSpec{ + ClusterRef: platformv1alpha1.LocalObjectRef{ + Name: tc.Name, + Namespace: tc.Namespace, + }, + Operation: platformv1alpha1.NodeMaintenanceOperationHardeningApply, + HardeningProfileRef: &platformv1alpha1.LocalObjectRef{ + Name: tc.Spec.HardeningProfileRef.Name, + Namespace: hpNS, + }, + }, + } + if err := r.Client.Create(ctx, nm); err != nil && !apierrors.IsAlreadyExists(err) { + return ctrl.Result{}, fmt.Errorf("ensureBootstrapHardening: create NodeMaintenance: %w", err) + } + + platformv1alpha1.SetCondition( + &tc.Status.Conditions, + platformv1alpha1.ConditionTypeHardeningApplied, + metav1.ConditionFalse, + platformv1alpha1.ReasonHardeningPending, + "Bootstrap NodeMaintenance created, hardening-apply pending.", + tc.Generation, + ) + return ctrl.Result{RequeueAfter: hardeningRequeueInterval}, nil +} diff --git a/internal/controller/taloscluster_bootstrap_hardening_test.go b/internal/controller/taloscluster_bootstrap_hardening_test.go new file mode 100644 index 0000000..f9fa0b7 --- /dev/null +++ b/internal/controller/taloscluster_bootstrap_hardening_test.go @@ -0,0 +1,254 @@ +package controller + +import ( + "context" + "testing" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + clientgoscheme "k8s.io/client-go/kubernetes/scheme" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + + platformv1alpha1 "github.com/ontai-dev/platform/api/v1alpha1" + seamcorev1alpha1 "github.com/ontai-dev/seam-core/api/v1alpha1" +) + +// buildHardeningTestScheme registers all types needed for ensureBootstrapHardening tests. +func buildHardeningTestScheme(t *testing.T) *runtime.Scheme { + t.Helper() + s := runtime.NewScheme() + if err := clientgoscheme.AddToScheme(s); err != nil { + t.Fatalf("add clientgo: %v", err) + } + if err := seamcorev1alpha1.AddToScheme(s); err != nil { + t.Fatalf("add seamcore: %v", err) + } + if err := platformv1alpha1.AddToScheme(s); err != nil { + t.Fatalf("add platform: %v", err) + } + return s +} + +// makeValidHardeningProfile returns a HardeningProfile with Valid=True. +func makeValidHardeningProfile(name, ns string) *platformv1alpha1.HardeningProfile { + hp := &platformv1alpha1.HardeningProfile{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: ns, + ResourceVersion: "1", + }, + Spec: platformv1alpha1.HardeningProfileSpec{ + SysctlParams: map[string]string{"kernel.dmesg_restrict": "1"}, + }, + Status: platformv1alpha1.HardeningProfileStatus{ + Conditions: []metav1.Condition{ + { + Type: platformv1alpha1.ConditionTypeHardeningProfileValid, + Status: metav1.ConditionTrue, + Reason: "ProfileValid", + LastTransitionTime: metav1.Now(), + }, + }, + }, + } + return hp +} + +// makeTalosClusterWithHardeningRef returns a TalosCluster in seam-system with +// spec.hardeningProfileRef pointing to the given name/namespace. +func makeTalosClusterWithHardeningRef(clusterName, hpName, hpNS string) *platformv1alpha1.TalosCluster { + return &platformv1alpha1.TalosCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: clusterName, + Namespace: "seam-system", + ResourceVersion: "1", + }, + Spec: platformv1alpha1.TalosClusterSpec{ + Mode: platformv1alpha1.TalosClusterModeImport, + HardeningProfileRef: &platformv1alpha1.LocalObjectRef{ + Name: hpName, + Namespace: hpNS, + }, + }, + } +} + +// TestEnsureBootstrapHardening_NilRef returns without action when HardeningProfileRef is nil. +func TestEnsureBootstrapHardening_NilRef(t *testing.T) { + s := buildHardeningTestScheme(t) + tc := &platformv1alpha1.TalosCluster{ + ObjectMeta: metav1.ObjectMeta{Name: "ccs-dev", Namespace: "seam-system", ResourceVersion: "1"}, + Spec: platformv1alpha1.TalosClusterSpec{Mode: platformv1alpha1.TalosClusterModeImport}, + } + c := fake.NewClientBuilder().WithScheme(s).WithObjects(tc).Build() + r := &TalosClusterReconciler{Client: c} + + result, err := r.ensureBootstrapHardening(context.Background(), tc) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if result.RequeueAfter != 0 { + t.Errorf("expected no requeue, got %v", result.RequeueAfter) + } + + // No HardeningApplied condition should be set. + cond := platformv1alpha1.FindCondition(tc.Status.Conditions, platformv1alpha1.ConditionTypeHardeningApplied) + if cond != nil { + t.Errorf("expected no HardeningApplied condition, found: %+v", cond) + } + + // No NodeMaintenance should exist. + nmList := &platformv1alpha1.NodeMaintenanceList{} + if err := c.List(context.Background(), nmList); err != nil { + t.Fatalf("list NodeMaintenance: %v", err) + } + if len(nmList.Items) != 0 { + t.Errorf("expected 0 NodeMaintenance, got %d", len(nmList.Items)) + } +} + +// TestEnsureBootstrapHardening_CreatesNodeMaintenance verifies that a NodeMaintenance is +// created with the correct label and operation when no bootstrap NodeMaintenance exists. +func TestEnsureBootstrapHardening_CreatesNodeMaintenance(t *testing.T) { + s := buildHardeningTestScheme(t) + hp := makeValidHardeningProfile("baseline-hardening", "seam-system") + tc := makeTalosClusterWithHardeningRef("ccs-dev", "baseline-hardening", "seam-system") + + c := fake.NewClientBuilder().WithScheme(s).WithObjects(tc, hp).Build() + r := &TalosClusterReconciler{Client: c} + + result, err := r.ensureBootstrapHardening(context.Background(), tc) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if result.RequeueAfter == 0 { + t.Error("expected non-zero RequeueAfter while NodeMaintenance is pending") + } + + // HardeningApplied=False with reason HardeningPending. + cond := platformv1alpha1.FindCondition(tc.Status.Conditions, platformv1alpha1.ConditionTypeHardeningApplied) + if cond == nil { + t.Fatal("expected HardeningApplied condition") + } + if cond.Status != metav1.ConditionFalse { + t.Errorf("expected HardeningApplied=False, got %s", cond.Status) + } + if cond.Reason != platformv1alpha1.ReasonHardeningPending { + t.Errorf("expected reason %q, got %q", platformv1alpha1.ReasonHardeningPending, cond.Reason) + } + + // One NodeMaintenance with the bootstrap label. + nmList := &platformv1alpha1.NodeMaintenanceList{} + if err := c.List(context.Background(), nmList); err != nil { + t.Fatalf("list NodeMaintenance: %v", err) + } + if len(nmList.Items) != 1 { + t.Fatalf("expected 1 NodeMaintenance, got %d", len(nmList.Items)) + } + nm := nmList.Items[0] + if nm.Labels[hardeningBootstrapLabel] != hardeningBootstrapLabelValue { + t.Errorf("expected label %s=%s", hardeningBootstrapLabel, hardeningBootstrapLabelValue) + } + if nm.Spec.Operation != platformv1alpha1.NodeMaintenanceOperationHardeningApply { + t.Errorf("expected operation hardening-apply, got %q", nm.Spec.Operation) + } + if nm.Spec.HardeningProfileRef == nil || nm.Spec.HardeningProfileRef.Name != "baseline-hardening" { + t.Errorf("unexpected HardeningProfileRef: %v", nm.Spec.HardeningProfileRef) + } +} + +// TestEnsureBootstrapHardening_NoDuplicate verifies that no second NodeMaintenance is +// created when a bootstrap NodeMaintenance already exists (pending). +func TestEnsureBootstrapHardening_NoDuplicate(t *testing.T) { + s := buildHardeningTestScheme(t) + hp := makeValidHardeningProfile("baseline-hardening", "seam-system") + tc := makeTalosClusterWithHardeningRef("ccs-dev", "baseline-hardening", "seam-system") + + // Pre-create a pending bootstrap NodeMaintenance. + existingNM := &platformv1alpha1.NodeMaintenance{ + ObjectMeta: metav1.ObjectMeta{ + Name: "ccs-dev-bootstrap-hardening-abc", + Namespace: "seam-tenant-ccs-dev", + ResourceVersion: "1", + Labels: map[string]string{hardeningBootstrapLabel: hardeningBootstrapLabelValue}, + }, + Spec: platformv1alpha1.NodeMaintenanceSpec{ + ClusterRef: platformv1alpha1.LocalObjectRef{Name: "ccs-dev", Namespace: "seam-system"}, + Operation: platformv1alpha1.NodeMaintenanceOperationHardeningApply, + }, + } + + c := fake.NewClientBuilder().WithScheme(s).WithObjects(tc, hp, existingNM).Build() + r := &TalosClusterReconciler{Client: c} + + result, err := r.ensureBootstrapHardening(context.Background(), tc) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if result.RequeueAfter == 0 { + t.Error("expected non-zero RequeueAfter while NodeMaintenance is pending") + } + + // Still only one NodeMaintenance. + nmList := &platformv1alpha1.NodeMaintenanceList{} + if err := c.List(context.Background(), nmList); err != nil { + t.Fatalf("list NodeMaintenance: %v", err) + } + if len(nmList.Items) != 1 { + t.Errorf("expected 1 NodeMaintenance (no duplicate), got %d", len(nmList.Items)) + } +} + +// TestEnsureBootstrapHardening_SetsAppliedWhenReady verifies that HardeningApplied=True +// is set when the existing bootstrap NodeMaintenance has Ready=True. +func TestEnsureBootstrapHardening_SetsAppliedWhenReady(t *testing.T) { + s := buildHardeningTestScheme(t) + hp := makeValidHardeningProfile("baseline-hardening", "seam-system") + tc := makeTalosClusterWithHardeningRef("ccs-dev", "baseline-hardening", "seam-system") + + // Pre-create a Ready bootstrap NodeMaintenance. + readyNM := &platformv1alpha1.NodeMaintenance{ + ObjectMeta: metav1.ObjectMeta{ + Name: "ccs-dev-bootstrap-hardening-abc", + Namespace: "seam-tenant-ccs-dev", + ResourceVersion: "1", + Labels: map[string]string{hardeningBootstrapLabel: hardeningBootstrapLabelValue}, + }, + Spec: platformv1alpha1.NodeMaintenanceSpec{ + ClusterRef: platformv1alpha1.LocalObjectRef{Name: "ccs-dev", Namespace: "seam-system"}, + Operation: platformv1alpha1.NodeMaintenanceOperationHardeningApply, + }, + Status: platformv1alpha1.NodeMaintenanceStatus{ + Conditions: []metav1.Condition{ + { + Type: platformv1alpha1.ConditionTypeNodeMaintenanceReady, + Status: metav1.ConditionTrue, + Reason: "JobComplete", + LastTransitionTime: metav1.Now(), + }, + }, + }, + } + + c := fake.NewClientBuilder().WithScheme(s).WithStatusSubresource(readyNM).WithObjects(tc, hp, readyNM).Build() + r := &TalosClusterReconciler{Client: c} + + result, err := r.ensureBootstrapHardening(context.Background(), tc) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if result.RequeueAfter != 0 { + t.Errorf("expected no requeue when hardening complete, got %v", result.RequeueAfter) + } + + cond := platformv1alpha1.FindCondition(tc.Status.Conditions, platformv1alpha1.ConditionTypeHardeningApplied) + if cond == nil { + t.Fatal("expected HardeningApplied condition") + } + if cond.Status != metav1.ConditionTrue { + t.Errorf("expected HardeningApplied=True, got %s", cond.Status) + } + if cond.Reason != platformv1alpha1.ReasonHardeningApplied { + t.Errorf("expected reason %q, got %q", platformv1alpha1.ReasonHardeningApplied, cond.Reason) + } +} diff --git a/internal/controller/taloscluster_controller.go b/internal/controller/taloscluster_controller.go index 21d032e..39a8879 100644 --- a/internal/controller/taloscluster_controller.go +++ b/internal/controller/taloscluster_controller.go @@ -228,6 +228,24 @@ func (r *TalosClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request return routeResult, routeErr } + // Step G -- Bootstrap hardening (ONT-native path only). When hardeningProfileRef is + // set and the cluster is currently Ready, ensure the bootstrap NodeMaintenance exists + // in seam-tenant-{cluster} and set HardeningApplied once it reaches Ready=True. + // Idempotent: the label check prevents duplicate NodeMaintenance creation. + // CAPI path: HardeningApplied is set in reconcileCAPIPath (patches baked in at boot). + if tc.Spec.HardeningProfileRef != nil && (tc.Spec.CAPI == nil || !tc.Spec.CAPI.Enabled) { + currentReady := platformv1alpha1.FindCondition(tc.Status.Conditions, platformv1alpha1.ConditionTypeReady) + if currentReady != nil && currentReady.Status == metav1.ConditionTrue { + hardenResult, hardenErr := r.ensureBootstrapHardening(ctx, tc) + if hardenErr != nil { + return ctrl.Result{}, fmt.Errorf("reconcile: ensureBootstrapHardening: %w", hardenErr) + } + if hardenResult.RequeueAfter > 0 { + return hardenResult, nil + } + } + } + // Step F -- PKI expiry check and annotation-triggered rotation. // Only executed when the cluster was already in Ready state before this // reconcile pass (stable-Ready). Non-fatal: failures are logged and result @@ -341,15 +359,6 @@ func (r *TalosClusterReconciler) reconcileDirectBootstrap(ctx context.Context, t return result, nil } - // Copy the generated kubeconfig to target-cluster-kubeconfig in - // seam-tenant-{cluster} for all import clusters regardless of role. - // conductor-execute Jobs for both management-cluster and tenant-cluster - // PackExecutions mount this Secret — the same name, the same namespace, - // no role-specific divergence. platform-schema.md §9. - if err := r.ensureTenantKubeconfigCopy(ctx, tc); err != nil { - return ctrl.Result{}, fmt.Errorf("reconcileDirectBootstrap: copy kubeconfig to tenant namespace: %w", err) - } - // Role=tenant on the direct path: create the seam-tenant namespace and // register the cluster for RBAC and pack delivery. CP-INV-004: Platform is // the sole namespace creation authority. WS5. @@ -545,12 +554,23 @@ func (r *TalosClusterReconciler) reconcileCAPIPath(ctx context.Context, tc *plat return ctrl.Result{}, fmt.Errorf("reconcileCAPIPath: ensure CAPI Cluster: %w", err) } - // Step 4 — Ensure TalosConfigTemplate exists (with CNI=none + Cilium BPF params). - // CP-INV-009: every TalosConfigTemplate includes cluster.network.cni.name: none - // and the Cilium-required BPF kernel parameters. + // Step 4 — Ensure TalosConfigTemplate exists (with CNI=none + Cilium BPF params, + // plus HardeningProfile patches when hardeningProfileRef is set). CP-INV-009. if err := r.ensureTalosConfigTemplate(ctx, tc); err != nil { return ctrl.Result{}, fmt.Errorf("reconcileCAPIPath: ensure TalosConfigTemplate: %w", err) } + // Patches are baked into the template at creation time. Mark HardeningApplied when + // the profile is referenced (the template may already exist from a previous pass). + if tc.Spec.HardeningProfileRef != nil { + platformv1alpha1.SetCondition( + &tc.Status.Conditions, + platformv1alpha1.ConditionTypeHardeningApplied, + metav1.ConditionTrue, + platformv1alpha1.ReasonHardeningApplied, + "HardeningProfile patches merged into TalosConfigTemplate at provisioning time.", + tc.Generation, + ) + } // Step 5 — Ensure TalosControlPlane exists. if err := r.ensureTalosControlPlane(ctx, tc); err != nil { @@ -628,12 +648,30 @@ func (r *TalosClusterReconciler) reconcileCAPIPath(ctx context.Context, tc *plat // CAPI-bootstrapped cluster: origin is bootstrapped. tc.Status.Origin = platformv1alpha1.TalosClusterOriginBootstrapped + // Step 8.5 — Normalize CAPI-generated secrets to canonical platform names and + // register the cluster for RBAC and pack delivery. These three steps run once + // after CAPI Running is confirmed and are idempotent on subsequent passes. + // TALM writes {cluster}-talosconfig; translate to seam-mc-{cluster}-talosconfig + // so ensureExecutorTalosconfig finds the source when distributing to day-2 Jobs. + if err := r.ensureCAPITalosconfig(ctx, tc); err != nil { + return ctrl.Result{}, fmt.Errorf("reconcileCAPIPath: ensure CAPI talosconfig: %w", err) + } + // CAPI writes {cluster}-kubeconfig; translate to seam-mc-{cluster}-kubeconfig + // so EnsureRemoteConductorBootstrap and all conductor-execute Jobs read one name. + if err := r.ensureCAPIKubeconfig(ctx, tc); err != nil { + return ctrl.Result{}, fmt.Errorf("reconcileCAPIPath: ensure CAPI kubeconfig: %w", err) + } + // Register in RBACPolicy/RBACProfiles, create LocalQueue, platform-executor and + // wrapper-runner SA/Role/RoleBinding, distribute talosconfig to day-2 namespaces. + if err := r.ensureTenantOnboarding(ctx, tc); err != nil { + return ctrl.Result{}, fmt.Errorf("reconcileCAPIPath: tenant onboarding: %w", err) + } + // Step 9 — Check Cilium PackInstance Ready status. if tc.Spec.CAPI.CiliumPackRef == nil { // No Cilium pack configured — skip Cilium gate (development mode). logger.Info("no CiliumPackRef configured — skipping Cilium gate (development mode)", "name", tc.Name) - // Ensure Conductor Deployment exists and is Available. Gap 27. return r.ensureConductorReadyAndTransition(ctx, tc) } diff --git a/internal/controller/taloscluster_helpers.go b/internal/controller/taloscluster_helpers.go index 9499c8d..2b62ca0 100644 --- a/internal/controller/taloscluster_helpers.go +++ b/internal/controller/taloscluster_helpers.go @@ -473,22 +473,51 @@ func (r *TalosClusterReconciler) ensureTalosConfigTemplate(ctx context.Context, // not blocked by the kernel JIT hardening security gate. // kernel.unprivileged_bpf_disabled=0: allow non-privileged BPF, required for // Cilium's host networking and L3/L4 policy enforcement datapath. + baseSysctls := map[string]interface{}{ + "net.core.bpf_jit_harden": "0", + "kernel.unprivileged_bpf_disabled": "0", + } + + var hardeningPatches []interface{} + if tc.Spec.HardeningProfileRef != nil { + hpNS := tc.Spec.HardeningProfileRef.Namespace + if hpNS == "" { + hpNS = tc.Namespace + } + hp := &platformv1alpha1.HardeningProfile{} + if err := r.Client.Get(ctx, types.NamespacedName{ + Name: tc.Spec.HardeningProfileRef.Name, + Namespace: hpNS, + }, hp); err != nil { + return fmt.Errorf("ensureTalosConfigTemplate: get HardeningProfile: %w", err) + } + for k, v := range hp.Spec.SysctlParams { + baseSysctls[k] = v + } + for _, patchStr := range hp.Spec.MachineConfigPatches { + var patchObj map[string]interface{} + if err := json.Unmarshal([]byte(patchStr), &patchObj); err != nil { + return fmt.Errorf("ensureTalosConfigTemplate: parse HardeningProfile patch: %w", err) + } + hardeningPatches = append(hardeningPatches, patchObj) + } + } + machineConfigPatches := []interface{}{ map[string]interface{}{ "op": "replace", "path": "/cluster/network/cni/name", "value": "none", }, - // Cilium-required BPF kernel parameters. CP-INV-009. + // Cilium-required BPF kernel parameters merged with HardeningProfile sysctlParams. CP-INV-009. map[string]interface{}{ "op": "add", "path": "/machine/sysctls", - "value": map[string]interface{}{ - "net.core.bpf_jit_harden": "0", - "kernel.unprivileged_bpf_disabled": "0", - }, + "value": baseSysctls, }, } + machineConfigPatches = append(machineConfigPatches, hardeningPatches...) + if err := unstructured.SetNestedField(tct.Object, map[string]interface{}{ "generateType": "worker", "talosVersion": tc.Spec.CAPI.TalosVersion, @@ -759,17 +788,10 @@ func (r *TalosClusterReconciler) EnsureRemoteConductorBootstrap( tenantNS := "seam-tenant-" + tc.Name - // Determine kubeconfig Secret name from cluster mode. - // - Import clusters: kubeconfig is at tenantKubeconfigSecretName ("target-cluster-kubeconfig"), - // written by ensureTenantKubeconfigCopy. platform-schema.md §12. - // - CAPI clusters: kubeconfig is at "{cluster-name}-kubeconfig", written by CAPI after - // the cluster reaches Running state. - var kubeSecretName string - if tc.Spec.Mode == platformv1alpha1.TalosClusterModeImport { - kubeSecretName = tenantKubeconfigSecretName - } else { - kubeSecretName = tc.Name + "-kubeconfig" - } + // Both import and CAPI clusters: kubeconfig is at seam-mc-{cluster}-kubeconfig in + // seam-tenant-{cluster}. Import path writes it via ensureKubeconfigSecret. + // CAPI path writes it via ensureCAPIKubeconfig after the cluster reaches Running. + kubeSecretName := kubeconfigSecretName(tc.Name) // Get the kubeconfig Secret for the target cluster. kubeconfigSecret := &corev1.Secret{} @@ -1573,8 +1595,11 @@ func (r *TalosClusterReconciler) ensureExecutorTalosconfig(ctx context.Context, return fmt.Errorf("ensureExecutorTalosconfig: get source Secret %s/%s: %w", srcNS, srcName, err) } - // Copy to ont-system (Conductor agent Jobs) and to seam-tenant-{cluster} (day-2 executor Jobs). - for _, dstNS := range []string{bootstrapRunnerConfigNamespace, "seam-tenant-" + tc.Name} { + // Copy to seam-tenant-{cluster} (day-2 executor Jobs). The Job namespace is always + // seam-tenant-{clusterName}; operational_job_base.go mounts from the Job namespace. + // ont-system is NOT a destination: the conductor agent Deployment reads its talosconfig + // via TALOSCONFIG_PATH from the enable bundle manifest, not via this copy. + for _, dstNS := range []string{"seam-tenant-" + tc.Name} { dst := &corev1.Secret{} if err := r.Client.Get(ctx, types.NamespacedName{Name: dstName, Namespace: dstNS}, dst); err == nil { continue // already exists @@ -1809,3 +1834,83 @@ func (r *TalosClusterReconciler) ensureWrapperRunnerResources(ctx context.Contex return nil } + +// ensureCAPIKubeconfig copies the CAPI-generated kubeconfig Secret to the canonical +// seam-mc-{cluster}-kubeconfig name in seam-tenant-{cluster}. CAPI writes +// {cluster}-kubeconfig in the cluster namespace after the cluster reaches Running state. +// All platform operations (EnsureRemoteConductorBootstrap, PKI rotation, conductor-execute +// Jobs) read from the canonical name. Idempotent. Called from reconcileCAPIPath after +// CAPI Cluster reaches Running. +func (r *TalosClusterReconciler) ensureCAPIKubeconfig(ctx context.Context, tc *platformv1alpha1.TalosCluster) error { + tenantNS := "seam-tenant-" + tc.Name + dstName := kubeconfigSecretName(tc.Name) + + if err := r.Client.Get(ctx, types.NamespacedName{Name: dstName, Namespace: tenantNS}, &corev1.Secret{}); err == nil { + return nil + } else if !apierrors.IsNotFound(err) { + return fmt.Errorf("ensureCAPIKubeconfig: check %s/%s: %w", tenantNS, dstName, err) + } + + srcName := tc.Name + "-kubeconfig" + src := &corev1.Secret{} + if err := r.Client.Get(ctx, types.NamespacedName{Name: srcName, Namespace: tenantNS}, src); err != nil { + if apierrors.IsNotFound(err) { + return nil // CAPI not yet written; reconcile will retry + } + return fmt.Errorf("ensureCAPIKubeconfig: get source %s/%s: %w", tenantNS, srcName, err) + } + + dst := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: dstName, + Namespace: tenantNS, + Labels: map[string]string{"platform.ontai.dev/cluster": tc.Name}, + }, + Type: corev1.SecretTypeOpaque, + Data: src.Data, + } + if err := r.Client.Create(ctx, dst); err != nil && !apierrors.IsAlreadyExists(err) { + return fmt.Errorf("ensureCAPIKubeconfig: create %s/%s: %w", tenantNS, dstName, err) + } + return nil +} + +// ensureCAPITalosconfig copies the TALM-generated talosconfig Secret to the canonical +// seam-mc-{cluster}-talosconfig name in seam-tenant-{cluster}. TALM writes +// {cluster}-talosconfig in the cluster namespace. The canonical name is what +// ensureExecutorTalosconfig reads as its source, so day-2 executor Jobs receive +// the correct talosconfig in seam-tenant-{cluster}. Idempotent. Called from +// reconcileCAPIPath after CAPI Cluster reaches Running. +func (r *TalosClusterReconciler) ensureCAPITalosconfig(ctx context.Context, tc *platformv1alpha1.TalosCluster) error { + tenantNS := "seam-tenant-" + tc.Name + dstName := talosconfigSecretName(tc.Name) + + if err := r.Client.Get(ctx, types.NamespacedName{Name: dstName, Namespace: tenantNS}, &corev1.Secret{}); err == nil { + return nil + } else if !apierrors.IsNotFound(err) { + return fmt.Errorf("ensureCAPITalosconfig: check %s/%s: %w", tenantNS, dstName, err) + } + + srcName := tc.Name + "-talosconfig" + src := &corev1.Secret{} + if err := r.Client.Get(ctx, types.NamespacedName{Name: srcName, Namespace: tenantNS}, src); err != nil { + if apierrors.IsNotFound(err) { + return nil // TALM not yet written; reconcile will retry + } + return fmt.Errorf("ensureCAPITalosconfig: get source %s/%s: %w", tenantNS, srcName, err) + } + + dst := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: dstName, + Namespace: tenantNS, + Labels: map[string]string{"platform.ontai.dev/cluster": tc.Name}, + }, + Type: corev1.SecretTypeOpaque, + Data: src.Data, + } + if err := r.Client.Create(ctx, dst); err != nil && !apierrors.IsAlreadyExists(err) { + return fmt.Errorf("ensureCAPITalosconfig: create %s/%s: %w", tenantNS, dstName, err) + } + return nil +} diff --git a/internal/controller/taloscluster_import_helpers.go b/internal/controller/taloscluster_import_helpers.go index e839212..926edb0 100644 --- a/internal/controller/taloscluster_import_helpers.go +++ b/internal/controller/taloscluster_import_helpers.go @@ -187,56 +187,3 @@ func (r *TalosClusterReconciler) ensureKubeconfigSecret(ctx context.Context, tc return ctrl.Result{}, nil } -// tenantKubeconfigSecretName is the name of the kubeconfig Secret written to the -// seam-tenant-{clusterName} namespace for role=tenant clusters on the direct import path. -// Operators look for this name when routing target cluster API access. WS5. -const tenantKubeconfigSecretName = "target-cluster-kubeconfig" - -// ensureTenantKubeconfigCopy copies the generated kubeconfig Secret into -// seam-tenant-{clusterName} as "target-cluster-kubeconfig". Called for all import -// clusters regardless of role. conductor-execute Jobs for both management-cluster and -// tenant-cluster PackExecutions mount this Secret by name. Idempotent. WS5. -func (r *TalosClusterReconciler) ensureTenantKubeconfigCopy(ctx context.Context, tc *platformv1alpha1.TalosCluster) error { - tenantNS := "seam-tenant-" + tc.Name - - // Idempotency: skip if the copy already exists. - existing := &corev1.Secret{} - if err := r.Client.Get(ctx, types.NamespacedName{ - Name: tenantKubeconfigSecretName, - Namespace: tenantNS, - }, existing); err == nil { - return nil // already present - } else if !apierrors.IsNotFound(err) { - return fmt.Errorf("ensureTenantKubeconfigCopy: check existing secret %s/%s: %w", - tenantNS, tenantKubeconfigSecretName, err) - } - - // Read the kubeconfig from seam-tenant-{cluster}. - sourceName := kubeconfigSecretName(tc.Name) - source := &corev1.Secret{} - secretsNS := importSecretsNamespace(tc.Name) - if err := r.Client.Get(ctx, types.NamespacedName{ - Name: sourceName, - Namespace: secretsNS, - }, source); err != nil { - return fmt.Errorf("ensureTenantKubeconfigCopy: read source kubeconfig %s/%s: %w", - secretsNS, sourceName, err) - } - - // Write the copy to seam-tenant-{clusterName}. - copy := &corev1.Secret{ - ObjectMeta: metav1.ObjectMeta{ - Name: tenantKubeconfigSecretName, - Namespace: tenantNS, - Labels: map[string]string{ - importClusterLabel: tc.Name, - }, - }, - Data: source.Data, - } - if err := r.Client.Create(ctx, copy); err != nil && !apierrors.IsAlreadyExists(err) { - return fmt.Errorf("ensureTenantKubeconfigCopy: create kubeconfig copy %s/%s: %w", - tenantNS, tenantKubeconfigSecretName, err) - } - return nil -} diff --git a/test/e2e/day2/pkirotation_e2e_test.go b/test/e2e/day2/pkirotation_e2e_test.go index a444fe1..e068676 100644 --- a/test/e2e/day2/pkirotation_e2e_test.go +++ b/test/e2e/day2/pkirotation_e2e_test.go @@ -96,32 +96,18 @@ var _ = Describe("TENANT-PKI-ROTATE: PKIRotation on import-mode cluster", func() "PKIRotation Ready must be True after pki-rotate completes") }, pkiRotationTimeout, pollInterval).Should(Succeed()) - // Verify that pkiRotateHandler wrote the refreshed kubeconfig Secret. - // The Secret name is target-cluster-kubeconfig in seam-tenant-{cluster}. - // platform-schema.md §13: pkiRotateHandler writes two Secrets on success. + // Verify that pkiRotateHandler wrote the refreshed seam-mc-{cluster}-kubeconfig + // Secret. This is the canonical kubeconfig name; target-cluster-kubeconfig no + // longer exists. platform-schema.md §13. Eventually(func(g Gomega) { secret := &corev1.Secret{} g.Expect(mgmtClient.Get(mgmtCtx, types.NamespacedName{ - Name: "target-cluster-kubeconfig", + Name: "seam-mc-" + cluster + "-kubeconfig", Namespace: tenantNS, - }, secret)).To(Succeed(), "target-cluster-kubeconfig Secret not found in %s", tenantNS) + }, secret)).To(Succeed(), "seam-mc-%s-kubeconfig Secret not found in %s", cluster, tenantNS) g.Expect(secret.Data).NotTo(BeEmpty(), - "target-cluster-kubeconfig Secret must have non-empty data") + "seam-mc-%s-kubeconfig Secret must have non-empty data", cluster) }, 30*time.Second, pollInterval).Should(Succeed()) - - // Best-effort: verify the seam-mc-{cluster}-kubeconfig Secret was also refreshed. - // This Secret is written with best-effort semantics by pkiRotateHandler -- - // its absence does not indicate a failed rotation. - func() { - secret := &corev1.Secret{} - if err := mgmtClient.Get(mgmtCtx, types.NamespacedName{ - Name: "seam-mc-" + cluster + "-kubeconfig", - Namespace: tenantNS, - }, secret); err == nil { - Expect(secret.Data).NotTo(BeEmpty(), - "seam-mc-%s-kubeconfig Secret exists but has empty data", cluster) - } - }() }) }) diff --git a/test/integration/day2/suite_test.go b/test/integration/day2/suite_test.go index b9cb559..bd402cc 100644 --- a/test/integration/day2/suite_test.go +++ b/test/integration/day2/suite_test.go @@ -6,10 +6,10 @@ // the unit tests' fake client cannot replicate (no SSA merge semantics, no etcd // visibility, no watch event propagation). // -// envtest binaries required: +// envtest binaries required. From the ontai root: // -// setup-envtest use --bin-dir /tmp/envtest-bins -// export KUBEBUILDER_ASSETS=/tmp/envtest-bins/k8s/1.35.0-linux-amd64 +// make envtest-setup +// export KUBEBUILDER_ASSETS=$(make -s envtest-path) // // All tests skip automatically when KUBEBUILDER_ASSETS is absent. package day2_integration_test