diff --git a/docs/en/solutions/How_to_Migrate_NFS_PersistentVolumes_to_OceanStor_Dorado_and_Rebind_PVCs_in_Place.md b/docs/en/solutions/How_to_Migrate_NFS_PersistentVolumes_to_OceanStor_Dorado_and_Rebind_PVCs_in_Place.md new file mode 100644 index 000000000..de73104af --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_NFS_PersistentVolumes_to_OceanStor_Dorado_and_Rebind_PVCs_in_Place.md @@ -0,0 +1,401 @@ +--- +products: + - Alauda Container Platform +kind: + - Solution +ProductsVersion: + - 4.x +--- + +# How to Migrate NFS PersistentVolumes to OceanStor Dorado and Rebind PVCs in Place + +## Overview + +`PersistentVolumeClaim.spec.volumeName` is immutable. An existing PVC cannot be updated to reference a different PersistentVolume. To migrate a workload to a new NFS volume without changing the PVC name, delete and recreate the PVC with the same name, and pre-bind the new PV to that claim. + +This guide describes how to copy the data, reserve the new PV for the original PVC name, and restore the workload. The procedure requires a maintenance window. It is validated for Deployment and StatefulSet workloads. + +For a StatefulSet, the PVC names generated from `volumeClaimTemplates` remain unchanged. After migration, the StatefulSet must also be recreated with an updated template so that later scale-out operations use the new StorageClass. + +## Environment + +| Component | Version | +|-----------|---------| +| Container Platform | ACP 4.x (validated on 4.3.1) | +| Node Operating System | Micro OS 5.5 | +| Storage Device | OceanStor Dorado 6.1.9 | +| OceanStor CSI driver For Dorado | v4.12.0 | +| Source storage | NFS, provisioned by `nfs.csi.k8s.io` | +| Target storage | Dorado NFS (`volumeType: fs`) | +| Validated workloads | Deployment (RWX), StatefulSet (RWO, 2 ordinals) | + +> **Note**: The data-copy procedure is for `Filesystem`-mode volumes. A `Block`-mode volume requires a block-level copy method. The PVC rebinding procedure uses Kubernetes objects only and can be applied to other storage types. + +## Prerequisites + +- An ACP 4.x cluster with the OceanStor CSI driver For Dorado installed and a target NFS StorageClass available. +- `kubectl` access with permission to patch PersistentVolumes, which are cluster-scoped resources. +- A maintenance window in which the workload can be stopped. +- Sufficient capacity on the target array for the new volumes. +- An image that contains `rsync`, or another copy tool that preserves the required file metadata. +- A current backup and a verified rollback plan for the workload. + +The following placeholders are used throughout this guide. Replace them with values from your environment: + +| Placeholder | Description | +|-------------|-------------| +| `` | Workload namespace | +| `` | Existing PVC name, which is preserved | +| `` | Temporary PVC used to provision the target volume | +| `` | Target StorageClass backed by the new array | +| `` | Workload resource, for example `deploy/app` or `sts/web` | +| `` | Label selector for the workload Pods | +| `` | PV provisioned for the temporary target PVC | + +## Resolution + +### 1. Review the PVC rebinding requirements + +The migration has two operations: + +- Copy data from the source volume to the target volume. +- Rebind the original PVC name to the target PV. + +The PVC must be deleted and recreated because `PersistentVolumeClaim.spec.volumeName` cannot be changed. The workload manifest can continue to reference the original PVC name. + +The PV records its PVC binding in `PersistentVolume.spec.claimRef`: + +| Field | Meaning | +|-------|---------| +| `namespace`, `name` | Namespace and name of the claim that owns the volume | +| `uid` | UID of the specific PVC object; a recreated PVC has a different UID | +| `resourceVersion` | Optimistic concurrency token for the referenced object | + +These fields determine the PV state: + +| `claimRef` condition | PV state and behavior | +|----------------------|-----------------------| +| No `claimRef` | `Available`; any matching PVC can bind | +| `namespace` and `name` are present, but `uid` is absent | `Available`; reserved for the named PVC | +| Complete `claimRef`, and the referenced PVC exists | `Bound` | +| Complete `claimRef`, but the PVC with that `uid` no longer exists | `Released` | + +A `Released` PV does not return to `Available` automatically. Reserving the new PV therefore requires two merge patches. The first patch sets the target `namespace` and `name`. Because a merge patch retains unspecified fields, the stale `uid` remains and the PV stays `Released`. The second patch must explicitly set both `uid` and `resourceVersion` to `null`. + +Before starting, confirm that the final PVC will match the target PV in all of the following areas: + +- The requested capacity does not exceed the PV capacity. +- The PVC access modes are supported by the PV. +- `storageClassName` matches the PV exactly. + +### 2. Provision the target volume + +Read the source PVC so that the temporary PVC uses compatible access modes and capacity: + +```bash +kubectl -n get pvc \ + -o custom-columns=\ +NAME:.metadata.name,MODES:.spec.accessModes,SIZE:.spec.resources.requests.storage +``` + +Create a temporary PVC on the target StorageClass: + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: + namespace: +spec: + # Set these to the source PVC's access modes, read in the previous command. + # ReadWriteMany matches the validated Deployment; the validated StatefulSet + # uses ReadWriteOnce. Do not widen the modes unintentionally. + accessModes: ["ReadWriteMany"] + storageClassName: + resources: + requests: + # Must be >= the source volume. The rebinding step later creates the + # final claim against this same volume, so its capacity is fixed here. + storage: 2Gi +``` + +Wait until the PVC is bound: + +```bash +kubectl -n wait --for=jsonpath='{.status.phase}'=Bound \ + pvc/ --timeout=180s +``` + +For a StatefulSet, create one target PVC for each ordinal. + +### 3. Copy the initial data + +Create a temporary Pod that mounts the source and target PVCs: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: migrator + namespace: +spec: + restartPolicy: Never + containers: + - name: m + image: + command: ["sh", "-c", "sleep infinity"] + volumeMounts: + - {name: src, mountPath: /src} + - {name: dst, mountPath: /dst} + volumes: + - name: src + persistentVolumeClaim: {claimName: } + - name: dst + persistentVolumeClaim: {claimName: } +``` + +Run an initial copy while the application is still running. This pass moves most of the data before the maintenance window: + +```bash +kubectl -n exec migrator -- \ + rsync -aHAX --numeric-ids --delete --exclude='/.snapshot' /src/ /dst/ +``` + +Use `rsync` where possible so that the final pass transfers only changed data. Note the following options and their limits: + +- `-H` preserves hard links, `-A` preserves POSIX ACLs, and `-X` preserves extended attributes. +- `--numeric-ids` synchronizes UID/GID by numeric value instead of by user and group name, which keeps ownership correct when the source and target hosts have different name-to-ID mappings. It does **not** override NFS squash: `root_squash` and `all_squash` are applied by the NFS server to the client's credentials, independent of any `rsync` flag. +- `-aHAX` does not include `--sparse`. If the data contains sparse files, such as virtual machine images or preallocated database files, add `-S` so that the holes are not written as real data, and size the target volume accordingly. +- `--exclude='/.snapshot'` prevents `--delete` from trying to remove the target volume's read-only `.snapshot` directory. The leading slash limits the match to the volume root. + +Without the exclusion, `rsync` can report the following errors and exit with a non-zero status: + +```text +rmdir: '/data/.snapshot': Permission denied +rm: can't remove '/data/.snapshot': Permission denied +``` + +Because the copy runs across NFS, the copy process must be able to read every source file and to set ownership on the target. Run the migrator as UID 0, confirm that the source export does not squash root, and confirm that the target export allows `chown`. Otherwise, `0600` files may be unreadable or ownership cannot be restored. Test with a representative file before the full copy. + +If an image with `rsync` is not available, the following command runs inside the Pod and preserves permissions, ownership, and symbolic links, but transfers all data on every pass: + +```bash +kubectl -n exec migrator -- sh -c 'tar -C /src -cf - . | tar -C /dst -xpf -' +``` + +Plain `tar` without `--xattrs --acls` does not preserve POSIX ACLs or extended attributes, and expands sparse files to their full size. Select the copy method according to the workload's metadata requirements. + +The source PVC is provisioned by `nfs.csi.k8s.io`, whose `CSIDriver.spec.attachRequired` is `false`. NFS volumes are not attached through a `VolumeAttachment`, so mounting the source PVC on the migrator Pod while the application also mounts it does not cause a `Multi-Attach error`, even across nodes. The migrator can run on any node. For an attach-based driver such as iSCSI or RBD, where `attachRequired` is `true`, a `ReadWriteOnce` source instead requires the migrator and the application to share a node. In all cases, evaluate application consistency before copying a volume that is still being written. + +Because both volumes use NFS, the copy can alternatively run on a host that can reach both NFS servers. This avoids the in-cluster Pod entirely but requires suitable network access outside the cluster. + +### 4. Protect both PVs with the Retain policy + +Set variables for the remaining procedure and record the target and source PV names: + +```bash +NS=; PVC=; TMP=; WORKLOAD= +NEWPV=$(kubectl -n $NS get pvc $TMP -o jsonpath='{.spec.volumeName}') +OLDPV=$(kubectl -n $NS get pvc $PVC -o jsonpath='{.spec.volumeName}') +``` + +Before deleting either PVC, change both PVs to `Retain`: + +```bash +kubectl patch pv $NEWPV -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +kubectl patch pv $OLDPV -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +``` + +> **Warning**: Do not continue until both PVs use `Retain`. Deleting a PVC while its PV still uses `reclaimPolicy: Delete` can delete the underlying volume. The new PV must be retained to protect the migrated data, and the old PV must be retained to preserve the rollback path. This is the only checkpoint whose omission can cause irreversible data loss. + +Verify both policies: + +```bash +kubectl get pv $NEWPV $OLDPV \ + -o custom-columns=NAME:.metadata.name,POLICY:.spec.persistentVolumeReclaimPolicy +``` + +### 5. Stop the workload and complete the final synchronization + +Record the replica count, stop the workload, and wait for its Pods to be deleted: + +```bash +REPLICAS=$(kubectl -n $NS get $WORKLOAD -o jsonpath='{.spec.replicas}') +kubectl -n $NS scale $WORKLOAD --replicas=0 +kubectl -n $NS wait --for=delete pod -l --timeout=300s +``` + +Run the final synchronization and compare file checksums: + +```bash +kubectl -n $NS exec migrator -- \ + rsync -aHAX --numeric-ids --delete --exclude='/.snapshot' /src/ /dst/ + +kubectl -n $NS exec migrator -- sh -c ' + set -eu + for d in /src /dst; do + ( cd "$d" && find . -type f -not -path "./.snapshot/*" -print0 \ + | sort -z | xargs -0 -r md5sum ) > /tmp/$(basename "$d").sum + done + diff /tmp/src.sum /tmp/dst.sum && echo CONTENT_OK' +``` + +`set -eu` and NUL-delimited (`-print0` / `sort -z` / `xargs -0`) handling are required: without them, a filename containing spaces splits into separate arguments and a failed `md5sum` only writes to stderr, so two empty `.sum` files can compare equal and print `CONTENT_OK` even when the data differs. This check is immediately followed by PVC deletion, so a false positive is not acceptable. + +This check compares regular-file contents only. Ownership, permissions, ACLs, extended attributes, symbolic-link targets, and hard-link relationships are not verified here; check them separately according to the workload's requirements (see the FAQ). Do not continue unless the checksums match. + +### 6. Delete and recreate the PVC under its original name + +Delete the migrator Pod and both PVCs: + +```bash +kubectl -n $NS delete pod migrator +kubectl -n $NS delete pvc $TMP $PVC +``` + +Both PVs enter the `Released` state and retain their data. + +Reserve the new PV for the original PVC name. Run both merge patches in the following order: + +```bash +kubectl patch pv $NEWPV --type merge -p \ + "{\"spec\":{\"claimRef\":{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"namespace\":\"$NS\",\"name\":\"$PVC\"}}}" + +kubectl patch pv $NEWPV --type merge -p \ + '{"spec":{"claimRef":{"uid":null,"resourceVersion":null}}}' +``` + +The second merge patch is required. Without it, the stale `uid` remains in `claimRef` and the PV stays `Released`. + +Confirm that the new PV is `Available` and reserved for the original PVC name: + +```bash +kubectl get pv $NEWPV -o custom-columns=\ +NAME:.metadata.name,STATUS:.status.phase,CLAIM:.spec.claimRef.name,UID:.spec.claimRef.uid +``` + +```text +NAME STATUS CLAIM UID +pvc-0464141b-… Available app-data +``` + +Recreate the PVC with its original name and pre-bind it to the new PV: + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: # unchanged + namespace: +spec: + accessModes: ["ReadWriteMany"] # match the source PVC (ReadWriteOnce for the StatefulSet) + storageClassName: # must match the new PV exactly + resources: + requests: + storage: 2Gi # must be <= the PV capacity + volumeName: +``` + +Wait for the PVC to become `Bound` before restarting the workload. + +### 7. Restart and verify the workload + +Restore the recorded replica count: + +```bash +kubectl -n $NS scale $WORKLOAD --replicas=$REPLICAS +``` + +Confirm that the workload uses the target NFS mount and verify the application data: + +```bash +kubectl -n $NS exec -- sh -c 'mount | grep " /data "' +``` + +For the validated Deployment with `ReadWriteMany`, the following checks succeeded: + +| Check | Result | +|-------|--------| +| File checksums | Matched the pre-migration baseline | +| Permissions and ownership | Preserved, including a mode `600` file owned by `1000:1000` | +| Symbolic links | Preserved, including relative targets | +| Deployment manifest | Unchanged and continued to reference the original PVC name | +| Mount point | Changed to the target NFS export | +| Concurrent writers | Two replicas and one external host observed each other's writes | + +For the validated StatefulSet with `ReadWriteOnce` and two ordinals, each ordinal retained its own data and PVC name, and both mount points changed to the target array without cross-volume data. + +### 8. Recreate a StatefulSet with the target StorageClass + +Apply the migration separately to each StatefulSet ordinal, such as `data-web-0` and `data-web-1`. Each ordinal has its own source and target volume. + +Rebinding the PVCs does not update the StatefulSet's `volumeClaimTemplates`. If the template still contains the old StorageClass, a later scale-out creates new PVCs on the old storage and splits the StatefulSet across storage systems: + +```text +data-web-0 Bound <- migrated +data-web-1 Bound <- migrated +data-web-2 Bound <- newly created from the template +``` + +`volumeClaimTemplates` cannot be changed in place: + +```text +The StatefulSet "web" is invalid: spec: Forbidden: updates to statefulset spec for +fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', +'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' +are forbidden +``` + +Delete the StatefulSet with `--cascade=orphan`, update `storageClassName` in `volumeClaimTemplates`, and recreate it: + +```bash +kubectl -n delete sts --cascade=orphan +# edit storageClassName in volumeClaimTemplates, then: +kubectl apply -f .yaml +``` + +The orphan deletion leaves the Pods and PVCs in place. The recreated StatefulSet adopts the existing Pods through its label selector. In the validation, Pod age remained continuous and restart counts stayed at zero, confirming that the Pods did not restart. Perform this step immediately after migration so that future scale-out uses the target StorageClass. + +### 9. Review risk and rollback actions + +| Step | Consequence of failure | Rollback | +|------|------------------------|----------| +| Initial data copy | No application interruption; the source remains available | Delete the temporary PVC and retry | +| Set both PVs to `Retain` | Omitting this protection can destroy a volume when its PVC is deleted | No recovery after deletion; this is the only irreversible checkpoint | +| Stop the workload | Downtime begins | Restore the recorded replica count | +| Final sync and verification | A checksum mismatch indicates an incomplete copy | Restore the workload; the source remains unchanged | +| Delete the PVCs | PVs become `Released`; data remains intact because both use `Retain` | Recreate the original PVC and bind it to the old PV | +| Reserve the new PV | Binding does not complete; no data is deleted | Reapply the two merge patches | +| Recreate the PVC | The PVC stays `Pending` if its fields do not match | Delete the PVC, correct it, and recreate it | +| Restart the workload | The volume does not mount correctly | Scale to zero and rebind the original PVC name to the old PV | +| Recreate the StatefulSet | The controller does not adopt existing Pods if selectors do not match | Correct the manifest and recreate the StatefulSet; orphaned Pods remain running | + +After both PVs are confirmed as `Retain`, every subsequent migration step is reversible. The old PV provides the rollback path. To restore it, reserve the old PV for the original PVC name with the same two-patch procedure, and recreate the PVC with `volumeName` set to the old PV. + +Before deleting either PVC, run this check and confirm that both rows show `Retain`: + +```bash +kubectl get pv $NEWPV $OLDPV \ + -o custom-columns=NAME:.metadata.name,POLICY:.spec.persistentVolumeReclaimPolicy +``` + +### 10. Finalize after a successful migration + +Both PVs are left as `Retain`, which leaves cleanup decisions to the administrator. After the workload has run on the target volume long enough to close the rollback window: + +- The old PV stays `Released`. Once a backup is confirmed and the rollback path is no longer needed, delete the old PV and reclaim the underlying volume on the source system. Deleting the PV object does not free the source-side storage. +- Decide the target PV's final reclaim policy. It remains `Retain`, so a later PVC deletion does not remove the array-side volume. Keep `Retain` for volumes that are shared or consumed externally; the volume must then be cleaned up by hand. Restore `Delete` only if the volume's lifecycle should follow its PVC. + +## FAQ + +### What must match when the PVC is recreated? + +The PVC request must not exceed the PV capacity. Its access modes must be a subset of the modes supported by the PV. Its `storageClassName` must match the PV exactly. For a static PV, set `storageClassName: ""` on both the PV and PVC; omitting the PVC field can cause Kubernetes to substitute the default StorageClass. + +### Why does the PV remain in Released after the first patch? + +The first merge patch changes `claimRef.namespace` and `claimRef.name` but preserves the stale `uid` because unspecified fields are not removed. The second merge patch must set both `uid` and `resourceVersion` to `null`. The PV can then become `Available` while remaining reserved for the named PVC. + +### How should file ownership be checked after migration? + +Source and target NFS servers can apply different squash policies, and that mapping happens on the server, not in `rsync`. Run the copy as UID 0 against a source export that does not squash root and a target export that allows `chown`. Use `--numeric-ids` so that IDs are matched by value rather than by name, and verify representative files after the copy. Confirm UID/GID values, permissions, ACLs, extended attributes, hard links, symbolic links, and sparse-file handling according to the application's requirements. diff --git a/docs/en/solutions/How_to_Share_OceanStor_Dorado_NFS_Volumes_with_Hosts_Outside_the_Cluster.md b/docs/en/solutions/How_to_Share_OceanStor_Dorado_NFS_Volumes_with_Hosts_Outside_the_Cluster.md new file mode 100644 index 000000000..fee4c6033 --- /dev/null +++ b/docs/en/solutions/How_to_Share_OceanStor_Dorado_NFS_Volumes_with_Hosts_Outside_the_Cluster.md @@ -0,0 +1,295 @@ +--- +products: + - Alauda Container Platform +kind: + - Solution +ProductsVersion: + - 4.x +--- + +# How to Share OceanStor Dorado NFS Volumes with Hosts Outside the Cluster + +## Overview + +An NFS PersistentVolume provisioned by the OceanStor CSI driver For Dorado is stored as a filesystem and an NFS share on the storage array. A host outside the cluster can mount the share directly if it can reach the NFS portal and is allowed by the access-client rule. + +This guide describes two supported approaches: + +- Create the filesystem and NFS share on the array, and then connect the existing volume to Kubernetes as a static volume. This approach provides a fixed export path. +- Use a dynamically provisioned volume and obtain its generated export path from the PersistentVolume. + +Use a static volume when an external system requires a stable mount path. Use a dynamically provisioned volume when Kubernetes manages the volume lifecycle and external access is required only after provisioning. + +## Environment + +| Component | Version | +|-----------|---------| +| Container Platform | ACP 4.x (validated on 4.3.1) | +| Node Operating System | Micro OS 5.5 | +| Storage Device | OceanStor Dorado 6.1.9 | +| OceanStor CSI driver For Dorado | v4.12.0 | +| External host | CentOS 7 with `nfs-utils` | +| Validated Protocol | NFS (v4.1 / v4.2) | + +> **Note**: The procedures in this guide were validated with the versions listed above. Confirm CSI driver and storage firmware compatibility before applying them to another version combination. + +## Prerequisites + +- An ACP 4.x cluster with the OceanStor CSI driver For Dorado installed and an NFS backend configured. See *How to Install and Configure OceanStor CSI driver For Dorado on ACP*. +- A StorageClass with `volumeType: fs` and an `authClient` value that permits the external host. +- Layer-3 connectivity from the external host to the NFS data-plane portal. NFSv4.1/4.2 mounts require TCP port 2049 only. The `showmount` check in this guide additionally needs rpcbind (port 111) and the mountd port; if those are not open, the volume can still be mounted but `showmount` fails. +- `nfs-utils`, or an equivalent NFS client package, installed on the external host. +- For a static volume, administrative access to the storage REST API or DeviceManager UI and the storage pool information provided by the storage administrator. + +The following placeholders are used throughout this guide. Replace them with values from your environment: + +| Placeholder | Description | +|-------------|-------------| +| `` | NFS data-plane portal address | +| `` | Storage management plane address | +| `` | Host outside the cluster that mounts the volume | +| `` | CSI storage backend name, for example `backend-nfs` | +| `` | NFS StorageClass name | +| `` | Numeric storage pool ID, not the pool name | +| `` | Filesystem name you choose for the static volume, for example `acp_static_nfs` | +| `` | Storage device ID returned when the REST session is created | +| `` | Filesystem ID returned when the filesystem is created | +| `` | NFS share ID returned when the share is created | +| `` | CIDR range that is allowed to mount the share | +| `` | Namespace of the PVC | + +## Resolution + +### 1. Select the volume provisioning approach + +Each NFS volume has a filesystem, an NFS share, and one or more access-client rules on the array. The StorageClass `authClient` parameter is used for the access-client rule. For example, `authClient: "*"` allows every client that can reach the NFS portal to mount the share. + +No additional CSI configuration is required for external mounting. The provisioning approach determines the export path and lifecycle: + +| Item | Static volume | Dynamically provisioned volume | +|------|---------------|--------------------------------| +| Export path | Uses the filesystem name selected when the volume is created | Uses a generated name stored in the PV | +| External-host configuration | Can be prepared before the Kubernetes objects are created | Must be updated with the generated path after provisioning | +| Lifecycle | Created and controlled separately from dynamic provisioning | Normally controlled by the PVC and StorageClass reclaim policy | + +### 2. Create the static volume on the array + +Create the filesystem, NFS share, and access-client rule in this order. The following block shows the request bodies for the storage REST API; it is not a runnable script. Each request goes to `https:///deviceManager/rest//...` and requires an authenticated session: create a session first (`POST /deviceManager/rest/xxxxx/sessions`) to obtain `` and an `iBaseToken`, then send that token in the `iBaseToken` header of each subsequent request. The same objects can also be created in the DeviceManager UI. + +```text +# 1. Filesystem. CAPACITY is in 512-byte sectors, so 4194304 = 2 GiB. +# PARENTID is the numeric storage pool ID, not the pool name. +# POST /deviceManager/rest//filesystem +{ + "NAME": "", + "PARENTID": "", + "CAPACITY": 4194304, + "ALLOCTYPE": 1, + "SECTORSIZE": 16384 +} + +# 2. NFS share. Use the FS ID from the response to request 1 as . +# POST /deviceManager/rest//NFSHARE +{ "SHAREPATH": "//", "FSID": "", "vstoreId": "0" } + +# 3. Access client. Use the share ID from the response to request 2 as . +# These values match what the CSI driver sets on dynamically +# provisioned volumes, so behaviour stays consistent. +# POST /deviceManager/rest//NFS_SHARE_AUTH_CLIENT +{ + "NAME": "*", "PARENTID": "", + "ACCESSVAL": 1, "SYNC": 0, "ALLSQUASH": 1, + "ROOTSQUASH": 1, "SECURE": 1, "vstoreId": "0" +} +``` + +The `CAPACITY` value is measured in 512-byte sectors. `PARENTID` must be the numeric ID of the storage pool, not its name. `` comes from the filesystem-creation response, and `` from the NFS-share response. After creation, the export is available at `:/`. + +The static volume can be connected to Kubernetes by creating the PV yourself or by asking the CSI driver to manage the existing volume. Select one of the following methods. + +### 3. Connect the static volume by creating a PV and PVC + +Create a PV and pre-bind a PVC to it: + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: pv-static-nfs +spec: + # Empty string: this PV takes no part in dynamic provisioning + storageClassName: "" + volumeMode: Filesystem + accessModes: ["ReadWriteMany"] + capacity: + storage: 2Gi + # An administrator-provisioned volume must never be deleted by Kubernetes + persistentVolumeReclaimPolicy: Retain + # Not inherited from a StorageClass, so set it here or the NFS version is negotiated + mountOptions: + - nfsvers=4.1 + csi: + driver: csi.huawei.com + volumeHandle: . # format: . + volumeAttributes: + backend: + name: +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: static-a + namespace: +spec: + # Must also be an empty string, otherwise the default StorageClass is + # substituted and the claim will not bind + storageClassName: "" + accessModes: ["ReadWriteMany"] + resources: + requests: + storage: 2Gi + volumeName: pv-static-nfs +``` + +Set `storageClassName: ""` on both the PV and PVC. If the field is omitted from the PVC, Kubernetes can substitute the default StorageClass and the PVC will not bind to this PV. + +A static PV does not inherit `mountOptions` from a StorageClass. When `mountOptions` is omitted in this environment, NFS negotiation selects version 4.2. Setting `nfsvers=4.1` makes the mount use version 4.1. + +### 4. Connect the static volume by using CSI volume management + +Instead of creating the PV, add the volume-management annotations and required label to a PVC. The CSI driver imports the existing volume and generates the PV: + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: static-b + namespace: + annotations: + csi.huawei.com/manageVolumeName: # volume name on the array + csi.huawei.com/manageBackendName: + labels: + provisioner: csi.huawei.com # required +spec: + accessModes: ["ReadWriteMany"] + storageClassName: # a real StorageClass, not "" + resources: + requests: + storage: 2Gi # must match the array-side size +``` + +The PVC must contain both `csi.huawei.com/manageVolumeName` and `csi.huawei.com/manageBackendName`, and the `provisioner: csi.huawei.com` label. In this method, `storageClassName` refers to an existing StorageClass rather than an empty string. + +The generated PV is named `pvc-`. Its `volumeHandle` points to `.`, and the export path continues to use the filesystem name selected on the array. + +> **Important**: The generated PV inherits `reclaimPolicy` from the StorageClass. If the StorageClass uses `Delete`, deleting the PVC also deletes the manually created filesystem. Immediately after the PVC becomes `Bound`, change the generated PV to `Retain`: +> +> ```bash +> kubectl patch pv \ +> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +> ``` + +### 5. Mount and verify the static volume + +The external host can mount the path selected when the filesystem was created: + +```bash +mkdir -p /mnt/appdata +mount -t nfs -o vers=4.1 :/ /mnt/appdata +``` + +The two static-volume methods have the following behavior: + +| Item | Create the PV and PVC | Use CSI volume management | +|------|-----------------------|---------------------------| +| Kubernetes objects to create | PV and PVC | PVC only | +| `storageClassName` | Must be `""` on both objects | Existing StorageClass | +| PV name | Selected by the administrator | Generated as `pvc-` | +| Reclaim policy | Set directly on the PV; use `Retain` | Inherited from the StorageClass; change `Delete` to `Retain` immediately | +| `mountOptions` | Set directly on the PV | Inherited from the StorageClass | +| Validated PVC status | `Bound` | `Bound` | +| Validated NFS version | 4.2 without `mountOptions`; 4.1 when specified | 4.1 inherited from the StorageClass | +| External access | Fixed export path and bidirectional read/write | Fixed export path and bidirectional read/write | + +Creating the PV and PVC directly makes the reclaim policy and mount options explicit. CSI volume management requires fewer Kubernetes objects but requires an immediate reclaim-policy check. + +### 6. Mount a dynamically provisioned volume + +A volume created from a normal PVC can also be mounted outside the cluster. Its export path is generated and must be read from `.spec.csi.volumeAttributes.name`. The export path is not the PV name. The CSI driver changes the hyphens in the generated value to underscores. + +1. Read the export path from the PV: + + ```bash + kubectl get pv -o jsonpath='{.spec.csi.volumeAttributes.name}' + ``` + + ```text + pvc_0464141b_4e64_47b1_bc9b_a9f41c686bf6 + ``` + +2. Optionally confirm that the export is visible from the external host. This check relies on rpcbind and mountd; if only NFSv4 port 2049 is open, skip it and mount directly. + + ```bash + showmount -e + ``` + + ```text + /pvc_0464141b_4e64_47b1_bc9b_a9f41c686bf6 * + ``` + +3. Mount the export: + + ```bash + mkdir -p /mnt/appdata + mount -t nfs -o vers=4.1 \ + :/pvc_0464141b_4e64_47b1_bc9b_a9f41c686bf6 /mnt/appdata + ``` + +Files written by a Pod are immediately visible on the external host, and files written on the external host are visible in the Pod. + +The generated path changes if the volume is reprovisioned. Update the external-host configuration whenever this occurs. The volume also remains subject to the StorageClass reclaim policy. If an external host depends on it, set the PV to `Retain` before the PVC can be deleted: + +```bash +kubectl patch pv \ + -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +``` + +### 7. Configure permissions and exclude the snapshot directory + +The root directory of a new volume is `root:root` with mode `755`. Root is not squashed on these exports. As a result, root on an allowed external host can write to the volume, but a non-root user receives `Permission denied` unless its UID and GID have suitable permissions. + +To permit non-root access, align the UID/GID used by the Pod and external host, or set `fsPermission` in the StorageClass. The following value grants read, write, and execute permissions to all users and should be limited to environments where that access is acceptable: + +```yaml +parameters: + fsPermission: "777" +``` + +The array also creates a read-only `.snapshot` directory at the volume root. It cannot be deleted. Any backup or synchronization process that traverses the root must exclude it. When using `rsync --delete`, specify `--exclude='/.snapshot'`. The leading slash limits the exclusion to the volume-root directory and does not exclude identically named directories elsewhere in the data tree. + +### 8. Restrict NFS client access + +`authClient: "*"` allows every host that can reach the NFS portal to mount the share. Restrict this value to the network that requires access: + +```yaml +parameters: + authClient: "" +``` + +For a static volume, set the same restriction in the `NAME` field of the `NFS_SHARE_AUTH_CLIENT` rule instead of using `*`. + +Treat access to the NFS portal as data access because root is not squashed. On an array shared by multiple teams, confirm the correct storage pool and logical ports with the storage administrator. Use `reclaimPolicy: Retain` for every volume that has external consumers. + +## FAQ + +### Why does the PV name fail as an NFS export path? + +For a dynamically provisioned volume, the export path is stored in `.spec.csi.volumeAttributes.name`. It is not the PV name, and the generated path uses underscores where the corresponding generated identifier uses hyphens. Read the value from the PV before configuring the external host. + +### Why does an external non-root user receive `Permission denied`? + +The volume root is created as `root:root` with mode `755`. Root is not squashed, but other users do not have write permission by default. Align UID/GID values or configure an appropriate `fsPermission` value. + +### Why does `rsync --delete` fail on `.snapshot`? + +`.snapshot` is a read-only array-managed directory at the volume root and cannot be removed. Use `--exclude='/.snapshot'` so that `rsync --delete` does not attempt to delete it. diff --git a/docs/zh/solutions/How_to_Migrate_NFS_PersistentVolumes_to_OceanStor_Dorado_and_Rebind_PVCs_in_Place.md b/docs/zh/solutions/How_to_Migrate_NFS_PersistentVolumes_to_OceanStor_Dorado_and_Rebind_PVCs_in_Place.md new file mode 100644 index 000000000..79cd36b99 --- /dev/null +++ b/docs/zh/solutions/How_to_Migrate_NFS_PersistentVolumes_to_OceanStor_Dorado_and_Rebind_PVCs_in_Place.md @@ -0,0 +1,401 @@ +--- +products: + - Alauda Container Platform +kind: + - Solution +ProductsVersion: + - 4.x +--- + +# 如何将 NFS PersistentVolume 迁移到 OceanStor Dorado 并原地换绑 PVC + +## 概述 + +`PersistentVolumeClaim.spec.volumeName` 是不可变字段，不能通过更新现有 PVC 使其引用另一个 PersistentVolume。如果要在不改变 PVC 名称的情况下将工作负载迁移到新的 NFS 卷，必须删除并同名重建 PVC，同时将新 PV 预绑定到该 PVC。 + +本文介绍如何复制数据、将新 PV 预留给原 PVC 名称，以及恢复工作负载。整个过程需要一个停机窗口，已完成 Deployment 和 StatefulSet 工作负载验证。 + +对于 StatefulSet，由 `volumeClaimTemplates` 生成的 PVC 名称保持不变。迁移后还必须使用更新后的模板重建 StatefulSet，确保以后扩容时使用新的 StorageClass。 + +## 环境 + +| 组件 | 版本 | +|------|------| +| 容器平台 | ACP 4.x（在 4.3.1 上验证） | +| 节点操作系统 | Micro OS 5.5 | +| 存储设备 | OceanStor Dorado 6.1.9 | +| OceanStor CSI driver For Dorado | v4.12.0 | +| 源存储 | NFS，由 `nfs.csi.k8s.io` 制备 | +| 目标存储 | Dorado NFS（`volumeType: fs`） | +| 验证的工作负载 | Deployment（RWX）、StatefulSet（RWO，2 个 ordinal） | + +> **注意**：本文的数据复制操作适用于 `Filesystem` 模式的卷。`Block` 模式的卷需要使用块级复制方式。PVC 换绑操作只涉及 Kubernetes 对象，也可用于其他存储类型。 + +## 先决条件 + +- 一个 ACP 4.x 集群，已安装 OceanStor CSI driver For Dorado，并具备可用的目标 NFS StorageClass。 +- 具备 `kubectl` 访问权限，并有权 patch 属于集群级资源的 PersistentVolume。 +- 可以停止工作负载的维护窗口。 +- 目标阵列具有足够容量创建新卷。 +- 一个包含 `rsync` 的镜像，或其他能够保留所需文件元数据的复制工具。 +- 当前有效的备份，以及经过确认的工作负载回滚方案。 + +本文使用以下占位符。请替换为实际环境中的值： + +| 占位符 | 说明 | +|--------|------| +| `` | 工作负载所在命名空间 | +| `` | 迁移后保持不变的现有 PVC 名称 | +| `` | 用于制备目标卷的临时 PVC | +| `` | 由新阵列提供的目标 StorageClass | +| `` | 工作负载资源，例如 `deploy/app` 或 `sts/web` | +| `` | 工作负载 Pod 的标签选择器 | +| `` | 为临时目标 PVC 制备的 PV | + +## 解决方案 + +### 1. 确认 PVC 换绑要求 + +迁移包含两项操作： + +- 将数据从源卷复制到目标卷。 +- 将原 PVC 名称换绑到目标 PV。 + +由于 `PersistentVolumeClaim.spec.volumeName` 不能修改，必须删除并重建 PVC。工作负载清单仍可引用原 PVC 名称。 + +PV 通过 `PersistentVolume.spec.claimRef` 记录 PVC 绑定关系： + +| 字段 | 含义 | +|------|------| +| `namespace`、`name` | 卷所属 PVC 的命名空间和名称 | +| `uid` | 具体 PVC 对象的 UID；重建后的 PVC 使用新的 UID | +| `resourceVersion` | 被引用对象的乐观并发控制标识 | + +这些字段决定 PV 状态： + +| `claimRef` 条件 | PV 状态和行为 | +|------------------|---------------| +| 没有 `claimRef` | `Available`；任何匹配的 PVC 都可以绑定 | +| 存在 `namespace` 和 `name`，但没有 `uid` | `Available`；仅预留给指定名称的 PVC | +| `claimRef` 完整，并且引用的 PVC 存在 | `Bound` | +| `claimRef` 完整，但对应 `uid` 的 PVC 已不存在 | `Released` | + +`Released` 状态的 PV 不会自动恢复为 `Available`。因此，将新 PV 预留给原 PVC 时必须执行两次 merge patch。第一次设置目标 `namespace` 和 `name`。merge patch 会保留未指定的字段，因此旧 `uid` 仍然存在，PV 会停留在 `Released`。第二次必须将 `uid` 和 `resourceVersion` 显式设置为 `null`。 + +开始迁移前，请确认最终 PVC 与目标 PV 满足以下要求： + +- 请求容量不超过 PV 容量。 +- PV 支持 PVC 的访问模式。 +- `storageClassName` 与 PV 完全一致。 + +### 2. 制备目标卷 + +先读取源 PVC，确保临时 PVC 使用兼容的访问模式和容量： + +```bash +kubectl -n get pvc \ + -o custom-columns=\ +NAME:.metadata.name,MODES:.spec.accessModes,SIZE:.spec.resources.requests.storage +``` + +在目标 StorageClass 上创建临时 PVC： + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: + namespace: +spec: + # 设置为源 PVC 的访问模式（用上一条命令读取）。 + # ReadWriteMany 对应已验证的 Deployment；已验证的 StatefulSet 使用 + # ReadWriteOnce。不要无意中放宽访问模式。 + accessModes: ["ReadWriteMany"] + storageClassName: + resources: + requests: + # 必须大于等于源卷。后续换绑步骤会针对同一个卷创建最终 PVC， + # 因此容量在这里就固定下来了。 + storage: 2Gi +``` + +等待 PVC 完成绑定： + +```bash +kubectl -n wait --for=jsonpath='{.status.phase}'=Bound \ + pvc/ --timeout=180s +``` + +对于 StatefulSet，需要为每个 ordinal 创建一个目标 PVC。 + +### 3. 执行首次数据复制 + +创建一个同时挂载源 PVC 和目标 PVC 的临时 Pod： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: migrator + namespace: +spec: + restartPolicy: Never + containers: + - name: m + image: + command: ["sh", "-c", "sleep infinity"] + volumeMounts: + - {name: src, mountPath: /src} + - {name: dst, mountPath: /dst} + volumes: + - name: src + persistentVolumeClaim: {claimName: } + - name: dst + persistentVolumeClaim: {claimName: } +``` + +在应用仍运行时执行首次复制，将大部分数据复制工作放在停机窗口之前完成： + +```bash +kubectl -n exec migrator -- \ + rsync -aHAX --numeric-ids --delete --exclude='/.snapshot' /src/ /dst/ +``` + +建议使用 `rsync`，这样最终同步只需传输发生变化的数据。注意以下参数及其适用边界： + +- `-H` 保留硬链接，`-A` 保留 POSIX ACL，`-X` 保留扩展属性。 +- `--numeric-ids` 按数字值而非用户名/组名同步 UID/GID，从而在源端和目标端主机的名称到 ID 映射不同时仍保持属主正确。它**不能**绕过 NFS squash：`root_squash` 和 `all_squash` 是 NFS 服务端对客户端凭据的映射，与任何 `rsync` 参数无关。 +- `-aHAX` 不包含 `--sparse`。如果数据中包含稀疏文件（例如虚拟机镜像、预分配的数据库文件），请加上 `-S`，避免把空洞按实际数据写入，并据此规划目标卷容量。 +- `--exclude='/.snapshot'` 防止 `--delete` 尝试删除目标卷中只读的 `.snapshot` 目录。前导斜杠表示只匹配卷根目录。 + +如果不排除该目录，`rsync` 可能报告以下错误并以非零状态码退出： + +```text +rmdir: '/data/.snapshot': Permission denied +rm: can't remove '/data/.snapshot': Permission denied +``` + +由于复制通过 NFS 进行，复制进程必须能读取每个源文件，并能在目标端设置属主。请以 UID 0 运行 migrator，确认源端导出不对 root 做 squash，且目标端导出允许 `chown`；否则 `0600` 文件可能读不到，或属主无法恢复。全量复制前先用一个代表性文件测试。 + +如果没有包含 `rsync` 的镜像，可以使用以下命令在 Pod 内复制，它会保留权限、属主和符号链接，但每次都会复制全部数据： + +```bash +kubectl -n exec migrator -- sh -c 'tar -C /src -cf - . | tar -C /dst -xpf -' +``` + +普通 `tar` 如果不使用 `--xattrs --acls`，不会保留 POSIX ACL 和扩展属性，还会将稀疏文件展开为完整大小。请根据工作负载的文件元数据要求选择复制方式。 + +源 PVC 由 `nfs.csi.k8s.io` 制备，其 `CSIDriver.spec.attachRequired` 为 `false`。NFS 卷不通过 `VolumeAttachment` 挂接，因此 migrator Pod 与应用同时挂载源 PVC 不会触发 `Multi-Attach error`，跨节点也不会。migrator 可以运行在任意节点。对于 iSCSI、RBD 这类 `attachRequired` 为 `true` 的挂接型驱动，`ReadWriteOnce` 源卷才要求 migrator 与应用位于同一节点。无论哪种驱动，复制一个仍在写入的卷之前都要评估应用一致性。 + +由于源卷和目标卷都是 NFS，也可以在一台能够同时访问两个 NFS 服务器的主机上执行复制。此方式完全不经过集群内的 Pod，但要求集群外具备相应的网络访问能力。 + +### 4. 将两个 PV 的回收策略设置为 Retain + +设置后续操作使用的变量，并记录目标 PV 和源 PV 名称： + +```bash +NS=; PVC=; TMP=; WORKLOAD= +NEWPV=$(kubectl -n $NS get pvc $TMP -o jsonpath='{.spec.volumeName}') +OLDPV=$(kubectl -n $NS get pvc $PVC -o jsonpath='{.spec.volumeName}') +``` + +删除任一 PVC 前，必须将两个 PV 都改为 `Retain`： + +```bash +kubectl patch pv $NEWPV -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +kubectl patch pv $OLDPV -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +``` + +> **警告**：确认两个 PV 都使用 `Retain` 后才能继续。PV 仍使用 `reclaimPolicy: Delete` 时删除 PVC，可能会删除底层卷。新 PV 使用 `Retain` 是为了保护已迁移的数据，旧 PV 使用 `Retain` 是为了保留回滚路径。遗漏此检查点是整个流程中唯一可能造成不可逆数据丢失的情况。 + +检查两个 PV 的回收策略： + +```bash +kubectl get pv $NEWPV $OLDPV \ + -o custom-columns=NAME:.metadata.name,POLICY:.spec.persistentVolumeReclaimPolicy +``` + +### 5. 停止工作负载并完成最终同步 + +记录副本数，停止工作负载，并等待其 Pod 删除： + +```bash +REPLICAS=$(kubectl -n $NS get $WORKLOAD -o jsonpath='{.spec.replicas}') +kubectl -n $NS scale $WORKLOAD --replicas=0 +kubectl -n $NS wait --for=delete pod -l --timeout=300s +``` + +执行最终同步并比较文件校验和： + +```bash +kubectl -n $NS exec migrator -- \ + rsync -aHAX --numeric-ids --delete --exclude='/.snapshot' /src/ /dst/ + +kubectl -n $NS exec migrator -- sh -c ' + set -eu + for d in /src /dst; do + ( cd "$d" && find . -type f -not -path "./.snapshot/*" -print0 \ + | sort -z | xargs -0 -r md5sum ) > /tmp/$(basename "$d").sum + done + diff /tmp/src.sum /tmp/dst.sum && echo CONTENT_OK' +``` + +`set -eu` 与 NUL 分隔（`-print0` / `sort -z` / `xargs -0`）是必需的：否则含空格的文件名会被拆成多个参数，而 `md5sum` 失败只写到 stderr，于是两个空的 `.sum` 文件可能比较相等并输出 `CONTENT_OK`，即使数据不同。该检查之后紧接着就是删除 PVC，因此绝不能容许假阳性。 + +该检查只比对普通文件的内容。属主、权限、ACL、扩展属性、符号链接目标和硬链接关系不在此校验范围内，需按工作负载要求单独核对（见常见问题）。只有校验和一致时才能继续。 + +### 6. 删除 PVC 并使用原名称重建 + +删除 migrator Pod 和两个 PVC： + +```bash +kubectl -n $NS delete pod migrator +kubectl -n $NS delete pvc $TMP $PVC +``` + +两个 PV 都会进入 `Released` 状态，并保留数据。 + +将新 PV 预留给原 PVC 名称。必须按以下顺序执行两次 merge patch： + +```bash +kubectl patch pv $NEWPV --type merge -p \ + "{\"spec\":{\"claimRef\":{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"namespace\":\"$NS\",\"name\":\"$PVC\"}}}" + +kubectl patch pv $NEWPV --type merge -p \ + '{"spec":{"claimRef":{"uid":null,"resourceVersion":null}}}' +``` + +第二次 merge patch 不能省略。否则 `claimRef` 中仍保留旧 `uid`，PV 会一直停留在 `Released`。 + +确认新 PV 已变为 `Available`，并预留给原 PVC 名称： + +```bash +kubectl get pv $NEWPV -o custom-columns=\ +NAME:.metadata.name,STATUS:.status.phase,CLAIM:.spec.claimRef.name,UID:.spec.claimRef.uid +``` + +```text +NAME STATUS CLAIM UID +pvc-0464141b-… Available app-data +``` + +使用原名称重建 PVC，并将其预绑定到新 PV： + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: # 保持不变 + namespace: +spec: + accessModes: ["ReadWriteMany"] # 与源 PVC 一致（StatefulSet 为 ReadWriteOnce） + storageClassName: # 必须与新 PV 完全一致 + resources: + requests: + storage: 2Gi # 必须小于等于 PV 容量 + volumeName: +``` + +PVC 变为 `Bound` 后才能恢复工作负载。 + +### 7. 恢复并验证工作负载 + +恢复之前记录的副本数： + +```bash +kubectl -n $NS scale $WORKLOAD --replicas=$REPLICAS +``` + +确认工作负载使用目标 NFS 挂载点，并检查应用数据： + +```bash +kubectl -n $NS exec -- sh -c 'mount | grep " /data "' +``` + +在经过验证的 `ReadWriteMany` Deployment 场景中，以下检查均通过： + +| 检查项 | 结果 | +|--------|------| +| 文件校验和 | 与迁移前基线一致 | +| 权限和属主 | 保持不变，包括权限为 `600`、属主为 `1000:1000` 的文件 | +| 符号链接 | 保持不变，包括相对路径目标 | +| Deployment 清单 | 未修改，仍引用原 PVC 名称 | +| 挂载点 | 已切换到目标 NFS 导出 | +| 并发写入 | 两个副本和一台集群外主机可以互相看到写入内容 | + +在经过验证的 `ReadWriteOnce`、两个 ordinal 的 StatefulSet 场景中，每个 ordinal 都保留了各自的数据和 PVC 名称，两个挂载点均切换到目标阵列，并且没有发生串卷。 + +### 8. 使用目标 StorageClass 重建 StatefulSet + +需要为每个 StatefulSet ordinal 分别执行迁移，例如 `data-web-0` 和 `data-web-1`。每个 ordinal 都有独立的源卷和目标卷。 + +换绑 PVC 不会更新 StatefulSet 的 `volumeClaimTemplates`。如果模板仍使用旧 StorageClass，以后扩容时会在旧存储上创建新 PVC，导致同一个 StatefulSet 使用不同的存储系统： + +```text +data-web-0 Bound <- 已迁移 +data-web-1 Bound <- 已迁移 +data-web-2 Bound <- 新建，来自模板 +``` + +`volumeClaimTemplates` 不能原地修改： + +```text +The StatefulSet "web" is invalid: spec: Forbidden: updates to statefulset spec for +fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', +'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' +are forbidden +``` + +使用 `--cascade=orphan` 删除 StatefulSet，修改 `volumeClaimTemplates` 中的 `storageClassName`，然后重建： + +```bash +kubectl -n delete sts --cascade=orphan +# 修改 volumeClaimTemplates 中的 storageClassName，然后： +kubectl apply -f .yaml +``` + +使用 `--cascade=orphan` 删除 StatefulSet 时，原有 Pod 和 PVC 会保留。重建后的 StatefulSet 通过标签选择器收养现有 Pod。验证过程中，Pod 存活时长保持连续，重启次数始终为 0，说明 Pod 没有重启。请在迁移完成后立即执行此步骤，确保以后扩容时使用目标 StorageClass。 + +### 9. 检查风险和回滚操作 + +| 步骤 | 出错后果 | 回滚方式 | +|------|----------|----------| +| 首次数据复制 | 应用不受影响，源卷仍可用 | 删除临时 PVC 后重试 | +| 将两个 PV 设置为 `Retain` | 遗漏此保护会在删除 PVC 时销毁卷 | 删除后无法恢复；这是唯一不可逆的检查点 | +| 停止工作负载 | 开始停机 | 恢复之前记录的副本数 | +| 最终同步和校验 | 校验和不一致表示复制不完整 | 恢复工作负载；源卷未发生变化 | +| 删除 PVC | 两个 PV 变为 `Released`；由于使用 `Retain`，数据仍然保留 | 重建原 PVC，并绑定到旧 PV | +| 预留新 PV | 无法完成绑定，但不会删除数据 | 重新执行两次 merge patch | +| 重建 PVC | 字段不匹配时 PVC 停留在 `Pending` | 删除 PVC，修正后重建 | +| 恢复工作负载 | 卷无法正确挂载 | 缩容到零，将原 PVC 名称换绑回旧 PV | +| 重建 StatefulSet | 标签选择器不匹配时，控制器无法收养现有 Pod | 修正清单并重建 StatefulSet；孤儿 Pod 会继续运行 | + +确认两个 PV 都使用 `Retain` 后，后续迁移步骤均可回退。旧 PV 是回滚路径。需要恢复时，使用相同的两次 patch 操作将旧 PV 预留给原 PVC 名称，再将 PVC 的 `volumeName` 设置为旧 PV 并重建。 + +删除任一 PVC 前，请执行以下检查，并确认两行都显示 `Retain`： + +```bash +kubectl get pv $NEWPV $OLDPV \ + -o custom-columns=NAME:.metadata.name,POLICY:.spec.persistentVolumeReclaimPolicy +``` + +### 10. 迁移成功后的收尾 + +两个 PV 都保留为 `Retain`，清理决策交由管理员完成。当工作负载在目标卷上运行足够长、回滚窗口关闭之后： + +- 旧 PV 仍处于 `Released`。在确认已有备份、不再需要回滚路径后，删除旧 PV 并在源系统上回收其底层卷。删除 PV 对象并不会释放源端存储。 +- 决定目标 PV 的最终回收策略。它保持 `Retain`，因此后续删除 PVC 不会删除阵列上的卷。对于共享或对外提供的卷,保留 `Retain`，此时卷需要手工清理。只有当卷的生命周期应跟随 PVC 时，才恢复为 `Delete`。 + +## 常见问题 + +### 重建 PVC 时必须匹配哪些字段？ + +PVC 请求容量不得超过 PV 容量。PVC 的访问模式必须是 PV 所支持模式的子集。`storageClassName` 必须与 PV 完全一致。对于静态 PV，PV 和 PVC 都应设置 `storageClassName: ""`；如果省略 PVC 中的该字段，Kubernetes 可能会填入默认 StorageClass。 + +### 为什么第一次 patch 后 PV 仍处于 Released？ + +第一次 merge patch 修改了 `claimRef.namespace` 和 `claimRef.name`，但未指定的旧 `uid` 会被保留。第二次 merge patch 必须将 `uid` 和 `resourceVersion` 都设置为 `null`。完成后，PV 可以恢复为 `Available`，同时仍预留给指定名称的 PVC。 + +### 迁移后应如何检查文件属主？ + +源端和目标端 NFS 服务器可能使用不同的 squash 策略，而该映射发生在服务端,不在 `rsync` 中。请以 UID 0 运行复制,针对不对 root 做 squash 的源端导出和允许 `chown` 的目标端导出;使用 `--numeric-ids` 让 ID 按数值而非名称匹配,并在复制后检查代表性文件。根据应用要求确认 UID/GID、权限、ACL、扩展属性、硬链接、符号链接以及稀疏文件处理结果。 diff --git a/docs/zh/solutions/How_to_Share_OceanStor_Dorado_NFS_Volumes_with_Hosts_Outside_the_Cluster.md b/docs/zh/solutions/How_to_Share_OceanStor_Dorado_NFS_Volumes_with_Hosts_Outside_the_Cluster.md new file mode 100644 index 000000000..d45befcac --- /dev/null +++ b/docs/zh/solutions/How_to_Share_OceanStor_Dorado_NFS_Volumes_with_Hosts_Outside_the_Cluster.md @@ -0,0 +1,294 @@ +--- +products: + - Alauda Container Platform +kind: + - Solution +ProductsVersion: + - 4.x +--- + +# 如何让集群外主机共享 OceanStor Dorado 的 NFS 卷 + +## 概述 + +OceanStor CSI driver For Dorado 通过 NFS 制备的 PersistentVolume，在存储阵列上对应一个文件系统和一个 NFS 共享。集群外主机只要能够访问 NFS 门户，并且符合访问客户端规则，就可以直接挂载该共享。 + +本文介绍以下两种接入方式： + +- 先在阵列上创建文件系统和 NFS 共享，再将已有卷作为静态卷接入 Kubernetes。这种方式可以使用固定的导出路径。 +- 使用动态制备的卷，从 PersistentVolume 中获取系统生成的导出路径。 + +如果外部系统需要固定的挂载路径，请使用静态卷。如果卷的生命周期由 Kubernetes 管理，并且只需在制备后提供外部访问，可以使用动态制备的卷。 + +## 环境 + +| 组件 | 版本 | +|------|------| +| 容器平台 | ACP 4.x（在 4.3.1 上验证） | +| 节点操作系统 | Micro OS 5.5 | +| 存储设备 | OceanStor Dorado 6.1.9 | +| OceanStor CSI driver For Dorado | v4.12.0 | +| 集群外主机 | CentOS 7，已安装 `nfs-utils` | +| 验证的协议 | NFS（v4.1 / v4.2） | + +> **注意**：本文操作已使用上表中的版本完成验证。用于其他版本组合前，请确认 CSI 驱动与存储固件兼容。 + +## 先决条件 + +- 一个 ACP 4.x 集群，已安装 OceanStor CSI driver For Dorado 并配置 NFS 后端。参见《如何在 ACP 上安装和配置 OceanStor CSI driver For Dorado》。 +- 一个配置了 `volumeType: fs` 的 StorageClass，并且 `authClient` 允许集群外主机访问。 +- 集群外主机与 NFS 数据平面门户之间具有三层网络连接。NFSv4.1/4.2 挂载只需 TCP 2049 端口。本文中的 `showmount` 检查还需要 rpcbind（111 端口）和 mountd 端口；若这些端口未放通，卷仍可挂载，但 `showmount` 会失败。 +- 集群外主机已安装 `nfs-utils` 或等效的 NFS 客户端软件包。 +- 使用静态卷时，具备存储 REST API 或 DeviceManager 界面的管理权限，并已从存储管理员处获取存储池信息。 + +本文使用以下占位符。请替换为实际环境中的值： + +| 占位符 | 说明 | +|--------|------| +| `` | NFS 数据平面门户地址 | +| `` | 存储管理平面地址 | +| `` | 挂载卷的集群外主机 | +| `` | CSI 存储后端名称，例如 `backend-nfs` | +| `` | NFS StorageClass 名称 | +| `` | 存储池数字 ID，不是存储池名称 | +| `` | 为静态卷选择的文件系统名称，例如 `acp_static_nfs` | +| `` | 创建 REST 会话时返回的存储设备 ID | +| `` | 创建文件系统时返回的文件系统 ID | +| `` | 创建 NFS 共享时返回的共享 ID | +| `` | 允许挂载共享的网段 | +| `` | PVC 所在命名空间 | + +## 解决方案 + +### 1. 选择卷制备方式 + +每个 NFS 卷在阵列上都包含一个文件系统、一个 NFS 共享以及一条或多条访问客户端规则。StorageClass 的 `authClient` 参数用于设置访问客户端规则。例如，`authClient: "*"` 允许所有能够访问 NFS 门户的客户端挂载该共享。 + +集群外主机挂载卷不需要额外的 CSI 配置。两种制备方式的导出路径和生命周期不同： + +| 项目 | 静态卷 | 动态制备卷 | +|------|--------|------------| +| 导出路径 | 使用创建卷时指定的文件系统名称 | 使用 PV 中记录的系统生成名称 | +| 集群外主机配置 | 可以在创建 Kubernetes 对象前完成 | 制备后必须使用生成的路径更新配置 | +| 生命周期 | 在动态制备流程之外单独创建和管理 | 通常由 PVC 和 StorageClass 的回收策略管理 | + +### 2. 在阵列上创建静态卷 + +按顺序创建文件系统、NFS 共享和访问客户端规则。下面展示的是存储 REST API 的请求体结构，不是可直接执行的脚本。每个请求都发往 `https:///deviceManager/rest//...`，且需要已认证的会话：先创建会话（`POST /deviceManager/rest/xxxxx/sessions`）以获得 `` 和 `iBaseToken`，再在后续每个请求的 `iBaseToken` 请求头中带上该令牌。也可以在 DeviceManager 界面中创建相同的对象。 + +```text +# 1. 文件系统。CAPACITY 的单位是 512 字节扇区，所以 4194304 = 2 GiB。 +# PARENTID 是存储池的数字 ID，不是存储池名称。 +# POST /deviceManager/rest//filesystem +{ + "NAME": "", + "PARENTID": "", + "CAPACITY": 4194304, + "ALLOCTYPE": 1, + "SECTORSIZE": 16384 +} + +# 2. NFS 共享。 取自请求 1 的响应。 +# POST /deviceManager/rest//NFSHARE +{ "SHAREPATH": "//", "FSID": "", "vstoreId": "0" } + +# 3. 访问客户端。 取自请求 2 的响应。 +# 这些取值与 CSI 驱动程序在动态制备卷上设置的一致， +# 以保证行为不产生差异。 +# POST /deviceManager/rest//NFS_SHARE_AUTH_CLIENT +{ + "NAME": "*", "PARENTID": "", + "ACCESSVAL": 1, "SYNC": 0, "ALLSQUASH": 1, + "ROOTSQUASH": 1, "SECURE": 1, "vstoreId": "0" +} +``` + +`CAPACITY` 的单位是 512 字节扇区。`PARENTID` 必须填写存储池的数字 ID，不能填写存储池名称。`` 取自文件系统创建响应，`` 取自 NFS 共享创建响应。创建完成后，导出路径为 `:/`。 + +静态卷可以通过自行创建 PV，或使用 CSI 卷纳管功能接入 Kubernetes。请选择以下一种方式。 + +### 3. 通过创建 PV 和 PVC 接入静态卷 + +创建 PV，并将 PVC 预绑定到该 PV： + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: pv-static-nfs +spec: + # 空字符串：该 PV 不参与任何动态制备 + storageClassName: "" + volumeMode: Filesystem + accessModes: ["ReadWriteMany"] + capacity: + storage: 2Gi + # 管理员预建的卷绝不能被 Kubernetes 删除 + persistentVolumeReclaimPolicy: Retain + # 不会从 StorageClass 继承，因此需在此指定，否则 NFS 版本由协商决定 + mountOptions: + - nfsvers=4.1 + csi: + driver: csi.huawei.com + volumeHandle: . # 格式：. + volumeAttributes: + backend: + name: +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: static-a + namespace: +spec: + # 同样必须是空字符串，否则会被替换成默认 StorageClass，导致绑定失败 + storageClassName: "" + accessModes: ["ReadWriteMany"] + resources: + requests: + storage: 2Gi + volumeName: pv-static-nfs +``` + +PV 和 PVC 都必须设置 `storageClassName: ""`。如果 PVC 省略该字段，Kubernetes 可能会填入默认 StorageClass，导致 PVC 无法与此 PV 绑定。 + +静态 PV 不会从 StorageClass 继承 `mountOptions`。在本文验证环境中，不设置 `mountOptions` 时，NFS 协商结果为 4.2；设置 `nfsvers=4.1` 后，挂载使用 NFS 4.1。 + +### 4. 通过 CSI 卷纳管功能接入静态卷 + +如果不自行创建 PV，可以在 PVC 上添加卷纳管注解和必需的标签。CSI 驱动会导入已有卷并生成 PV： + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: static-b + namespace: + annotations: + csi.huawei.com/manageVolumeName: # 阵列上的卷名 + csi.huawei.com/manageBackendName: + labels: + provisioner: csi.huawei.com # 必需 +spec: + accessModes: ["ReadWriteMany"] + storageClassName: # 使用真实 StorageClass，不是 "" + resources: + requests: + storage: 2Gi # 必须与阵列上卷的实际大小一致 +``` + +PVC 必须同时包含 `csi.huawei.com/manageVolumeName`、`csi.huawei.com/manageBackendName` 和 `provisioner: csi.huawei.com` 标签。此方式的 `storageClassName` 应填写已有 StorageClass，不能使用空字符串。 + +系统生成的 PV 名称为 `pvc-`，其 `volumeHandle` 指向 `.`，导出路径仍使用阵列上指定的文件系统名称。 + +> **重要**：系统生成的 PV 会继承 StorageClass 的 `reclaimPolicy`。如果 StorageClass 使用 `Delete`，删除 PVC 时也会删除手工创建的文件系统。PVC 变为 `Bound` 后，请立即将生成的 PV 改为 `Retain`： +> +> ```bash +> kubectl patch pv \ +> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +> ``` + +### 5. 挂载并验证静态卷 + +集群外主机可以直接挂载创建文件系统时指定的路径： + +```bash +mkdir -p /mnt/appdata +mount -t nfs -o vers=4.1 :/ /mnt/appdata +``` + +两种静态卷接入方式的行为如下： + +| 项目 | 创建 PV 和 PVC | 使用 CSI 卷纳管功能 | +|------|----------------|----------------------| +| 需要创建的 Kubernetes 对象 | PV 和 PVC | 仅 PVC | +| `storageClassName` | 两个对象都必须为 `""` | 已有 StorageClass | +| PV 名称 | 由管理员指定 | 生成为 `pvc-` | +| 回收策略 | 直接在 PV 上设置，应使用 `Retain` | 继承 StorageClass，若为 `Delete` 必须立即改为 `Retain` | +| `mountOptions` | 直接在 PV 上设置 | 继承 StorageClass | +| 验证的 PVC 状态 | `Bound` | `Bound` | +| 验证的 NFS 版本 | 未设置 `mountOptions` 时为 4.2；设置后为 4.1 | 从 StorageClass 继承 4.1 | +| 集群外访问 | 使用固定导出路径，可双向读写 | 使用固定导出路径，可双向读写 | + +自行创建 PV 和 PVC 可以明确指定回收策略和挂载参数。使用 CSI 卷纳管功能需要创建的 Kubernetes 对象更少，但必须立即检查回收策略。 + +### 6. 挂载动态制备的卷 + +通过普通 PVC 动态制备的卷也可以在集群外主机上挂载。导出路径由系统生成，必须从 `.spec.csi.volumeAttributes.name` 读取。导出路径不是 PV 名称，CSI 驱动会将生成值中的连字符替换为下划线。 + +1. 从 PV 读取导出路径： + + ```bash + kubectl get pv -o jsonpath='{.spec.csi.volumeAttributes.name}' + ``` + + ```text + pvc_0464141b_4e64_47b1_bc9b_a9f41c686bf6 + ``` + +2. 可选：在集群外主机上确认可以看到该导出。此检查依赖 rpcbind 和 mountd；若只放通了 NFSv4 的 2049 端口，可跳过此步直接挂载。 + + ```bash + showmount -e + ``` + + ```text + /pvc_0464141b_4e64_47b1_bc9b_a9f41c686bf6 * + ``` + +3. 挂载导出： + + ```bash + mkdir -p /mnt/appdata + mount -t nfs -o vers=4.1 \ + :/pvc_0464141b_4e64_47b1_bc9b_a9f41c686bf6 /mnt/appdata + ``` + +Pod 写入的文件会立即显示在集群外主机上，集群外主机写入的文件也会显示在 Pod 中。 + +重新制备卷后，系统生成的路径会发生变化，此时需要更新集群外主机配置。该卷仍受 StorageClass 回收策略控制。如果集群外主机依赖此卷，请在 PVC 可能被删除前将 PV 设置为 `Retain`： + +```bash +kubectl patch pv \ + -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +``` + +### 7. 配置目录权限并排除快照目录 + +新卷的根目录属主和属组为 `root:root`，权限为 `755`。这些 NFS 导出不会 squash root。因此，允许访问的集群外主机上的 root 用户可以写入，但非 root 用户在 UID 和 GID 没有相应权限时会收到 `Permission denied`。 + +如需允许非 root 用户访问，可以统一 Pod 与集群外主机使用的 UID/GID，或在 StorageClass 中设置 `fsPermission`。以下配置向所有用户授予读、写和执行权限，只能用于允许此访问范围的环境： + +```yaml +parameters: + fsPermission: "777" +``` + +阵列还会在卷根目录创建只读的 `.snapshot` 目录，该目录无法删除。备份或同步程序遍历卷根目录时必须排除它。使用 `rsync --delete` 时，应指定 `--exclude='/.snapshot'`。前导斜杠表示只排除卷根目录下的 `.snapshot`，不会排除数据目录中其他同名目录。 + +### 8. 限制 NFS 客户端访问范围 + +`authClient: "*"` 允许所有能够访问 NFS 门户的主机挂载共享。请将其限制为实际需要访问的网段： + +```yaml +parameters: + authClient: "" +``` + +对于静态卷，请在 `NFS_SHARE_AUTH_CLIENT` 规则的 `NAME` 字段中设置相同的限制，不要使用 `*`。 + +由于 root 不会被 squash，应将 NFS 门户的网络访问权限视为数据访问权限。在多团队共用的阵列上，请与存储管理员确认正确的存储池和逻辑端口。所有存在集群外使用者的卷都应使用 `reclaimPolicy: Retain`。 + +## 常见问题 + +### 为什么不能使用 PV 名称作为 NFS 导出路径？ + +对于动态制备的卷，导出路径保存在 `.spec.csi.volumeAttributes.name` 中。该值不是 PV 名称，并且系统生成路径使用下划线替代对应标识符中的连字符。配置集群外主机前，请从 PV 中读取此值。 + +### 为什么集群外的非 root 用户会收到 `Permission denied`？ + +卷根目录创建为 `root:root`，权限为 `755`。root 不会被 squash，但其他用户默认没有写权限。请统一 UID/GID，或设置合适的 `fsPermission`。 + +### 为什么 `rsync --delete` 会在 `.snapshot` 目录上失败？ + +`.snapshot` 是阵列在卷根目录中维护的只读目录，无法删除。请使用 `--exclude='/.snapshot'`，避免 `rsync --delete` 尝试删除该目录。