Issue with Migrating CephFS Volumes Between Clusters

[arezvani]

We are in the process of migrating our Kubernetes workloads from an existing Ceph cluster ("origin") to a new Ceph cluster ("remote"). Our goal is to transition the CephFS-based Persistent Volumes (PVs) with minimal changes, primarily by updating the CSI configuration in Kubernetes.

Migration Approach:

• Utilize a Ceph client that has access to both the origin and remote clusters.

• Create a CephFS volume in the origin cluster using the CSI driver connected to the origin monitors.

• Terminate the associated pod to unmount the volume.

• Employ rsync with the -avpPX options to copy the volume data from the origin to the remote cluster.

• Update the CSI configuration in Kubernetes to point to the remote cluster's monitors.

• Deploy the pod, expecting it to mount the volume from the remote cluster

After performing the above steps, when the pod is deployed with the updated CSI configuration pointing to the remote cluster, it fails to start. The error observed is related to extended attributes (xattr), specifically a getxattr error.

This suggests that the metadata associated with the files, such as inodes and other extended attributes, are not being preserved during the rsync operation.

Actions Taken:

• CephFS Snapshot Mirroring: We considered using CephFS snapshot mirroring to replicate the data between clusters. However, this approach requires the fsid of the origin and remote clusters to be different. In our case, due to constraints with RBD mirroring, we have set the same fsid for both clusters, making this approach unfeasible.

• RADOS Object Export/Import: We attempted to export and import the RADOS objects at the pool level for both data and metadata pools. While the objects appear to be present in the remote cluster, the CephFS filesystem on the remote cluster does not recognize them, indicating a mismatch or missing metadata.

Request for Guidance:

Given the constraints:

• The need to maintain the same fsid across clusters due to RBD mirroring requirements.

• The desire to minimize changes in Kubernetes, ideally only updating the CSI configuration.

• The necessity to preserve file metadata and extended attributes during migration.

I also get this error:

rpc error: code = Internal desc = rpc error: code = Internal desc = rados: ret=-61, No data available: "error in getxattr"

unable to read omap keys: pool or key missiong

What would be the recommended approach to migrate CephFS volumes between clusters under these conditions? Are there tools or methods that can facilitate this migration while ensuring data integrity and minimal disruption?

Any insights or suggestions would be greatly appreciated.

[nixpanic]

The easiest way is to use a tool like VolSync. Create a new volume with a StorageClass that references the new Ceph cluster, and have VolSync copy the contents for you.

It is not only required to copy the data of the volumes themselves, Ceph-CSI stores some metadata in RADOS directly (the 'journal') too. Without the contents of the journal, Ceph-CSI will not be able to manage the volumes anymore.

[arezvani]

Hi,
Thanks @nixpanic

Ok how can I migrate metadata in RADOS (journal) too? What should I do to not change storageClass and not create new storageClass?
I mean I want to just change mons in ceph csi config and can work with previous storageClass but it works with new ceph cluster.
I also try to have same things that we can set like fsid and cluster id and ... in new ceph cluster be like old cluster.

[Ganesh-403]

@arezvani Based on the maintainer's reply, I don't think migrating only the CephFS data is sufficient.

ceph-csi stores its own metadata (the CSI journal) in RADOS. This metadata tracks the relationship between Kubernetes resources (PVC/PV/VolumeID) and the underlying CephFS subvolumes. If the journal isn't present in the target cluster, ceph-csi won't be able to locate or manage the existing volumes correctly, which matches errors such as:

error in getxattr
unable to read omap keys: pool or key missing

The omap error is a strong indication that ceph-csi is looking for metadata that doesn't exist in the destination cluster. :contentReference[oaicite:0]{index=0}

Regarding your follow-up questions:

How can I migrate the RADOS journal too?

From what I can tell, ceph-csi doesn't currently provide a supported tool to export/import or migrate its journal metadata between independent clusters. The maintainer's recommendation to use VolSync avoids this problem by provisioning a new CSI-managed volume on the destination cluster and copying only the file contents, allowing ceph-csi to create a fresh journal automatically. :contentReference[oaicite:1]{index=1}

Can I keep the same StorageClass and only change the monitors?

I don't think that's sufficient. Even if you keep the same:

• fsid
• cluster ID
• StorageClass
• monitor addresses

the destination cluster still won't contain the ceph-csi journal entries unless they are recreated or migrated. Those metadata objects are separate from the CephFS file data and metadata managed by the MDS. Simply matching cluster identifiers doesn't recreate the CSI metadata. :contentReference[oaicite:2]{index=2}

So unless you're performing a full cluster replication that preserves both the CephFS metadata and the ceph-csi journal, I'd expect ceph-csi to continue failing to resolve the existing VolumeIDs.

Given your requirement of keeping the same PVCs and StorageClass, it may be worth opening a feature request asking whether ceph-csi could support journal export/import or a metadata migration utility. As far as I could find, no such migration tool is currently documented.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.