nfs: implement ControllerModifyVolume to change NFS-server hostname

PendingReleaseNotes.md

@@ -4,4 +4,6 @@
 
 ## Features
 
+- nfs: allow changing NFS-server through ControllerModifyVolume [PR](https://github.com/ceph/ceph-csi/pull/5829)
+
 ## NOTE

README.md

@@ -115,6 +115,7 @@ for its support details.
 |        | Creating and deleting snapshot                            | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.17.0         |
 |        | Provision volume from snapshot                            | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.17.0         |
 |        | Provision volume from another volume                      | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.16.0         |
+|        | Modify volume parameters with ControllerModifyVolume      | Alpha          | >= v3.17.0         | >= v1.12.0       | Pacific (>=v16.2.0)  | >= v1.34.0         |
 
 `NOTE`: The `Alpha` status reflects possible non-backward
 compatible changes in the future, and is thus not recommended

build.env

@@ -61,10 +61,11 @@ ROOK_VERSION=v1.16.4
 ROOK_CEPH_CLUSTER_IMAGE=quay.io/ceph/ceph:v19.2.2
 
 # CSI sidecar version
+K8S_IMAGE_REPO=gcr.io/k8s-staging-sig-storage
 CSI_ATTACHER_VERSION=v4.10.0
 CSI_SNAPSHOTTER_VERSION=v8.4.0
-CSI_RESIZER_VERSION=v2.0.0
-CSI_PROVISIONER_VERSION=v6.0.0
+CSI_RESIZER_VERSION=canary
+CSI_PROVISIONER_VERSION=canary
 CSI_NODE_DRIVER_REGISTRAR_VERSION=v2.15.0
 
 # e2e settings

deploy/nfs/kubernetes/csi-nfsplugin-provisioner.yaml

@@ -73,7 +73,7 @@ spec:
             - name: keys-tmp-dir
               mountPath: /tmp/csi/keys
         - name: csi-provisioner
-          image: registry.k8s.io/sig-storage/csi-provisioner:v6.0.0
+          image: gcr.io/k8s-staging-sig-storage/csi-provisioner:canary
           args:
             - "--csi-address=$(ADDRESS)"
             - "--v=1"
@@ -99,7 +99,7 @@ spec:
             - name: socket-dir
               mountPath: /csi
         - name: csi-resizer
-          image: registry.k8s.io/sig-storage/csi-resizer:v2.0.0
+          image: gcr.io/k8s-staging-sig-storage/csi-resizer:canary
           args:
             - "--csi-address=$(ADDRESS)"
             - "--v=1"

deploy/nfs/kubernetes/csi-nfsplugin.yaml

@@ -73,6 +73,8 @@ spec:
               mountPath: /etc/ceph/
             - name: ceph-csi-config
               mountPath: /etc/ceph-csi-config/
+            - name: keys-tmp-dir
+              mountPath: /tmp/csi/keys
         - name: driver-registrar
           # This is necessary only for systems with SELinux, where
           # non-privileged sidecar containers cannot access unix domain socket
@@ -133,3 +135,7 @@ spec:
         - name: ceph-csi-config
           configMap:
             name: ceph-csi-config
+        - name: keys-tmp-dir
+          emptyDir: {
+            medium: "Memory"
+          }

deploy/nfs/kubernetes/csi-provisioner-rbac.yaml

@@ -60,6 +60,9 @@ rules:
   - apiGroups: ["snapshot.storage.k8s.io"]
     resources: ["volumesnapshots"]
     verbs: ["get", "list"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["volumeattributesclasses"]
+    verbs: ["get", "list", "watch"]
 
 ---
 kind: ClusterRoleBinding

docs/volumeattributesclass.md

@@ -0,0 +1,66 @@
+# VolumeAttributesClass for Volume Modification
+
+Kubernetes offers a method for modifying Volume parameters after they have been
+created. This is done through VolumeAttributesClasses and is described in a
+[blog
+post](https://kubernetes.io/blog/2025/09/08/kubernetes-v1-34-volume-attributes-class).
+
+## Prerequisites
+
+- Kubernetes 1.34 is the first release where support for
+  VolumeAttributesClasses (the `ControllerModifyVolume` CSI procedure) is GA.
+  Older versions of Kubernetes may not work reliably.
+- The Kubernetes CSI external-provisioner (`csi-provisioner` sidecar) release
+  needs to be higher or equal to v6.1 (support for secrets).
+- The Kubernetes CSI external-resizer (`csi-resizer` sidecar) release
+  needs to be higher or equal to v2.1 (support for secrets).
+
+## Secret references in the StorageClass
+
+When setting a VolumeAttributesClass on a PersistentVolumeClaim, the
+`ControllerModifyVolume` CSI procedure is called on the provisioner. This
+procedure needs secrets (that contain the Ceph credentials) in order to
+communicate with the Ceph cluster. Below are the keys that should be set in the
+StorageClass' `parameters`:
+
+- `csi.storage.k8s.io/controller-modify-secret-name`
+- `csi.storage.k8s.io/controller-modify-secret-namespace`
+
+In addition to the execution of `ControllerModifyVolume`, the Node-plugin needs
+access to the Volume on the Ceph cluster to fetch the updated parameters.
+Usually NFS and NVMe-oF do not need any Ceph credentials for the Node-plugin,
+but for using VolumeAttributesClasses to modify Volumes this is a requirement.
+
+Depending on the storage backend, Volumes are optionally _Staged_ before
+getting _Published_. NVMe-oF uses the staging process, and needs credentials
+there, hence these parameters should be set:
+
+- `csi.storage.k8s.io/node-stage-secret-name`
+- `csi.storage.k8s.io/node-stage-secret-namespace`
+
+NFS does not use the staging process, and only needs the credentials during the
+publishing process:
+
+- `csi.storage.k8s.io/node-publish-secret-name`
+- `csi.storage.k8s.io/node-publish-secret-namespace`
+
+## Adding support to existing Volumes
+
+PersistentVolumeClaims that have not been created with the right secret
+references in the StorageClass will not be modifiable with a
+VolumeAttributesClass without manual intervention.
+
+In order to be able to modify parameters with a VolumeAttributesClass,
+annotations should be added to the PersistentVolume that is _Bound_ to the
+PersistentVolumeClaim. These annotations should refer to the namespace and
+secret where the credentials are available (just like the namespace and secret
+that would be referenced in the StorageClass).
+
+- `volume.kubernetes.io/controller-modify-secret-name`
+- `volume.kubernetes.io/controller-modify-secret-namespace`
+
+Note that these annotations only make it possible for the provisioner to modify
+parameters. This does not allow the Node-plugin accessing the Ceph cluster to
+fetch updated parameters when the Volume is _staged_ or _published_. The
+secrets for staging and publishing can not (easily) be updated after the fact,
+these are part of the fixed parameters in the PersistentVolume.

e2e/go.mod

@@ -18,7 +18,7 @@ require (
 	// when updating k8s.io modules, update the 'replace' section below too
 	k8s.io/api v0.35.0
 	k8s.io/apimachinery v0.35.0
-	k8s.io/client-go v12.0.0+incompatible
+	k8s.io/client-go v0.35.0
 	k8s.io/cloud-provider v0.35.0
 	k8s.io/kubernetes v1.35.0
 	k8s.io/pod-security-admission v0.35.0
@@ -36,6 +36,8 @@ replace (
 	k8s.io/kubelet => k8s.io/kubelet v0.35.0
 )
 
+exclude k8s.io/client-go v12.0.0+incompatible
+
 require (
 	cel.dev/expr v0.24.0 // indirect
 	cyphar.com/go-pathrs v0.2.1 // indirect
@@ -123,7 +125,7 @@ require (
 	k8s.io/klog/v2 v2.130.1 // indirect
 	k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 // indirect
 	k8s.io/kubectl v0.0.0 // indirect
-	k8s.io/kubelet v0.33.2 // indirect
+	k8s.io/kubelet v0.0.0 // indirect
 	k8s.io/mount-utils v0.35.0 // indirect
 	k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 // indirect
 	sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 // indirect

e2e/nfs.go

@@ -178,8 +178,8 @@ func createNFSStorageClass(
 	sc.Parameters["csi.storage.k8s.io/controller-expand-secret-namespace"] = cephCSINamespace
 	sc.Parameters["csi.storage.k8s.io/controller-expand-secret-name"] = cephFSProvisionerSecretName
 
-	sc.Parameters["csi.storage.k8s.io/node-stage-secret-namespace"] = cephCSINamespace
-	sc.Parameters["csi.storage.k8s.io/node-stage-secret-name"] = cephFSNodePluginSecretName
+	sc.Parameters["csi.storage.k8s.io/node-publish-secret-namespace"] = cephCSINamespace
+	sc.Parameters["csi.storage.k8s.io/node-publish-secret-name"] = cephFSNodePluginSecretName
 
 	if enablePool {
 		sc.Parameters["pool"] = "myfs-replicated"
@@ -227,6 +227,79 @@ func createNFSStorageClass(
 	})
 }
 
+func createNFSVolumeAttributesClass(
+	c clientset.Interface,
+	f *framework.Framework,
+	params map[string]string,
+) error {
+	vacPath := fmt.Sprintf("%s/%s", nfsExamplePath, "volumeattributesclass.yaml")
+	vac, err := getVolumeAttributesClass(vacPath)
+	if err != nil {
+		return err
+	}
+
+	// overload any parameters that were passed
+	if params == nil {
+		// create an empty params, so that params["clusterID"] below
+		// does not panic
+		params = map[string]string{}
+	}
+	for param, value := range params {
+		vac.Parameters[param] = value
+	}
+
+	vac.DriverName = nfsDriverName
+
+	timeout := time.Duration(deployTimeout) * time.Minute
+
+	return wait.PollUntilContextTimeout(context.TODO(), poll, timeout, true, func(ctx context.Context) (bool, error) {
+		_, err = c.StorageV1().VolumeAttributesClasses().Create(ctx, &vac, metav1.CreateOptions{})
+		if err != nil {
+			framework.Logf("error creating VolumeAttributesClass %q: %v", vac.Name, err)
+			if apierrs.IsAlreadyExists(err) {
+				return true, nil
+			}
+			if isRetryableAPIError(err) {
+				return false, nil
+			}
+
+			return false, fmt.Errorf("failed to create VolumeAttributesClass %q: %w", vac.Name, err)
+		}
+
+		return true, nil
+	})
+}
+
+func deleteNFSVolumeAttributesClass(
+	c clientset.Interface,
+	f *framework.Framework,
+) error {
+	vacPath := fmt.Sprintf("%s/%s", nfsExamplePath, "volumeattributesclass.yaml")
+	vac, err := getVolumeAttributesClass(vacPath)
+	if err != nil {
+		return err
+	}
+
+	timeout := time.Duration(deployTimeout) * time.Minute
+
+	return wait.PollUntilContextTimeout(context.TODO(), poll, timeout, true, func(ctx context.Context) (bool, error) {
+		err = c.StorageV1().VolumeAttributesClasses().Delete(ctx, vac.Name, metav1.DeleteOptions{})
+		if err != nil {
+			framework.Logf("error deleting VolumeAttributesClass %q: %v", vac.Name, err)
+			if apierrs.IsNotFound(err) {
+				return true, nil
+			}
+			if isRetryableAPIError(err) {
+				return false, nil
+			}
+
+			return false, fmt.Errorf("failed to delete VolumeAttributesClass %q: %w", vac.Name, err)
+		}
+
+		return true, nil
+	})
+}
+
 // unmountNFSVolume unmounts a NFS volume mounted on a pod.
 func unmountNFSVolume(f *framework.Framework, appName, pvcName string) error {
 	pod, err := f.ClientSet.CoreV1().Pods(f.UniqueName).Get(context.TODO(), appName, metav1.GetOptions{})
@@ -500,6 +573,60 @@ var _ = Describe("nfs", func() {
 				}
 			})
 
+			By("create a storageclass with relocated server and a PVC then bind it to an app", func() {
+				if !k8sVersionGreaterEquals(c, 1, 34) {
+					framework.Logf("skipping VolumeAttributesClass test, needs Kubernetes >= 1.34")
+
+					return
+				}
+
+				err := createNFSStorageClass(f.ClientSet, f, false, map[string]string{
+					"server": "relocated.example.net", // mounting will fail without vac
+				})
+				if err != nil {
+					logAndFail("failed to create NFS storageclass: %v", err)
+				}
+				err = createNFSVolumeAttributesClass(f.ClientSet, f, map[string]string{
+					"server": "rook-ceph-nfs-my-nfs-a." + rookNamespace + ".svc.cluster.local",
+				})
+				if err != nil {
+					logAndFail("failed to create NFS voluemattributesclass: %v", err)
+				}
+
+				pvc, err := loadPVC(pvcPath)
+				if err != nil {
+					logAndFail("Could not create PVC: 1 %v", err)
+				}
+				pvc.Namespace = f.UniqueName
+				vacName := "updated-parameters"
+				pvc.Spec.VolumeAttributesClassName = &vacName
+
+				app, err := loadApp(appPath)
+				if err != nil {
+					logAndFail("failed to load application: %v", err)
+				}
+				app.Namespace = f.UniqueName
+				app.Spec.Volumes[0].PersistentVolumeClaim.ClaimName = pvc.Name
+				err = createPVCAndApp("", f, pvc, app, deployTimeout)
+				if err != nil {
+					logAndFail("failed to create PVC or application: %v", err)
+				}
+
+				// delete PVC and app
+				err = deletePVCAndApp("", f, pvc, app)
+				if err != nil {
+					logAndFail("failed to delete PVC or application: %v", err)
+				}
+				err = deleteResource(nfsExamplePath + "storageclass.yaml")
+				if err != nil {
+					logAndFail("failed to delete NFS storageclass: %v", err)
+				}
+				err = deleteNFSVolumeAttributesClass(f.ClientSet, f)
+				if err != nil {
+					logAndFail("failed to delete NFS voluemattributesclass: %v", err)
+				}
+			})
+
 			By("create a storageclass with sys,krb5i security and a PVC then bind it to an app", func() {
 				err := createNFSStorageClass(f.ClientSet, f, false, map[string]string{
 					"secTypes": "sys,krb5i",

e2e/utils.go

@@ -369,6 +369,13 @@ func getStorageClass(path string) (scv1.StorageClass, error) {
 	return sc, err
 }
 
+func getVolumeAttributesClass(path string) (scv1.VolumeAttributesClass, error) {
+	vac := scv1.VolumeAttributesClass{}
+	err := unmarshal(path, &vac)
+
+	return vac, err
+}
+
 func getSecret(path string) (v1.Secret, error) {
 	sc := v1.Secret{}
 	err := unmarshal(path, &sc)

e2e/vendor/modules.txt

@@ -757,7 +757,7 @@ k8s.io/apiserver/pkg/util/feature
 k8s.io/apiserver/pkg/util/webhook
 k8s.io/apiserver/pkg/util/x509metrics
 k8s.io/apiserver/pkg/warning
-# k8s.io/client-go v12.0.0+incompatible => k8s.io/client-go v0.35.0
+# k8s.io/client-go v0.35.0 => k8s.io/client-go v0.35.0
 ## explicit; go 1.25.0
 k8s.io/client-go/applyconfigurations/admissionregistration/v1
 k8s.io/client-go/applyconfigurations/admissionregistration/v1alpha1
@@ -1110,7 +1110,7 @@ k8s.io/kube-openapi/pkg/validation/strfmt/bson
 ## explicit; go 1.25.0
 k8s.io/kubectl/pkg/scale
 k8s.io/kubectl/pkg/util/podutils
-# k8s.io/kubelet v0.33.2 => k8s.io/kubelet v0.35.0
+# k8s.io/kubelet v0.0.0 => k8s.io/kubelet v0.35.0
 ## explicit; go 1.25.0
 k8s.io/kubelet/pkg/apis
 k8s.io/kubelet/pkg/apis/stats/v1alpha1

examples/nfs/storageclass.yaml

@@ -38,8 +38,11 @@ parameters:
   csi.storage.k8s.io/provisioner-secret-namespace: default
   csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: default
-  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
-  csi.storage.k8s.io/node-stage-secret-namespace: default
+  csi.storage.k8s.io/controller-modify-secret-name: csi-cephfs-secret
+  csi.storage.k8s.io/controller-modify-secret-namespace: default
+  # publish-secret is needed for parameters set by a VolumeAttributeClass
+  csi.storage.k8s.io/node-publish-secret-name: csi-cephfs-secret
+  csi.storage.k8s.io/node-publish-secret-namespace: default
 
   # (optional) Prefix to use for naming subvolumes.
   # If omitted, defaults to "csi-vol-".

examples/nfs/volumeattributesclass.yaml

@@ -0,0 +1,11 @@
+---
+# apiVersion: storage.k8s.io/v1beta1 for k8s version 1.33
+apiVersion: storage.k8s.io/v1
+kind: VolumeAttributesClass
+metadata:
+  name: updated-parameters
+driverName: nfs.csi.ceph.com
+parameters:
+  # "server" can be an alternative NFS-server that should be used when the
+  # volume is attached the next time to a node.
+  server: to-be-deployed.example.net

internal/journal/voljournal.go

@@ -828,7 +828,7 @@ func (conn *Connection) FetchAttribute(ctx context.Context, pool, reservedUUID,
 
 	value, ok := values[key]
 	if !ok {
-		return "", fmt.Errorf("failed to find key %q in returned map: %v", key, values)
+		return "", fmt.Errorf("%w: missing key %q: %v", util.ErrKeyNotFound, key, values)
 	}
 
 	return value, nil