Add cgroup v2 QoS support for RBD volumes

api/deploy/kubernetes/csi-config-map.go

@@ -60,6 +60,9 @@ type RBD struct {
 	// ControllerPublishSecretRef contains the secret reference for controller
 	// publish operations.
 	ControllerPublishSecretRef corev1.SecretReference `json:"controllerPublishSecretRef"`
+	// NodePublishSecretRef contains the secret reference for node publish
+	// operations. Used for retrieving QoS metadata during NodePublishVolume.
+	NodePublishSecretRef corev1.SecretReference `json:"nodePublishSecretRef"`
 }
 
 type NFS struct {

deploy/csi-config-map-sample.yaml

@@ -72,6 +72,10 @@ data:
            "netNamespaceFilePath": "<kubeletRootPath>/plugins/rbd.csi.ceph.com/net",
            "radosNamespace": "<rados-namespace>",
            "mirrorDaemonCount": 1,
+           "nodePublishSecretRef": {
+             "name": "<secret-name>",
+             "namespace": "<secret-namespace>"
+           }
         },
         "monitors": [
           "<MONValue1>",

docs/design/proposals/rbd-qos.md

@@ -93,60 +93,38 @@ above `kubepods-besteffort.slice` or `kubepods-burstable.slice` or
 `kubepods.slice` (Guaranteed QoS) cgroup. The 3 QoS classes are defined
 [here](https://kubernetes.io/docs/concepts/workloads/pods/pod-QoS/#quality-of-service-classes)
 
-To identify the right cgroup file, we need pod UUID and container UUID from the
-`pod yaml` output
+To identify the right cgroup file, we need pod UUID from the `pod yaml` output
 
 ```bash
 [$]kubectl get po csi-rbd-demo-pod -oyaml |grep uid
   uid: cdf7b785-4eb7-44f7-99cc-ef53890f4dfd
-[$]kubectl get po csi-rbd-demo-pod -oyaml |grep -i containerID
-  - containerID: cri-o://77e57fbbc0f0630f41f9f154f4b5fe368b6dcf7bef7dcd75a9c4b56676f10bc9
 [$]kubectl get po csi-rbd-demo-pod -oyaml |grep -i qosClass
   qosClass: BestEffort
 ```
 
 Now check in the `kubepods-besteffort.slice` and identify the right path using
-pod UID and container UID
+pod UID (hyphens replaced with underscores).
 
-Before that check `io.max` on the application pod and see if there is any limit
+Come back to the Node and navigate to the pod's cgroup slice
 
 ```bash
-[$]kubectl exec -it csi-rbd-demo-pod -- sh
-sh-4.4# cat /sys/fs/cgroup/io.max
-sh-4.4#
-```
-
-Come back to the Node and find the right cgroup scope
-
-```bash
-sh-5.1# cd kubepods-besteffort.slice/kubepods-besteffort-podcdf7b785_4eb7_44f7_99cc_ef53890f4dfd.slice/crio-77e57fbbc0f0630f41f9f154f4b5fe368b6dcf7bef7dcd75a9c4b56676f10bc9.scope/
-
-
+sh-5.1# cd kubepods-besteffort.slice/kubepods-besteffort-podcdf7b785_4eb7_44f7_99cc_ef53890f4dfd.slice/
 sh-5.1# echo "252:0 wbps=1048576" > io.max
 sh-5.1# cat io.max
 252:0 rbps=max wbps=1048576 riops=max wiops=max
 ```
 
-Now go back to the application pod and check if we have the right limit set
-
-```bash
-[$]kubectl exec -it csi-rbd-demo-pod -- sh
-sh-4.4# cat /sys/fs/cgroup/io.max
-252:0 rbps=max wbps=1048576 riops=max wiops=max
-sh-4.4#
-```
-
 Note:- We can only support the QoS that cgroup v2 io controller supports, this
 means that cumulative read+write QoS limits won't be supported.
 
 Below are the configurations that will be supported
 
 |  Parameter     |  Description     |
 |  ---  |  ---  |
-|  MaxReadIOPS     | Max read IO operations per second      |
-|  MaxWriteIOPS     | Max write IO operations per second      |
-|  MaxReadBytesPerSecond     |  Max read bytes per second     |
-|  MaxWriteBytesPerSecond     |  Max write bytes per second     |
+|  maxReadIops     | Max read IO operations per second      |
+|  maxWriteIops     | Max write IO operations per second      |
+|  maxReadBps     |  Max read bytes per second     |
+|  maxWriteBps     |  Max write bytes per second     |
 
 ## Implementation Approach
 
@@ -160,10 +138,10 @@ kind: VolumeAttributesClass
 metadata:
   name: silver
 parameters:
-  MaxReadIOPS: ""
-  MaxWriteIOPS: ""
-  MaxReadBytesPerSecond: ""
-  MaxWriteBytesPerSecond: ""
+  maxReadIops: ""
+  maxWriteIops: ""
+  maxReadBps: ""
+  maxWriteBps: ""
 ```
 
 VolumeAttributesClassName is a new parameter in the PVC object the user can
@@ -182,41 +160,52 @@ QoS at the storage level which means setting some configuration at the storage
 1. During NodePublishVolume operation retrieve the QoS from image metadata
 1. Whenever a new pod comes in apply the QoS
 
-#### Container Discovery and QoS Application
+#### Pod-Level QoS Application
 
 When kubelet invokes the NodePublishVolume RPC call, it provides the pod UUID
-as part of the request. Ceph-CSI will use this pod UUID to locate the correct
-cgroup hierarchy path, following the same approach demonstrated in the manual
-steps above.
+as part of the request. Ceph-CSI uses this pod UUID to locate the pod's cgroup
+hierarchy path and applies QoS limits at the pod level.
+
+##### Key Design Decision: Pod-Level io.max
 
-Since Ceph-CSI cannot determine which specific container within the pod the
-RBD volume is being mounted to, the QoS limits (io.max) must be applied to
-**all containers** found in the pod's cgroup directory. This ensures that the
-QoS limits are enforced regardless of which container is using the volume.
+QoS limits are applied to the pod's io.max file, not individual container
+io.max files. This design choice provides several benefits:
 
-The container discovery process follows these steps:
+1. **Automatic Propagation**: cgroup v2 hierarchical design ensures pod-level
+   limits automatically apply to all containers within the pod
+1. **Simplified Implementation**: Single write operation instead of discovering
+   and updating multiple container cgroups
+1. **Timing Independence**: No dependency on container creation timing - works
+   even if containers start after volume mount
+1. **Consistent Behavior**: Same enforcement regardless of container runtime
+   (CRI-O, containerd, etc.)
+
+The QoS application process follows these steps:
 
 1. Kubelet provides pod UUID in NodePublishVolume request (via pod info in
    volume context)
 1. Ceph-CSI identifies the pod's QoS class (BestEffort, Burstable, or
-   Guaranteed) from the cgroup hierarchy
+   Guaranteed) by probing the cgroup hierarchy
 1. Navigate to the appropriate kubepods slice based on QoS class:
    * `kubepods-besteffort.slice` for BestEffort
    * `kubepods-burstable.slice` for Burstable
    * `kubepods.slice` for Guaranteed
-1. Locate the pod-specific slice using the pod UUID:
-   `kubepods-<qos>-pod<uuid>.slice/`
-1. Enumerate all container scopes (e.g., `crio-<container-id>.scope/`) within
-   the pod slice
-1. Apply io.max limits to each container's cgroup by writing to
-   `<container-scope>/io.max`
-
-Example path construction:
-`/sys/fs/cgroup/kubepods-besteffort.slice/kubepods-besteffort-podcdf7b785_4eb7_44f7_99cc_ef53890f4dfd.slice/crio-77e57fbbc0f0630f41f9f154f4b5fe368b6dcf7bef7dcd75a9c4b56676f10bc9.scope/io.max`
-
-This approach ensures QoS enforcement across all containers in the pod,
-addressing the limitation that the specific target container is not known at
-NodePublishVolume time.
+1. Locate the pod-specific slice using the normalized pod UUID (hyphens
+   replaced with underscores): `kubepods-<qos>-pod<normalized-uuid>.slice/`
+1. Apply io.max limits to the pod's cgroup by writing to `<pod-slice>/io.max`
+1. cgroup v2 automatically enforces these limits on all containers in the pod
+
+##### Example path construction
+
+For a BestEffort pod with UUID `cdf7b785-4eb7-44f7-99cc-ef53890f4dfd`:
+
+```text
+/sys/fs/cgroup/kubepods-besteffort.slice/kubepods-besteffort-podcdf7b785_4eb7_44f7_99cc_ef53890f4dfd.slice/io.max
+```
+
+This pod-level approach leverages cgroup v2's hierarchical design to
+automatically enforce QoS limits on all containers, eliminating the need for
+container discovery and individual container updates.
 
 #### Secret Management
 

docs/volumeattributesclass.md

@@ -65,9 +65,22 @@ fetch updated parameters when the Volume is _staged_ or _published_. The
 secrets for staging and publishing can not (easily) be updated after the fact,
 these are part of the fixed parameters in the PersistentVolume.
 
-## Use VolumeAttributesClass for rbd-nbd volume qos
+## RBD Volume QoS with VolumeAttributesClass
 
-### Create a VolumeAttributesClass
+Ceph-CSI supports two types of QoS for RBD volumes:
+
+1. **Traditional QoS** (for rbd-nbd mounter only)
+1. **Cgroup v2 QoS** (for krbd/kernel mounter, requires cgroup v2)
+
+**Note**: You cannot mix traditional QoS parameters and cgroup v2 QoS parameters
+in the same VolumeAttributesClass.
+
+### Traditional QoS for rbd-nbd
+
+Traditional QoS is applied at the RBD device level using rbd-nbd's built-in
+QoS capabilities. This approach only works with the rbd-nbd mounter.
+
+#### Create a VolumeAttributesClass for rbd-nbd QoS
 
 - Define a VolumeAttributesClass
 
@@ -183,3 +196,127 @@ $ kubectl get pvc
 NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
 rbd-pvc-vac   Bound    pvc-8b2fcb47-233a-4bcf-bb94-f94e9aa1150a   30Gi       RWO            csi-rbd-sc        gold                    2m
 ```
+
+### Cgroup v2 QoS for krbd
+
+Cgroup v2 QoS applies io.max limits at the container cgroup level, providing
+QoS for volumes mapped using the krbd (kernel RBD) mounter. This approach works
+on systems with cgroup v2 enabled.
+
+#### Prerequisites for Cgroup v2 QoS
+
+- Cgroup v2 must be enabled on all Kubernetes nodes
+- `podInfoOnMount` must be enabled in the CSIDriver spec (enabled by default)
+- NodePublishVolume secret must be configured in StorageClass parameters
+  (`csi.storage.k8s.io/node-publish-secret-name`) or CSI ConfigMap
+  (`rbd.nodePublishSecretRef`)
+
+#### Create a VolumeAttributesClass for Cgroup v2 QoS
+
+- Define a VolumeAttributesClass
+
+```console
+---
+apiVersion: storage.k8s.io/v1
+kind: VolumeAttributesClass
+metadata:
+  name: cgroup-qos
+driverName: rbd.csi.ceph.com
+parameters:
+  # Cgroup v2 QoS parameters
+  maxReadIops: "1000"         # 1000 read IOPS
+  maxWriteIops: "2000"        # 2000 write IOPS
+  maxReadBps: "104857600"     # 100 MiB/s
+  maxWriteBps: "209715200"    # 200 MiB/s
+```
+
+```console
+kubectl create -f volumeattributesclass-cgroup.yaml
+```
+
+- Verify VolumeAttributesClass has been created
+
+```console
+$ kubectl get vac
+NAME          DRIVERNAME                   AGE
+cgroup-qos    rbd.csi.ceph.com             2s
+```
+
+#### Configure NodePublishVolume Secret
+
+##### Option 1: StorageClass Parameters (Recommended)
+
+Add the following parameters to your StorageClass:
+
+```yaml
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: csi-rbd-sc
+provisioner: rbd.csi.ceph.com
+parameters:
+  clusterID: <cluster-id>
+  pool: <rbd-pool-name>
+  # ... other parameters ...
+  csi.storage.k8s.io/node-publish-secret-name: csi-rbd-secret
+  csi.storage.k8s.io/node-publish-secret-namespace: default
+```
+
+##### Option 2: CSI ConfigMap Fallback
+
+Alternatively, configure the default secret in the CSI ConfigMap:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ceph-csi-config
+data:
+  config.json: |-
+    [
+      {
+        "clusterID": "<cluster-id>",
+        "monitors": ["<mon1>", "<mon2>", "<mon3>"],
+        "rbd": {
+          "nodePublishSecretRef": {
+            "name": "csi-rbd-secret",
+            "namespace": "default"
+          }
+        }
+      }
+    ]
+```
+
+#### Create PVC with Cgroup v2 QoS
+
+```console
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: rbd-pvc-cgroup-qos
+spec:
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi
+  storageClassName: csi-rbd-sc
+  volumeMode: Block
+  volumeAttributesClassName: cgroup-qos
+```
+
+```console
+$ kubectl create -f pvc.yaml
+persistentvolumeclaim/rbd-pvc-cgroup-qos created
+```
+
+#### How Cgroup v2 QoS Works
+
+1. QoS parameters are stored in RBD image metadata during
+  `CreateVolume` or `ControllerModifyVolume`.
+1. During `NodePublishVolume`, Ceph-CSI retrieves the QoS metadata
+1. The device major:minor number is determined from the mapped RBD device
+1. io.max limits are applied to the pod's cgroup
+1. QoS limits can be dynamically modified by updating the VolumeAttributesClass
+1. Pod need to be restart to get the new QoS limits applied.