rbd: (filesystem) add async health checks to NodeGetVolumeStats by SanjalKatiyar · Pull Request #6382 · ceph/ceph-csi

SanjalKatiyar · 2026-07-06T15:40:21Z

Pre-requisite PR: #6418

Use the shared health-check manager in the RBD node server so NodeGetVolumeStats reports async health state before calling os.Stat().
Avoids stat hangs and aligns RBD behaviour with CephFS.
Start/Stop shared checkers during stage/unstage and incorporated fallback in NodeGetVolumeStats for node-plugin restart scenarios.
Only supported for filesystem-mode volumes.

Testing:

Logs of `openshift-storage.rbd.csi.ceph.com-nodeplugin-<HASH>` pod:

Returns "volume_condition":{"message":"block-mode health checking is not supported"} for RBD block.
Returns "volume_condition":{"message":"volume is in a healthy condition"} for RBD filesystem.

I0727 13:08:51.304966 1545666 utils.go:368] ID: 3410 GRPC response: {"capabilities":[{"rpc":{"type":"STAGE_UNSTAGE_VOLUME"}},{"rpc":{"type":"GET_VOLUME_STATS"}},{"rpc":{"type":"VOLUME_CONDITION"}},{"rpc":{"type":"EXPAND_VOLUME"}},{"rpc":{"type":"SINGLE_NODE_MULTI_WRITER"}}]}
I0727 13:08:51.305872 1545666 utils.go:361] ID: 3411 GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0727 13:08:51.305900 1545666 utils.go:362] ID: 3411 GRPC request: {"volume_id":"0001-0011-openshift-storage-0000000000000002-dec53106-24a3-4f4d-8ed4-865e614f04ae","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1049141e-3ce1-434d-83b3-c10e270a18e3/dev/ca34f0f7-fc11-4552-9dd8-bc91714ef737"}
I0727 13:08:51.312332 1545666 omap.go:87] ID: 3411 got omap values by keys: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.dec53106-24a3-4f4d-8ed4-865e614f04ae"): map[csi.imageid:723a5aa4d498 csi.imagename:csi-vol-dec53106-24a3-4f4d-8ed4-865e614f04ae csi.volname:pvc-1049141e-3ce1-434d-83b3-c10e270a18e3 csi.volume.owner:volume-health-test]
I0727 13:08:51.349269 1545666 rbd_util.go:2435] ID: 3411 rbd: running DiffIterate over image ocs-storagecluster-cephblockpool/csi-vol-dec53106-24a3-4f4d-8ed4-865e614f04ae of size 1073741824 bytes
I0727 13:08:51.350479 1545666 rbd_util.go:2456] ID: 3411 DiffIterate on image ocs-storagecluster-cephblockpool/csi-vol-dec53106-24a3-4f4d-8ed4-865e614f04ae of size 1073741824 bytes took 0.00118902 seconds
I0727 13:08:51.350814 1545666 utils.go:368] ID: 3411 GRPC response: {"usage":[{"available":1073741824,"total":1073741824,"unit":"BYTES"}],"volume_condition":{"message":"block-mode health checking is not supported"}}
I0727 13:08:54.234071 1545666 utils.go:361] ID: 3412 GRPC call: /csi.v1.Node/NodeGetCapabilities
I0727 13:08:54.234097 1545666 utils.go:362] ID: 3412 GRPC request: {}
I0727 13:08:54.234192 1545666 utils.go:368] ID: 3412 GRPC response: {"capabilities":[{"rpc":{"type":"STAGE_UNSTAGE_VOLUME"}},{"rpc":{"type":"GET_VOLUME_STATS"}},{"rpc":{"type":"VOLUME_CONDITION"}},{"rpc":{"type":"EXPAND_VOLUME"}},{"rpc":{"type":"SINGLE_NODE_MULTI_WRITER"}}]}
I0727 13:08:54.234936 1545666 utils.go:361] ID: 3413 GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0727 13:08:54.234969 1545666 utils.go:362] ID: 3413 GRPC request: {"volume_id":"0001-0011-openshift-storage-0000000000000002-a52055c9-b783-46b8-8e0f-8972915a7796","volume_path":"/var/lib/kubelet/pods/a765e803-7650-45b5-a73d-7c432b182b1a/volumes/kubernetes.io~csi/pvc-3103d574-581f-4399-92ad-901bebbd1353/mount"}
I0727 13:08:54.235097 1545666 utils.go:368] ID: 3413 GRPC response: {"usage":[{"available":88815403008,"total":98747928576,"unit":"BYTES","used":9915748352},{"available":5475380,"total":6160384,"unit":"INODES","used":685004}],"volume_condition":{"message":"volume is in a healthy condition"}}

nixpanic · 2026-07-30T08:14:40Z

 		} else if !isNotMnt {
 			log.DebugLog(ctx, "rbd: volume %s is already mounted to %s, skipping", volID, stagingTargetPath)

+			ns.startSharedHealthChecker(ctx, volID, stagingTargetPath)


Don't start the health-checker when the volume-mode is Block.

One way to prevent that, is by adding the volume-mode as parameter to startSharedHealthChecker() and conditionally select the StatCheckerType for filesystem mode, with no checker to start otherwise. This makes it easy to extend later on for Block mode.

@nixpanic initially, I was doing that, but the issue is that even if we block the shared checker in the NodeStageVolume RPC, NodeGetVolumeStats will still end up starting a non-shared checker later. Right now we use a synchronous os.Stat call to determine whether the volume is block-mode or filesystem-mode, and that call is what hangs when the volume is unhealthy.

For now, though, as you suggested, I will update and at least prevent the shared checker from being started.

But, in order to extend later, we will need a better way for NodeGetVolumeStats to determine the volume mode without relying on os.Stat.

Yes. When we have a Block-mode health-checker we can start it in startSharedHealthChecker() too. Preparing for that, and logging a warning that Block-mode is not supported would be nice.

Filesystem-mode or Block-mode is part of the CSI Volume specification. There are statements in nodeserver.go like these already:

isBlock := req.GetVolumeCapability().GetBlock() != nil

nixpanic · 2026-07-31T07:51:16Z

+		// NOTE: rbd.getStagingPath() uses os.Stat() internally which
+		// if called synchronously, could block indefinitely.
+
+		// TODO: Add health checker for block-mode volumes.


I think we always run into this on block-mode volumes?

Maybe we can introduce a no-op health checker, that just always returns healthy with a message?

@nixpanic actually this is what I was concerned about here in this comment: #6382 (comment).

We determine whether the volume is block-mode or filesystem-mode later in the flow, using os.Stat (synchronously).
At this point in the flow, we are still determining the volume's health and don't yet know its mode, so we can't rely on os.Stat for that info. We need a better way to determine the volume mode earlier in the flow, right ?? Which is why I left this as a TODO comment.

os.Stat() will always work on block-mode volumes. The information that is returned is about the local device node (/dev/rdbX) and does not trigger any I/O towards the Ceph cluster.

For filesystem-mode volumes, os.Stat() will not be reached if the volume is unhealthy (baring races between checking in the background and NodeGetVolumeStat calls).

Starting a No-Op checker for block-mode should not make the situation any worse than that it is now? But a No-Op checker should always return true, nil so that it isn't getting started on every NodeGetVolumeStat request.

My doubt is how would we know whether to start a no-op checker or a stat checker ??

Ex: err = ns.healthChecker.StartChecker(volumeId, targetPath, hc.StatCheckerType)

Here we need to pass an argument hc.NoOpCheckerType and in order to do that we need to know if the volume is block-mode or filesystem-mode. We can't do synchronous os.Stat yet, both filesystem and block volumes shares the same flow, so even if os.Stat works for block, it will hang for filesystem.

right, that is indeed a good point.

This code is expected to only get executed when the node-plugin restarted while a volume was already attached (no health-checker running yet). In this case, I think it it acceptable to call os.Stat() to figure out what type of volume it is. There is an unlikely/rare case where the volume is not healthy and the node-plugin is restarted, we can live with this potential hang.

We can do one more thing, since we can determine the mode during NodeStageVolume, we can start a shared no-op checker at that point.

For NodeGetVolumeStats, we can keep it as currently proposed in this PR, that is, always a non-shared StatChecker for now (as u pointed out that this flow will be triggered less often) to avoid performing an early os.Stat check, which could potentially hang (though, I know it might look like introducing several workarounds that make the code more complex and reduce its overall readability).

If there is a real issue on the Ceph cluster, Ceph-CSI won't work correctly in that case anyway (not only NodeGetVolumeStat, but any CSI procedure). SlowGRPCRestart causes a complete cleanup/retry of everything, I don't think we need to care about it specifically.

There is an alternative where you can store/retrieve the mode of the volume. Extend stashRBDImageMetadata() and struct rbdImageMetadataStash. Those details are stored during NodeStageVolume on a host-path, and can be retrieved after a restart of the node-plugin. This introduces a little more work, but that way you can start a selected health-checker without calling os.Stat().

@nixpanic thanks for the suggestion. I checked, but it won't solve our problem, issue is currently metadata is stashed at "stagingPath", but NodeGetVolumeStats won't have access to "stagingPath" as easily as expected.
I know we have getStagingPath util already, but that function internally uses os.Stat itself, so we are looping back to the same issue again which we were discussing above :)

If it's okay, I will stash the volume mode at new path indexed at "volumeID" ??

using the volume-id makes sense to me. But what would the full path be then? The staging path is a known location that can be used, not sure what other location suits the purpose.

It all gets quite tricky... Maybe get a simple approach (with known issues) in with this PR, and improve with a next one?

There are now 3 commits:

no-op checker (cherry-picked from your fork).

updating rbd server to use no-op checker for block based on os.Stat, with a comment about known bug.

fixing the known bug via stashing the mode on the host, indexed by volumeID.

I still need to test the changes, will let you know once done.

Block-mode volumes can not be health-checked yet, the implementation is not trivial. Because a health-checker needs to be started for all volumes managed by a backend, this no-op checker can be started for RBD block-mode volumes until we have a better way. Assisted-by: AskBob <askbob@ibm.com> Signed-off-by: Niels de Vos <ndevos@ibm.com>

Use the shared health-check manager in the RBD node server so NodeGetVolumeStats reports async health state before calling os.Stat(). Avoids stat hangs and aligns RBD behaviour with CephFS. Start/Stop shared checkers during stage/unstage and incorporated fallback in NodeGetVolumeStats for node-plugin restart scenarios. Only supported for filesystem-mode volumes. Signed-off-by: SanjalKatiyar <sanjaldhir@gmail.com>

NodeStageVolume stashes each volume's block/filesystem mode to /csi/volume-info/<volumeID>.json. NodeGetVolumeStats uses this on node-plugin restart to pick the correct checker type (NoOp for block, Stat for filesystem) without a synchronous os.Stat() on the target path, which could hang. NodeUnstageVolume removes the stash on both of its success paths, including the early "already unstaged" return, to avoid leaking stale files. Assisted-by: Claude Sonnet 5 and Claude Opus 4.6 Signed-off-by: SanjalKatiyar <sanjaldhir@gmail.com>

SanjalKatiyar marked this pull request as draft July 6, 2026 15:40

mergify Bot added the component/rbd Issues related to RBD label Jul 6, 2026

SanjalKatiyar marked this pull request as ready for review July 6, 2026 15:41

SanjalKatiyar marked this pull request as draft July 6, 2026 15:42

nixpanic reviewed Jul 6, 2026

View reviewed changes

Comment thread internal/rbd/nodeserver.go Outdated

SanjalKatiyar requested a review from nixpanic July 23, 2026 12:11

SanjalKatiyar force-pushed the volume_condition_rbd branch from b4aa66c to 97ec187 Compare July 27, 2026 08:38

SanjalKatiyar changed the title ~~rbd: add async health checks to NodeGetVolumeStats~~ rbd (filesystem): add async health checks to NodeGetVolumeStats Jul 27, 2026

SanjalKatiyar marked this pull request as ready for review July 27, 2026 13:21

SanjalKatiyar marked this pull request as draft July 27, 2026 14:01

SanjalKatiyar force-pushed the volume_condition_rbd branch 2 times, most recently from 23620ff to df76b06 Compare July 28, 2026 10:45

SanjalKatiyar marked this pull request as ready for review July 28, 2026 14:10

SanjalKatiyar force-pushed the volume_condition_rbd branch from df76b06 to cc5ca83 Compare July 29, 2026 07:13

nixpanic reviewed Jul 30, 2026

View reviewed changes

SanjalKatiyar force-pushed the volume_condition_rbd branch from cc5ca83 to 1b24596 Compare July 31, 2026 05:09

SanjalKatiyar requested a review from nixpanic July 31, 2026 05:10

SanjalKatiyar force-pushed the volume_condition_rbd branch from 1b24596 to 2356f5c Compare July 31, 2026 07:40

SanjalKatiyar changed the title ~~rbd (filesystem): add async health checks to NodeGetVolumeStats~~ rbd: (filesystem) add async health checks to NodeGetVolumeStats Jul 31, 2026

nixpanic reviewed Jul 31, 2026

View reviewed changes

nixpanic and others added 2 commits July 31, 2026 18:49

SanjalKatiyar force-pushed the volume_condition_rbd branch from 2356f5c to 0ba92e8 Compare August 1, 2026 12:29

Uh oh!

Conversation

SanjalKatiyar commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing:

Logs of openshift-storage.rbd.csi.ceph.com-nodeplugin-<HASH> pod:

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SanjalKatiyar Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SanjalKatiyar Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SanjalKatiyar Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SanjalKatiyar Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SanjalKatiyar commented Jul 6, 2026 •

edited

Loading

Logs of `openshift-storage.rbd.csi.ceph.com-nodeplugin-<HASH>` pod:

SanjalKatiyar Jul 30, 2026 •

edited

Loading

SanjalKatiyar Jul 31, 2026 •

edited

Loading

SanjalKatiyar Jul 31, 2026 •

edited

Loading

SanjalKatiyar Jul 31, 2026 •

edited

Loading