RBD - processes stuck in "disk sleep"

[remram44]

I tried benchmarking RBD volumes by running fio in a Pod. Unfortunately, multiple times fio got stuck, and trying to terminate it left its worker subprocesses in disk sleep forever. In this situation I am unable to stop the processes or the container (stays Terminating, cannot stop container: tried to kill container, but did not receive an exit event). My only move is to reboot the whole node, which is very disruptive.

I should add that the mount is not completely broken in that situation, I can still issue writes to other files without hanging. But the hung processes never recover or exit.

I am very worried about running production workloads on the cluster if enough sustained disk operations can apparently crash the node.

How can I recover when this happens? How can I prevent it from happening?

I am using Ubuntu 20.04, Linux 5.4.0-109

[Ganesh-403]

@remram44 Processes stuck in D (uninterruptible disk sleep) are generally a symptom of the Linux kernel waiting on an I/O operation—they usually can't be killed with SIGKILL because they're blocked in kernel space rather than in userspace.

From what you've described, this doesn't immediately point to a ceph-csi bug. Instead, I'd first suspect one of:

• a hung RBD client (krbd),
• an underlying Ceph cluster issue (slow/unresponsive OSDs),
• a kernel bug in the RBD client,
• or an interaction between the kernel version and the Ceph client.

A few things I'd recommend checking when this happens:

1. Cluster health

   ceph -s
   ceph health detail

   If the cluster has slow requests, degraded PGs, or unavailable OSDs, the client can block waiting for I/O to complete. The Ceph documentation recommends checking cluster health first when operations appear hung. :contentReference[oaicite:0]{index=0}

2. Kernel logs

   Immediately after the hang, collect:

   dmesg -T
   journalctl -k

   Look for messages involving:

   • rbd
   • libceph
   • blocked tasks
   • I/O timeouts
   • network errors

3. Ceph CSI / RBD configuration

   Are you using:

   • the default krbd mounter, or
   • rbd-nbd?

   There have historically been differences in behavior and feature support between the two, and ceph-csi has added logic over time to fall back to rbd-nbd when appropriate. :contentReference[oaicite:1]{index=1}

4. Kernel version

   You mentioned:

   Ubuntu 20.04
   Linux 5.4.0-109
   

   That's a fairly old kernel. I couldn't find documentation tying your exact symptom to that kernel version, but there have been numerous fixes to the Linux RBD client in later kernel releases. If this is reproducible, it would be worth testing with a newer HWE or LTS kernel before concluding it's a ceph-csi issue.

One question that would help narrow it down:

• Which versions of Ceph and ceph-csi are you running?
• Does the problem occur only during heavy fio workloads, or have you seen it under normal application I/O as well?

If the cluster remains healthy while only a single client gets stuck in D state, I'd be inclined to investigate the client kernel/RBD path. On the other hand, if ceph -s shows slow requests or unhealthy OSDs at the same time, the root cause is more likely on the storage side than in ceph-csi itself.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.