Expose the overlayfs `volatile` mount option for the kubernetes containerd snapshotter

[almson]

What I'd like

A setting to make containerd mount container rootfs overlays with volatile, for example:

[settings.container-runtime]
overlay-volatile = true

Containerd already supports this via its overlay snapshotter config (mount_options = ['volatile']), so this is only about exposing it. (Alternatively, let user drop-ins survive in /etc/containerd/config.d; today files written there by bootstrap containers are removed by the settings render.)

Why

With volatile, images pull 20% faster and various other things (eg warming caches at boot, or just writing heavily) proceed at their CPU and network limits, and happily use the page cache.

Without volatile, unmounting any container rootfs (eg when startup containers run or images are pulled) causes a syncfs on the whole backing filesystem. Because EC2 filesystems are often very slow, this causes noticeable slowdowns. The intent is durability to system crashes, yet container rootfs resilience is not something that is important on Kubernetes. I'm struggling to even imagine what the syncfs is trying to achieve. Being sure that a container's state has been saved to disk in a happens-before relationship with the stop command return? There's no state. This isn't desktop Docker. The rootfs syncfs doesn't even protect real data, which is typically on other filesystems. Honestly, volatile should simply be the default.

Related

• https://www.redhat.com/en/blog/container-volatile-overlay-mounts
• Overlayfs volatile functionality containerd/containerd#6406
• Setting containerd configurations using userdata #2408 (custom containerd configuration via userdata — closed, custom-AMI only)

Background

From the RedHat link above:

What is syncing, and why is it important?

In Linux, when you write to a file or directory, the kernel does not instantly write the data to disk. Instead, it buffers up a bunch of writes and then periodically saves the data to disk to increase performance. This is called a sync. The problem with this is that a process thinks the data was saved when the write completes, but it really isn't until the kernel syncs that data. This means that if you wrote data and the kernel crashed, there is a chance that the data was never saved.

Because of this, lots of file systems sync regularly, and tools can request syncing to happen often. When a sync occurs, the kernel stops processing data with a lock and syncs all of the data to disk. Of course, this causes poorer performance. If you have a process that causes syncs frequently, your job’s performance can really be hurt. Certain tools like RPM call for a sync after every file is written to disk, causing all the dirty pages for that file to be flushed, and it is a considerable overhead.

Containers may not need syncing

In the container world, we have many use cases where we don’t care if the data is saved. If the kernel crashed, we would not use the written data anyway.

...

In the Kubernetes world, CRI-O is the container engine. Kubernetes is almost always set up to remove all containers at boot time. Basically, it wants to start with a clean state. This means if the kernel crashed while data was being written to the overlay mount, this data would be destroyed as soon as the system boots. It is also safe to use such configurations with stateful containers because the data is usually written to external volumes that won’t be affected by the “volatile” flag at runtime.