Description
Hey team,
Following up on the discussion in #14222 where @ayushr2 pointed us at the filesystem checkpoint API. We are exploring it as potential option for the future (thanks for the pointer @ayushr2, it was really helpful). We however hit a different wall and we would like to raise it as a feature request.
What we see
fscheckpoint --leave-running pauses the whole sandbox for the entire duration of the save. From reading pkg/sentry/kernel/fscheckpoint.go: the save runs inside quiescePausedAnd, and the async pages-file writer is drained before that callback returns, so k.Unpause() only happens after the last page has been written to the image files. On top of the I/O there's the zero-page scan in MemoryFile.SaveTo, which reads every allocated page inside the same window. Net effect: the pause is roughly dirty_bytes / image_path_write_speed, with an O(dirty_bytes) memory scan as the floor even if the disk were infinitely fast.
Measured on an EKS node under the containerd shim with --leave-running and the image path on local NVMe: 1 GiB dirty in the root overlay, disk writing ~106 MB/s, guest frozen for about 10s. That matches dirty_bytes / write_speed, and we've confirmed it scales that way on slower disks too. Extrapolating, a 20 GiB workspace is minutes of freeze even on good hardware. For us the entire point of --leave-running is that the sandbox stays usable, and a multi-minute freeze mostly defeats that.
Why we think the long pause is not fundamental
The pause only needs to be long enough to capture a consistent state of the filesystem, not to copy the data out. Other systems with the same requirement already work this way. LVM implements snapshots by blocking writes just long enough to set up copy-on-write metadata, then lets the workload continue against shared blocks. QEMU implements live migration by copying memory while the guest runs, pausing only for the final round of dirty pages. Nothing about a filesystem checkpoint obviously requires the copy itself to happen while the sandbox is stopped.
One observation that seems specific to gVisor and makes us think this could be cheap: with the default overlay2=root:self, the data being checkpointed already lives in a disk-backed filestore file on the host. On a reflink-capable filesystem (XFS with reflink, Btrfs), a single FICLONE of that file captures all of its contents in constant time, extents shared copy-on-write by the host filesystem. At that point the only work that genuinely needs the sandbox frozen is the tmpfs metadata walk, which is proportional to file count, not bytes. We might be naive about pgalloc details here (offset stability across the clone, what restore would need), but from the outside it looks like most of what the pause currently pays for is avoidable when the backing filesystem cooperates.
The ask
Would you be open to a save mode where the pause covers only establishing the consistency point? Three shapes come to mind, roughly in order of how much machinery they need:
- Filestore cloning: on reflink-capable filesystems, capture page contents by cloning the disk-backed filestore instead of streaming pages, and keep only the metadata walk inside the pause. Cheapest to build, but conditional on the host filesystem.
- Copy-on-write in the sentry: pause briefly, write-protect the checkpointed MemoryFile mappings (the platform already owns the stub page tables, so the fault plumbing exists), snapshot the fs metadata, unpause, then stream pages lazily and copy a page aside before allowing a write to it. Works on any filesystem, costs a fault storm and transient memory.
- Pre-copy: dirty tracking per MemoryFile range, stream everything live, re-stream the delta, pause only for the final delta plus metadata. No write-protect machinery, but has the usual convergence problem under heavy write rates.
Even an intermediate that freezes only the checkpointed filesystems rather than the whole kernel (fsfreeze semantics) would be a real improvement on its own, since compute- and network-bound guests would keep running through the capture.
Happy to help with any of this. We put together a POC for approach 1 with the help of Claude. It captures same 1 GiB dirty overlay in ~10ms and restores it correctly. If there's alignment on the direction, we're happy to put in the work to write it properly for upstream. We are also happy to hear any alternative approaches to overcome the extended pause. My colleague @jackowfish has been in the adjacent tar rootfs-upper code recently as well (#14222).
Environment
- runsc release-20260727.0 (also reproduced on a build of current master)
- EKS (AL2023) node, containerd shim, systrap platform, directfs on, systemd-cgroup on
- root overlay with the default self-backed filestore, which is what fscheckpoint captures
- image path on local instance NVMe (XFS)
Thanks!
Written with the help of Claude
Description
Hey team,
Following up on the discussion in #14222 where @ayushr2 pointed us at the filesystem checkpoint API. We are exploring it as potential option for the future (thanks for the pointer @ayushr2, it was really helpful). We however hit a different wall and we would like to raise it as a feature request.
What we see
fscheckpoint --leave-runningpauses the whole sandbox for the entire duration of the save. From readingpkg/sentry/kernel/fscheckpoint.go: the save runs insidequiescePausedAnd, and the async pages-file writer is drained before that callback returns, sok.Unpause()only happens after the last page has been written to the image files. On top of the I/O there's the zero-page scan inMemoryFile.SaveTo, which reads every allocated page inside the same window. Net effect: the pause is roughlydirty_bytes / image_path_write_speed, with an O(dirty_bytes) memory scan as the floor even if the disk were infinitely fast.Measured on an EKS node under the containerd shim with
--leave-runningand the image path on local NVMe: 1 GiB dirty in the root overlay, disk writing ~106 MB/s, guest frozen for about 10s. That matchesdirty_bytes / write_speed, and we've confirmed it scales that way on slower disks too. Extrapolating, a 20 GiB workspace is minutes of freeze even on good hardware. For us the entire point of--leave-runningis that the sandbox stays usable, and a multi-minute freeze mostly defeats that.Why we think the long pause is not fundamental
The pause only needs to be long enough to capture a consistent state of the filesystem, not to copy the data out. Other systems with the same requirement already work this way. LVM implements snapshots by blocking writes just long enough to set up copy-on-write metadata, then lets the workload continue against shared blocks. QEMU implements live migration by copying memory while the guest runs, pausing only for the final round of dirty pages. Nothing about a filesystem checkpoint obviously requires the copy itself to happen while the sandbox is stopped.
One observation that seems specific to gVisor and makes us think this could be cheap: with the default
overlay2=root:self, the data being checkpointed already lives in a disk-backed filestore file on the host. On a reflink-capable filesystem (XFS with reflink, Btrfs), a singleFICLONEof that file captures all of its contents in constant time, extents shared copy-on-write by the host filesystem. At that point the only work that genuinely needs the sandbox frozen is the tmpfs metadata walk, which is proportional to file count, not bytes. We might be naive about pgalloc details here (offset stability across the clone, what restore would need), but from the outside it looks like most of what the pause currently pays for is avoidable when the backing filesystem cooperates.The ask
Would you be open to a save mode where the pause covers only establishing the consistency point? Three shapes come to mind, roughly in order of how much machinery they need:
Even an intermediate that freezes only the checkpointed filesystems rather than the whole kernel (fsfreeze semantics) would be a real improvement on its own, since compute- and network-bound guests would keep running through the capture.
Happy to help with any of this. We put together a POC for approach 1 with the help of Claude. It captures same 1 GiB dirty overlay in ~10ms and restores it correctly. If there's alignment on the direction, we're happy to put in the work to write it properly for upstream. We are also happy to hear any alternative approaches to overcome the extended pause. My colleague @jackowfish has been in the adjacent tar rootfs-upper code recently as well (#14222).
Environment
Thanks!
Written with the help of Claude