Support consuming CVMFS in sind containers

[dennisklein]

Summary

Add a storage.cvmfs boolean flag to the cluster config that makes CVMFS repositories available on demand inside all sind containers at /cvmfs. Two backends provide the mount — sind picks the right one automatically.

User perspective

kind: Cluster
name: test

storage:
  cvmfs: true

nodes:
  - role: controller
  - role: submitter
  - role: worker
    count: 3

$ sind create cluster -c cluster.yaml
$ sind enter test
[user@submitter data]$ ls /cvmfs/software.cern.ch/
bin  etc  lib  share

No repository list needed. Repos mount on demand when first accessed inside the container, exactly like on a bare-metal node with autofs.

Backend A: Host bind-mount with rslave propagation

If CVMFS is already mounted on the host (autofs or otherwise), sind bind-mounts /cvmfs into containers with slave mount propagation:

--mount type=bind,source=/cvmfs,target=/cvmfs,readonly,bind-propagation=rslave

With rslave, the container's /cvmfs is a slave of the host's /cvmfs. Directory lookups from inside the container still hit the host's autofs superblock, triggering the host's automount daemon. The resulting submount propagates into the container via slave propagation. Autofs idle unmounts also propagate — re-access triggers remount.

Requirements:

• CVMFS installed and autofs running on the host
• The host's /cvmfs must be in a shared mount peer group (default on systemd-based hosts, verify with findmnt -o TARGET,PROPAGATION /cvmfs)
• No container privileges needed

Backend B: Docker volume plugin

If a cvmfs Docker volume driver is installed, sind uses it instead:

--mount type=volume,volume-driver=cvmfs,source=cvmfs,target=/cvmfs,readonly

The plugin runs as a privileged daemon on the host, manages its own CVMFS client and autofs/automount, and serves repositories on demand. Containers stay fully unprivileged.

Requirements:

• cvmfs Docker volume plugin installed (docker plugin install ...)
• No container privileges needed

Backend selection logic

When storage.cvmfs: true:

1. Check if the cvmfs Docker volume driver is installed (docker plugin ls --format '{{.Name}}' or docker plugin inspect cvmfs)
2. If yes → use volume plugin backend (B)
3. If no → check if /cvmfs exists on the host (stat or mountpoint check)
4. If yes → use rslave bind-mount backend (A)
5. If neither → fail with an error message suggesting either installing CVMFS on the host or installing the Docker volume plugin

The selected backend should be logged during cluster creation.

Implementation

Config changes (pkg/config/config.go)

Add a CVMFS field to Storage:

type Storage struct {
    DataStorage DataStorage `json:"dataStorage,omitempty"`
    CVMFS       bool        `json:"cvmfs,omitempty"`
}

RunConfig changes

Propagate the flag through to RunConfig so BuildRunArgs() can act on it.

BuildRunArgs changes (pkg/cluster/node.go)

When CVMFS is enabled, append the appropriate --mount flag based on the detected backend. Use --mount (not -v) because bind-propagation options require the long-form mount syntax.

Backend detection

Implement backend detection as a function called once during cluster creation (not per-node). Cache the result and pass it through to BuildRunArgs(). Suggested location: pkg/cluster/cvmfs.go or similar.

Error handling

• If storage.cvmfs: true but neither backend is available, fail early during cluster creation with a clear message
• If backend A is selected but /cvmfs is not a shared mount, warn (the mount will work but autofs propagation won't — repos must be pre-mounted on the host)

Docker volume plugin packaging

The volume plugin should live in a standalone repository (e.g. GSI-HPC/docker-volume-cvmfs), not inside sind. Reasons:

• Independent lifecycle: The plugin has its own versioning, releases, and users beyond sind. Other Docker users may want CVMFS volumes without sind.
• Different runtime model: The plugin runs as a persistent privileged daemon managed by Docker's plugin system. sind is a CLI tool. Coupling them would be awkward.
• Different dependencies: The plugin needs cvmfs2, fusermount, and CVMFS config/keys baked into its rootfs. sind's build has none of these.
• Docker managed plugin packaging: Docker plugins are OCI images with a config.json manifest, built via docker plugin create. This is its own build/release pipeline.

Plugin implementation sketch

~300 lines of Go using github.com/docker/go-plugins-helpers/volume:

• Create: Validate repo name, store in state map
• Mount: Write CVMFS config file, run cvmfs2 <repo> <mountpoint> -o config=<path>, return mountpoint. Reference-count mounts (multiple containers may use the same repo).
• Unmount: Decrement reference count, fusermount -u when count reaches zero
• Remove: Ensure unmounted, clean up config and cache

Reference pattern: vieux/docker-volume-sshfs (canonical example of wrapping a FUSE filesystem as a Docker volume plugin).

Plugin configuration

Expose settings via docker plugin set:

• HTTP_PROXY → CVMFS_HTTP_PROXY
• QUOTA_LIMIT → CVMFS_QUOTA_LIMIT (default 4000 MB)
• CACHE_BASE → CVMFS_CACHE_BASE
• CVMFS_KEYS_DIR → for custom repository signing keys

Plugin rootfs

The plugin's Docker image needs:

• cvmfs2 binary and libraries (from CVMFS RPM or cvmfsexec's makedist)
• fusermount / fuse3
• Default CVMFS config and keys
• The Go plugin binary as entrypoint

Out of scope

• Testing CVMFS config management (autofs/mount unit setup inside containers): This requires CAP_SYS_ADMIN + /dev/fuse inside the container and is a separate feature (see Support CVMFS #37).
• Per-repo configuration (custom proxy, keys, etc.): Can be added later as storage.cvmfs evolves from a boolean to an object if needed.

References

• cvmfsexec (unprivileged CVMFS mounting): https://github.com/cvmfs/cvmfsexec
• CVMFS CSI driver (Kubernetes, prior art): https://github.com/cvmfs-contrib/cvmfs-csi
• Docker volume plugin Go helpers: https://github.com/docker/go-plugins-helpers
• Docker managed plugin docs: https://docs.docker.com/engine/extend/
• mount propagation: mount_namespaces(7), shared/slave/private semantics
• Support CVMFS #37 — original CVMFS discussion