EPERM when pulling image in privileged userns using overlayfs

[r10r]

Description

Steps to reproduce the issue:

I'm trying to run buildah pull from within a container running in kubernetes in privileged userns. The relevant kubernetes deployment configuration is:

    metadata:
      labels:
        app: devcontainer
      annotations:
        io.kubernetes.cri-o.userns-mode: "auto:size=65536"
    spec:
      containers:
        - name: devcontainer
          image: drachenfels.de/user.ruben/devcontainer:dev
          securityContext:
            privileged: true

References

• Podman is gaining rootless overlay support
• How to use Podman inside of Kubernetes

Describe the results you received:

When trying to pull an image

buildah --log-level=trace pull --policy=always docker.io/library/postgres:14.4-bullseye

I'll get the error

DEBU[0006] Error pulling candidate docker.io/library/postgres:14.4-bullseye: writing blob: adding layer with blob "sha256:8d6943e62c5432f1fddd4b062713d5f400098bbff34365f5bb27bfabc25d4e2f": Error processing tar file(exit status 1): operation not permitted
Error processing tar file(exit status 1): operation not permitted
adding layer with blob "sha256:8d6943e62c5432f1fddd4b062713d5f400098bbff34365f5bb27bfabc25d4e2f"
github.com/containers/image/v5/storage.(*storageImageDestination).commitLayer
/workdir/vendor/github.com/containers/image/v5/storage/storage_image.go:1019
github.com/containers/image/v5/storage.(*storageImageDestination).queueOrCommit
/workdir/vendor/github.com/containers/image/v5/storage/storage_image.go:841
github.com/containers/image/v5/storage.(*storageImageDestination).PutBlobWithOptions
/workdir/vendor/github.com/containers/image/v5/storage/storage_image.go:470
github.com/containers/image/v5/copy.(*copier).copyBlobFromStream
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1570
github.com/containers/image/v5/copy.(*imageCopier).copyLayerFromStream
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1306
github.com/containers/image/v5/copy.(*imageCopier).copyLayer.func3
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1241
github.com/containers/image/v5/copy.(*imageCopier).copyLayer
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1273
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:940
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1571
writing blob
github.com/containers/image/v5/copy.(*copier).copyBlobFromStream
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1572
github.com/containers/image/v5/copy.(*imageCopier).copyLayerFromStream
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1306
github.com/containers/image/v5/copy.(*imageCopier).copyLayer.func3
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1241
github.com/containers/image/v5/copy.(*imageCopier).copyLayer
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1273
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:940
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1571
DEBU[0006] shutting down the store

This seems to be closely related to https://github.com/containers/buildah/blob/main/troubleshooting.md#6-rootless-buildah-build-fails-when-using-overlayfs

But I have the privilege to create whiteout files

devcontainer-0:/var/lib/containers# mknod foo c 0 0
devcontainer-0:/var/lib/containers# stat foo 
  File: foo
  Size: 0               Blocks: 0          IO Block: 4096   character special file
Device: 202,33  Inode: 266302      Links: 1     Device type: 0,0
Access: (0644/crw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-07-15 08:43:47.829761885 +0000
Modify: 2022-07-15 08:43:47.829761885 +0000
Change: 2022-07-15 08:43:47.829761885 +0000
 Birth: -

and mounting overlayfs is possible too

devcontainer-0:/var/lib/containers/ovtest# mkdir upper work lower merged
devcontainer-0:/var/lib/containers/ovtest# mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work merged
devcontainer-0:/var/lib/containers/ovtest# 

Describe the results you expected:

The image should be pulled sucessfully.

Output of buildah version:

devcontainer-0:/var/lib/containers# buildah version
Version:         1.26.1
Go Version:      go1.18.3
Image Spec:      1.0.2-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        1.0.0
libcni Version:  v1.1.0
image Version:   5.21.1
Git Commit:      
Built:           Tue Jun 14 14:57:18 2022
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of cat /etc/containers/storage.conf:

[storage]
driver = "overlay"
graphroot = "/var/lib/containers/storage"
runroot = "/run/containers"

More debugging output:

devcontainer-0:/var/lib/containers/ovtest# stat -f /run/containers/
  File: "/run/containers/"
    ID: 8f602ae14c2e8243 Namelen: 255     Type: tmpfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 1531389    Free: 1531361    Available: 1531361
Inodes: Total: 1531389    Free: 1531306

devcontainer-0:~# stat -f /var/lib/containers/
  File: "/var/lib/containers/"
    ID: 6887898cdcff9c6 Namelen: 255     Type: ext2/ext3
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 7696617    Free: 7695050    Available: 7297751
Inodes: Total: 1966080    Free: 1965527

devcontainer-0:~# ls -l /run/containers/overlay
total 4
-rw-r--r-- 1 root root   0 Jul 15 08:29  idmapped-lower-dir-false
-rw-r--r-- 1 root root   0 Jul 15 08:29 'metacopy()-false'
-rw-r--r-- 1 root root 237 Jul 15 08:29 'native-diff()-false'
-rw-r--r-- 1 root root   0 Jul 15 08:29  overlay-true

devcontainer-0:~# cat /proc/self/uid_map 
         0       2000      65536
devcontainer-0:~# cat /proc/self/gid_map 
         0       2000      65536

cluster node details

Details of the cluster node that schedules the container buildah is running in.

[ruben@k8s-cluster9-node2 disk]$ crio --version
crio version 1.23.3
Version:          1.23.3
GitCommit:        2cf9dbf1a447d25931008c1815de407d9e5749f7
GitTreeState:     dirty
BuildDate:        1980-01-01T00:00:00Z
GoVersion:        go1.17.4
Compiler:         gc
Platform:         linux/amd64
Linkmode:         static
BuildTags:        apparmor, exclude_graphdriver_devicemapper, seccomp, selinux
SeccompEnabled:   true
AppArmorEnabled:  false

[ruben@k8s-cluster9-node2 disk]$ uname -a
Linux k8s-cluster9-node2 5.15.24-1-lts #1 SMP Wed, 16 Feb 2022 16:04:21 +0000 x86_64 GNU/Linux

[ruben@k8s-cluster9-node2 disk]$ cat /etc/subuid 
containers:2000:268435456
[ruben@k8s-cluster9-node2 disk]$ cat /etc/subgid 
containers:2000:268435456

[rhatdan]

@giuseppe PTAL

[giuseppe]

you can't use overlay on top of overlay.

Are you using a volume for the containers storage?

Alternatively, is fuse-overlayfs installed in the parent container?

[r10r]

you can't use overlay on top of overlay.

Are you using a volume for the containers storage?

Yes the containers storage path /var/lib/containers is local storage from the node
which is mounted as a kubernetes volume. I have successfully mounted an overlay filesystem on top of it.
You can see the output above under More debugging output.

Alternatively, is fuse-overlayfs installed in the parent container?

No, and we would like to use native overlayfs instead.

[giuseppe]

can you please strace the buildah process so we can see what syscall is failing?

something like:

# strace -f -s 1000 -v -Z buildah --log-level=trace pull --policy=always docker.io/library/postgres:14.4-bullseye