EPERM when pulling image in privileged userns using overlayfs #4121

r10r · 2022-07-15T09:02:50Z

r10r
Jul 15, 2022

Description

Steps to reproduce the issue:

I'm trying to run buildah pull from within a container running in kubernetes in privileged userns. The relevant kubernetes deployment configuration is:

    metadata:
      labels:
        app: devcontainer
      annotations:
        io.kubernetes.cri-o.userns-mode: "auto:size=65536"
    spec:
      containers:
        - name: devcontainer
          image: drachenfels.de/user.ruben/devcontainer:dev
          securityContext:
            privileged: true

References

Describe the results you received:

When trying to pull an image

buildah --log-level=trace pull --policy=always docker.io/library/postgres:14.4-bullseye

I'll get the error

DEBU[0006] Error pulling candidate docker.io/library/postgres:14.4-bullseye: writing blob: adding layer with blob "sha256:8d6943e62c5432f1fddd4b062713d5f400098bbff34365f5bb27bfabc25d4e2f": Error processing tar file(exit status 1): operation not permitted
Error processing tar file(exit status 1): operation not permitted
adding layer with blob "sha256:8d6943e62c5432f1fddd4b062713d5f400098bbff34365f5bb27bfabc25d4e2f"
github.com/containers/image/v5/storage.(*storageImageDestination).commitLayer
/workdir/vendor/github.com/containers/image/v5/storage/storage_image.go:1019
github.com/containers/image/v5/storage.(*storageImageDestination).queueOrCommit
/workdir/vendor/github.com/containers/image/v5/storage/storage_image.go:841
github.com/containers/image/v5/storage.(*storageImageDestination).PutBlobWithOptions
/workdir/vendor/github.com/containers/image/v5/storage/storage_image.go:470
github.com/containers/image/v5/copy.(*copier).copyBlobFromStream
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1570
github.com/containers/image/v5/copy.(*imageCopier).copyLayerFromStream
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1306
github.com/containers/image/v5/copy.(*imageCopier).copyLayer.func3
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1241
github.com/containers/image/v5/copy.(*imageCopier).copyLayer
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1273
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:940
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1571
writing blob
github.com/containers/image/v5/copy.(*copier).copyBlobFromStream
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1572
github.com/containers/image/v5/copy.(*imageCopier).copyLayerFromStream
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1306
github.com/containers/image/v5/copy.(*imageCopier).copyLayer.func3
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1241
github.com/containers/image/v5/copy.(*imageCopier).copyLayer
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:1273
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1
/workdir/vendor/github.com/containers/image/v5/copy/copy.go:940
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1571
DEBU[0006] shutting down the store

This seems to be closely related to https://github.com/containers/buildah/blob/main/troubleshooting.md#6-rootless-buildah-build-fails-when-using-overlayfs

But I have the privilege to create whiteout files

devcontainer-0:/var/lib/containers# mknod foo c 0 0
devcontainer-0:/var/lib/containers# stat foo 
  File: foo
  Size: 0               Blocks: 0          IO Block: 4096   character special file
Device: 202,33  Inode: 266302      Links: 1     Device type: 0,0
Access: (0644/crw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-07-15 08:43:47.829761885 +0000
Modify: 2022-07-15 08:43:47.829761885 +0000
Change: 2022-07-15 08:43:47.829761885 +0000
 Birth: -

and mounting overlayfs is possible too

devcontainer-0:/var/lib/containers/ovtest# mkdir upper work lower merged
devcontainer-0:/var/lib/containers/ovtest# mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work merged
devcontainer-0:/var/lib/containers/ovtest#

Describe the results you expected:

The image should be pulled sucessfully.

Output of buildah version:

devcontainer-0:/var/lib/containers# buildah version
Version:         1.26.1
Go Version:      go1.18.3
Image Spec:      1.0.2-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        1.0.0
libcni Version:  v1.1.0
image Version:   5.21.1
Git Commit:      
Built:           Tue Jun 14 14:57:18 2022
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of cat /etc/containers/storage.conf:

[storage]
driver = "overlay"
graphroot = "/var/lib/containers/storage"
runroot = "/run/containers"

More debugging output:

devcontainer-0:/var/lib/containers/ovtest# stat -f /run/containers/
  File: "/run/containers/"
    ID: 8f602ae14c2e8243 Namelen: 255     Type: tmpfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 1531389    Free: 1531361    Available: 1531361
Inodes: Total: 1531389    Free: 1531306

devcontainer-0:~# stat -f /var/lib/containers/
  File: "/var/lib/containers/"
    ID: 6887898cdcff9c6 Namelen: 255     Type: ext2/ext3
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 7696617    Free: 7695050    Available: 7297751
Inodes: Total: 1966080    Free: 1965527

devcontainer-0:~# ls -l /run/containers/overlay
total 4
-rw-r--r-- 1 root root   0 Jul 15 08:29  idmapped-lower-dir-false
-rw-r--r-- 1 root root   0 Jul 15 08:29 'metacopy()-false'
-rw-r--r-- 1 root root 237 Jul 15 08:29 'native-diff()-false'
-rw-r--r-- 1 root root   0 Jul 15 08:29  overlay-true

devcontainer-0:~# cat /proc/self/uid_map 
         0       2000      65536
devcontainer-0:~# cat /proc/self/gid_map 
         0       2000      65536

cluster node details

Details of the cluster node that schedules the container buildah is running in.

[ruben@k8s-cluster9-node2 disk]$ crio --version
crio version 1.23.3
Version:          1.23.3
GitCommit:        2cf9dbf1a447d25931008c1815de407d9e5749f7
GitTreeState:     dirty
BuildDate:        1980-01-01T00:00:00Z
GoVersion:        go1.17.4
Compiler:         gc
Platform:         linux/amd64
Linkmode:         static
BuildTags:        apparmor, exclude_graphdriver_devicemapper, seccomp, selinux
SeccompEnabled:   true
AppArmorEnabled:  false

[ruben@k8s-cluster9-node2 disk]$ uname -a
Linux k8s-cluster9-node2 5.15.24-1-lts #1 SMP Wed, 16 Feb 2022 16:04:21 +0000 x86_64 GNU/Linux

[ruben@k8s-cluster9-node2 disk]$ cat /etc/subuid 
containers:2000:268435456
[ruben@k8s-cluster9-node2 disk]$ cat /etc/subgid 
containers:2000:268435456

rhatdan · 2022-07-15T10:31:32Z

rhatdan
Jul 15, 2022
Maintainer

@giuseppe PTAL

0 replies

giuseppe · 2022-07-18T07:54:39Z

giuseppe
Jul 18, 2022
Maintainer

you can't use overlay on top of overlay.

Are you using a volume for the containers storage?

Alternatively, is fuse-overlayfs installed in the parent container?

0 replies

r10r · 2022-07-18T13:10:45Z

r10r
Jul 18, 2022
Author

you can't use overlay on top of overlay.

Are you using a volume for the containers storage?

Yes the containers storage path /var/lib/containers is local storage from the node
which is mounted as a kubernetes volume. I have successfully mounted an overlay filesystem on top of it.
You can see the output above under More debugging output.

Alternatively, is fuse-overlayfs installed in the parent container?

No, and we would like to use native overlayfs instead.

0 replies

giuseppe · 2022-07-18T13:37:28Z

giuseppe
Jul 18, 2022
Maintainer

can you please strace the buildah process so we can see what syscall is failing?

something like:

# strace -f -s 1000 -v -Z buildah --log-level=trace pull --policy=always docker.io/library/postgres:14.4-bullseye

2 replies

r10r Jul 19, 2022
Author

Hmm, running buildah pull --policy=always docker.io/library/alpine:latest works fine.

I've looked at the strace output of buildah pull docker.io/library/postgres:14.4-bullseye and noticed the following

devpod-userns-0:~# grep xattr strace2.txt
time="2022-07-19T13:18:04Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: failed to set opaque flag on middle layer: lsetxattr /var/lib/containers/storage/overlay/opaque-bug-check2772648804/l2/d: operation not permitted"
[pid 37511] setxattr("/etc/X11", "trusted.overlay.opaque", "y", 1, 0) = -1 EPERM (Operation not permitted)

Could that be the root cause ? Here is the full strace output strace2.txt

giuseppe Jul 19, 2022
Maintainer

yes that seems to be the reason. The trusted.overlay.opaque xattr is needed for opaque directories

giuseppe · 2022-07-18T13:38:17Z

giuseppe
Jul 18, 2022
Maintainer

converted to a discussion, it doesn't seem to be an issue in buildah so far

0 replies

Navigation Menu

Uh oh!

EPERM when pulling image in privileged userns using overlayfs #4121

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 5 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

EPERM when pulling image in privileged userns using overlayfs #4121

Uh oh!

Uh oh!

r10r Jul 15, 2022

cluster node details

Replies: 5 comments · 2 replies

Uh oh!

rhatdan Jul 15, 2022 Maintainer

Uh oh!

giuseppe Jul 18, 2022 Maintainer

Uh oh!

r10r Jul 18, 2022 Author

Uh oh!

giuseppe Jul 18, 2022 Maintainer

Uh oh!

Uh oh!

r10r Jul 19, 2022 Author

Uh oh!

giuseppe Jul 19, 2022 Maintainer

Uh oh!

giuseppe Jul 18, 2022 Maintainer

r10r
Jul 15, 2022

Replies: 5 comments 2 replies

rhatdan
Jul 15, 2022
Maintainer

giuseppe
Jul 18, 2022
Maintainer

r10r
Jul 18, 2022
Author

giuseppe
Jul 18, 2022
Maintainer

r10r Jul 19, 2022
Author

giuseppe Jul 19, 2022
Maintainer

giuseppe
Jul 18, 2022
Maintainer