Error Creating Mountpoint, rootless buildah, overlayfs enabled(5.15 kernel)

[SaurabhAhuja1983]

We are migrating from docker to buildah for building images.
One of the build is failing with following error message

error running subprocess: error creating mountpoint "/var/tmp/buildah1588703561/mnt/rootfs/var/run/mysqld" in mount namespace: mkdir /var/tmp/buildah1588703561/mnt/rootfs/var/run/mysqld: permission denied
error building at STEP 

I am running buildah inside a jenkins_agent running in kubernetes cluster

_BUILDAH_STARTED_IN_USERNS="" BUILDAH_ISOLATION=chroot
/usr/local/bin/buildah build --format=docker --layers=true --skip-unused-stages=false --ulimit nofile=4096 --memory-swap -1 --memory 15032385536 --build-arg .....
--network=host .........

Dockerfile

Looks like binary is trying to mount a path and getting permission denied.

[rhatdan]

Can you provide a Containerfile, with a reproducer?

[SaurabhAhuja1983]

yeah.. Thanks.. Let me isolate the problem and work on a reproducer.. will update

[SaurabhAhuja1983]

Able to isolate and reproduce the problem. Please see below @rhatdan

[SaurabhAhuja1983]

$ cat Dockerfile
FROM ubuntu as default-image
VOLUME /var/lib/mysql /var/run/mysqld /var/log/mysql

FROM default-image as test-image

RUN ls /var/lib/mysql
RUN ls /var/run/mysqld

foo@abcde-95c7bd7bc-8bzss:/tmp/tmp.Vcrfm31POX/test$ buildah bud .
[1/2] STEP 1/2: FROM ubuntu AS default-image
[1/2] STEP 2/2: VOLUME /var/lib/mysql /var/run/mysqld /var/log/mysql
Getting image source signatures
Copying blob f4a670ac65b6 skipped: already exists
Copying blob f2b890b9c476 done
Copying config c941ef72df done
Writing manifest to image destination
Storing signatures
--> c941ef72df6
[2/2] STEP 1/3: FROM c941ef72df6bdaa6ec2e9a4da17bdc84b07c465c613a20fa0ba03872dbb4c8a9 AS test-image
[2/2] STEP 2/3: RUN ls /var/lib/mysql
error running subprocess: error creating mountpoint "/var/tmp/buildah2143072700/mnt/rootfs/var/run/mysqld" in mount namespace: mkdir /var/tmp/buildah2143072700/mnt/rootfs/var/run/mysqld: permission denied
error building at STEP "RUN ls /var/lib/mysql": exit status 1

[rhatdan]

VOLUME /var/lib/mysql /var/run/mysqld /var/log/mysql
Your issue is you added a d to /var/run/mysql

[SaurabhAhuja1983]

Thank You.. @rhatdan
Can you please elaborate, what exactly is happening here by having a suffix 'd' at the end of the name of the volume

[rhatdan]

You are checking in your run command for /var/run/mysql (NO d) But you are creating it with the d

[SaurabhAhuja1983]

@rhatdan Thank you for pointing out the typo but problem is not there.. if you carefully look the error, it's at STEP 2 error creating mountpoint. Also as i mentioned below /var/run is symlink to /run
I am not able to use anything inside /var/run/, i tried /var/run/foo etc.

[2/2] STEP 2/3: RUN ls /var/lib/mysql
error running subprocess: error creating mountpoint "/var/tmp/buildah2143072700/mnt/rootfs/var/run/mysqld" in mount namespace: mkdir /var/tmp/buildah2143072700/mnt/rootfs/var/run/mysqld: permission denied
error building at STEP "RUN ls /var/lib/mysql": exit status 1

[SaurabhAhuja1983]

One of the thing that i observed is /var/run is symlink to /run and maybe it's permission issue mounting rootfs?

[rhatdan]

Could you attach the Containerfile you are using?