Rootless Podman on Mac M4 - on reboot, containers do not have access to remote host unless podman machine is restarted

[mperriwater]

Rootless Podman on Mac M4 - on reboot, machine and containers do not have access to remote host unless podman machine is restarted with 'podman machine stop' then' podman machine start' from zsh terminal.

Running rootless podman as a local user on Mac. Loads a gitlab runner image. After reboot, everything loads, but machine and container's network is messed up and is denied access from 10.88.x to a remote host.

but if i do:
podman machine stop
podman machine start

Everything works fine. (can access remote hosts after this)

Mac M4 proc
Latest OS
Podman 5.8
Removed Desktop app
recreated podman machine, recreated runners, etc
using slirpy4netns (if i don't, then resolv.conf is messed up)

This is happening on 2 separate macs

I have delayed podman start on boot by using local user crontab. Tried Launchtl, tried using Automator, same thing. Tried to start, stop, then start again from cron.

Summary:

1. Starting podman from cront, or launchtcl, on reboot does not load container network correctly?
2. Simple podman machine stop && podman machine start fixes it
3. podman network reload -a does not work
4. Setting machine rootful did not seem to help

Thoughts? Suggestions?

[manimovassagh]

This is a known pain point with Podman machine on macOS. The issue is a timing/ordering problem during automatic boot startup -- the VM's networking stack is not fully ready when your cron/launchd job starts it.

Why stop+start fixes it

When you manually run podman machine stop && podman machine start, the full VM teardown and re-initialization happens, including a fresh network setup with gvproxy (the user-mode networking proxy). On a cold boot, if the machine starts before the macOS networking stack is fully initialized, gvproxy binds to an interface or route that changes moments later, and the VM's network ends up broken.

The fix: delay the start and do a clean cycle

Create a LaunchAgent that waits for network availability before starting the machine:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.user.podman-machine-start</string>
    <key>ProgramArguments</key>
    <array>
        <string>/bin/bash</string>
        <string>-c</string>
        <string>/usr/local/bin/podman machine stop 2>/dev/null; sleep 5; /usr/local/bin/podman machine start</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <dict>
        <key>NetworkState</key>
        <true/>
    </dict>
    <key>ThrottleInterval</key>
    <integer>30</integer>
    <key>StandardOutPath</key>
    <string>/tmp/podman-machine-start.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/podman-machine-start.err</string>
</dict>
</plist>

Save this to ~/Library/LaunchAgents/com.user.podman-machine-start.plist and load it:

launchctl load ~/Library/LaunchAgents/com.user.podman-machine-start.plist

The NetworkState key tells launchd to only trigger the job when the network is up. The explicit stop before start ensures a clean VM networking state.

Alternative: switch to pasta networking

You mentioned using slirp4netns. If you are on Podman 5.x, try pasta instead (the new default):

podman machine stop
podman machine rm
podman machine init --rootful
podman machine start

Pasta handles network namespace setup differently and may be more resilient to the boot timing issue. It is also faster than slirp4netns.

Also check gvproxy logs

If the problem persists, check gvproxy logs after a broken boot:

cat ~/Library/Logs/podman-machine-*.log

This should show whether gvproxy failed to bind or encountered a routing error during startup.

[mperriwater]

@manimovassagh

Thank you reaching out. It helped me down the correct path.

I could not get your example plist file nor the one at:
https://github.com/containers/podman/blob/main/docs/tutorials/macos_autostart.md to work.

However, i have found a work around for rootless

There is something wrong with gvproxy when trying to load podman from launchtcl. (And from cron for that matter). If you print the environment from launchctl vs cron vs logged in terminal, they are all different. (trying to make env from term the same in plist failed for me)

So, you can have user auto login if you want, and this plist file actually opens a Terminal and runs the bash script like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.podman.machine.default</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/open</string>
        <string>-a</string>
        <string>Terminal</string>
        <string>/Users/username/podmanStuff/start_podman.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>AbandonProcessGroup</key>
    <true/>
</dict>
</plist>

but it still starts too quickly, so just put a sleep in the start_podman.sh

#!/bin/zsh
sleep 30
/opt/podman/bin/podman machine start podman-machine-default

Doing it the other way, podman machine may load, but the network is all messed up. And the gvproxy log is blank, because it thinks it is fine maybe? But from inside the podman machine, you can tell something is a messed up because resolve.conf is wrong, and even though you can ping external servers, you can't get to say port 3306? (You get a connection refused)

Hack for now, but when i have some time, i will try to put gvproxy in debug mode, and look at logs.

This is curiously happening on several macs, but they all have two network interfaces, And if you have the wifi turned off.. it really makes things worse.

However, the above hack does work, and i can clean it up later, but just wanted to put it out because i know others are having problems.

Machine is MacOS (latest), only CLI of podman (5.8) is installed, running from non root user, rootless podman, no need to try slirp4netns.

Thanks for all your help!