Description
After any VM/daemon restart (colima restart, Docker Desktop restart, host reboot), network-node fails to start with:
Error response from daemon: failed to set up container networking: Address already in use
The message reads like a port conflict, but nothing is listening on 50211/50212/9999 — it is an IP conflict. network-node reserves the static address 172.27.0.3:
IPAMConfig: {"IPv4Address": "172.27.0.3"}
172.27.0.3 sits inside the range Docker allocates dynamically, and network-node has restart: no while its siblings have unless-stopped. So on restart the siblings come up first and one of them takes .3, leaving the node unable to bind its reserved address.
Confirmed it is a race, not one misbehaving container — three different squatters across three restarts: json-rpc-relay, then json-rpc-relay-ws, then mirror-node-monitor.
Steps to reproduce
hedera start -d
- Restart the Docker VM (
colima restart, or restart Docker Desktop)
docker start network-node → Address already in use
docker network inspect hedera-network-node --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}' | grep '0\.3/' → some other container holds it
Expected behavior
Either assign static IPs to every service in the network (so none can be handed out dynamically), or place the reserved addresses outside the dynamic allocation range (ipam.config.ip_range), or give network-node a restart policy so it participates in the same startup wave.
Workaround
squatter=$(docker network inspect hedera-network-node \
--format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}' \
| awk '$2 ~ /^172\.27\.0\.3\// {print $1}')
docker stop "$squatter" && docker start network-node && docker start "$squatter"
Then wait for swirlds.log to reach newStatus":"ACTIVE" — the container being Up is not sufficient (see #1391).
Environment
- @hashgraph/hedera-local 2.40.2, macOS 26.5 (Apple Silicon), colima 0.10.1
Description
After any VM/daemon restart (
colima restart, Docker Desktop restart, host reboot),network-nodefails to start with:The message reads like a port conflict, but nothing is listening on 50211/50212/9999 — it is an IP conflict.
network-nodereserves the static address172.27.0.3:172.27.0.3sits inside the range Docker allocates dynamically, andnetwork-nodehasrestart: nowhile its siblings haveunless-stopped. So on restart the siblings come up first and one of them takes.3, leaving the node unable to bind its reserved address.Confirmed it is a race, not one misbehaving container — three different squatters across three restarts:
json-rpc-relay, thenjson-rpc-relay-ws, thenmirror-node-monitor.Steps to reproduce
hedera start -dcolima restart, or restart Docker Desktop)docker start network-node→Address already in usedocker network inspect hedera-network-node --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}' | grep '0\.3/'→ some other container holds itExpected behavior
Either assign static IPs to every service in the network (so none can be handed out dynamically), or place the reserved addresses outside the dynamic allocation range (
ipam.config.ip_range), or givenetwork-nodea restart policy so it participates in the same startup wave.Workaround
Then wait for
swirlds.logto reachnewStatus":"ACTIVE"— the container beingUpis not sufficient (see #1391).Environment