Skip to content

network-node cannot restart after VM restart: "Address already in use" is an IP conflict on static 172.27.0.3, not a port conflict #1395

Description

@SirsiMaster

Description

After any VM/daemon restart (colima restart, Docker Desktop restart, host reboot), network-node fails to start with:

Error response from daemon: failed to set up container networking: Address already in use

The message reads like a port conflict, but nothing is listening on 50211/50212/9999 — it is an IP conflict. network-node reserves the static address 172.27.0.3:

IPAMConfig: {"IPv4Address": "172.27.0.3"}

172.27.0.3 sits inside the range Docker allocates dynamically, and network-node has restart: no while its siblings have unless-stopped. So on restart the siblings come up first and one of them takes .3, leaving the node unable to bind its reserved address.

Confirmed it is a race, not one misbehaving container — three different squatters across three restarts: json-rpc-relay, then json-rpc-relay-ws, then mirror-node-monitor.

Steps to reproduce

  1. hedera start -d
  2. Restart the Docker VM (colima restart, or restart Docker Desktop)
  3. docker start network-nodeAddress already in use
  4. docker network inspect hedera-network-node --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}' | grep '0\.3/' → some other container holds it

Expected behavior

Either assign static IPs to every service in the network (so none can be handed out dynamically), or place the reserved addresses outside the dynamic allocation range (ipam.config.ip_range), or give network-node a restart policy so it participates in the same startup wave.

Workaround

squatter=$(docker network inspect hedera-network-node \
  --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}' \
  | awk '$2 ~ /^172\.27\.0\.3\// {print $1}')
docker stop "$squatter" && docker start network-node && docker start "$squatter"

Then wait for swirlds.log to reach newStatus":"ACTIVE" — the container being Up is not sufficient (see #1391).

Environment

  • @hashgraph/hedera-local 2.40.2, macOS 26.5 (Apple Silicon), colima 0.10.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions