devops: release-node external IPs detach on every MIG recreate, and nothing detects it

[alchemydc]

What happens

The single-node release MIGs use stateful configuration that preserves the cache disk and the container declaration across a recreate. The instance template carries no access config, so the external IP is not preserved. Any recreate — a roll, a resize, a preemption, or the drain/restore inside the chain-state snapshot pipeline — brings the instance back with no external address.

The node stays RUNNING, /ready and /healthy both return ok, and it keeps announcing an address that no longer routes to it. From every health signal we have, it looks fine. It is simply unreachable inbound.

Why it does not self-heal

The only code path that reattaches a reserved address lives inside the chain-state snapshot pipeline, and only runs during a snapshot cut. Nothing in the deploy or recreate path reattaches, so outside of a cut the condition persists indefinitely.

The existing repair helper cannot fix it alone: it reads the instance's current external IP and exits early if there is not one. It repairs the announced address once an IP is attached, but cannot perform the attach. That two-step requirement is not obvious from its error message, which just reports no external IP found.

Nothing detects it

• The fleet status tool prints a "N/M P2P open" count but exits 0 regardless, so it cannot be alerted on.
• The health endpoints stay green throughout, so any check based on them passes.

The result is a node that is silently undiscoverable for an unbounded period.

Observations

Four of six release nodes were found in this state during unrelated work. They were repaired by hand. It then recurred twice more the same day following MIG recreates, most recently 2026-08-25 — repaired by hand again each time. Three occurrences in one day is the reason this needs a real fix rather than another manual pass.

Notably, the peer/seed MIGs do not show this, because they carry a stateful external-IP policy. The release MIGs appear to lack that policy, which looks like the underlying asymmetry.

Suggested directions

1. Make recreates preserve the address. Add stateful external-IP policy to the release MIGs, matching the peer MIGs. This removes the failure mode rather than repairing it after the fact, and is likely the smallest correct change.
2. Failing that, move the reattach into the deploy/recreate path so it is not exclusive to the snapshot pipeline.
3. Make the condition alertable. An opt-in flag on the fleet status tool that exits non-zero when a node has no external address, so a scheduled check can page on it. This is worth doing regardless of 1 or 2, because "RUNNING and healthy but undiscoverable" is currently invisible.

Also worth improving the repair helper's error message to state that an address must be attached first, and how.

[alchemydc]

Recurred again on 2026-09-04. Two testnet release nodes and one mainnet release node were all running with no external address, announcing addresses that no longer routed to them, with health endpoints green throughout. Found incidentally while doing unrelated work on the snapshot pipeline — nothing surfaced it, which is exactly the gap this issue describes.

Two things worth adding to the report.

The repair helper stamps the wrong port on testnet. It defaults the announced P2P port to the mainnet port with no network awareness, so running it against a testnet node writes <ip>:8233 instead of <ip>:18233 — and reports success. The documented two-step repair therefore leaves a testnet node still unreachable unless the operator knows to override the port, and the success message gives no hint that anything is wrong. I hit this directly: the first repair applied the mainnet port and had to be redone. Deriving the default from the network, rather than defaulting to mainnet, would remove a step that is easy to get wrong precisely when someone is repairing under pressure.

The repair does not always persist. On one of the two testnet nodes the helper reported that it could not pin the stateful declaration and applied a mutable-only fix. That repair will be lost on the next recreate, so for that node the condition is guaranteed to return rather than merely likely to.

This now blocks scheduled work. The snapshot pipeline gained a pre-drain health floor that refuses to drain a node unless enough siblings are reachable, and it correctly refused here ("only 1 other node RUNNING with P2P open, need 2"). That guard is working as intended, but it also means any move to run snapshot cuts on a schedule is blocked for as long as this condition persists undetected — an unattended job cannot repair the fleet, and it should not drain a degraded tier. Detection is the part that matters most for automation: the durable fix for the attach itself is still the right end state, but an alert that fires on inbound unreachability would at least turn a silent degradation into a visible one.

[alchemydc]

One more finding from finishing the repair, and it is the most important one here: the documented two-step repair is self-defeating on a MIG-managed node.

The repair helper resets the instance so the container picks up the new announced address. On a MIG-managed instance that reset causes the group to re-project the instance from its template — and since the template carries no access config, the reset strips the external address that step one just attached. The helper reports success, and the node ends up exactly where it started: running, healthy, and unreachable.

The evidence across the three nodes I repaired is consistent. On the one node whose declaration actually changed, the helper stamped the address, triggered a reset, and the address was detached again afterwards (the reserved address returned to RESERVED, the instance had no access configs, and the instance's creation timestamp confirmed it was reset rather than recreated). On the two nodes whose declaration was already correct, the helper short-circuited with "already correct", no reset fired, and the attached address stuck.

So the working order is three steps, not two: attach the address, run the helper (which stamps and resets), then attach the address again once the reset has settled. Only the last attach survives. Anyone following the current two-step procedure will believe they have fixed a node they have not fixed, which compounds the detection gap this issue is about — the repair path produces the same false-green signal as the original fault.

A helper that performed the attach itself, after the reset rather than before, would collapse this back into one idempotent step and remove the trap entirely. That is likely the same change needed to make repair safe to run unattended.

[alchemydc]

There is a proper fix for this, and part of the fleet has quietly been running it all along.

The MIGs use a stateful policy that preserves the cache disk, but says nothing about external IPs. That is the whole gap. Managed instance groups support preserving external IPs as well, via a per-instance configuration:

gcloud compute instance-groups managed instance-configs create <mig> \
  --zone <zone> --instance <instance> \
  --stateful-external-ip=address=<reserved-address-url>,interface-name=nic0,auto-delete=never

The documentation is explicit that this works even in our situation: "You can preserve the IP address that's specified in a network interface for a specific managed instance, even if that network interface is not defined in the group's instance template." That last clause matters, because it is exactly why the current templates lose the address. It also states that if the given address is already reserved, the group assigns that reservation to the instance — so this restores a detached IP as well as preserving it, with no separate attach step.

The evidence that it works is already in the fleet. Two of the six release MIGs — one per network — turned out to already have this per-instance configuration, set at some point and never rolled out to the rest. Those two are precisely the nodes that have never lost their address. The other four had no such configuration, and are precisely the nodes that keep detaching. That is a natural experiment running over months, and it is much better evidence than anything I could stage: pinned nodes survive recreates, unpinned nodes do not.

I have now applied the same configuration to the remaining four. Both networks are fully reachable.

Why this recurs so quickly. The release nodes are Spot instances, so recreates are routine rather than exceptional — one node was preempted and recreated roughly two hours after I had manually repaired it. Any repair that is not held in the template or in stateful configuration has a half-life of hours, which is why manual reattachment has felt like a losing battle. It was.

This also explains the repair helper's blind spot. The helper pins the container declaration into stateful configuration so a recreate restores it, which is correct as far as it goes — but it never pins the IP, and the IP is the half that keeps getting lost. Adding the stateful external IP to whatever provisions these MIGs would make the helper's remaining job much smaller, and would mean new nodes inherit the fix instead of needing a follow-up.

One caveat, stated plainly. Three of the four configurations I created still report APPLYING rather than EFFECTIVE. The difference tracks whether the instance needed a change at the time: the one node that had no address when I applied the config converged immediately, because the group performed the attach itself. The others already had an address attached by hand, which the group did not itself place there, so the running instance does not yet derive from the preserved state. A refresh is not enough to converge it; only a replace would be, and forcing a replace on production nodes purely to clear a status field is not worth the disruption. The next preemption converges them for free, and that is also the only moment the setting matters. Worth confirming after the next one rather than assuming.