Skip to content

BF-3: Stale keepalived routes persist after container removal #36

Description

@szachovy

Problem

The mysql-mgmt init container runs `ifmetric` and keepalived adds a route to the host routing table (because `network_mode: host`). When the container is removed, this route persists. On the next deployment, the stale metric-1 route takes precedence, outgoing packets use the VIP as source IP, and host-level iptables `PREROUTING DROP` rules block inter-node connectivity entirely.

File: `services/mysql-mgmt/docker_compose.yml`, initcontainer

Impact

After any failed run, both mgmt nodes become unreachable from each other, making re-deployment impossible without manual `nsenter` cleanup on every mgmt node.

Fix Options

  1. Add a shutdown hook (SIGTERM trap) in the main container that runs `ip route del ... proto keepalived` and `ip addr del /... dev ` before exit.
  2. Add a pre-start cleanup step in the init container that removes any leftover keepalived routes before proceeding.
  3. Add cleanup to the `start_cluster()` `finally` block so failed runs clean up after themselves.

Acceptance Criteria

  • Stale keepalived routes are cleaned up automatically on container shutdown or next deployment start
  • Re-deployment after failure does not require manual `nsenter` cleanup on mgmt nodes
  • One of the fix options implemented: SIGTERM trap in main container, pre-start cleanup in init container, or cleanup in `start_cluster()` finally block

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions