Skip to content

Expand keepalived healthcheck to cover all critical components #66

Description

@szachovy

Problem

Keepalived only tracks the mysqlrouter process via killall -0 mysqlrouter. If other critical components fail (Superset, Redis, Docker Swarm), the VIP stays on the unhealthy node and traffic is routed to broken services. The Docker healthcheck in docker_compose.yml only affects container status — it does not trigger VRRP failover.

File: services/mysql-mgmt/keepalived.conf.tpl

Impact

Component failed Detected before Detected after VIP failover?
MySQL Router process
MySQL Router unresponsive
Docker daemon
Superset service
Redis

Fix

Create services/mysql-mgmt/healthcheck.sh:

#!/bin/bash
killall -0 mysqlrouter 2>/dev/null || exit 1
mysqladmin --login-path=superset --port=6446 ping 2>/dev/null || exit 1
docker info > /dev/null 2>&1 || exit 1
docker service inspect superset --format '{{.Spec.Name}}' 2>/dev/null | grep -q superset || exit 1
docker exec redis redis-cli ping 2>/dev/null | grep -q PONG || exit 1
exit 0

Update keepalived.conf.tpl to use it with interval=5, weight=-20, fall=3, rise=2.

Acceptance Criteria

  • healthcheck.sh created covering: mysqlrouter process, mysqladmin ping via Router, docker daemon, superset swarm service, redis ping
  • keepalived.conf.tpl updated to use healthcheck.sh with interval=5, weight=-20, fall=3, rise=2
  • Dockerfile copies healthcheck.sh; docker_compose.yml init container copies it to named volume
  • Failure of Docker daemon, Superset service, or Redis triggers VRRP failover to BACKUP node

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions