Skip to content

DS-2: Parallelize Superset startup on management nodes #45

Description

@szachovy

Problem

Superset is started sequentially on node-0 then node-1. Each node pulls the large superset-service image (~2 GB), initializes Docker Swarm, starts Redis, and waits for the Superset healthcheck (60s start + up to 14×60s retries = up to 15 min per node). Sequential startup means ~30 min for both nodes.

File: src/initialize.py, start_superset(), lines 249–281

Impact / Caveat

node-0 and node-1 run independent Docker Swarm instances (Docker-in-Docker), so they CAN run in parallel. However, verify there is no dependency on shared MySQL state before parallelizing.

Fix

Same ThreadPoolExecutor pattern as DS-1.

Acceptance Criteria

  • start_superset() uses ThreadPoolExecutor to start node-0 and node-1 in parallel
  • No shared MySQL state dependency is broken by parallelization (verified)
  • Superset startup phase completes in ~15 min instead of ~30 min

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions