Add --cleanup CLI flag with idempotent deploy

Make deploy idempotent by checking component health before acting:
- start_mysql_servers() skips nodes where mysql is already healthy
- start_mysql_mgmt() skips nodes where mysql-mgmt is already healthy
- start_superset() skips nodes where superset service already exists
- credentials() handles idempotent directory creation

Cleanup via --cleanup tears down all containers, volumes, networks,
routes, deployed files, and VIP addresses. Teardown commands use
conditional logic (if/then guards, xargs -r) instead of || true,
with all commands per node executed in a single SSH call and exit
status checked.

Test suite extended with redeploy and cleanup stages:
- redeploy.yml: restarts disaster-stopped nodes, runs cleanup then
  fresh deploy, verifies cluster fully functional
- cleanup.yml: runs --cleanup, verifies no containers, volumes,
  directories, .pyc files, or VIP addresses remain

Closes #49

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: szachovy

CHANGELOG.md

@@ -15,17 +15,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 * Dependabot updates for `github-actions` and `terraform` ecosystems. (#29)
 * CI workflow to build and push service images to GitHub Container Registry on master merge. (#97)
 * Pull-first with local build fallback for container images during deployment. (#97)
-
-### Added
-
-* `--cleanup` CLI flag and `deploy`/`cleanup` actions with proper teardown. (#49)
+* `--cleanup` CLI flag with idempotent deploy and cleanup actions. (#49)
 * Redeploy and cleanup test stages with post-disaster recovery verification. (#49)
 
 ### Changed
 
 * Completed [ARCHITECTURE.md](./docs/ARCHITECTURE.md) (#93)
 * Migrated CI from self-hosted to GitHub-hosted runners with Docker-in-Docker test infrastructure. (#94)
 * Test workflow always builds service images locally for reproducibility. (#97)
+* Moved `.pyc` bytecode files from `/opt` to `/opt/superset-cluster` for cleaner file organization.
 
 ### Fixed
 

src/initialize.py

@@ -164,6 +164,21 @@ def get_mylogin_cnf(self, node: remote.RemoteConnection) -> bytes:
 
     def start_mysql_servers(self) -> None:
         for node in self.mysql_nodes:
+            _, stdout, _ = node.ssh_client.exec_command(
+                "docker container inspect mysql > /dev/null 2>&1 && echo exists"
+            )
+            if stdout.read().decode().strip() == "exists":
+                _, stdout, _ = node.ssh_client.exec_command(
+                    "for i in $(seq 1 60); do "
+                    "docker inspect --format='{{.State.Health.Status}}'"
+                    " mysql 2>/dev/null"
+                    " | grep -qx healthy && exit 0; sleep 5; done; exit 1"
+                )
+                if stdout.channel.recv_exit_status() != 0:
+                    raise RuntimeError(
+                        f"MySQL on {node.node} exists but did not become healthy"
+                    )
+                continue
             node.upload_directory(
                 local_directory_path="./services/mysql-server",
                 remote_directory_path="/opt/superset-cluster/mysql-server"
@@ -195,6 +210,21 @@ def start_mysql_servers(self) -> None:
             )
 
     def start_mysql_mgmt(self, node: remote.RemoteConnection, state: str, priority: int) -> None:
+        _, stdout, _ = node.ssh_client.exec_command(
+            "docker container inspect mysql-mgmt > /dev/null 2>&1 && echo exists"
+        )
+        if stdout.read().decode().strip() == "exists":
+            _, stdout, _ = node.ssh_client.exec_command(
+                "for i in $(seq 1 40); do "
+                "docker inspect --format='{{.State.Health.Status}}'"
+                " mysql-mgmt 2>/dev/null"
+                " | grep -qx healthy && exit 0; sleep 5; done; exit 1"
+            )
+            if stdout.channel.recv_exit_status() != 0:
+                raise RuntimeError(
+                    f"mysql-mgmt on {node.node} exists but did not become healthy"
+                )
+            return
         node.upload_directory(
             local_directory_path="./services/mysql-mgmt",
             remote_directory_path="/opt/superset-cluster/mysql-mgmt"
@@ -251,6 +281,38 @@ def start_mysql_mgmt(self, node: remote.RemoteConnection, state: str, priority:
 
     def start_superset(self) -> None:
         for node in self.mgmt_nodes:
+            _, redis_out, _ = node.ssh_client.exec_command(
+                "docker container inspect redis > /dev/null 2>&1 && echo exists"
+            )
+            _, svc_out, _ = node.ssh_client.exec_command(
+                "docker service inspect superset > /dev/null 2>&1 && echo exists"
+            )
+            redis_exists = redis_out.read().decode().strip() == "exists"
+            superset_exists = svc_out.read().decode().strip() == "exists"
+            if redis_exists != superset_exists:
+                raise RuntimeError(
+                    f"Partial state on {node.node}: "
+                    f"redis={'exists' if redis_exists else 'missing'}, "
+                    f"superset={'exists' if superset_exists else 'missing'}. "
+                    f"Run --cleanup first"
+                )
+            if redis_exists and superset_exists:
+                _, stdout, _ = node.ssh_client.exec_command(
+                    "for i in $(seq 1 90); do "
+                    "docker inspect --format='{{.State.Health.Status}}'"
+                    " redis 2>/dev/null | grep -qx healthy"
+                    " && docker service ps superset"
+                    " --format='{{.CurrentState}}'"
+                    " --filter desired-state=running 2>/dev/null"
+                    " | head -1 | grep -q Running"
+                    " && exit 0; sleep 5; done; exit 1"
+                )
+                if stdout.channel.recv_exit_status() != 0:
+                    raise RuntimeError(
+                        f"Superset stack on {node.node} exists"
+                        f" but did not become healthy"
+                    )
+                continue
             node.upload_directory(
                 local_directory_path="./services/superset",
                 remote_directory_path='/opt/superset-cluster/superset'
@@ -283,27 +345,6 @@ def start_superset(self) -> None:
                           mysql_superset_password=self.mysql_superset_password)
             )
 
-    def _close_connections(self) -> None:
-        for node in list(itertools.chain(self.mysql_nodes, self.mgmt_nodes)):
-            node.ssh_client.close()
-            node.sftp_client.close()
-
-    def _run_ssh_command(self, node: remote.RemoteConnection, command: str) -> None:
-        _, stdout, stderr = node.ssh_client.exec_command(command)
-        exit_status = stdout.channel.recv_exit_status()
-        if exit_status != 0:
-            raise RuntimeError(
-                f"Command failed on {node.node} (exit {exit_status}): "
-                f"{command}\n{stderr.read().decode().strip()}"
-            )
-
-    def _prepare_nodes(self) -> None:
-        for node in itertools.chain(self.mysql_nodes, self.mgmt_nodes):
-            try:
-                node.sftp_client.mkdir('/opt/superset-cluster')
-            except IOError:
-                pass
-
     def teardown_node(self, node: remote.RemoteConnection, is_mgmt: bool = False) -> None:
         commands = [
             "if docker info --format '{{.Swarm.LocalNodeState}}' 2>/dev/null"
@@ -316,7 +357,7 @@ def teardown_node(self, node: remote.RemoteConnection, is_mgmt: bool = False) ->
             " | xargs -r docker network rm",
             "if [ -d /opt/superset-cluster ]; then"
             " rm -rf /opt/superset-cluster; fi",
-            "find /opt -maxdepth 1 -name '*.pyc' -delete",
+            "find /opt/superset-cluster -maxdepth 1 -name '*.pyc' -delete",
         ]
         if is_mgmt:
             commands.extend([
@@ -330,34 +371,35 @@ def teardown_node(self, node: remote.RemoteConnection, is_mgmt: bool = False) ->
                 " ip route del {vip}; fi".format(
                     vip=self.virtual_ip_address),
             ])
-        for cmd in commands:
-            self._run_ssh_command(node, cmd)
-
-    def teardown_cluster(self) -> None:
-        for node in self.mgmt_nodes:
-            self.teardown_node(node, is_mgmt=True)
-        for node in self.mysql_nodes:
-            self.teardown_node(node, is_mgmt=False)
+        _, stdout, stderr = node.ssh_client.exec_command(" && ".join(commands))
+        exit_status = stdout.channel.recv_exit_status()
+        if exit_status != 0:
+            raise RuntimeError(
+                f"Teardown failed on {node.node} (exit {exit_status}): "
+                f"{stderr.read().decode().strip()}"
+            )
 
     def start_cluster(self) -> None:
-        self.start_mysql_servers()
-        self.start_mysql_mgmt(node=self.mgmt_nodes[0], state="MASTER", priority=100)
-        self.start_mysql_mgmt(node=self.mgmt_nodes[1], state="BACKUP", priority=90)
-        self.start_superset()
-
-    def deploy(self) -> None:
         try:
-            self.teardown_cluster()
-            self._prepare_nodes()
-            self.start_cluster()
+            self.start_mysql_servers()
+            self.start_mysql_mgmt(node=self.mgmt_nodes[0], state="MASTER", priority=100)
+            self.start_mysql_mgmt(node=self.mgmt_nodes[1], state="BACKUP", priority=90)
+            self.start_superset()
         finally:
-            self._close_connections()
+            for node in list(itertools.chain(self.mysql_nodes, self.mgmt_nodes)):
+                node.ssh_client.close()
+                node.sftp_client.close()
 
     def cleanup(self) -> None:
         try:
-            self.teardown_cluster()
+            for node in self.mgmt_nodes:
+                self.teardown_node(node, is_mgmt=True)
+            for node in self.mysql_nodes:
+                self.teardown_node(node, is_mgmt=False)
         finally:
-            self._close_connections()
+            for node in list(itertools.chain(self.mysql_nodes, self.mgmt_nodes)):
+                node.ssh_client.close()
+                node.sftp_client.close()
 
 
 if __name__ == "__main__":
@@ -371,4 +413,4 @@ def cleanup(self) -> None:
     if action == "cleanup":
         controller.cleanup()
     else:
-        controller.deploy()
+        controller.start_cluster()

src/remote.py

@@ -128,8 +128,8 @@ def run_python_container_command(self, command: str) -> dict:
             pyc_file = io.BytesIO()
             pyc_file.write(b'o\r\r\n\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
             marshal.dump(code_object, pyc_file)
-            self.upload_file(content=pyc_file.getvalue(), remote_file_path=f'/opt/{nonce}.pyc')
-        _, stdout, stderr = self.ssh_client.exec_command(f"python3 /opt/{nonce}.pyc")
+            self.upload_file(content=pyc_file.getvalue(), remote_file_path=f'/opt/superset-cluster/{nonce}.pyc')
+        _, stdout, stderr = self.ssh_client.exec_command(f"python3 /opt/superset-cluster/{nonce}.pyc")
         return {
             "output": stdout.read().decode(),
             "error": stderr.read().decode()

tests/testsuite/deploy.yml

@@ -23,6 +23,41 @@
     - name: "Run functional tasks from testing role"
       ansible.builtin.include_role: {name: "testing", tasks_from: "functional"}
 
+- name: "Restore test infrastructure after disaster simulation"
+  connection: "local"
+  hosts: "testing"
+  any_errors_fatal: true
+  tasks:
+    - name: "Restart stopped nodes"
+      community.docker.docker_container:
+        name: "{{ item }}"
+        state: "started"
+      loop:
+        - "{{ node_prefix }}-0"
+        - "{{ node_prefix }}-2"
+
+    - name: "Start services in restarted nodes"
+      community.docker.docker_container_exec:
+        container: "{{ item }}"
+        command: "/bin/bash -c 'service ssh start && service docker start'"
+      loop:
+        - "{{ node_prefix }}-0"
+        - "{{ node_prefix }}-2"
+      changed_when: false
+
+    - name: "Wait for Docker daemon readiness in restarted nodes"
+      community.docker.docker_container_exec:
+        container: "{{ item }}"
+        command: >-
+          /bin/bash -c
+          'for i in $(seq 1 30); do
+          docker info >/dev/null 2>&1 && exit 0; sleep 2; done; exit 1'
+      loop:
+        - "{{ node_prefix }}-0"
+        - "{{ node_prefix }}-2"
+      register: "docker_readiness"
+      changed_when: false
+
 - name: "Redeploy testing"
   connection: "local"
   hosts: "testing"

tests/testsuite/roles/testing/tasks/cleanup.yml

@@ -53,12 +53,12 @@
     - "{{ node_prefix }}-4"
   register: "no_deploy_dir"
 
-- name: "Verify no .pyc files in /opt on any node"
+- name: "Verify no .pyc files in /opt/superset-cluster on any node"
   community.docker.docker_container_exec:
     container: "{{ item }}"
     command: >-
       /bin/bash -c
-      'test $(find /opt -maxdepth 1 -name "*.pyc" | wc -l) -eq 0'
+      'test $(find /opt/superset-cluster -maxdepth 1 -name "*.pyc" 2>/dev/null | wc -l) -eq 0'
   loop:
     - "{{ node_prefix }}-0"
     - "{{ node_prefix }}-1"

tests/testsuite/roles/testing/tasks/redeploy.yml

@@ -1,35 +1,5 @@
 ---
-- name: "Restart stopped nodes for redeploy testing"
-  community.docker.docker_container:
-    name: "{{ item }}"
-    state: "started"
-  loop:
-    - "{{ node_prefix }}-0"
-    - "{{ node_prefix }}-2"
-
-- name: "Start services in restarted nodes"
-  community.docker.docker_container_exec:
-    container: "{{ item }}"
-    command: "/bin/bash -c 'service ssh start && service docker start'"
-  loop:
-    - "{{ node_prefix }}-0"
-    - "{{ node_prefix }}-2"
-  changed_when: false
-
-- name: "Wait for Docker daemon readiness in restarted nodes"
-  community.docker.docker_container_exec:
-    container: "{{ item }}"
-    command: >-
-      /bin/bash -c
-      'for i in $(seq 1 30); do
-      docker info >/dev/null 2>&1 && exit 0; sleep 2; done; exit 1'
-  loop:
-    - "{{ node_prefix }}-0"
-    - "{{ node_prefix }}-2"
-  register: "docker_readiness"
-  changed_when: false
-
-- name: "Run redeploy (deploy cleans existing state first)"
+- name: "Run redeploy"
   ansible.builtin.shell: |
     set -euxo pipefail
     {
@@ -43,7 +13,7 @@
   args:
     executable: "/bin/bash"
     chdir: "../../"
-  async: 2100
+  async: 4200
   poll: 60
   register: "redeploy_command"
   changed_when: "redeploy_command.rc != 0"