Add --cleanup CLI flag with idempotent deploy

Make deploy idempotent by checking component health before acting:
- start_mysql_servers() skips nodes where mysql is already healthy
- start_mysql_mgmt() skips nodes where mysql-mgmt is already healthy
- start_superset() skips nodes where superset service already exists
- credentials() handles idempotent directory creation

Cleanup via --cleanup tears down all containers, volumes, networks,
routes, deployed files, and VIP addresses. Teardown commands use
conditional logic (if/then guards, xargs -r) instead of || true,
with all commands per node executed in a single SSH call and exit
status checked.

Test suite extended with redeploy and cleanup stages:
- redeploy.yml: restarts disaster-stopped nodes, runs cleanup then
  fresh deploy, verifies cluster fully functional
- cleanup.yml: runs --cleanup, verifies no containers, volumes,
  directories, .pyc files, or VIP addresses remain

Closes #49

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: szachovy

CHANGELOG.md

@@ -15,17 +15,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 * Dependabot updates for `github-actions` and `terraform` ecosystems. (#29)
 * CI workflow to build and push service images to GitHub Container Registry on master merge. (#97)
 * Pull-first with local build fallback for container images during deployment. (#97)
-
-### Added
-
-* `--cleanup` CLI flag and `deploy`/`cleanup` actions with proper teardown. (#49)
-* Redeploy and cleanup test stages with post-disaster recovery verification. (#49)
+* `--cleanup` CLI flag with idempotent deploy and cleanup actions. (#49)
+* Cleanup test stage with post-cleanup state verification. (#49)
 
 ### Changed
 
 * Completed [ARCHITECTURE.md](./docs/ARCHITECTURE.md) (#93)
 * Migrated CI from self-hosted to GitHub-hosted runners with Docker-in-Docker test infrastructure. (#94)
 * Test workflow always builds service images locally for reproducibility. (#97)
+* Moved `.pyc` bytecode files from `/opt` to `/opt/superset-cluster` for cleaner file organization.
 
 ### Fixed
 

src/initialize.py

@@ -164,6 +164,13 @@ def get_mylogin_cnf(self, node: remote.RemoteConnection) -> bytes:
 
     def start_mysql_servers(self) -> None:
         for node in self.mysql_nodes:
+            _, stdout, _ = node.ssh_client.exec_command(
+                "docker inspect --format='{{.State.Health.Status}}'"
+                " mysql 2>/dev/null"
+            )
+            if stdout.read().decode().strip() == "healthy":
+                continue
+            node.ssh_client.exec_command("docker rm -f mysql 2>/dev/null")
             node.upload_directory(
                 local_directory_path="./services/mysql-server",
                 remote_directory_path="/opt/superset-cluster/mysql-server"
@@ -195,6 +202,13 @@ def start_mysql_servers(self) -> None:
             )
 
     def start_mysql_mgmt(self, node: remote.RemoteConnection, state: str, priority: int) -> None:
+        _, stdout, _ = node.ssh_client.exec_command(
+            "docker inspect --format='{{.State.Health.Status}}'"
+            " mysql-mgmt 2>/dev/null"
+        )
+        if stdout.read().decode().strip() == "healthy":
+            return
+        node.ssh_client.exec_command("docker rm -f mysql-mgmt 2>/dev/null")
         node.upload_directory(
             local_directory_path="./services/mysql-mgmt",
             remote_directory_path="/opt/superset-cluster/mysql-mgmt"
@@ -251,6 +265,23 @@ def start_mysql_mgmt(self, node: remote.RemoteConnection, state: str, priority:
 
     def start_superset(self) -> None:
         for node in self.mgmt_nodes:
+            _, redis_out, _ = node.ssh_client.exec_command(
+                "docker inspect --format='{{.State.Health.Status}}'"
+                " redis 2>/dev/null"
+            )
+            _, svc_out, _ = node.ssh_client.exec_command(
+                "docker service ps superset"
+                " --format='{{.CurrentState}}'"
+                " --filter desired-state=running 2>/dev/null"
+            )
+            redis_healthy = redis_out.read().decode().strip() == "healthy"
+            superset_running = "Running" in svc_out.read().decode()
+            if redis_healthy and superset_running:
+                continue
+            node.ssh_client.exec_command(
+                "docker service rm superset 2>/dev/null;"
+                " docker rm -f redis 2>/dev/null"
+            )
             node.upload_directory(
                 local_directory_path="./services/superset",
                 remote_directory_path='/opt/superset-cluster/superset'
@@ -283,27 +314,6 @@ def start_superset(self) -> None:
                           mysql_superset_password=self.mysql_superset_password)
             )
 
-    def _close_connections(self) -> None:
-        for node in list(itertools.chain(self.mysql_nodes, self.mgmt_nodes)):
-            node.ssh_client.close()
-            node.sftp_client.close()
-
-    def _run_ssh_command(self, node: remote.RemoteConnection, command: str) -> None:
-        _, stdout, stderr = node.ssh_client.exec_command(command)
-        exit_status = stdout.channel.recv_exit_status()
-        if exit_status != 0:
-            raise RuntimeError(
-                f"Command failed on {node.node} (exit {exit_status}): "
-                f"{command}\n{stderr.read().decode().strip()}"
-            )
-
-    def _prepare_nodes(self) -> None:
-        for node in itertools.chain(self.mysql_nodes, self.mgmt_nodes):
-            try:
-                node.sftp_client.mkdir('/opt/superset-cluster')
-            except IOError:
-                pass
-
     def teardown_node(self, node: remote.RemoteConnection, is_mgmt: bool = False) -> None:
         commands = [
             "if docker info --format '{{.Swarm.LocalNodeState}}' 2>/dev/null"
@@ -316,7 +326,6 @@ def teardown_node(self, node: remote.RemoteConnection, is_mgmt: bool = False) ->
             " | xargs -r docker network rm",
             "if [ -d /opt/superset-cluster ]; then"
             " rm -rf /opt/superset-cluster; fi",
-            "find /opt -maxdepth 1 -name '*.pyc' -delete",
         ]
         if is_mgmt:
             commands.extend([
@@ -330,34 +339,35 @@ def teardown_node(self, node: remote.RemoteConnection, is_mgmt: bool = False) ->
                 " ip route del {vip}; fi".format(
                     vip=self.virtual_ip_address),
             ])
-        for cmd in commands:
-            self._run_ssh_command(node, cmd)
-
-    def teardown_cluster(self) -> None:
-        for node in self.mgmt_nodes:
-            self.teardown_node(node, is_mgmt=True)
-        for node in self.mysql_nodes:
-            self.teardown_node(node, is_mgmt=False)
+        _, stdout, stderr = node.ssh_client.exec_command(" && ".join(commands))
+        exit_status = stdout.channel.recv_exit_status()
+        if exit_status != 0:
+            raise RuntimeError(
+                f"Teardown failed on {node.node} (exit {exit_status}): "
+                f"{stderr.read().decode().strip()}"
+            )
 
     def start_cluster(self) -> None:
-        self.start_mysql_servers()
-        self.start_mysql_mgmt(node=self.mgmt_nodes[0], state="MASTER", priority=100)
-        self.start_mysql_mgmt(node=self.mgmt_nodes[1], state="BACKUP", priority=90)
-        self.start_superset()
-
-    def deploy(self) -> None:
         try:
-            self.teardown_cluster()
-            self._prepare_nodes()
-            self.start_cluster()
+            self.start_mysql_servers()
+            self.start_mysql_mgmt(node=self.mgmt_nodes[0], state="MASTER", priority=100)
+            self.start_mysql_mgmt(node=self.mgmt_nodes[1], state="BACKUP", priority=90)
+            self.start_superset()
         finally:
-            self._close_connections()
+            for node in list(itertools.chain(self.mysql_nodes, self.mgmt_nodes)):
+                node.ssh_client.close()
+                node.sftp_client.close()
 
     def cleanup(self) -> None:
         try:
-            self.teardown_cluster()
+            for node in self.mgmt_nodes:
+                self.teardown_node(node, is_mgmt=True)
+            for node in self.mysql_nodes:
+                self.teardown_node(node, is_mgmt=False)
         finally:
-            self._close_connections()
+            for node in list(itertools.chain(self.mysql_nodes, self.mgmt_nodes)):
+                node.ssh_client.close()
+                node.sftp_client.close()
 
 
 if __name__ == "__main__":
@@ -371,4 +381,4 @@ def cleanup(self) -> None:
     if action == "cleanup":
         controller.cleanup()
     else:
-        controller.deploy()
+        controller.start_cluster()

src/remote.py

@@ -128,8 +128,8 @@ def run_python_container_command(self, command: str) -> dict:
             pyc_file = io.BytesIO()
             pyc_file.write(b'o\r\r\n\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
             marshal.dump(code_object, pyc_file)
-            self.upload_file(content=pyc_file.getvalue(), remote_file_path=f'/opt/{nonce}.pyc')
-        _, stdout, stderr = self.ssh_client.exec_command(f"python3 /opt/{nonce}.pyc")
+            self.upload_file(content=pyc_file.getvalue(), remote_file_path=f'/opt/superset-cluster/{nonce}.pyc')
+        _, stdout, stderr = self.ssh_client.exec_command(f"python3 /opt/superset-cluster/{nonce}.pyc")
         return {
             "output": stdout.read().decode(),
             "error": stderr.read().decode()

tests/testsuite/deploy.yml

@@ -23,14 +23,6 @@
     - name: "Run functional tasks from testing role"
       ansible.builtin.include_role: {name: "testing", tasks_from: "functional"}
 
-- name: "Redeploy testing"
-  connection: "local"
-  hosts: "testing"
-  any_errors_fatal: true
-  tasks:
-    - name: "Run redeploy tasks from testing role"
-      ansible.builtin.include_role: {name: "testing", tasks_from: "redeploy"}
-
 - name: "Cleanup testing"
   connection: "local"
   hosts: "testing"

tests/testsuite/roles/testing/tasks/cleanup.yml

@@ -1,4 +1,34 @@
 ---
+- name: "Restart nodes stopped during disaster simulation"
+  community.docker.docker_container:
+    name: "{{ item }}"
+    state: "started"
+  loop:
+    - "{{ node_prefix }}-0"
+    - "{{ node_prefix }}-2"
+
+- name: "Start services in restarted nodes"
+  community.docker.docker_container_exec:
+    container: "{{ item }}"
+    command: "/bin/bash -c 'service ssh start && service docker start'"
+  loop:
+    - "{{ node_prefix }}-0"
+    - "{{ node_prefix }}-2"
+  changed_when: false
+
+- name: "Wait for Docker daemon readiness in restarted nodes"
+  community.docker.docker_container_exec:
+    container: "{{ item }}"
+    command: >-
+      /bin/bash -c
+      'for i in $(seq 1 30); do
+      docker info >/dev/null 2>&1 && exit 0; sleep 2; done; exit 1'
+  loop:
+    - "{{ node_prefix }}-0"
+    - "{{ node_prefix }}-2"
+  register: "docker_readiness"
+  changed_when: false
+
 - name: "Run cleanup"
   ansible.builtin.shell: |
     set -euxo pipefail
@@ -53,12 +83,12 @@
     - "{{ node_prefix }}-4"
   register: "no_deploy_dir"
 
-- name: "Verify no .pyc files in /opt on any node"
+- name: "Verify no .pyc files in /opt/superset-cluster on any node"
   community.docker.docker_container_exec:
     container: "{{ item }}"
     command: >-
       /bin/bash -c
-      'test $(find /opt -maxdepth 1 -name "*.pyc" | wc -l) -eq 0'
+      'test $(find /opt/superset-cluster -maxdepth 1 -name "*.pyc" 2>/dev/null | wc -l) -eq 0'
   loop:
     - "{{ node_prefix }}-0"
     - "{{ node_prefix }}-1"