Handle transient Docker API errors in wait_until_healthy()

Wrap health check polling in try/except to catch APIError, KeyError,
and IndexError. These transient errors occur during Docker Swarm
stabilization and should be treated as not-healthy-yet rather than
crashing the deployment.

Closes #43

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: szachovy

CHANGELOG.md

@@ -26,6 +26,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 * Fixed `run_mysql_server()` not instantiating `MySQLServer` class. (#94)
 * Disabled MD060 markdownlint rule to fix table column style false positives in documentation. (#94)
+* Handled transient Docker API errors in `wait_until_healthy()` retry loop. (#43)
 
 ## 1.0 - 2024-10-13
 

src/container.py

@@ -198,12 +198,16 @@ def wait_until_healthy(self, cls: typing.Type[ContainerInstance]) -> str:
         cls.run()  # type: ignore[call-arg]
         time.sleep(cls.healthcheck_start_period)
         for _ in range(cls.healthcheck_retries):
-            if self.container == "superset":
-                if self.client.api.tasks(filters={"service": "superset"})[0]["Status"]["State"] == "running":
-                    return f"{self.get_logs()}\nContainer {self.container} is healthy"
-            else:
-                if self.client.containers.get(self.container).attrs["State"]["Health"]["Status"] == "healthy":
-                    return f"{self.get_logs()}\nContainer {self.container} is healthy"
+            try:
+                if self.container == "superset":
+                    tasks = self.client.api.tasks(filters={"service": "superset"})
+                    if tasks and tasks[0]["Status"]["State"] == "running":
+                        return f"{self.get_logs()}\nContainer {self.container} is healthy"
+                else:
+                    if self.client.containers.get(self.container).attrs["State"]["Health"]["Status"] == "healthy":
+                        return f"{self.get_logs()}\nContainer {self.container} is healthy"
+            except (docker.errors.APIError, KeyError, IndexError):
+                pass
             time.sleep(cls.healthcheck_interval)
         return f"{self.get_logs()}\nTimeout while waiting for {self.container} healthcheck to be healthy"