fix(triple-store): harden Fuseki/TDB2 against boot-time 500s (#1057)

Dr0p42 · web-flow · commit 97c437193d01 · 2026-07-07T14:32:02.000+02:00
* fix(triple-store): harden Fuseki/TDB2 against boot-time 500s

A persistent HTTP 500 on the trivial boot probe (`ASK { ?s ?p ?o }`) was
crashing the whole engine/Dagster import. Root cause: the `ds` TDB2 dataset
becomes unreadable after an unclean shutdown (OOM-kill or disk-full mid-write),
while the container healthcheck only probed the web root — so Fuseki reported
healthy and the app booted against a dead store.

Adapter (naas-abi-core):
- `ApacheJenaTDB2._test_connection` now retries transient 500/503/connection
  errors with backoff and raises a domain `RequestError` carrying Fuseki's real
  response body, instead of a bare `HTTPError` that killed the import.
- 4 new unit tests (29 pass).

Compose (root dev stack + naas-abi-cli deploy template):
- Healthcheck now probes the dataset (authenticated `ASK` against `/ds/query`)
  so a broken TDB2 dataset marks the container unhealthy; adds `start_period`.
- `JVM_ARGS=-Xmx2g` + `mem_limit: 4g` so an OOM can't Docker-kill the JVM
  mid-write (the unclean kill that corrupts TDB2).
- Dropped `pull_policy: always` so a restart uses the cached image.
- Image kept on `stain/jena-fuseki:latest` (Apache publishes no prebuilt image;
  moving off it is a separate backup + dump-and-reload migration).

Recoverability + deploy propagation (naas-abi-cli):
- New `scripts/fuseki_backup.sh` and a deploy-template
  `docker/fuseki/backup.sh` (TDB2 backup + compaction via the admin API) so
  `abi deploy local --regenerate` ships the helper into deployments.
- Regression test asserts the rendered compose carries the hardening (12 pass).

* fix(triple-store): lower Fuseki healthcheck start_period to 20s
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -268,15 +268,28 @@ services:
   # Stores structured knowledge as RDF triples for semantic reasoning
   # TDB2 provides transactional storage with SPARQL query/update endpoints
   fuseki:
+    # Staying on stain/jena-fuseki (the older community image) for now — this is
+    # only about making the *current* store reliable. Apache does NOT publish a
+    # prebuilt Fuseki image on Docker Hub (the official path is the build-your-own
+    # jena-fuseki-docker kit), so moving off this is a real migration (build an
+    # image + back up + dump-and-reload the TDB2 data), tracked separately.
+    # `pull_policy: always` dropped on purpose: a boot now uses the locally cached
+    # image instead of requiring a registry pull every start (one less thing that
+    # can fail a restart). Re-add it if you'd rather always re-pull.
     image: stain/jena-fuseki:latest
-    pull_policy: always
     restart: unless-stopped
     ports:
       - 3030:3030
     volumes:
       - fuseki_data:/fuseki
+    # Cap Docker memory so the JVM can't balloon and get OOM-killed mid-write —
+    # an unclean kill is what leaves TDB2's node table/journal corrupt. Keep
+    # mem_limit comfortably above JVM_ARGS -Xmx: TDB2 memory-maps its files
+    # off-heap, so it needs headroom on top of the Java heap. Tune to host/data.
+    mem_limit: 4g
     environment:
       - ADMIN_PASSWORD=${FUSEKI_ADMIN_PASSWORD}
+      - JVM_ARGS=-Xmx2g
     command: >-
       bash -c '
       mkdir -p /fuseki/configuration /fuseki/databases/ds &&
@@ -309,10 +322,21 @@ services:
       " > /fuseki/configuration/ds.ttl &&
       /jena-fuseki/fuseki-server'
     healthcheck:
-      test: ["CMD-SHELL", "curl -fsS http://localhost:3030/ >/dev/null || exit 1"]
+      # Probe the DATASET, not just the web root. `curl /` returns 200 even when
+      # the `ds` dataset failed to attach or is corrupt, which lets dependents
+      # start against a dead store and then 500 on the first query. `ASK { ?s ?p
+      # ?o }` short-circuits at the first triple (cheap) but opens TDB2 storage,
+      # so a broken dataset correctly marks the container unhealthy. Authenticated
+      # because the server runs with ADMIN_PASSWORD; `$$` is a literal `$` in the
+      # container shell (Compose would otherwise treat `$ADMIN_PASSWORD` as a
+      # host-side variable).
+      test: ["CMD-SHELL", "curl -fsS -u admin:$$ADMIN_PASSWORD 'http://localhost:3030/ds/query?query=ASK%20%7B%20%3Fs%20%3Fp%20%3Fo%20%7D' >/dev/null || exit 1"]
       interval: 10s
-      timeout: 5s
+      timeout: 10s
       retries: 10
+      # Grace window for TDB2 to attach/recover before probes count; the
+      # interval×retries budget below still covers slower recoveries.
+      start_period: 20s
     networks:
       - abi-network
 
diff --git a/libs/naas-abi-cli/naas_abi_cli/cli/deploy/local_test.py b/libs/naas-abi-cli/naas_abi_cli/cli/deploy/local_test.py
@@ -211,3 +211,45 @@ def test_setup_local_deploy_regenerate_without_backup(tmp_path: Path) -> None:
     )
 
     assert not (tmp_path / ".abi-backups").exists()
+
+
+def _service_block(compose_text: str, service: str, next_service: str) -> str:
+    start = compose_text.index(f"\n  {service}:")
+    end = compose_text.index(f"\n  {next_service}:", start)
+    return compose_text[start:end]
+
+
+def test_setup_local_deploy_hardens_fuseki_for_reliability(tmp_path: Path) -> None:
+    # Guards that `abi deploy local` (and therefore `--regenerate`, which re-renders
+    # this same template) carries the Fuseki reliability hardening into a
+    # deployment's compose, so existing deployments can be patched by regenerating.
+    setup_local_deploy(str(tmp_path), base_domain="localhost")
+
+    compose_text = (tmp_path / "docker-compose.yml").read_text(encoding="utf-8")
+    fuseki = _service_block(compose_text, "fuseki", "yasgui")
+    lines = [line.strip() for line in fuseki.splitlines()]
+
+    # Image intentionally unchanged for now (staying on the community image;
+    # migrating off it is a separate backup + dump-and-reload job).
+    assert [line for line in lines if line.startswith("image:")] == [
+        "image: stain/jena-fuseki:latest"
+    ]
+
+    # Heap + memory caps so an OOM can't Docker-kill the JVM mid-write (the
+    # unclean kill that corrupts TDB2).
+    assert "mem_limit: 4g" in fuseki
+    assert "JVM_ARGS=-Xmx2g" in fuseki
+
+    # Healthcheck probes the dataset, not just the web root, so a broken TDB2
+    # dataset marks the container unhealthy instead of falsely reporting ready.
+    assert "/ds/query?query=ASK" in fuseki
+    assert "start_period: 20s" in fuseki
+
+    # pull_policy: always removed (the explanatory comment mentions it, so assert
+    # on real directive lines rather than a naive substring check).
+    assert not any(line.startswith("pull_policy") for line in lines)
+
+    # The TDB2 backup/compaction helper ships into the deployment's .deploy/.
+    backup_script = tmp_path / ".deploy" / "docker" / "fuseki" / "backup.sh"
+    assert backup_script.exists()
+    assert "--compact" in backup_script.read_text(encoding="utf-8")
diff --git a/libs/naas-abi-cli/naas_abi_cli/cli/deploy/templates/local/docker-compose.yml b/libs/naas-abi-cli/naas_abi_cli/cli/deploy/templates/local/docker-compose.yml
@@ -206,15 +206,28 @@ services:
   # Stores structured knowledge as RDF triples for semantic reasoning
   # TDB2 provides transactional storage with SPARQL query/update endpoints
   fuseki:
+    # Staying on stain/jena-fuseki (the older community image) for now — this is
+    # only about making the *current* store reliable. Apache does NOT publish a
+    # prebuilt Fuseki image on Docker Hub (the official path is the build-your-own
+    # jena-fuseki-docker kit), so moving off this is a real migration (build an
+    # image + back up + dump-and-reload the TDB2 data), tracked separately.
+    # `pull_policy: always` dropped on purpose: a boot now uses the locally cached
+    # image instead of requiring a registry pull every start (one less thing that
+    # can fail a restart). Re-add it if you'd rather always re-pull.
     image: stain/jena-fuseki:latest
-    pull_policy: always
     restart: unless-stopped
     ports:
       - 3030:3030
     volumes:
       - fuseki_data:/fuseki
+    # Cap Docker memory so the JVM can't balloon and get OOM-killed mid-write —
+    # an unclean kill is what leaves TDB2's node table/journal corrupt. Keep
+    # mem_limit comfortably above JVM_ARGS -Xmx: TDB2 memory-maps its files
+    # off-heap, so it needs headroom on top of the Java heap. Tune to host/data.
+    mem_limit: 4g
     environment:
       - ADMIN_PASSWORD=${FUSEKI_ADMIN_PASSWORD}
+      - JVM_ARGS=-Xmx2g
     command: >-
       bash -c '
       mkdir -p /fuseki/configuration /fuseki/databases/ds &&
@@ -247,10 +260,21 @@ services:
       " > /fuseki/configuration/ds.ttl &&
       /jena-fuseki/fuseki-server'
     healthcheck:
-      test: ["CMD-SHELL", "curl -fsS http://localhost:3030/ >/dev/null || exit 1"]
+      # Probe the DATASET, not just the web root. `curl /` returns 200 even when
+      # the `ds` dataset failed to attach or is corrupt, which lets dependents
+      # start against a dead store and then 500 on the first query. `ASK { ?s ?p
+      # ?o }` short-circuits at the first triple (cheap) but opens TDB2 storage,
+      # so a broken dataset correctly marks the container unhealthy. Authenticated
+      # because the server runs with ADMIN_PASSWORD; `$$` is a literal `$` in the
+      # container shell (Compose would otherwise treat `$ADMIN_PASSWORD` as a
+      # host-side variable).
+      test: ["CMD-SHELL", "curl -fsS -u admin:$$ADMIN_PASSWORD 'http://localhost:3030/ds/query?query=ASK%20%7B%20%3Fs%20%3Fp%20%3Fo%20%7D' >/dev/null || exit 1"]
       interval: 10s
-      timeout: 5s
+      timeout: 10s
       retries: 10
+      # Grace window for TDB2 to attach/recover before probes count; the
+      # interval×retries budget below still covers slower recoveries.
+      start_period: 20s
     networks:
       - abi-network
 
diff --git a/libs/naas-abi-cli/naas_abi_cli/cli/deploy/templates/local/docker/fuseki/backup.sh b/libs/naas-abi-cli/naas_abi_cli/cli/deploy/templates/local/docker/fuseki/backup.sh
@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+#
+# Fuseki / TDB2 backup + compaction helper (rendered into a deployment's
+# .deploy/docker/fuseki/backup.sh by `abi deploy local`).
+#
+# Why this exists
+# ---------------
+# TDB2 corruption after an unclean shutdown (OOM-kill, disk-full mid-write) is
+# the failure mode behind the intermittent/boot-time HTTP 500s on this store.
+# Two server-side operations make that recoverable and less likely:
+#
+#   * backup  -> writes a consistent N-Quads dump of the dataset to the Fuseki
+#                data volume ($FUSEKI_BASE/backups/). If the live TDB2 files are
+#                ever wedged, you reload from the most recent good dump instead
+#                of losing the graph.
+#   * compact -> rewrites the TDB2 database into a fresh, defragmented copy and
+#                (with deleteOld=true) drops the old generation. Reclaims the
+#                disk that append-only TDB2 growth eats, and a freshly written
+#                copy is far less prone to the states that precede corruption.
+#
+# Both run through Fuseki's admin HTTP API — no container exec required.
+#
+# Usage (run from the deployment root, where the .env lives)
+# ----------------------------------------------------------
+#   set -a; . ./.env; set +a
+#   bash .deploy/docker/fuseki/backup.sh            # backup only
+#   bash .deploy/docker/fuseki/backup.sh --compact  # backup + compact
+#
+# Environment
+# -----------
+#   FUSEKI_ADMIN_PASSWORD  (required) admin password (from the deployment .env)
+#   FUSEKI_URL             base URL, default http://localhost:3030
+#   FUSEKI_DATASET         dataset name, default ds
+#   FUSEKI_ADMIN_USER      admin user, default admin
+#
+# Scheduling
+# ----------
+# Run this on a schedule (nightly is sensible). Since the stack already runs
+# Dagster, a Dagster schedule that shells out here is the most native option; a
+# host cron entry works too:
+#   0 3 * * *  cd /path/to/deployment && set -a && . ./.env && set +a && \
+#              bash .deploy/docker/fuseki/backup.sh --compact
+#
+set -euo pipefail
+
+FUSEKI_URL="${FUSEKI_URL:-http://localhost:3030}"
+FUSEKI_DATASET="${FUSEKI_DATASET:-ds}"
+FUSEKI_ADMIN_USER="${FUSEKI_ADMIN_USER:-admin}"
+DO_COMPACT=0
+
+for arg in "$@"; do
+  case "$arg" in
+    --compact) DO_COMPACT=1 ;;
+    -h|--help) sed -n '2,45p' "$0"; exit 0 ;;
+    *) echo "Unknown argument: $arg" >&2; exit 2 ;;
+  esac
+done
+
+if [[ -z "${FUSEKI_ADMIN_PASSWORD:-}" ]]; then
+  echo "ERROR: FUSEKI_ADMIN_PASSWORD is not set (source the deployment .env first)." >&2
+  exit 1
+fi
+
+AUTH=(-u "${FUSEKI_ADMIN_USER}:${FUSEKI_ADMIN_PASSWORD}")
+BASE="${FUSEKI_URL%/}"
+
+# Trigger an async admin task and echo the task id it returns.
+_trigger() {
+  local path="$1"
+  curl -fsS -X POST "${AUTH[@]}" "${BASE}${path}" \
+    | grep -o '"taskId"[^,}]*' | grep -o '[0-9]\+' | head -n1
+}
+
+# Poll /$/tasks/<id> until the task reports it finished; fail on error.
+_wait_for_task() {
+  local task_id="$1" label="$2"
+  [[ -z "$task_id" ]] && { echo "  (no task id returned for ${label}; assuming synchronous completion)"; return 0; }
+  echo "  ${label}: task ${task_id} running…"
+  for _ in $(seq 1 120); do
+    local body
+    body="$(curl -fsS "${AUTH[@]}" "${BASE}/\$/tasks/${task_id}")" || true
+    if echo "$body" | grep -q '"finished"'; then
+      if echo "$body" | grep -qi '"success"[[:space:]]*:[[:space:]]*false'; then
+        echo "  ${label}: FAILED — ${body}" >&2
+        return 1
+      fi
+      echo "  ${label}: done."
+      return 0
+    fi
+    sleep 5
+  done
+  echo "  ${label}: timed out waiting for completion" >&2
+  return 1
+}
+
+echo "==> Backing up dataset '${FUSEKI_DATASET}' at ${BASE}"
+BACKUP_TASK="$(_trigger "/\$/backup/${FUSEKI_DATASET}")"
+_wait_for_task "$BACKUP_TASK" "backup"
+echo "    Backup written to the Fuseki data volume under \$FUSEKI_BASE/backups/"
+
+if [[ "$DO_COMPACT" -eq 1 ]]; then
+  echo "==> Compacting dataset '${FUSEKI_DATASET}' (deleteOld=true)"
+  COMPACT_TASK="$(_trigger "/\$/compact/${FUSEKI_DATASET}?deleteOld=true")"
+  _wait_for_task "$COMPACT_TASK" "compact"
+fi
+
+echo "==> Done."
diff --git a/libs/naas-abi-core/naas_abi_core/services/triple_store/adaptors/secondary/ApacheJenaTDB2.py b/libs/naas-abi-core/naas_abi_core/services/triple_store/adaptors/secondary/ApacheJenaTDB2.py
@@ -180,20 +180,91 @@ def __init__(
 
         logger.info("ApacheJenaTDB2 adapter initialized: %s", self.jena_tdb2_url)
 
-    def _test_connection(self):
-        response = self._session.get(
-            self.query_endpoint,
-            params={"query": "ASK { ?s ?p ?o }"},
-            timeout=self.timeout,
-        )
-        response.raise_for_status()
+    def _test_connection(self) -> None:
+        """Verify the dataset is queryable, tolerating a slow/starting server.
+
+        The probe query is ``ASK { ?s ?p ?o }`` — it short-circuits at the first
+        triple (so it stays cheap even on a huge dataset) while still opening and
+        reading TDB2 storage, which is what makes it a genuine *corruption*
+        detector rather than a mere liveness ping.
+
+        A freshly (re)started Fuseki can take several seconds to attach a large
+        TDB2 dataset, during which it may refuse the connection or answer 500/503.
+        Those are transient, so we retry with the same exponential back-off used
+        by the read/write paths instead of letting a single boot-time blip crash
+        the whole engine import (which manifests as the Dagster code-server
+        failing to load). On a *persistent* failure we raise a domain
+        ``RequestError`` carrying Fuseki's own response body (e.g. a TDB2 recovery
+        stack trace), so the cause is diagnosable rather than a bare
+        ``HTTPError: 500 Server Error``.
+        """
+        last_error: Optional[Exceptions.RequestError] = None
+        for attempt in range(self._CONNECT_MAX_RETRIES + 1):
+            try:
+                response = self._session.get(
+                    self.query_endpoint,
+                    params={"query": "ASK { ?s ?p ?o }"},
+                    timeout=self.timeout,
+                )
+            except requests.RequestException as exc:
+                # Connection refused/reset/read-timeout while Fuseki is still
+                # coming up. Treat as transient and retry.
+                last_error = Exceptions.RequestError(
+                    operation="connect",
+                    message=(
+                        f"Could not reach Fuseki at {self.query_endpoint}: {exc}"
+                    ),
+                    endpoint=self.query_endpoint,
+                    attempts=attempt + 1,
+                )
+            else:
+                if response.status_code not in self._RETRYABLE_STATUS_CODES:
+                    # Success (<400) or a non-retryable error (e.g. 401/404):
+                    # surface it immediately with the server's body.
+                    self._raise_for_status(
+                        response,
+                        operation="connect",
+                        endpoint=self.query_endpoint,
+                        attempts=attempt + 1,
+                    )
+                    return
+                # Retryable 500/503: capture the body in case this is the last
+                # attempt, then fall through to back-off.
+                try:
+                    self._raise_for_status(
+                        response,
+                        operation="connect",
+                        endpoint=self.query_endpoint,
+                        attempts=attempt + 1,
+                    )
+                except Exceptions.RequestError as exc:
+                    last_error = exc
+
+            if attempt < self._CONNECT_MAX_RETRIES:
+                delay = self.retry_delay * (2 ** attempt) + random.uniform(0, 0.1)
+                logger.warning(
+                    "Fuseki connectivity check failed (attempt %d/%d); "
+                    "retrying in %.2fs",
+                    attempt + 1,
+                    self._CONNECT_MAX_RETRIES + 1,
+                    delay,
+                )
+                time.sleep(delay)
+
+        assert last_error is not None  # loop always sets it before exhausting
+        raise last_error
 
     # ------------------------------------------------------------------
     # Internal HTTP helpers with retry
     # ------------------------------------------------------------------
 
     _RETRYABLE_STATUS_CODES = frozenset({500, 503})
 
+    # Boot-time connectivity probe retries. A restarting Fuseki can take a few
+    # seconds to attach a large TDB2 dataset; we tolerate that many transient
+    # failures before declaring the store unreachable.
+    _CONNECT_MAX_RETRIES = 5
+
     # Upper bound on how many characters of the server's error body we keep.
     # Fuseki error responses are usually a short message + Java stack trace;
     # this is plenty to diagnose OOM / disk-full / lock aborts without bloating
diff --git a/libs/naas-abi-core/naas_abi_core/services/triple_store/adaptors/secondary/ApacheJenaTDB2_test.py b/libs/naas-abi-core/naas_abi_core/services/triple_store/adaptors/secondary/ApacheJenaTDB2_test.py
diff --git a/scripts/fuseki_backup.sh b/scripts/fuseki_backup.sh
