Retry transient download failures and keep testing healthy binaries

The v7.9.0 release run lost one step to a one-off CERTIFICATE_VERIFY_FAILED from a Windows runner's TLS-intercepting proxy while the tool runner fetched gh, which skipped the windows-x64 attestation sidecar upload; the binary itself shipped and its attestation is registered server-side. The verification seam now retries network-level failures (TLS errors, timeouts, truncated bodies) up to 3 times before surfacing. The same run also exposed test-binaries skipping wholesale when a sibling build cell fails, leaving every healthy binary untested: the job now gates with always() and fail-fast off, so only the failed target's cell errors out on its missing artifact. The v7.9.0 changelog section records the missing sidecar.

Author: kdeldycke

.github/workflows/_release-engine.yaml

@@ -387,12 +387,19 @@ jobs:
     # job re-validates the artifact on a pristine VM, through the same
     # upload/download round-trip a user's binary takes. Also skipped for
     # branches that don't affect code (e.g., update-mailmap, format-markdown).
+    # always() keeps it running when a sibling build failed: without it, one
+    # red build cell skips this whole job and every healthy binary ships with
+    # no runtime test at all (v7.9.0 shipped exactly that way).
     if: >
-      fromJSON(needs.metadata.outputs.metadata).nuitka_matrix
+      always()
+      && fromJSON(needs.metadata.outputs.metadata).nuitka_matrix
       && !(fromJSON(needs.metadata.outputs.metadata).skip_binary_build || false)
       && fromJSON(needs.metadata.outputs.metadata).release_commits_matrix
     strategy:
-      # Advisory matrix (default fail-fast): see docs/workflows.md "Matrix fail-fast strategy".
+      # Info-gathering matrix: see docs/workflows.md "Matrix fail-fast strategy".
+      # fail-fast off so the cell of a target whose build failed (its artifact
+      # download errors out) does not cancel the healthy targets' tests.
+      fail-fast: false
       # Fall back to an empty `{"include":[]}` matrix when nuitka_matrix is null
       # (binary builds disabled via nuitka.enabled = false). GitHub resolves
       # strategy.matrix during job setup even when `if:` will skip the job, and a

changelog.md

@@ -6,9 +6,14 @@
 > This version is **not released yet** and is under active development.
 
 - Teach the `repomatic-ship` skill to reconcile bundled skills and agents as a third pass, judge a false-positive autofix PR against current `main` before writing a lint rule for it, and tell a superseded intra-cycle measurement from a genuine contradiction.
+- The tool runner now retries a download up to 3 times on transient network failures, instead of failing the job on a one-off TLS or truncation error.
+- Standalone binary tests now run for every healthy target when a sibling build fails, instead of being skipped wholesale.
 
 ## [`7.9.0` (2026-08-10)](https://github.com/kdeldycke/repomatic/compare/v7.8.0...v7.9.0)
 
+> [!WARNING]
+> The `windows-x64` binary shipped without its `.attestation.json` sidecar: a transient TLS failure on the runner skipped the upload, and immutable releases lock the asset list. The attestation itself is registered, so `gh attestation verify repomatic-7.9.0-windows-x64.exe --repo kdeldycke/repomatic` still verifies against GitHub's attestation service.
+
 - **Breaking:** `REPOMATIC_PAT` now requires `Administration: Read-only`, and `lint-repo` fails without it. Regenerate the token with the setup guide's pre-filled link; steps that cannot be verified now say so instead of vanishing.
 - New `[tool.repomatic] nuitka.dev-targets` option: an ordinary push now compiles binaries only for a canary subset, `["linux-arm64"]` by default. Release commits, a new weekly schedule and manual dispatches build the full 6-target fleet.
 - `lint-repo` now also verifies versionless `releases/latest/download` URLs in `docs/install.md`, catching a renamed asset that leaves the guide pointing at a 404.

docs/workflows.md

@@ -801,8 +801,8 @@ For how to *choose* what the matrix tests (covering the shipped config broadly w
 Whether a matrix job overrides the default `fail-fast: true` depends on what the cells produce, not on which workflow they live in. Three categories:
 
 1. **Asset-producing matrices that feed an immutable downstream artifact.** Each cell builds something the next job ships and cannot retroactively fix. Override to `fail-fast: false` so a transient runner crash on one cell does not cancel siblings whose output was already valid: shipping partial coverage is strictly better than shipping nothing. Downstream gates must accept `result != 'skipped'` (not `== 'success'`) so partial-success runs still flow through. **Applies to:** `compile-binaries` (binaries attached to the draft release before [§ Immutable releases](#immutable-releases) locks them).
-2. **Info-gathering matrices.** Each cell collects diagnostic data and the value of the run scales with how many cells reported. Override to `fail-fast: false` so a single failure does not erase the rest of the snapshot. **Applies to:** `tests` (per-cell `continue-on-error` already decides what fails the workflow), `dump-context`.
-3. **Advisory or single-cell matrices.** Tests that do not gate publication, validations, or matrices that typically run with one cell. Keep the default `fail-fast: true`: cancelling siblings on the first failure saves runner minutes, and a real regression is resolved by fixing the underlying code (then re-running) or, for already-published releases, by skipping that version (see [§ Immutable releases](#immutable-releases)) rather than by exhaustively diagnosing every platform up front. **Applies to:** `test-binaries`, `validate-arch`, and the single-cell publish-pipeline matrices (`build-package`, `create-tag`, `publish-pypi`, `create-release`, `publish-release`, `scan-virustotal`).
+2. **Info-gathering matrices.** Each cell collects diagnostic data and the value of the run scales with how many cells reported. Override to `fail-fast: false` so a single failure does not erase the rest of the snapshot. **Applies to:** `tests` (per-cell `continue-on-error` already decides what fails the workflow), `dump-context`, and `test-binaries` (gated with `always()` besides, so one failed build cell neither skips nor cancels the healthy targets' tests: its own cell fails on the missing artifact, which the advisory nature tolerates).
+3. **Advisory or single-cell matrices.** Tests that do not gate publication, validations, or matrices that typically run with one cell. Keep the default `fail-fast: true`: cancelling siblings on the first failure saves runner minutes, and a real regression is resolved by fixing the underlying code (then re-running) or, for already-published releases, by skipping that version (see [§ Immutable releases](#immutable-releases)) rather than by exhaustively diagnosing every platform up front. **Applies to:** `validate-arch` and the single-cell publish-pipeline matrices (`build-package`, `create-tag`, `publish-pypi`, `create-release`, `publish-release`, `scan-virustotal`).
 
 GitHub resolves a job's `strategy.matrix` during setup even when the job's `if:` guard will skip it, so a matrix expression that resolves to an empty or null value can abort the entire run with `Unexpected value ''` before `if:` is ever checked. This surfaces when a project disables binary builds (`nuitka.enabled = false` makes `nuitka_matrix` null), turning every non-release push red. Two triggers exist: a `workflow_call` output read as a bare string (an empty `release_commits_matrix` becomes `fromJSON('')`), and a matrix-derived `runs-on: ${{ matrix.os }}` that cannot resolve against an absent matrix. The fix is a fallback to a valid empty matrix. `matrix: ${{ ... || fromJSON('{"include":[]}') }}` expands the job to zero runs, so it skips cleanly instead of failing the workflow. `compile-binaries`, `test-binaries`, and the caller's `publish-pypi` job carry this fallback; a job that pins a static `runs-on` and has no other matrix-derived fields (like `create-tag`) already skips cleanly on a null matrix and needs none.
 

repomatic/tool_runner.py

@@ -44,6 +44,7 @@
 import sys
 import tarfile
 import tempfile
+import time
 import zipfile
 from contextlib import nullcontext
 from dataclasses import replace
@@ -305,6 +306,20 @@ def resolve_config(
 for small JSON API responses.
 """
 
+_DOWNLOAD_ATTEMPTS = 3
+"""Attempts at downloading one artifact before giving up.
+
+```{note}
+Runner egress is flaky in ways that clear on the next connection: `v7.9.0`
+lost its windows-x64 attestation sidecar to a one-off
+`CERTIFICATE_VERIFY_FAILED: self-signed certificate` from a Windows runner's
+TLS-intercepting proxy, and short bodies surface as the truncation guard's
+`OSError`. One artifact download aborting a whole release job over a
+transient is the wrong trade, so the verification seam retries with a short
+pause and lets only a repeated failure surface.
+```
+"""
+
 _DOWNLOAD_CHUNK_SIZE = 65536
 """Read size for streaming downloads and incremental hashing."""
 
@@ -386,15 +401,33 @@ def _download_and_verify(
 ) -> None:
     """Download a file and verify its SHA-256 checksum.
 
+    Network-level failures (TLS errors, timeouts, truncated bodies: all
+    `OSError` subclasses, `URLError` included) are retried up to
+    {data}`_DOWNLOAD_ATTEMPTS` times; a checksum mismatch on a complete body
+    is not, since identical bytes would fail identically.
+
     :param url: URL to download.
     :param expected_sha256: Expected lowercase hex SHA-256 digest.
         `None` skips verification (logs the computed digest for reference).
     :param dest_path: Where to write the downloaded file.
     :param label: Progress bar label. Defaults to the destination filename.
-    :raises OSError: If the body is shorter than the advertised `Content-Length`.
+    :raises OSError: If every attempt fails at the network level, including a
+        body shorter than the advertised `Content-Length`.
     :raises ValueError: If the checksum does not match.
     """
-    actual = download_to(url, dest_path, label=label)
+    for attempt in range(1, _DOWNLOAD_ATTEMPTS + 1):
+        try:
+            actual = download_to(url, dest_path, label=label)
+            break
+        except OSError as ex:
+            dest_path.unlink(missing_ok=True)
+            if attempt == _DOWNLOAD_ATTEMPTS:
+                raise
+            logging.warning(
+                f"Download attempt {attempt}/{_DOWNLOAD_ATTEMPTS} of {url}"
+                f" failed ({ex}); retrying."
+            )
+            time.sleep(2 * attempt)
     if expected_sha256 is None:
         logging.info("SHA-256 of %s: %s (not verified).", url, actual)
         return

tests/test_tool_runner.py

@@ -31,6 +31,7 @@
 from itertools import combinations
 from pathlib import Path
 from unittest.mock import MagicMock, patch
+from urllib.error import URLError
 
 import pytest
 import tomlrt
@@ -732,22 +733,69 @@ def test_download_and_verify_mismatch(tmp_path):
     assert not dest.exists()
 
 
+def test_download_and_verify_retries_transient_failure(tmp_path):
+    """A transient network failure is retried and the next attempt succeeds."""
+    content = b"hello binary world"
+    expected = hashlib.sha256(content).hexdigest()
+    dest = tmp_path / "downloaded"
+
+    with (
+        patch(
+            "repomatic.tool_runner.urlopen",
+            side_effect=[
+                URLError("certificate verify failed: self-signed certificate"),
+                _urlopen_response(content),
+            ],
+        ) as fake_urlopen,
+        patch("repomatic.tool_runner.time.sleep") as fake_sleep,
+    ):
+        _download_and_verify("https://example.com/file", expected, dest)
+
+    assert fake_urlopen.call_count == 2
+    assert fake_sleep.call_count == 1
+    assert dest.read_bytes() == content
+
+
+def test_download_and_verify_gives_up_after_attempts(tmp_path):
+    """A persistent network failure surfaces after the last attempt."""
+    dest = tmp_path / "downloaded"
+
+    with (
+        patch(
+            "repomatic.tool_runner.urlopen",
+            side_effect=URLError("connection reset"),
+        ) as fake_urlopen,
+        patch("repomatic.tool_runner.time.sleep"),
+        pytest.raises(URLError, match="connection reset"),
+    ):
+        _download_and_verify("https://example.com/file", "0" * 64, dest)
+
+    assert fake_urlopen.call_count == 3
+    assert not dest.exists()
+
+
 def test_download_and_verify_truncated(tmp_path):
     """A body shorter than Content-Length raises OSError, not a mismatch.
 
     A truncated transfer (proxy hiccup, dropped connection) hashes to a wrong
     digest, so it used to be reported as a SHA-256 mismatch: that reads as a
     stale pin or a tampered artifact when the registry checksum is correct.
+    Truncation is retried like any network failure, so the error only
+    surfaces once every attempt came up short.
     """
     content = b"hello binary world"
     dest = tmp_path / "downloaded"
 
     with (
         patch(
             "repomatic.tool_runner.urlopen",
-            # Advertise more bytes than the body delivers.
-            return_value=_urlopen_response(content, advertised_length=len(content) + 7),
+            # Advertise more bytes than the body delivers, on every attempt.
+            side_effect=[
+                _urlopen_response(content, advertised_length=len(content) + 7)
+                for _ in range(3)
+            ],
         ),
+        patch("repomatic.tool_runner.time.sleep"),
         pytest.raises(OSError, match="Truncated download .* got 18 of 25 bytes"),
     ):
         _download_and_verify(