Skip to content

ci(release): drop GHA build cache from docker push (unblock releases) - #1043

Merged
ch4r10t33r merged 1 commit into
mainfrom
fix/release-drop-gha-cache
Jul 14, 2026
Merged

ci(release): drop GHA build cache from docker push (unblock releases)#1043
ch4r10t33r merged 1 commit into
mainfrom
fix/release-drop-gha-cache

Conversation

@ch4r10t33r

Copy link
Copy Markdown
Contributor

Problem

The devnet5 release workflow has failed the last two releases (v0.5.8 and v0.5.9) at the docker push step:

#16 ERROR: failed to push docker.io/.../zeam:0.5.9-amd64: ... error writing layer blob: failed to reserve cache
ERROR: failed to build: failed to solve: error writing layer blob: failed to reserve cache

The tags get created but no image is published (create-manifest and create-github-release are then skipped).

Root cause

The Build and push Docker image step in auto-release.yml uses:

cache-from: type=gha
cache-to: type=gha,mode=max

failed to reserve cache is the GitHub Actions cache backend rejecting a reserveCache call. mode=max exports every layer of the large zeam image (the ~240 MB prover staticlib plus rust deps), and both matrix legs (amd64 + arm64) write to the same cache at once, exhausting the repo's 10 GB GHA cache quota. When the cache export fails, buildkit aborts the whole build-and-push solve, so the image never pushes. It is not transient flakiness: it fires at the same step on every run.

Fix

Remove the GHA build cache from the release push. Dockerfile.prebuilt only packages an already-built binary, so the layer cache saves almost nothing here, and a release must never be blocked by a best-effort build cache.

Follow-up (completing v0.5.9)

The v0.5.9 / Devnet5 git tags already exist from the failed run, and create-tags hard-fails if a version tag already exists. So after this merges, the stuck v0.5.9 release needs to be re-driven with the fixed workflow, either by deleting the dangling v0.5.9/Devnet5 tags and re-opening the release PR, or by cutting it as a new version label. I can do that once this is in.

The release build-and-push step used cache-to: type=gha,mode=max. On the
large zeam image (prover staticlib + rust deps), with both matrix legs
(amd64/arm64) writing concurrently, the GHA cache reservation fails once
the repo's cache quota is exhausted, and buildkit aborts the entire
build-and-push solve with:

  error writing layer blob: failed to reserve cache

That fails the image push and blocks the release entirely. v0.5.8 and
v0.5.9 both failed here. Dockerfile.prebuilt only packages an
already-built binary, so the layer cache saves almost nothing; a release
push must not be gated on a best-effort build cache. Remove it.
@ch4r10t33r
ch4r10t33r merged commit eff96a0 into main Jul 14, 2026
14 checks passed
@ch4r10t33r
ch4r10t33r deleted the fix/release-drop-gha-cache branch July 14, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants