cloud-build-docker: reproducible image digests via buildx#34
Open
jwbron wants to merge 2 commits into
Open
Conversation
Pass SOURCE_DATE_EPOCH=0 to the BuildKit build so the resulting image digest is a pure function of content. Without it, the Dockerfile frontend stamps the image config — the top-level `created` field and every `history[].created` entry — with the current wall-clock time. Two builds of byte-identical content (identical layers, even a 100% cache hit) therefore produce different image *config* blobs and thus different manifest digests. Anything that pins images by digest then sees a phantom change on every rebuild. In Khan/internal-services this kept the GitHub Actions Runner Cloud Run job's image digest "drifting" between every terraform plan: the runner image was rebuilt on each plan, got a new digest purely from the timestamp, and the apply -> chained re-plan loop turned each one into a fresh "image digest update" PR even though nothing in the image had actually changed. Clamping the epoch to a constant makes the config timestamps deterministic, so the digest changes only when image content changes. This addresses the config/history non-determinism specifically (which is what was observed). Layer file mtimes for *rebuilt* layers are also clamped by SOURCE_DATE_EPOCH; full base-image timestamp rewriting would additionally need the BuildKit `rewrite-timestamp` image-exporter option, which requires a buildx migration and is out of scope here.
The earlier commit on this branch only passed --build-arg
SOURCE_DATE_EPOCH=0. Testing showed that is insufficient: it clamps the
image config and history timestamps, but file mtimes inside rebuilt
layer blobs still drift, so two builds of identical content still
produce different digests on any cache-miss rebuild.
Full reproducibility additionally needs the image exporter's
rewrite-timestamp option, which clamps layer file mtimes. That option
requires the docker-container buildx driver (the default "docker"
driver does not support it), so this reworks the build from
`docker build` to `docker buildx build`:
- docker-container driver, so rewrite-timestamp is available
- SOURCE_DATE_EPOCH=0 + rewrite-timestamp=true: clamp every timestamp
(config, history, and layer file mtimes) to a constant
- --provenance=false --sbom=false: attestations embed build metadata
that is non-deterministic and would defeat the clamping
- --cache-from type=registry / --cache-to type=inline: replaces the
old `docker pull` + BUILDKIT_INLINE_CACHE build-arg; same
branch-based layer caching
- --output type=image,push=true: replaces the `images:` block; only
$_IMAGE_TAG is ever pushed (no cache-tag clobber)
Verified locally: two --no-cache builds, and a rebuild with the local
cache pruned and re-imported from the registry, all produce the
identical pushed manifest digest.
jwbron
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks
cloud-build-docker'scloudbuild.ymlso the image digest it produces is a pure function of build content — two builds of identical sources yield the identical pushed digest.BuildKit by default stamps the image config, per-step
historyentries, and file mtimes inside layers with wall-clock time, so byte-identical content produces a different digest on every build. Anything that pins images by digest (e.g. a Cloud Run job) then sees spurious "updates" on every rebuild.What changed
Switches from
docker buildtodocker buildx buildwith the docker-container driver, and:--build-arg SOURCE_DATE_EPOCH=0— clamps the image config +history[].createdtimestamps--output type=image,...,rewrite-timestamp=true— clamps file mtimes inside the layer blobs (requires the docker-container driver; the defaultdockerdriver can't do it)--provenance=false --sbom=false— attestations embed build metadata (timestamps, build IDs) that is non-deterministic and would defeat the clamping--cache-from type=registry/--cache-to type=inline— replacesdocker pull+ theBUILDKIT_INLINE_CACHEbuild-arg; same branch-based layer caching--output type=image,push=true— replaces theimages:block; only$_IMAGE_TAGis ever pushed (so no cache-tag clobber — supersedes the earlier clobber fix)Why the first commit wasn't enough
The first commit on this branch only added
--build-arg SOURCE_DATE_EPOCH=0. Testing showed that clamps config/history timestamps but not layer file mtimes — two--no-cachebuilds still diverged (theRUN-layer blob differed).rewrite-timestampis the missing piece, and it forces thebuildxrework.Verification (local, Docker 29.4 / BuildKit)
SOURCE_DATE_EPOCHbuild-arg onlySOURCE_DATE_EPOCH+rewrite-timestamp+--provenance=falseCACHEDTest plan
cloud-build-docker-v0.5.0) after merge.gcr.io/cloud-builders/dockerin Cloud Build providesdocker buildxand thatdocker buildx create --driver docker-containerworks in that environment.--output ...,push=truefrom the docker-container builder authenticates to Artifact Registry (buildx forwards the docker CLI credentials).