build(base-image): make apt install resilient to transient mirror failures - #42221
Draft
subrata71 wants to merge 1 commit into
Draft
build(base-image): make apt install resilient to transient mirror failures#42221subrata71 wants to merge 1 commit into
subrata71 wants to merge 1 commit into
Conversation
…lures ubuntu:24.04 ships no apt retry config (default is 0 retries), so a single dropped connection to the Ubuntu mirrors fails the whole base-image build. Recent flake: appsmith-ee run 34570186320 (IPv6 unreachable + IPv4 timeouts to archive.ubuntu.com), and a same-SHA run failed then passed on 2026-09-05. - Add a build-scoped apt drop-in (Acquire::Retries 3 + http/https Timeout 30s), removed in the same layer's cleanup so the shipped image is unchanged. - Add --retry/--retry-connrefused/--connect-timeout/--retry-max-time to the MongoDB and PostgreSQL GPG-key curls; add --fail to the PostgreSQL one so an HTTP error body is never piped into apt-key. Reduces flake frequency for transient mirror blips; does not rescue a sustained egress outage. CE-only: apt block is byte-identical CE/EE, sync carries to EE. Linear: APP-15960
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
TL;DR: Make the base-image
aptinstall resilient to transient Ubuntu-mirror connection failures, so a momentary network blip on the build host doesn't red the wholeDocker Base Imagebuild.Background / root cause
The
Docker Base Imageworkflow (deploy/docker/base.dockerfile) intermittently fails at the apt dependency-install layer when the builder briefly can't reach the Ubuntu mirrors. Most recent example: appsmith-ee run 34570186320 failed twice withconnect (101: Network is unreachable)(IPv6) andconnection timed out(IPv4) toarchive.ubuntu.com/security.ubuntu.com. The same base-image build has flaked on apt before (a same-SHA run failed, then passed, on 2026-09-05).Verified cause:
ubuntu:24.04ships no apt retry configuration —apt-config dumpshows noAcquire::Retriesand there is no drop-in in/etc/apt/apt.conf.d/, so the compiled default of 0 retries applies. A single dropped connection fails the build. The two GPG-keycurlfetches also had no retry, and the PostgreSQL one lacked--fail(so an HTTP error body could be piped intoapt-key).Changes (
deploy/docker/base.dockerfile, apt layer only)Acquire::Retries "3"+Acquire::http(s)::Timeout "30". It is deleted in the same layer's cleanup (rm -rf), so the shipped image's apt behavior is unchanged.--retry 3 --retry-connrefused --connect-timeout 15 --retry-max-time 60to the MongoDB and PostgreSQL key-fetch curls; add--failto the PostgreSQL one.Acquire::ForceIPv4(IPv4 also timed out in the incident, so it wouldn't help), and no change to the deprecatedapt-keyusage (out of scope).This matches the retry pattern already used in this file (the Keycloak jar overlay uses
curl --fail --retry 3 --connect-timeout 15).Scope / honest limitation
This reduces flake frequency for transient mirror blips. It will not rescue a sustained multi-minute total egress outage — retries only help if egress recovers within the retry window. That class of failure is infra, not the Dockerfile.
Verification
ubuntu:24.04default confirmed: noAcquire::Retries, no apt.conf.d retry drop-in → default 0.\-continuation +printfdrop-in + cleanup) withdocker build:apt-config dumpreportsAcquire::Retries "3"; the drop-in is removed by cleanup (test ! -fpasses); build prints success. BuildKit strips the inline#comment lines before the shell runs (same idiom already in this file).curlflags accepted byubuntu:24.04's curl (connect failure exit 7 with 3 retries observed; no unknown-option error).Impact on existing instances
None. The drop-in is created and deleted within the same build layer, so the produced image is byte-equivalent in apt configuration to before. Fresh install, upgrade-from-default, upgrade-from-customized, and rollback are all unaffected (this only changes how the base image is built, not its contents).
Reviewers / second opinion
Approach independently reviewed by GPT-5.6 sol and reconciled: retries tuned to 3 (not 5),
Acquire::Retries::Delaydropped as redundant, drop-in build-scoped rather than persisted,--failnot duplicated on the mongo curl (already has-f), curl retries bounded, IPv4 not forced.Linear: https://linear.app/appsmith/issue/APP-15960
Automation
/ok-to-test tags="@tag.All"
Communication
Should the DevRel and Marketing teams inform users about this change?