Move docker-builds to OSDC runners (#22106)

Part 1 of moving ExecuTorch's Linux CI to
[OSDC](https://github.com/pytorch/test-infra/blob/main/docs/osdc_runners.md).
The `linux_job_v2` -> `v3` migration follows in #22245 onward; #22108
covers the native jobs.

OSDC pods have no docker daemon, so the builds move to the in-cluster
BuildKit pool: `build.sh` switches to `docker buildx build --push` under
`REMOTE_BUILDKIT`, driven by test-infra's `docker-build-remote-buildkit`
action. The tag is computed in the workflow now, but it is the same
string `calculate-docker-image` produced, so jobs still on v2 keep
resolving. That action only retries failures from before BuildKit
starts, so the three retries `calculate-docker-image` wrapped the build
in are kept around the SDK downloads.

sccache resolved its S3 credentials from EC2 instance metadata, which a
BuildKit pod has none of, so every compiler invocation died on a
metadata timeout. They now reach the build as a BuildKit secret, and
`install_pytorch.sh` caches locally when it is absent. Dropping the S3
backend would have been the smaller change, but the PyTorch build layer
takes ~19 minutes only because sccache serves nearly all of it, and
uncached it would not fit the job timeout.

`pull_request` gives way to `ciflow/docker`, since a fork PR gets no
OIDC token and could never push. The tag is registered in
`pytorch-probot.yml` and auto-applied on the paths that need a rebuild,
so such a PR still rebuilds its images without anyone having to remember
— the same rule pytorch has.

Last commit adds `_docker-image.yml`, the consumer-side counterpart: it
emits `git rev-parse HEAD:.ci/docker` for the v3 jobs to name their
image with. Nothing calls it yet.

All 16 images build and push green on OSDC in [run
32799247406](https://github.com/pytorch/executorch/actions/runs/32799247406):
16–48 min each against 37–56 on EC2, arm64 included.

Authored with Claude Code.

Author: huydhn

.ci/docker/build.sh

@@ -116,7 +116,16 @@ fi
 # Copy requirements-lintrunner.txt from root to here
 cp ../../requirements-lintrunner.txt ./
 
-docker build \
+# OSDC runners have no docker daemon, so the build runs on the in-cluster
+# BuildKit pool via a remote buildx builder. That builder has nowhere to load
+# an image into, so the result has to go straight to the registry.
+if [[ -n "${REMOTE_BUILDKIT:-}" ]]; then
+  BUILD_CMD=(docker buildx build --push)
+else
+  BUILD_CMD=(docker build)
+fi
+
+"${BUILD_CMD[@]}" \
   --no-cache \
   --progress=plain \
   --build-arg "OS_VERSION=${OS_VERSION}" \

.ci/docker/common/install_pytorch.sh

@@ -10,6 +10,19 @@ set -ex
 # shellcheck source=/dev/null
 source "$(dirname "${BASH_SOURCE[0]}")/utils.sh"
 
+# The compiler stubs run through sccache, which fails hard rather than
+# compiling uncached when it cannot reach its S3 bucket. Use the credentials
+# docker-builds mounts, or fall back to a local cache so a build without them
+# still works.
+SCCACHE_CREDENTIALS=/run/secrets/aws-credentials
+if [[ -s "${SCCACHE_CREDENTIALS}" ]]; then
+  export AWS_SHARED_CREDENTIALS_FILE="${SCCACHE_CREDENTIALS}"
+else
+  echo "No sccache credentials; caching compiler output locally" >&2
+  unset SCCACHE_BUCKET SCCACHE_S3_KEY_PREFIX
+  export SCCACHE_DIR=/tmp/sccache
+fi
+
 install_domains() {
   echo "Install torchvision and torchaudio"
   pip_install --no-build-isolation --user "git+https://github.com/pytorch/audio.git@${TORCHAUDIO_VERSION}"

.ci/docker/ubuntu/Dockerfile

@@ -70,7 +70,12 @@ ARG SKIP_PYTORCH
 ARG PYTORCH_BUILD_MAX_JOBS
 COPY ./common/install_pytorch.sh install_pytorch.sh
 COPY ./common/utils.sh utils.sh
-RUN if [ -z "${SKIP_PYTORCH}" ]; then bash ./install_pytorch.sh; fi && rm install_pytorch.sh utils.sh
+# A host docker build let sccache reach its S3 bucket with the EC2 instance
+# role; a BuildKit pod has no instance metadata, so docker-builds passes the
+# credentials in as this secret. World-readable because the sccache server runs
+# as ci-user; a secret mount never lands in a layer.
+RUN --mount=type=secret,id=aws-credentials,mode=0444 \
+    if [ -z "${SKIP_PYTORCH}" ]; then bash ./install_pytorch.sh; fi && rm install_pytorch.sh utils.sh
 
 ARG LINTRUNNER
 # Install lintrunner if needed

.github/labeler.yml

@@ -13,3 +13,12 @@ ciflow/trunk:
   - any-glob-to-any-file:
     - 'backends/arm/**'
     - 'examples/arm/**'
+
+# Same paths docker-builds triggers on. It has no pull_request trigger, so this
+# label is what rebuilds the images for a PR that changes them.
+ciflow/docker:
+- changed-files:
+  - any-glob-to-any-file:
+    - '.ci/docker/**'
+    - '.github/workflows/docker-builds.yml'
+    - 'requirements-lintrunner.txt'

.github/pytorch-probot.yml

@@ -5,6 +5,7 @@ ciflow_push_tags:
 - ciflow/apple
 - ciflow/cuda
 - ciflow/cuda-perf
+- ciflow/docker
 - ciflow/metal
 - ciflow/mlx
 - ciflow/rocm

.github/workflows/_docker-image.yml

@@ -0,0 +1,40 @@
+name: Resolve CI docker image
+
+# linux_job_v3 passes `docker-image` straight to the runner pod's container, which
+# is pulled before any step runs, so nothing inside the job can derive it. Callers
+# depend on this workflow and name the image as
+# 308535385114.dkr.ecr.us-east-1.amazonaws.com/executorch/ci-image:<name>-<hash>.
+#
+# Nothing here waits for docker-builds. A commit that changes .ci/docker needs the
+# ciflow/docker label to build the images it asks for; without them the pod fails
+# to pull and the job has to be re-run once they land.
+
+on:
+  workflow_call:
+    outputs:
+      ci-docker-hash:
+        description: Tag suffix shared by every executorch CI image for this commit.
+        value: ${{ jobs.resolve.outputs.ci-docker-hash }}
+
+permissions:
+  contents: read
+
+jobs:
+  resolve:
+    name: resolve
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    outputs:
+      ci-docker-hash: ${{ steps.hash.outputs.ci-docker-hash }}
+
+    steps:
+      - name: Checkout ExecuTorch
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
+
+      - name: Compute the docker tag
+        id: hash
+        run: |
+          set -eu
+          echo "ci-docker-hash=$(git rev-parse HEAD:.ci/docker)" >> "$GITHUB_OUTPUT"

.github/workflows/docker-builds.yml

@@ -1,16 +1,17 @@
 name: docker-builds
 
+# No pull_request trigger: a fork PR gets no OIDC token, so the build could not
+# assume role/arc to push and every run would fail. A PR that needs fresh images
+# gets the ciflow/docker label instead, which tags the PR head and fires the push
+# trigger below with the PR's own SHA. Matches pytorch/pytorch.
 on:
   workflow_dispatch:
-  pull_request:
-    paths:
-      - .ci/docker/**
-      - .github/workflows/docker-builds.yml
-      - requirements-lintrunner.txt
   push:
     branches:
       - main
       - release/*
+    tags:
+      - ciflow/docker/*
     paths:
       - .ci/docker/**
       - .github/workflows/docker-builds.yml
@@ -19,19 +20,30 @@ on:
     - cron: 1 3 * * 3
 
 concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref_name }}-${{ github.ref_type == 'branch' && github.sha }}-${{ github.event_name == 'workflow_dispatch' }}-${{ github.event_name == 'schedule' }}
+  group: ${{ github.workflow }}-${{ github.ref_name }}-${{ github.ref_type == 'branch' && github.sha }}-${{ github.event_name == 'workflow_dispatch' }}-${{ github.event_name == 'schedule' }}
   cancel-in-progress: true
 
 env:
   AWS_DEFAULT_REGION: us-east-1
+  ECR_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
+  ECR_REPOSITORY: executorch/ci-image
+
+permissions:
+  id-token: write
+  contents: read
 
 jobs:
   docker-build:
     timeout-minutes: 240
     strategy:
       fail-fast: false
       matrix:
-        runner: [linux.4xlarge]
+        # BuildKit runs out of cluster on the OSDC BuildKit pool; this runner is
+        # only an orchestrator that streams the (tiny) .ci/docker context and
+        # waits, so it can be small. The architecture still has to match, because
+        # the helper picks the per-architecture buildkitd address off `uname -m`,
+        # and on arm64 this is the smallest label that actually schedules.
+        runner: [mt-l-x86iavx512-8-64]
         docker-image-name: [
           executorch-ubuntu-22.04-gcc11,
           executorch-ubuntu-22.04-gcc9-nopytorch,
@@ -45,49 +57,90 @@ jobs:
           executorch-ubuntu-22.04-clang12-android,
           executorch-ubuntu-24.04-gcc14,
           executorch-ubuntu-26.04-gcc14,
+          # Built on x86: this image only cross-compiles for CUDA on Windows,
+          # so building it never needed the GPU runner it used to run on.
+          executorch-ubuntu-22.04-cuda-windows,
         ]
         include:
           - docker-image-name: executorch-ubuntu-22.04-gcc11-aarch64
-            runner: linux.arm64.2xlarge
+            runner: mt-l-arm64g4-16-62
           - docker-image-name: executorch-ubuntu-22.04-gcc11-aarch64-android
-            runner: linux.arm64.2xlarge
+            runner: mt-l-arm64g4-16-62
           - docker-image-name: executorch-ubuntu-22.04-gcc11-aarch64-arm-sdk
-            runner: linux.arm64.2xlarge
-          - docker-image-name: executorch-ubuntu-22.04-cuda-windows
-            runner: linux.g5.4xlarge.nvidia.gpu
+            runner: mt-l-arm64g4-16-62
 
-    runs-on: [self-hosted, "${{ matrix.runner }}"]
-    env:
-      DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/executorch/${{ matrix.docker-image-name }}
+    runs-on: ${{ matrix.runner }}
+    container:
+      image: ghcr.io/actions/actions-runner:latest
     steps:
-      - name: Clean workspace
-        shell: bash
-        run: |
-          echo "${GITHUB_WORKSPACE}"
-          sudo rm -rf "${GITHUB_WORKSPACE}"
-          mkdir "${GITHUB_WORKSPACE}"
+      - name: Checkout ExecuTorch
+        uses: actions/checkout@v4
 
-      - name: Setup SSH (Click me for login details)
-        uses: pytorch/test-infra/.github/actions/setup-ssh@main
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v4
         with:
-          github-secret: ${{ secrets.GITHUB_TOKEN }}
+          role-to-assume: arn:aws:iam::308535385114:role/arc
+          aws-region: us-east-1
+          role-duration-seconds: 18000
 
-      - name: Checkout Executorch
-        uses: actions/checkout@v3
+      # buildx forwards the client's registry auth to the remote builder, so the
+      # push at the end of build.sh authenticates with what this step writes.
+      - name: Login to ECR
+        uses: aws-actions/amazon-ecr-login@v2
+        with:
+          registries: "308535385114"
 
-      - name: Setup Linux
-        uses: pytorch/test-infra/.github/actions/setup-linux@main
+      # sccache cannot source S3 credentials from instance metadata inside a
+      # BuildKit pod, so hand the assumed-role ones to the build as a secret. An
+      # empty file is a valid outcome: install_pytorch.sh then caches locally.
+      - name: Stage sccache credentials
+        id: sccache-credentials
+        shell: bash
+        run: |
+          set -euo pipefail
+          creds="${RUNNER_TEMP}/aws-credentials"
+          install -m 600 /dev/null "${creds}"
+          if [[ -n "${AWS_ACCESS_KEY_ID:-}" ]]; then
+            {
+              echo "[default]"
+              echo "aws_access_key_id=${AWS_ACCESS_KEY_ID}"
+              echo "aws_secret_access_key=${AWS_SECRET_ACCESS_KEY}"
+              echo "aws_session_token=${AWS_SESSION_TOKEN}"
+            } > "${creds}"
+          else
+            echo "::warning::No AWS credentials; the image build will not use the sccache S3 cache"
+          fi
+          echo "path=${creds}" >> "${GITHUB_OUTPUT}"
 
-      - name: Build docker image
-        id: build-docker-image
-        timeout-minutes: 145
-        uses: pytorch/test-infra/.github/actions/calculate-docker-image@main
-        with:
-          docker-image-name: ci-image:${{ matrix.docker-image-name }}
-          always-rebuild: true
-          push: true
-          force-push: true
+      - name: Compute the image tag
+        id: tag
+        shell: bash
+        run: |
+          set -eux
+          git config --global --add safe.directory "${GITHUB_WORKSPACE}"
+          # Same tag that calculate-docker-image derives for consumers of
+          # `ci-image:<name>`, so jobs still on linux_job_v2 keep resolving.
+          DOCKER_TAG=$(git rev-parse HEAD:.ci/docker)
+          echo "docker-image=${ECR_REGISTRY}/${ECR_REPOSITORY}:${{ matrix.docker-image-name }}-${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
 
-      - name: Teardown Linux
-        uses: pytorch/test-infra/.github/actions/teardown-linux@main
-        if: always()
+      # No step timeout: the action retries a cold BuildKit pool for at least two
+      # hours, and the job's timeout-minutes is what bounds that.
+      - name: Build and push to ECR
+        uses: pytorch/test-infra/.github/actions/docker-build-remote-buildkit@main
+        with:
+          # calculate-docker-image wrapped the build in three retries because it
+          # "frequently fails with network error downloading various stuffs".
+          # The action only retries failures from before BuildKit starts, so keep
+          # a retry of our own around the SDK downloads.
+          command: |
+            cd .ci/docker
+            for attempt in 1 2 3; do
+              if REMOTE_BUILDKIT=1 ./build.sh ${{ matrix.docker-image-name }} \
+                  -t ${{ steps.tag.outputs.docker-image }} \
+                  --secret id=aws-credentials,src=${{ steps.sccache-credentials.outputs.path }}; then
+                exit 0
+              fi
+              echo "::warning::docker build attempt ${attempt} of 3 failed"
+              if [ "${attempt}" -lt 3 ]; then sleep 90; fi
+            done
+            exit 1