Move _unittest.yml to linux_job_v3 - #22245
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22245
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit c417bb7 with merge base b46bd04 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
a698a8f to
828a13d
Compare
Consumer-side counterpart of the docker-builds move, landing here so the linux_job_v3 migration can build on it. linux_job_v3 passes docker-image straight to the runner pod's container, which is pulled before any step runs, so v2's in-job calculate-docker-image -- which resolved ci-image:<name> to an ECR reference -- has nowhere left to run. This emits the same tag docker-builds computes and callers name the image with it. Like pytorch's OSDC jobs, nothing waits for the images: a commit that changes .ci/docker needs the ciflow/docker label, and without it the pod fails to pull. Nothing calls this yet; the callers arrive in #22245 onward. Authored with Claude Code.
Part 1 of moving ExecuTorch's Linux CI to [OSDC](https://github.com/pytorch/test-infra/blob/main/docs/osdc_runners.md). The `linux_job_v2` -> `v3` migration follows in #22245 onward; #22108 covers the native jobs. OSDC pods have no docker daemon, so the builds move to the in-cluster BuildKit pool: `build.sh` switches to `docker buildx build --push` under `REMOTE_BUILDKIT`, driven by test-infra's `docker-build-remote-buildkit` action. The tag is computed in the workflow now, but it is the same string `calculate-docker-image` produced, so jobs still on v2 keep resolving. That action only retries failures from before BuildKit starts, so the three retries `calculate-docker-image` wrapped the build in are kept around the SDK downloads. sccache resolved its S3 credentials from EC2 instance metadata, which a BuildKit pod has none of, so every compiler invocation died on a metadata timeout. They now reach the build as a BuildKit secret, and `install_pytorch.sh` caches locally when it is absent. Dropping the S3 backend would have been the smaller change, but the PyTorch build layer takes ~19 minutes only because sccache serves nearly all of it, and uncached it would not fit the job timeout. `pull_request` gives way to `ciflow/docker`, since a fork PR gets no OIDC token and could never push. The tag is registered in `pytorch-probot.yml` and auto-applied on the paths that need a rebuild, so such a PR still rebuilds its images without anyone having to remember — the same rule pytorch has. Last commit adds `_docker-image.yml`, the consumer-side counterpart: it emits `git rev-parse HEAD:.ci/docker` for the v3 jobs to name their image with. Nothing calls it yet. All 16 images build and push green on OSDC in [run 32799247406](https://github.com/pytorch/executorch/actions/runs/32799247406): 16–48 min each against 37–56 on EC2, arm64 included. Authored with Claude Code.
828a13d to
ba98bce
Compare
|
The two The GH runner image creates
Fix: pytorch/test-infra#8665. The last commit here temporarily points |
…8665) `Clean workspace`, the first step of every v3 job, dies with a non-root image: ``` rm: cannot remove '/__w/executorch/executorch': Permission denied ``` and every later step skips. The GH hook creates `GITHUB_WORKSPACE` as uid 1001 ([runner-images#10936](actions/runner-images#10936)), while CI images commonly default to a uid 1000 user mirroring `ec2-user` — pytorch's `USER jenkins` and executorch's `USER ci-user` both do. It only surfaces with a non-root image, which is why jobs on `pytorch/almalinux-builder` have never hit it. Reproducer: [pytorch/executorch#22245](pytorch/executorch#22245), the first v3 caller to pass a repo CI image. **An OSDC pod is ephemeral, so there is nothing to clean.** The workspace starts empty — the failing `rm -rfv` printed no `removed` lines before erroring — and the step is carried over from v2, where EC2 runners were reused between jobs. What a job actually needs is to be able to write into the directory the hook created, so this chmods it and drops the removal, which is what pytorch/pytorch does in [`.github/actions/setup-linux`](https://github.com/pytorch/pytorch/blob/d3b8bf0a11cba909aac9e8cdf8b81af8d59a251e/.github/actions/setup-linux/action.yml#L53-L63) for its ARC path ([pytorch/pytorch#178973](pytorch/pytorch#178973)). Worth recording: adding a chmod *before* the `rm` does not work, and the test below caught it. Unlinking the workspace needs write on its parent `/__w/<repo>`, which chmodding the workspace does not grant — the chmod succeeded and the `rm` failed anyway. Root images can already write and often ship no `sudo` (`nvidia/cuda`, bare distro images), so the chmod tolerates failure rather than being gated on a user check. ## Tests `test_linux_job_v3.yml` covers both paths: - `test-docker-image` (`ghcr.io/pytorch/test-infra:cpu-x86_64-latest`) asserts `id -u` is 0, pinning the root path that must not require sudo. - `test-non-root-docker-image` is new, on `ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3.10-clang21` — a uid 1000 `USER jenkins` image. It asserts a non-root user, probes that the workspace is writable, and checks the checkout produced a git repo. It fails on `main` today. Authored with Claude Code.
6f3b947 to
ba98bce
Compare
ba98bce to
7b62802
Compare
First caller of _docker-image.yml. pull.yml and trunk.yml call this workflow, so they exercise the resolution on every PR. _docker-image.yml also gains a docker-registry output, so the ECR account and region are written once rather than at every call site. An output rather than an env, which GitHub does not allow in a reusable workflow's `with`. It stops short of the ci-image repository so that callers can keep passing `ci-image:<name>` as their docker-image input, which keeps each PR in this stack landable on its own. Authored with Claude Code.
7b62802 to
176a6a8
Compare
|
The three
Under v2 the script ran through a Verified directly: Last commit drops the variable from the environment the tests hand to the CLI, so they assert what they mean wherever they run. I fixed the test rather than the script, since the script's behaviour is correct — writing to |
gather_benchmark_configs.py writes its matrix to $GITHUB_OUTPUT when that is set and only prints it when it is not, so the three CLI tests that assert against stdout depend on the variable being absent or its path unwritable. Under linux_job_v2 the script ran through a docker exec where $GITHUB_OUTPUT pointed at a host path the container could not see, so set_output fell back to printing and the tests passed. v3 runs the script in the same container as the runner, where the file is present and writable, so the matrix goes to the file and stdout has nothing to match. Drop the variable from the environment the tests hand to the CLI, so they assert what they mean regardless of where they run. Authored with Claude Code.
630fe05 to
c417bb7
Compare
First of six splitting up #22107, which was too large to review in one piece. Each PR stacks on the previous one.
_docker-image.yml, which this depends on, now lands in #22106 instead._unittest.ymlis its first caller, so pull.yml and trunk.yml exercise the resolution on every PR. Thedocker-imageinput still takesci-image:<name>, so its own callers are unchanged; this workflow expands that into the full ECR reference.Authored with Claude Code.