Skip to content

Move pull.yml to linux_job_v3 - #22247

Open
huydhn wants to merge 3 commits into
osdc/v3-reusablefrom
osdc/v3-pull
Open

Move pull.yml to linux_job_v3#22247
huydhn wants to merge 3 commits into
osdc/v3-reusablefrom
osdc/v3-pull

Conversation

@huydhn

@huydhn huydhn commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Third of six splitting up #22107. Stacked on #22246.

45 call sites in one file, all mechanical: the v2 -> v3 rename, EC2 runner labels swapped for their OSDC equivalents, use-custom-docker-registry dropped since v3 ignores it, and the ECR image spelled out against the hash _docker-image.yml resolves.

EC2 OSDC
linux.2xlarge, linux.2xlarge.memory mt-l-x86iavx512-8-64
linux.4xlarge.memory mt-l-x86iavx512-16-128
linux.24xlarge mt-l-x86iavx512-94-192
linux.24xlarge.memory mt-l-x86iavx512-94-768
linux.arm64.2xlarge mt-l-arm64g4-16-62
linux.g5.4xlarge.nvidia.gpu mt-l-x86aavx2-29-113-a10g

Authored with Claude Code.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

@pytorch-bot

pytorch-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22247

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Unrelated Failures, 1 Unclassified Failure

As of commit 01e69e9 with merge base f665a3b (image):

NEW FAILURE - The following job has failed:

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 28, 2026
@huydhn
huydhn requested a review from shoumikhin August 31, 2026 22:12
@huydhn
huydhn marked this pull request as ready for review August 31, 2026 22:13
@huydhn
huydhn requested a review from digantdesai as a code owner August 31, 2026 22:55
@github-actions github-actions Bot added ciflow/trunk module: arm Issues related to arm backend labels Aug 31, 2026
45 call sites, all mechanical: the v2 -> v3 rename, EC2 runner labels swapped
for their OSDC equivalents, and the ECR image spelled out against the hash
_docker-image.yml resolves.

Authored with Claude Code.
@huydhn

huydhn commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Last commit temporarily points pull.yml's 45 linux_job_v3 refs at pytorch/test-infra#8683 (osdc/hf-cache-writable) to exercise the HF cache mode fix. Reverts to @main before landing.

The job to watch is test-models-linux (ic4, portable, ...), which is one of the ones failing with Errno 30 Read-only file system on models--timm--inception_v4.tf_in1k.

Worth knowing what this run can and cannot prove: this PR triggers a pull_request run, where the label is already visible in the event payload, so it exercises the refresh path but not the ciflow-tag resolution #8683 adds. It also will not seed the shared cache — meta-pytorch/pytorch-gha-infra#1463 is still open, so the sync step fails assume-role and is skipped. So a pass here means "the refresh path downloads what it needs and the tests go green", not "the cache is fixed".

`auto` and `logical` both resolve through psutil.cpu_count(), which reports the
machine's cores and ignores the container's CPU limit. On an 8 vCPU OSDC pod that
meant 64 workers in test-arm-backend-no-driver, which exhausted the memory limit
and got the container OOMKilled, and enough workers in unittest-nxp-neutron to
starve the aot_neutron_compile tests past their 300 second subprocess timeout.

pytest-parallelism.sh exports PYTEST_XDIST_AUTO_NUM_WORKERS, which xdist checks
ahead of psutil, from nproc. nproc honours the pod's cpuset, which is what
pytorch relies on for OMP_NUM_THREADS on the same fleet (.ci/pytorch/test.sh,
USE_ARC block). Off OSDC it is the machine count, so this is a no-op there.

Sourced from the four scripts that let xdist size itself. The variable governs
`-n logical` as well, despite its name.

Authored with Claude Code.
Exercises the HF cache mode fix on test-models-linux, which is one of the jobs
failing on the read-only mount. Revert to @main before landing.

Authored with Claude Code.
@huydhn

huydhn commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Pushed the unittest-nxp-neutron fix, folded into the existing worker-cap commit rather than added on top, since it is the same fix reaching one more call site.

backends/nxp/run_unittests.sh runs pytest -c /dev/null -n "logical", and logical resolves through psutil.cpu_count() just like auto does — so it fanned out to the node's core count on an 8 vCPU pod and starved the aot_neutron_compile tests past their hardcoded 300s subprocess timeout. My original sweep grepped for auto only and missed this spelling.

Verified the variable governs logical too, despite its name:

-n logical, no env var           -> 16 workers
PYTEST_XDIST_AUTO_NUM_WORKERS=3  ->  3 workers

Re-grepped for both spellings across .ci, backends, examples and extension. The remaining hits are unittest-macos-cmake.sh (macOS, dedicated VMs, unaffected), a docstring in backends/test/suite/flows/nxp.py, and two READMEs — so all four executable call sites are now covered.

From the previous run on this branch, the two job families that were failing are green: test-models-linux across the whole matrix including ic4 (was Errno 30 Read-only file system), and all four test-arm-backend-no-driver cells on mt-l-x86iavx512-8-64 (was OOMKilled at 64 workers). The only other failure in that run was test-models-macos-cpu (resnet50), a 502 from gitlab.arm.com fetching KleidiAI — unrelated to the migration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-refresh-hf-cache ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: arm Issues related to arm backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant