Skip to content

CICD NeMo

CICD NeMo #18530

Manually triggered July 22, 2026 19:58
Status Success
Total duration 3h 37m 31s
Artifacts 51

cicd-main.yml

on: workflow_dispatch
pre-flight  /  pre-flight
11s
pre-flight / pre-flight
configure
4s
configure
Lint check
20s
Lint check
cicd-wait-in-queue
4s
cicd-wait-in-queue
cicd-compute-build-matrix
3s
cicd-compute-build-matrix
Matrix: cicd-container-build
Launch_Unit_Tests_Core
8m 11s
Launch_Unit_Tests_Core
Launch_Unit_Tests_Diffusion
3m 0s
Launch_Unit_Tests_Diffusion
generate-test-matrix
6s
generate-test-matrix
generate-gb200-test-matrix
0s
generate-gb200-test-matrix
Launch_Import_Check
1m 34s
Launch_Import_Check
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-flaky
Nemo_CICD_Test
9s
Nemo_CICD_Test
Coverage_Fake
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
svcnvidia-nemo-ci
approved Jul 22, 2026
test

Automatically approved by queue manager

Annotations

1 error, 8 warnings, and 51 notices
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: The directory '/home/runner/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
pre-flight / pre-flight
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, step-security/changed-files@v45.0.1. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@v5, docker/setup-buildx-action@v3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
L0_Launch_recipes_gemma4_vl
Back off 27.27 seconds before retry.
L0_Launch_recipes_gemma4_vl
Failed to download action 'https://codeload.github.com/actions/checkout/tar.gz/d23441a48e516b6c34aea4fa41551a30e30af803'. Error: Response status code does not indicate success: 429 (Too Many Requests). E794:1D0935:1E5FC:3294F:6A614819
L0_Launch_converter_generate
Back off 13.05 seconds before retry.
L0_Launch_converter_generate
Failed to download action 'https://codeload.github.com/actions/checkout/tar.gz/d23441a48e516b6c34aea4fa41551a30e30af803'. Error: Response status code does not indicate success: 429 (Too Many Requests). 3EF7:FA4F9:1EA48:32ECD:6A614831
L0_Launch_data
Back off 17.61 seconds before retry.
L0_Launch_data
Failed to download action 'https://codeload.github.com/actions/checkout/tar.gz/d23441a48e516b6c34aea4fa41551a30e30af803'. Error: Response status code does not indicate success: 429 (Too Many Requests). 4002:240C92:1FA14:346BF:6A61485F
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ L1_Launch_models_llada15 — PASSED
Result
✅ L1_Launch_models_qwen3_omni — PASSED
Result
✅ L1_Launch_models_gemma3 — PASSED
Result
✅ L1_Launch_models_qwen25_vl — PASSED
Result
✅ L1_Launch_recipes_glm_vl — PASSED
Result
✅ L1_Launch_models_glm_vl — PASSED
Result
✅ L1_Launch_models_stepfun — PASSED
Result
✅ L1_Launch_models_qwen35_vl — PASSED
Result
✅ L1_Launch_models_glm — PASSED
Result
✅ L0_Launch_converter_fsdp — PASSED
Result
✅ L1_Launch_recipes_gemma — PASSED
Result
✅ L1_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_megatron_fsdp — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L1_Launch_models_qwen2 — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L1_Launch_models_qwen3_vl — PASSED
Result
✅ L0_Launch_training_callbacks — PASSED
Result
✅ L1_Launch_models_gpt_oss — PASSED
Result
✅ L1_Launch_models_nemotron — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_models_gemma4_vl — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L1_Launch_models_ministral3 — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_converter_generate — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_models_nemotronh — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L1_Launch_recipes_llama_3b — PASSED
Result
✅ L1_Launch_recipes_ministral3 — PASSED
Result
✅ L1_Launch_recipes_nemotron_omni — PASSED
Result
✅ L1_Launch_recipes_gpt_oss — PASSED
Result
✅ L1_Launch_recipes_qwen — PASSED
Result
✅ L1_Launch_recipes_nemotronh — PASSED
Result
✅ L0_Launch_training_distributed_save — PASSED
Result
✅ L0_Launch_training_decentralized_pg — PASSED
Result
✅ L1_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_training_local_checkpointing — PASSED
Result
✅ L0_Launch_training_pretrain — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_training_resiliency — PASSED
Result
✅ L0_Launch_training_tensor_inspect — PASSED
Result
✅ L0_Launch_training_finetune — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L1_Launch_recipes_qwen_vl — PASSED

Artifacts

Produced during runtime
Name Size Digest
coverage-e2e-29953218008-01d733b3-5929-4c36-83e2-f22ebbffa552
743 KB
sha256:e6c49ce9bf715d4219531fcd63302998e1bb8c8d8f3e9b02cdd47a429863bd46
coverage-e2e-29953218008-05be9638-2cf6-4ef5-a1cc-34fcc101e625
316 KB
sha256:b313b2c2b4a0e6dc7234685aa43b8229ca62434d86ccaa8ea6ef0a152cd05ca4
coverage-e2e-29953218008-094d881d-a2e6-4f26-aeff-58ce6216caf5
384 KB
sha256:5e3c083b50f632a40a13741511824625850f86ab147a3111b742acd56b971f84
coverage-e2e-29953218008-0a8f59d8-e909-43a3-b635-b79dc63233ab
809 KB
sha256:eadb8940825693f0fada2c1bbdaa8acb8afef326f9c28c18163d68cff334f137
coverage-e2e-29953218008-0f325a08-b42a-4221-96e7-30d45b44f259
742 KB
sha256:7100b1f4b2f7cd09125c7c9e409260749d52166d74ad19a4579271598bf14c9d
coverage-e2e-29953218008-0fd6f7ed-a2b2-4247-ad8f-afd1bdbba35d
742 KB
sha256:fc92082de49591ceb8208cfed21dc5a84f142ebaff2dcc8fd74d6d8ab63e55d5
coverage-e2e-29953218008-23a07397-c17b-487c-a85e-be321cf31de1
341 KB
sha256:2d898a344f74866f82c6d1e0c54ff3a0e0ebf7db8c4d80f74ea8232e260eed69
coverage-e2e-29953218008-2493bf34-80cd-453b-9123-b221f3f7e557
249 KB
sha256:639990d6c896726fe045ad7d805ced138020d1894a03e5f8282071c67d9f877b
coverage-e2e-29953218008-2600bf5a-2252-4910-a09f-d11a0df86342
511 KB
sha256:110619bc080f9485b49802b885e8e8bf1695c0290f8a33ffedbbc9af520a6c11
coverage-e2e-29953218008-3c55bc25-8bce-4531-8d92-8dad3f3378bc
460 KB
sha256:2da93661d434f5ca1c97aacfe90c4e1bb4f3babbd303a12fc63ffa5d9772ccc9
coverage-e2e-29953218008-40546d7f-8ad8-4080-aed7-39555e2d6015
760 KB
sha256:180c62fada2eaf3a7172f46fcdf3f3ac5c318223dff4fd27135bedb7b580f6d6
coverage-e2e-29953218008-46699e21-a523-4651-a27f-ff7fd579fd64
385 KB
sha256:107889e69ef3b06cb9d3f396ec7a9fe3fb31f292944c9432b972f510dd8cfcdf
coverage-e2e-29953218008-510698cd-c9af-46a9-8f0b-756007c98386
287 KB
sha256:9db483c409ac439b4752774abd06fd6832747c8f5f4372636cec05525a5a8487
coverage-e2e-29953218008-514ad5b1-c3a2-4279-ac9c-138d2db6e467
271 KB
sha256:dedab6d4ae854253bbdcb163910d095fee8b43d745d46c84a4f17895501d8f20
coverage-e2e-29953218008-5792102f-1669-4e41-b4d6-4d6ad4552d1d
288 KB
sha256:35141da4caa23714cb901c6c45526ba54bc250c9323aba727ea346acf2750a91
coverage-e2e-29953218008-5a6943ee-dc24-41f3-a3e7-92a220ac9725
248 KB
sha256:ab03ab69e8d179d16f3a93b48ba0a18debed9ff14b23a606bb15b663f0ca3dfa
coverage-e2e-29953218008-60071260-b5f2-451e-92fc-9367c49bba5e
743 KB
sha256:c4192e36ffbc9a3cc3594d94379b8c644b0895f0dee316aec139491ab7f20e9e
coverage-e2e-29953218008-635b7d32-68db-40b9-806d-f1b0890456aa
248 KB
sha256:2966d715a190437e6dc2eed134ba4a7fe5bc9ad88f192f1740ad62f671a75120
coverage-e2e-29953218008-63bf7429-e97c-4827-ac9c-5b9ba5348383
743 KB
sha256:ff39d49161b1480006ce298f9ea06fca78168c26c5f93b218c68ef81090fb3e4
coverage-e2e-29953218008-6456a767-11cf-42e0-a695-6e63266ffbba
371 KB
sha256:43fb34f404c9a2c7903169813bad37d01553c2152cecb8b65cf57c253d6ac54f
coverage-e2e-29953218008-6724f674-3221-4b96-a026-7a4a3b39aad0
376 KB
sha256:77a6f345da622b6d24f24cad590604e06ba6256ea3a5b721047519fb4528239e
coverage-e2e-29953218008-6856885e-233d-42f5-bd2e-2e3dac73cf1b
744 KB
sha256:590034d6829e9679fbefc874bc1d2a775be8d21c28ba5829e1291bc11b74ce15
coverage-e2e-29953218008-773db566-cc15-489f-a3fd-94314be32715
339 KB
sha256:38ea3f7f645296dd79f84d9fb419cab66486c295466b06f3d6e3d76b300fd0bb
coverage-e2e-29953218008-7e30e097-761e-4d60-aa93-8270af8a52a6
295 KB
sha256:37455d437b70fe1441c840ed582d590883d05d960e6b95d10d58167f321c3033
coverage-e2e-29953218008-8390cb7f-8f94-43f3-9c1d-65eb8d669d68
807 KB
sha256:597be4918ece3413978b88a42edce031a9a6b9328bfb873b914d435b7bcdb013
coverage-e2e-29953218008-86793e91-133c-4232-995e-a276cf95b195
759 KB
sha256:44ed6dfe8a65d03a267d535d5b5f79a2559589f18f50f74c4c3edc6b3fb6f36e
coverage-e2e-29953218008-8e56fb71-6a38-45de-a433-873c79c2b797
809 KB
sha256:b9c37712173e31041bc4e8fdf0262693f0fa7c5e57c6f8bc94abf52baadc394e
coverage-e2e-29953218008-8ed76b89-7ebd-4800-a436-99f5e61d11ba
245 KB
sha256:fde2865804d4d7daeab6548e6b1501370cbd19b1984ec78a3a0ff1148f004a57
coverage-e2e-29953218008-a5425683-4123-4817-9212-a7563d94093c
351 KB
sha256:063b1a77ce139253cdb7a17200affd47f6cb9892f9efb14fa2762f7a1e282207
coverage-e2e-29953218008-a633314a-00c3-4d7c-8d29-d57a902e0dda
742 KB
sha256:8e9479cc149aa6701a0e04f4ee9b2f254e12d847654ebda5edd11fed767a1956
coverage-e2e-29953218008-aa045ea5-af8d-4e5b-978a-e5e58a803e57
288 KB
sha256:9f56f37f8390af8aac8c9fecb7f1a8a37c64b10b0d4ab305bf0d3ff42c41a993
coverage-e2e-29953218008-b3963b3c-dbbe-4dd1-99df-7ff2bdd71249
804 KB
sha256:4d4d055c01192c3c93a5e260648a34295932a1ac7dc34327ca021ffeacf577b7
coverage-e2e-29953218008-b46bf20f-d3db-477f-a6c6-9aa0a8592572
742 KB
sha256:0defaef140d185451677221e8ba329cb02bfc32e1a1e3932722790753ae88b50
coverage-e2e-29953218008-b71596f2-c14e-4798-a44c-9b9f2a06ae36
743 KB
sha256:e72cdb6be219b5e363c2cf19e6a623362f03360c0aba5104c401e24c7dc47792
coverage-e2e-29953218008-b97e2e8f-8825-436c-9605-44d994fcbff2
315 KB
sha256:08e30d56f2497de5256754db0e1ec8dd0bdc37f2ce282600267cd51bf412e343
coverage-e2e-29953218008-bd06a23f-ea3b-40d3-82d7-ce9765ae93b9
248 KB
sha256:2c807a5c79e80d65458e17b9a21219c9797e42d18a5263938bd71ef543994619
coverage-e2e-29953218008-bea91e41-a864-4b33-91ab-bae3e56d6ae6
248 KB
sha256:0f05bfe53c31a526969244b5ee3cc35425990d1a73af23d41801599f4fbd471c
coverage-e2e-29953218008-c7b395a2-9e92-499b-a0be-c6fd1bc681a1
811 KB
sha256:d361754bf6b46f4d305cf688db70a9956c93190dfb983214d1fd0cfcf6d8fb98
coverage-e2e-29953218008-cebfad9c-bddd-4e52-af9a-35121b65bc47
316 KB
sha256:d2a3d928565d2f5cacfde6adf955afe6ff18fe4d43b5931595fbe3cdadc4e2a3
coverage-e2e-29953218008-da4eea16-f5b5-4ef5-b71f-c4fc8f2c0697
249 KB
sha256:1381b5c02d661fdf23af3c3f7990e806377e5c9ed2c3caec403d5b40961c9d9d
coverage-e2e-29953218008-dc79e342-b653-402c-9b04-e5447b9b53ab
248 KB
sha256:a8ba25deebf2db2271ffa39a74af05c50c86ddb5cd65e066972ada79c31bc8ad
coverage-e2e-29953218008-e100221b-01a5-4a1e-bdc2-f33ca85cff1f
816 KB
sha256:5ce2f295f4ab3da8114c258fc9bc2b8f2dc35dc47acf4feda49461b7fe9bd693
coverage-e2e-29953218008-e1f36744-0ac9-4ac2-95d7-d63ea19f8472
807 KB
sha256:fbb89311ba1f41d4a46e4a98900d5608ffc2bc97044bda690c48c0d0e11520b8
coverage-e2e-29953218008-e580892b-a315-429b-9705-b8ddea7c1c07
382 KB
sha256:d6ee39a2d43ca5a8cef7f36f635590f4a116ba339e14e369b06110ff059ebe60
coverage-e2e-29953218008-ef277446-7799-40d9-a41c-735c2b81247c
249 KB
sha256:570ed18f4e9f3620ac7b7ac0b7d7e974517817f1092bc8601821f1e8d2ac71a2
coverage-e2e-29953218008-f05a67b7-00dc-4729-8a9e-e312e5ee32f8
332 KB
sha256:fc5ec5e3f25b01a7a19f0ab91141c733122ce0d3c6939a8cfbeac32b00c6a9fc
coverage-e2e-29953218008-f46ec5df-60a6-48c8-8431-6bfd4a58106e
1.29 KB
sha256:6ab0c0ca5ba3c2a301dcbab5e6c19a0b7e7053d765e789770aca378cac55bdbc
coverage-e2e-29953218008-fd03dbfb-c893-4b30-805c-6ab27dc4f3fe
282 KB
sha256:7d83cd884190c0ab5d776677f3f46b24165a261a6cbecd8b691d098d3bdcb6e3
coverage-e2e-29953218008-fd46a6dc-8609-4a61-832d-7e2b7ef8d4ba
808 KB
sha256:5fc95edfbe7f0751584fc28a4d1a425c855286c308d9baddce262c410ddfe999
coverage-unit-test-29953218008-011c1619-109a-4652-8c31-261864f8f457
337 KB
sha256:ff30791d2e52efe158e128e9bc3fb4b86d8816de111957a0639a9cb88c22cada
coverage-unit-test-29953218008-da4fc922-4a39-4fb1-8630-cb0a285e27ca
863 KB
sha256:7fcf4e5e2c285469501938598746ba3c634e9a84a9707d75557ea3f5415db485