Skip to content

CICD NeMo

CICD NeMo #18535

Manually triggered July 22, 2026 20:44
Status Success
Total duration 3h 43m 42s
Artifacts 51

cicd-main.yml

on: workflow_dispatch
pre-flight  /  pre-flight
15s
pre-flight / pre-flight
configure
4s
configure
Lint check
21s
Lint check
cicd-wait-in-queue
3s
cicd-wait-in-queue
cicd-compute-build-matrix
5s
cicd-compute-build-matrix
Matrix: cicd-container-build
Launch_Unit_Tests_Core
9m 19s
Launch_Unit_Tests_Core
Launch_Unit_Tests_Diffusion
3m 0s
Launch_Unit_Tests_Diffusion
generate-test-matrix
9s
generate-test-matrix
generate-gb200-test-matrix
0s
generate-gb200-test-matrix
Launch_Import_Check
1m 36s
Launch_Import_Check
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-flaky
Nemo_CICD_Test
8s
Nemo_CICD_Test
Coverage_Fake
0s
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
svcnvidia-nemo-ci
approved Jul 22, 2026
test

Automatically approved by queue manager

Annotations

1 error, 2 warnings, and 51 notices
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: The directory '/home/runner/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
pre-flight / pre-flight
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, step-security/changed-files@v45.0.1. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@v5, docker/setup-buildx-action@v3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ L1_Launch_models_nemotron — PASSED
Result
✅ L1_Launch_recipes_glm_vl — PASSED
Result
✅ L1_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_converter_fsdp — PASSED
Result
✅ L1_Launch_models_qwen35_vl — PASSED
Result
✅ L1_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L1_Launch_models_llada15 — PASSED
Result
✅ L0_Launch_training_callbacks — PASSED
Result
✅ L1_Launch_models_qwen25_vl — PASSED
Result
✅ L1_Launch_models_qwen2 — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L1_Launch_models_glm — PASSED
Result
✅ L1_Launch_models_gemma3 — PASSED
Result
✅ L0_Launch_megatron_fsdp — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_models_gemma4_vl — PASSED
Result
✅ L1_Launch_models_qwen3_omni — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L1_Launch_models_stepfun — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L1_Launch_models_ministral3 — PASSED
Result
✅ L1_Launch_models_glm_vl — PASSED
Result
✅ L0_Launch_models_nemotronh — PASSED
Result
✅ L1_Launch_models_qwen3_vl — PASSED
Result
✅ L0_Launch_converter_generate — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L1_Launch_recipes_gpt_oss — PASSED
Result
✅ L1_Launch_models_gpt_oss — PASSED
Result
✅ L1_Launch_recipes_llama_3b — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L1_Launch_recipes_ministral3 — PASSED
Result
✅ L0_Launch_training_decentralized_pg — PASSED
Result
✅ L1_Launch_recipes_nemotron_omni — PASSED
Result
✅ L0_Launch_training_distributed_save — PASSED
Result
✅ L1_Launch_recipes_qwen — PASSED
Result
✅ L0_Launch_training_finetune — PASSED
Result
✅ L1_Launch_recipes_nemotronh — PASSED
Result
✅ L1_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_training_local_checkpointing — PASSED
Result
✅ L0_Launch_training_pretrain — PASSED
Result
✅ L0_Launch_training_tensor_inspect — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_training_resiliency — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L1_Launch_recipes_qwen_vl — PASSED

Artifacts

Produced during runtime
Name Size Digest
coverage-e2e-29956412484-0204f8e0-a01a-4a1f-b968-ab593763781b
249 KB
sha256:410322229bb96882ddef85205d7057fecc76f1f3d578d7fd8de45140e1362b95
coverage-e2e-29956412484-030a2af4-80d4-41a2-804e-dba9c472f5c5
384 KB
sha256:736f4cbc33a4bf6252d0c100652925382a1d8f78340b5eb5084682dc5c9cf11a
coverage-e2e-29956412484-0f589c0c-4185-469b-96ba-e733c110f465
332 KB
sha256:58a2e9a8cbdf4e33bd34eca6109a94179aaa733607cd0bd2f2f5641229d6bc67
coverage-e2e-29956412484-10059d75-995a-4a48-93ce-9383889d101b
812 KB
sha256:12ba9b8eaa92f2d0479fc5e4234ba5be9f6f65aea19c3d3bca5aa88fd7bad70f
coverage-e2e-29956412484-1623541b-67ff-48d6-9ea8-a5ac51513145
816 KB
sha256:36a5fe9754be5805e87524a9190b84f8dec00dda7a4a7e9bcfc59ee096daa0ee
coverage-e2e-29956412484-179db6a9-42a1-42dd-aa48-66da125a4e79
341 KB
sha256:6b14a0165ec84fb10b6c345281ff9c690ce1785e6aef936d6dcbbd8bf0432a1b
coverage-e2e-29956412484-19cd57e4-bcfa-4ce3-9334-2e476afb399f
316 KB
sha256:6564c85483786c83e13981ed7ab72bd6982d677550fec47c2bb74a7200318fa1
coverage-e2e-29956412484-1bb0df44-8e97-432a-a6a2-7156bf007b2b
807 KB
sha256:17dee0231b408fb34c4c1dc5fe166bdbc1c6f947327909b1ac6036b9fcf26944
coverage-e2e-29956412484-1e8cf643-cd1e-4d68-8e21-2f7e41f0b130
744 KB
sha256:e450b1bcefdbdf2cd534a0abeca742d233400befe37a8b659bb8e43da50bf9b7
coverage-e2e-29956412484-282f04c6-93d5-49bc-beda-499813ac4fb9
742 KB
sha256:826ed8557bfa000c99451c99c9ca8652e12477a47762cacdde6e0981e27247f2
coverage-e2e-29956412484-2bffbea2-22f0-4dfe-8289-e2607dfcbbd0
249 KB
sha256:bce2bf8ec7d388bb41d77ca55e4179308fe063813d7d9074ef1da17614a7dcd0
coverage-e2e-29956412484-2e06eaba-7944-4aa4-be8c-1fd5ee63a812
743 KB
sha256:aeecb506b3ffcc6b5b6255a862a4b2f8f28a300bb5be44b73368dc880704e549
coverage-e2e-29956412484-3c61ea57-3c4f-44c3-af77-51c564deef9b
810 KB
sha256:03c993d20171fa68c6b1fd158db4df3872ffc8a413ee486874c9d51048c2ed84
coverage-e2e-29956412484-3f87c892-2c29-438b-9b12-042d22078bdc
340 KB
sha256:986c424ebeea493d87bce5b45b5d963d56355b9c7fe306fcfecebd0d41b22cd9
coverage-e2e-29956412484-49f030aa-29d2-4bf4-bbc9-3adac248da5c
804 KB
sha256:32e5289692b0327d6aab12bc600afa55b5893eee0d5a858ca59fc8930ad831fd
coverage-e2e-29956412484-5176fda8-c9f3-4d2e-8063-36fb3ddaf1c7
809 KB
sha256:f43c5eab96c952b2f70fb5c05694b8ffc755167676590221e9e03e5c25f04afc
coverage-e2e-29956412484-54030b83-ecd8-46a7-a87b-ea63d8942806
1.29 KB
sha256:00780403e20dadf4274898e5d0c63225f0b1c69749d497c9a05a6d5552cc2b49
coverage-e2e-29956412484-540adbca-5999-4632-ac71-c3a2403a0720
249 KB
sha256:7b2357088ae4779c032da3bae5803f34db493459d2c8cafb5c4f07764fc7fcfb
coverage-e2e-29956412484-577afe45-21eb-4ee1-82d0-20ad1a85d378
743 KB
sha256:d05ebe10de095ed0075eff0cea04ee578cad2f7545aa6efb49a0224e25efd6fa
coverage-e2e-29956412484-5951eb6c-63bb-4083-89a2-a0b991af26c8
742 KB
sha256:eeb42c80f37de627c0db2083ac582623f25008d82e6e1c3b2717c12d8574bf5e
coverage-e2e-29956412484-5b18f4a8-48c5-4d7f-8111-237e0e578990
249 KB
sha256:0a9d768db0e5ffc060e93285d9005ede078a3f78d3d8bad5235999de4822f443
coverage-e2e-29956412484-60bf4490-aa3b-4e28-94b0-4e89a52ae1f6
295 KB
sha256:b4810b361f6d4d9ec4668d158d385b22db511438803c0013a8fec4ce198dede4
coverage-e2e-29956412484-634aa824-206e-46c3-a6ed-32be835f2d35
288 KB
sha256:c282c0505cc962168e69b297843691f58267b4b450f2bf78bb6ccb496bc70d2e
coverage-e2e-29956412484-64c60381-c23b-4e4b-bb33-d9167c2a40e7
743 KB
sha256:3327a70dd5733d81e7ec354098e7bae60fa3123f3bbb210dd1ddf3abf92f8931
coverage-e2e-29956412484-6f4b40a4-d9f2-48d3-8565-f09645f71e84
807 KB
sha256:8cdbfc2c9776a60248d24344dd5362f9d29b530af23f2073dbd848b3906346a2
coverage-e2e-29956412484-74f44330-190e-400a-a076-5a714707ff69
317 KB
sha256:14b1f50f6a38283458da0586c3c18d69943832d53b67230808464b9b6eed0a9b
coverage-e2e-29956412484-7c6f6b25-f56b-4a78-84b6-769845a92b9f
315 KB
sha256:a06a5e31afb17932fb341b9a9a1599490cde1e37bff40d71d24141c8b160c0f4
coverage-e2e-29956412484-7e808a9b-26d3-43ab-9760-78a58e82c62d
385 KB
sha256:0f62e729fc17b240a3058cf62e14d7b18bc81ba80a38f393b2e7cfc340259b18
coverage-e2e-29956412484-827bd6b1-fc7a-4261-8ea6-6a22bcb3de17
248 KB
sha256:18fc96f79de41b1b6463d2175c48aa1fedafc2e596f30aa6a8e40d2ae73b8972
coverage-e2e-29956412484-90492214-0709-48fa-8e78-7684d544ad9d
808 KB
sha256:134245c34d27b8c6e14b96fa7c75e27f0bdccc40947f1b9101403831216be30e
coverage-e2e-29956412484-91220bd1-1e86-4c28-9d3d-05e51e00a544
288 KB
sha256:a4e489b57f2a44131c0f462b62273cc71de67f326337eff6c8b84366203ade11
coverage-e2e-29956412484-a54b8988-12bd-44c4-bf81-c84272ff346c
742 KB
sha256:d825aef04782611ce21bbd45bc59335c7805a5cd858124c23049d3b84f7215d4
coverage-e2e-29956412484-a6a586a2-0612-46c6-8388-e794f67cd428
248 KB
sha256:553d803672f3dfa97415af745421e34a6282be912bb089dbc209614a0c5fb964
coverage-e2e-29956412484-a70897e6-3062-4fa2-826c-51ab76386051
288 KB
sha256:0e56e59f987ce1e8673da33361aa3b2d9f0be657fceb241d317ce9ec9af7b481
coverage-e2e-29956412484-bf5664d1-ae44-40c4-a670-45fac8ead539
282 KB
sha256:da7bee3162db435ef828d3caee2d410e0647bb837d58d9f54065f8357018105f
coverage-e2e-29956412484-c0731b88-0d73-4b19-8768-a71bdba306cf
371 KB
sha256:070e360dafeacb7563379891f73217fb3dd769ec26567942178d24ededb2af7c
coverage-e2e-29956412484-c29c0707-2b75-4ef0-9fdd-cffe3016a8df
377 KB
sha256:44adf6f1cb2f20bbcfbcf12497eb57b3f6d02ac2ab5f1dfae37b054e9cb6be1e
coverage-e2e-29956412484-cf0445e6-5555-46cc-807f-7083b8d7171f
743 KB
sha256:a1dce6f5f960a8a85af0db12016765d72b35f8770d38f625ed2f7801ba334d61
coverage-e2e-29956412484-cf7fdebb-88ad-4abc-bfde-ba5d0165b700
761 KB
sha256:cccef3a85d52a4bf5f2a4ec41a3d1b644f45e47c6b7c433c16b0bf99329cf4c6
coverage-e2e-29956412484-da768d65-15d4-4aa5-9667-edc132ce8a2a
351 KB
sha256:056c770d50d5301676b9e6499dafd0bbb02196ae560dadd330ace9f072cbadad
coverage-e2e-29956412484-dd802c8f-60fc-48a4-87dd-f4f097da9e0c
271 KB
sha256:acd98ae0f807ea49be60407706770a31ade98013b8fa2e8d09cac492a4dd545e
coverage-e2e-29956412484-dec11d7a-171c-46c9-a42a-198248db0e94
245 KB
sha256:e509086a9b2a7c38ec337602da92c0f0dc8cc06e6547d7a6a2b351a777b82483
coverage-e2e-29956412484-e282d0f4-ef32-4314-83ec-96252acb2f40
759 KB
sha256:ec68e7ea7121823cde4e12bcc0dbd4f6c363637e43f128793d728c2ce00cce41
coverage-e2e-29956412484-f1a11dc3-a9eb-4321-956a-7294bd1e9e30
249 KB
sha256:bc606098b445d8982aa4776277f51df25b0d5a3495c27b257886f23f4ca98f20
coverage-e2e-29956412484-f68406fa-fbfb-43b7-9fdb-0ab82f3f470b
460 KB
sha256:da407739790c6ce8b5f6c73099f1aeaad5b25a3b016efd4b86e9a91181592201
coverage-e2e-29956412484-f71c1163-cce3-4dff-89bb-4aa54b41e1f5
248 KB
sha256:6ef2d5adfeab5a4a982c83236956138bed1b552915811764973b4d057b34b9c1
coverage-e2e-29956412484-f9f48506-c47f-48ca-b405-3a2f6e2b8863
382 KB
sha256:1a2806cdc5d76e30a42b90fb3270aa8babc3feae32299d7f78e88db769c73705
coverage-e2e-29956412484-fe73b59e-e109-426e-953e-e072e53b6085
743 KB
sha256:dd5962dbf1a122616038e35c14e7bbde4c938555c51561eafdd09c380ea33d07
coverage-e2e-29956412484-ff1e09af-7c7d-4cb0-a029-e43295b18d8b
512 KB
sha256:f16884dc7affb540b09745e607ebc61b9764b4c5515a76ccaefaa6711c626377
coverage-unit-test-29956412484-16b4903a-cede-40b0-b25f-8f0f10bf600c
337 KB
sha256:fc3d02e77d6a449930da2e3207a92132697927b61660a0ab0474ab20872dcef1
coverage-unit-test-29956412484-820b069d-a83a-4813-94e7-007bd5256d38
863 KB
sha256:7950a2dfad71c6e468f8899dd574c576f9007a66208b07253830281014d05a95