CICD NeMo #18535
cicd-main.yml
on: workflow_dispatch
pre-flight
/
pre-flight
15s
Matrix: cicd-container-build
Launch_Unit_Tests_Core
9m 19s
Launch_Unit_Tests_Diffusion
3m 0s
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-flaky
Coverage_Fake
0s
Matrix: Coverage
Deployment protection rules
Reviewers, timers, and other rules protecting deployments in this run
| Event | Environments |
|---|---|
|
svcnvidia-nemo-ci
approved
|
test |
Annotations
1 error, 2 warnings, and 51 notices
|
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: The directory '/home/runner/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
|
|
pre-flight / pre-flight
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, step-security/changed-files@v45.0.1. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@v5, docker/setup-buildx-action@v3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
|
|
Result
✅ Launch_Unit_Tests_Core — PASSED
|
|
Result
✅ L1_Launch_models_nemotron — PASSED
|
|
Result
✅ L1_Launch_recipes_glm_vl — PASSED
|
|
Result
✅ L1_Launch_models_kimi_k25_vl — PASSED
|
|
Result
✅ L0_Launch_converter_fsdp — PASSED
|
|
Result
✅ L1_Launch_models_qwen35_vl — PASSED
|
|
Result
✅ L1_Launch_recipes_gemma — PASSED
|
|
Result
✅ L0_Launch_recipes_perf_config — PASSED
|
|
Result
✅ L1_Launch_models_llada15 — PASSED
|
|
Result
✅ L0_Launch_training_callbacks — PASSED
|
|
Result
✅ L1_Launch_models_qwen25_vl — PASSED
|
|
Result
✅ L1_Launch_models_qwen2 — PASSED
|
|
Result
✅ L0_Launch_models_flux — PASSED
|
|
Result
✅ L0_Launch_data — PASSED
|
|
Result
✅ L1_Launch_models_glm — PASSED
|
|
Result
✅ L1_Launch_models_gemma3 — PASSED
|
|
Result
✅ L0_Launch_megatron_fsdp — PASSED
|
|
Result
✅ L0_Launch_recipes_llama_1b — PASSED
|
|
Result
✅ L0_Launch_models_nemotron_omni — PASSED
|
|
Result
✅ L0_Launch_models_gemma4_vl — PASSED
|
|
Result
✅ L1_Launch_models_qwen3_omni — PASSED
|
|
Result
✅ L0_Launch_recipes_deepseek — PASSED
|
|
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
|
|
Result
✅ L1_Launch_models_stepfun — PASSED
|
|
Result
✅ L0_Launch_models_deepseek — PASSED
|
|
Result
✅ L1_Launch_models_ministral3 — PASSED
|
|
Result
✅ L1_Launch_models_glm_vl — PASSED
|
|
Result
✅ L0_Launch_models_nemotronh — PASSED
|
|
Result
✅ L1_Launch_models_qwen3_vl — PASSED
|
|
Result
✅ L0_Launch_converter_generate — PASSED
|
|
Result
✅ L0_Launch_models_qwen3 — PASSED
|
|
Result
✅ L1_Launch_recipes_gpt_oss — PASSED
|
|
Result
✅ L1_Launch_models_gpt_oss — PASSED
|
|
Result
✅ L1_Launch_recipes_llama_3b — PASSED
|
|
Result
✅ L0_Launch_converter — PASSED
|
|
Result
✅ L1_Launch_recipes_ministral3 — PASSED
|
|
Result
✅ L0_Launch_training_decentralized_pg — PASSED
|
|
Result
✅ L1_Launch_recipes_nemotron_omni — PASSED
|
|
Result
✅ L0_Launch_training_distributed_save — PASSED
|
|
Result
✅ L1_Launch_recipes_qwen — PASSED
|
|
Result
✅ L0_Launch_training_finetune — PASSED
|
|
Result
✅ L1_Launch_recipes_nemotronh — PASSED
|
|
Result
✅ L1_Launch_vlm_inference — PASSED
|
|
Result
✅ L0_Launch_training_local_checkpointing — PASSED
|
|
Result
✅ L0_Launch_training_pretrain — PASSED
|
|
Result
✅ L0_Launch_training_tensor_inspect — PASSED
|
|
Result
✅ L0_Launch_utils — PASSED
|
|
Result
✅ L0_Launch_training_resiliency — PASSED
|
|
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
|
|
Result
✅ L1_Launch_recipes_qwen_vl — PASSED
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
coverage-e2e-29956412484-0204f8e0-a01a-4a1f-b968-ab593763781b
|
249 KB |
sha256:410322229bb96882ddef85205d7057fecc76f1f3d578d7fd8de45140e1362b95
|
|
|
coverage-e2e-29956412484-030a2af4-80d4-41a2-804e-dba9c472f5c5
|
384 KB |
sha256:736f4cbc33a4bf6252d0c100652925382a1d8f78340b5eb5084682dc5c9cf11a
|
|
|
coverage-e2e-29956412484-0f589c0c-4185-469b-96ba-e733c110f465
|
332 KB |
sha256:58a2e9a8cbdf4e33bd34eca6109a94179aaa733607cd0bd2f2f5641229d6bc67
|
|
|
coverage-e2e-29956412484-10059d75-995a-4a48-93ce-9383889d101b
|
812 KB |
sha256:12ba9b8eaa92f2d0479fc5e4234ba5be9f6f65aea19c3d3bca5aa88fd7bad70f
|
|
|
coverage-e2e-29956412484-1623541b-67ff-48d6-9ea8-a5ac51513145
|
816 KB |
sha256:36a5fe9754be5805e87524a9190b84f8dec00dda7a4a7e9bcfc59ee096daa0ee
|
|
|
coverage-e2e-29956412484-179db6a9-42a1-42dd-aa48-66da125a4e79
|
341 KB |
sha256:6b14a0165ec84fb10b6c345281ff9c690ce1785e6aef936d6dcbbd8bf0432a1b
|
|
|
coverage-e2e-29956412484-19cd57e4-bcfa-4ce3-9334-2e476afb399f
|
316 KB |
sha256:6564c85483786c83e13981ed7ab72bd6982d677550fec47c2bb74a7200318fa1
|
|
|
coverage-e2e-29956412484-1bb0df44-8e97-432a-a6a2-7156bf007b2b
|
807 KB |
sha256:17dee0231b408fb34c4c1dc5fe166bdbc1c6f947327909b1ac6036b9fcf26944
|
|
|
coverage-e2e-29956412484-1e8cf643-cd1e-4d68-8e21-2f7e41f0b130
|
744 KB |
sha256:e450b1bcefdbdf2cd534a0abeca742d233400befe37a8b659bb8e43da50bf9b7
|
|
|
coverage-e2e-29956412484-282f04c6-93d5-49bc-beda-499813ac4fb9
|
742 KB |
sha256:826ed8557bfa000c99451c99c9ca8652e12477a47762cacdde6e0981e27247f2
|
|
|
coverage-e2e-29956412484-2bffbea2-22f0-4dfe-8289-e2607dfcbbd0
|
249 KB |
sha256:bce2bf8ec7d388bb41d77ca55e4179308fe063813d7d9074ef1da17614a7dcd0
|
|
|
coverage-e2e-29956412484-2e06eaba-7944-4aa4-be8c-1fd5ee63a812
|
743 KB |
sha256:aeecb506b3ffcc6b5b6255a862a4b2f8f28a300bb5be44b73368dc880704e549
|
|
|
coverage-e2e-29956412484-3c61ea57-3c4f-44c3-af77-51c564deef9b
|
810 KB |
sha256:03c993d20171fa68c6b1fd158db4df3872ffc8a413ee486874c9d51048c2ed84
|
|
|
coverage-e2e-29956412484-3f87c892-2c29-438b-9b12-042d22078bdc
|
340 KB |
sha256:986c424ebeea493d87bce5b45b5d963d56355b9c7fe306fcfecebd0d41b22cd9
|
|
|
coverage-e2e-29956412484-49f030aa-29d2-4bf4-bbc9-3adac248da5c
|
804 KB |
sha256:32e5289692b0327d6aab12bc600afa55b5893eee0d5a858ca59fc8930ad831fd
|
|
|
coverage-e2e-29956412484-5176fda8-c9f3-4d2e-8063-36fb3ddaf1c7
|
809 KB |
sha256:f43c5eab96c952b2f70fb5c05694b8ffc755167676590221e9e03e5c25f04afc
|
|
|
coverage-e2e-29956412484-54030b83-ecd8-46a7-a87b-ea63d8942806
|
1.29 KB |
sha256:00780403e20dadf4274898e5d0c63225f0b1c69749d497c9a05a6d5552cc2b49
|
|
|
coverage-e2e-29956412484-540adbca-5999-4632-ac71-c3a2403a0720
|
249 KB |
sha256:7b2357088ae4779c032da3bae5803f34db493459d2c8cafb5c4f07764fc7fcfb
|
|
|
coverage-e2e-29956412484-577afe45-21eb-4ee1-82d0-20ad1a85d378
|
743 KB |
sha256:d05ebe10de095ed0075eff0cea04ee578cad2f7545aa6efb49a0224e25efd6fa
|
|
|
coverage-e2e-29956412484-5951eb6c-63bb-4083-89a2-a0b991af26c8
|
742 KB |
sha256:eeb42c80f37de627c0db2083ac582623f25008d82e6e1c3b2717c12d8574bf5e
|
|
|
coverage-e2e-29956412484-5b18f4a8-48c5-4d7f-8111-237e0e578990
|
249 KB |
sha256:0a9d768db0e5ffc060e93285d9005ede078a3f78d3d8bad5235999de4822f443
|
|
|
coverage-e2e-29956412484-60bf4490-aa3b-4e28-94b0-4e89a52ae1f6
|
295 KB |
sha256:b4810b361f6d4d9ec4668d158d385b22db511438803c0013a8fec4ce198dede4
|
|
|
coverage-e2e-29956412484-634aa824-206e-46c3-a6ed-32be835f2d35
|
288 KB |
sha256:c282c0505cc962168e69b297843691f58267b4b450f2bf78bb6ccb496bc70d2e
|
|
|
coverage-e2e-29956412484-64c60381-c23b-4e4b-bb33-d9167c2a40e7
|
743 KB |
sha256:3327a70dd5733d81e7ec354098e7bae60fa3123f3bbb210dd1ddf3abf92f8931
|
|
|
coverage-e2e-29956412484-6f4b40a4-d9f2-48d3-8565-f09645f71e84
|
807 KB |
sha256:8cdbfc2c9776a60248d24344dd5362f9d29b530af23f2073dbd848b3906346a2
|
|
|
coverage-e2e-29956412484-74f44330-190e-400a-a076-5a714707ff69
|
317 KB |
sha256:14b1f50f6a38283458da0586c3c18d69943832d53b67230808464b9b6eed0a9b
|
|
|
coverage-e2e-29956412484-7c6f6b25-f56b-4a78-84b6-769845a92b9f
|
315 KB |
sha256:a06a5e31afb17932fb341b9a9a1599490cde1e37bff40d71d24141c8b160c0f4
|
|
|
coverage-e2e-29956412484-7e808a9b-26d3-43ab-9760-78a58e82c62d
|
385 KB |
sha256:0f62e729fc17b240a3058cf62e14d7b18bc81ba80a38f393b2e7cfc340259b18
|
|
|
coverage-e2e-29956412484-827bd6b1-fc7a-4261-8ea6-6a22bcb3de17
|
248 KB |
sha256:18fc96f79de41b1b6463d2175c48aa1fedafc2e596f30aa6a8e40d2ae73b8972
|
|
|
coverage-e2e-29956412484-90492214-0709-48fa-8e78-7684d544ad9d
|
808 KB |
sha256:134245c34d27b8c6e14b96fa7c75e27f0bdccc40947f1b9101403831216be30e
|
|
|
coverage-e2e-29956412484-91220bd1-1e86-4c28-9d3d-05e51e00a544
|
288 KB |
sha256:a4e489b57f2a44131c0f462b62273cc71de67f326337eff6c8b84366203ade11
|
|
|
coverage-e2e-29956412484-a54b8988-12bd-44c4-bf81-c84272ff346c
|
742 KB |
sha256:d825aef04782611ce21bbd45bc59335c7805a5cd858124c23049d3b84f7215d4
|
|
|
coverage-e2e-29956412484-a6a586a2-0612-46c6-8388-e794f67cd428
|
248 KB |
sha256:553d803672f3dfa97415af745421e34a6282be912bb089dbc209614a0c5fb964
|
|
|
coverage-e2e-29956412484-a70897e6-3062-4fa2-826c-51ab76386051
|
288 KB |
sha256:0e56e59f987ce1e8673da33361aa3b2d9f0be657fceb241d317ce9ec9af7b481
|
|
|
coverage-e2e-29956412484-bf5664d1-ae44-40c4-a670-45fac8ead539
|
282 KB |
sha256:da7bee3162db435ef828d3caee2d410e0647bb837d58d9f54065f8357018105f
|
|
|
coverage-e2e-29956412484-c0731b88-0d73-4b19-8768-a71bdba306cf
|
371 KB |
sha256:070e360dafeacb7563379891f73217fb3dd769ec26567942178d24ededb2af7c
|
|
|
coverage-e2e-29956412484-c29c0707-2b75-4ef0-9fdd-cffe3016a8df
|
377 KB |
sha256:44adf6f1cb2f20bbcfbcf12497eb57b3f6d02ac2ab5f1dfae37b054e9cb6be1e
|
|
|
coverage-e2e-29956412484-cf0445e6-5555-46cc-807f-7083b8d7171f
|
743 KB |
sha256:a1dce6f5f960a8a85af0db12016765d72b35f8770d38f625ed2f7801ba334d61
|
|
|
coverage-e2e-29956412484-cf7fdebb-88ad-4abc-bfde-ba5d0165b700
|
761 KB |
sha256:cccef3a85d52a4bf5f2a4ec41a3d1b644f45e47c6b7c433c16b0bf99329cf4c6
|
|
|
coverage-e2e-29956412484-da768d65-15d4-4aa5-9667-edc132ce8a2a
|
351 KB |
sha256:056c770d50d5301676b9e6499dafd0bbb02196ae560dadd330ace9f072cbadad
|
|
|
coverage-e2e-29956412484-dd802c8f-60fc-48a4-87dd-f4f097da9e0c
|
271 KB |
sha256:acd98ae0f807ea49be60407706770a31ade98013b8fa2e8d09cac492a4dd545e
|
|
|
coverage-e2e-29956412484-dec11d7a-171c-46c9-a42a-198248db0e94
|
245 KB |
sha256:e509086a9b2a7c38ec337602da92c0f0dc8cc06e6547d7a6a2b351a777b82483
|
|
|
coverage-e2e-29956412484-e282d0f4-ef32-4314-83ec-96252acb2f40
|
759 KB |
sha256:ec68e7ea7121823cde4e12bcc0dbd4f6c363637e43f128793d728c2ce00cce41
|
|
|
coverage-e2e-29956412484-f1a11dc3-a9eb-4321-956a-7294bd1e9e30
|
249 KB |
sha256:bc606098b445d8982aa4776277f51df25b0d5a3495c27b257886f23f4ca98f20
|
|
|
coverage-e2e-29956412484-f68406fa-fbfb-43b7-9fdb-0ab82f3f470b
|
460 KB |
sha256:da407739790c6ce8b5f6c73099f1aeaad5b25a3b016efd4b86e9a91181592201
|
|
|
coverage-e2e-29956412484-f71c1163-cce3-4dff-89bb-4aa54b41e1f5
|
248 KB |
sha256:6ef2d5adfeab5a4a982c83236956138bed1b552915811764973b4d057b34b9c1
|
|
|
coverage-e2e-29956412484-f9f48506-c47f-48ca-b405-3a2f6e2b8863
|
382 KB |
sha256:1a2806cdc5d76e30a42b90fb3270aa8babc3feae32299d7f78e88db769c73705
|
|
|
coverage-e2e-29956412484-fe73b59e-e109-426e-953e-e072e53b6085
|
743 KB |
sha256:dd5962dbf1a122616038e35c14e7bbde4c938555c51561eafdd09c380ea33d07
|
|
|
coverage-e2e-29956412484-ff1e09af-7c7d-4cb0-a029-e43295b18d8b
|
512 KB |
sha256:f16884dc7affb540b09745e607ebc61b9764b4c5515a76ccaefaa6711c626377
|
|
|
coverage-unit-test-29956412484-16b4903a-cede-40b0-b25f-8f0f10bf600c
|
337 KB |
sha256:fc3d02e77d6a449930da2e3207a92132697927b61660a0ab0474ab20872dcef1
|
|
|
coverage-unit-test-29956412484-820b069d-a83a-4813-94e7-007bd5256d38
|
863 KB |
sha256:7950a2dfad71c6e468f8899dd574c576f9007a66208b07253830281014d05a95
|
|